Tuesday, March 11, 2008

Bluffer's Guide to Consumer-related Science Papers

I've been wanting to write something along these lines for a while and the opportunity seems to have presented itself at last. (Which is poor timing, as I won't have time to tell you about the teapot-worshipper who is going to prison for not abandoning any teachings contrary to Islam).

Wading through PR blurb and advertising copy that relate to 'sciencey' things can be an effort, but with a few key pointers, sense can be made of most of it - for instance, a company will put the strongest claim possible on its strapline, so woolly words like 'may', 'could', help', 'aid', 'reduce', 'some' all go some way to indicate how much of the products claims are real and how much are puff (perhaps a blog post for another day).

Things can get much trickier though, if the advert in question claims "proved by scientific research". If there is grounds for this, the company will no doubt give full access to the research, otherwise, it's fairly safe to assume that it is selling a bum steer.

So far, so straightforward. Let's say our imaginary company has a product, which claims to have scientific proof for its claims AND is presenting the research. Obviously, advertising will already have convinced/dissuaded a section of public; the "proved by research" strapline will convince another swathe and the presentation of gritty full scale science research paper with all the reader-friendly niceties (and pictures) removed, will certainly convince another swathe, especially if the reader is not used to dealing with scientific papers.

If you are like me (and if you are, I empathise with your shortcomings) then, depending on the product, the paper may or may not be read. It all depends on the claim. If a toothpaste claims to whiten my yellowing teeth, the paper may not get read. If a nutriceutical claims to alleviate my arthritis pain, then I am reaching for my intelligence glasses. (I don't wear glasses, but pretending to makes me feel more intelligent).

I have some experience in reading scientific papers. But only really in my area, outwith of that, it can be tough going. What hope is there for someone who is not used at all to dealing with the curt, unloving, perfunctory monotones of thick research text?

With a bit of persuasion, I hope to have here Dr* T's Bluffer's Guide to Published Research. This will probably be a work in progress, so I'm keen to hear of any additional points which will make it easier for people to wade through.

What I've decided to do is use a paper I've been reading recently as a case study.

The product in question is Life Mel Honey.

As a beekeeper, I was interested to learn the nation's most paranoid shopkeeper, Mohammad (Not The Teddy Bear) Al Fayed has started selling Life Mel honey in his large corner shop at the very reasonable price of £42 for a small jar.

The reason that this product commands a high price is because (apart from beeing in Harrod's) there is some research to show that Life Mel honey can help patients having chemotherapy avoid a complication called neutropenia. This is caused by a lack of neutrophils (a type of white blood cell which serve as a defence against infections), which when low (as can happen during chemotherapy) can compromise the whole immune system and can be life-threatening.

Sounds like good stuff - tell me more. Firstly, the advertising blurb - it's a nice website, we get directed to Holywell Health products, the distributor of Life Mel in the UK, which proudly claims the research published in Medical Oncology and has kindly put a link to the .pdf of the paper.

Zidan J, Shetver L, Gershuny A, Abzah A, Tamam S, Stein M, Friedman E.;
Prevention of chemotherapy-induced neutropenia by special honey intake.
Medical Oncology 2006;23(4):549-52.

So far, full marks. How do we tackle the paper?

What Journal is the paper published in and is it any good?

If I started up Dr*T's Journal of Stuff, I could print what I like, and apart from a silly name, no-one would know whether it was any good or not.

To find out a bit more about a journal, start at PubMed. Medical Oncology is on there, Dr*T's Journal of Stuff isn't. If the research journal is not on PubMed, I wouldn't bother reading the paper. Med. Oncol is a peer-reviewed journal, so no qualms there. On to the actual paper...

How many people were in the trial and how many dropouts were there?

(Technically two questions, but no matter). This is called the 'n'. For 'number'. Clever. How many patients took the treatment? The bigger the n the more reliable the result.

e.g. I flip a coin twice, it comes up heads both times.
Result: It is a two-headed coin. (n=2)

I flip a coin 100 times, it comes up heads 100 times.
Result: It is a two-headed coin (n=100).

There is no hard and fast rule to this, but I would reckon that the general consensus of sciencey types would say that n<100 is not much cop. It may be a small scale trial which could then justify a larger scale trial, but the trial by itself should not be relied on too heavily.

In this case, n=30, so the results in this paper are already hanging by a shoogly hook.

The dropout number is also important. Without going into details of bias, it's a bit like saying in our experiment above "I flipped the coin 100 times, but I only looked at the result on 10 occasions". In this case, as the patients were being given the honey as well as their medication, you wouldn't expect dropouts and indeed all 30 completed the course.

What was the control and what was the active ingredient?

It's important to know what is being tested. In this case, Life Mel honey is being tested. So (in my simple head) one would expect the basic trial to be set up so that some patients got Life Mel honey, some patients got 'every day' supermarket honey, some people got a sugar syrup and some people got nuthin'. Because this particular experiment is in addition to the regular medication and treatment that the patients should be getting, there's no reason why it couldn't be done cheaply and easily.

We need to know if an effect is seen whether it is because of a placebo effect (compare with people who got nothing and with sugar syrup). If not, is it because of something inherent in honey or something inherent in that honey?

This is where the paper crumbles. There is no control - all 30 patients in the trial received honey. So there is no way of knowing if the response seen is because of placebo effect or some other feature of the experiment unrelated to honey.

There are other questions to ask (which will get added here in time) but for me, this paper is passingly interesting but of no value without further research.

Bear in mind that we haven't really read the paper, or gone through the biology of the process or the analysed the methodology - just a few easy questions has weighed up this paper as being very much in the "nothing more than PR" camp.

It may be the case that further investigation will lead to show some effect with everyday honey, or even Life Mel honey, but with the evidence given, it isn't possible to make the call.

However, as a salesman, I think Life Mel have done just enough to market the product to the vast majority of people and have done well. The 20 people who will actually see the research as being of little value without follow up work (it was published 2 years ago) will not be the target market. Especially now they have celebrity endorsements and a full raft of papers running with the story, including The Daily Mail taking it to extremes - personal anecdotes sell product.

I wonder will we see the large scale trial being published in due course?

I'd be really interested if they do, but I am, as ever, sceptical.

If you have further questions that you think could be of benefit in weighing up a published paper, please feel free to leave commment - thanks.


  1. Do you think that a journal citation index is a good way to determine if a n article is worthwhile?
    I appreciate that Medical Oncology is a relatively small and specialist journal, but a paper in big hitting journals like Nature or Science would, one hopes, have sufficient peer review to consider the question 'is there a control?'.

  2. True, but Science/Nature type papers rarely (although I stand to be corrected) will put through trials based on a commercial product aimed at providing scientific backing to the product.

    Bear in mind the questions above are aimed at people who won't necessarily make head nor tail of 50% of the paper, but will still be able to make a decision on its worth based on a few simple markers.


  3. Oh, and thanks for leaving a comment :)

  4. It's funny you should bring this up today - I've just posted a blog myself looking at claims surrounding a Cuban product called Policosanol (see http://layscience.net/?q=node/73).

    It's particularly relevant to your post because various people on Wikipedia added about 10 references to scientific studies showing the product worked, which upped the debunking stakes a bit.

    If you read my article (go on!) there are three things worth adding to your list before the person even reads the paper itself.

    1) Who are the authors, and what interests to they have. For example, the studies supporting this drug turned out to be funded by a Cuban public body set up to promote the product abroad in order to raise funds for the Cuban health system.

    2) Is the paper actually a trial? Some of the papers cited simply reported or commented on existing results.

    3) Does the abstract support the assertion made when referencing the paper? For example, one of the papers given as supporting Policosanol actually concluded that it had no effect - made clear in the abstract.

    So I was able to demonstrate that the list of references supplied was invalid before I even got to actually reading any of the papers themselves.

    Hope this helps, keep up the good work.

  5. Thanks Martin - I'm really pleased you left a comment!

    I had already read your article on Policosanol this morning and thought they complemented each other well.

    The additional questions you ask are very relevant, especially when meddling in the messy world of CAM journals - the abstract can sometimes be at odds with the "positive" spin put on it by the reporter.


  6. Thank you very much for an excellent article. It very well addresses some of the major pitfalls in studies like the one about Life Mel. And you are absolutely right in your conclusions: a group of 30 is too small and the controls are definetly insufficient.

  7. Dr Teeeeeee
    Love your work. Always have. ;o)
    Incidentally, Id be very worried if some common bee's honey was found to increase neutrophil counts in neutropenia patients (however much they charge for it). Although boosting differentiation of neutrophils from bone marrow stem cells may be useful for patients who have none (from chemotherapy or for other reasons), it is possible to very quickly enter an oncogenic state with chronic dosing leading to, amongst other things, chronic mylogenous leukemia (as has been shown with the clinical treatment for neutropenia, recombinant GCSF (reference not included)). If this honey does indeed have an effect over and above placebo, its long-term study in healthy and neutropenic patients should be carefully assessed. Thinking is very dangerous...
    Cheers, Dr G

  8. Fantastic post - relevant and made me giggle.