I've been wanting to write something along these lines for a while and the opportunity seems to have presented itself at last. (Which is poor timing, as I won't have time to tell you about the teapot-worshipper who is going to prison for not abandoning any teachings contrary to Islam).
Wading through PR blurb and advertising copy that relate to 'sciencey' things can be an effort, but with a few key pointers, sense can be made of most of it - for instance, a company will put the strongest claim possible on its strapline, so woolly words like 'may', 'could', help', 'aid', 'reduce', 'some' all go some way to indicate how much of the products claims are real and how much are puff (perhaps a blog post for another day).
Things can get much trickier though, if the advert in question claims "proved by scientific research". If there is grounds for this, the company will no doubt give full access to the research, otherwise, it's fairly safe to assume that it is selling a bum steer.
So far, so straightforward. Let's say our imaginary company has a product, which claims to have scientific proof for its claims AND is presenting the research. Obviously, advertising will already have convinced/dissuaded a section of public; the "proved by research" strapline will convince another swathe and the presentation of gritty full scale science research paper with all the reader-friendly niceties (and pictures) removed, will certainly convince another swathe, especially if the reader is not used to dealing with scientific papers.
If you are like me (and if you are, I empathise with your shortcomings) then, depending on the product, the paper may or may not be read. It all depends on the claim. If a toothpaste claims to whiten my yellowing teeth, the paper may not get read. If a nutriceutical claims to alleviate my arthritis pain, then I am reaching for my intelligence glasses. (I don't wear glasses, but pretending to makes me feel more intelligent).
I have some experience in reading scientific papers. But only really in my area, outwith of that, it can be tough going. What hope is there for someone who is not used at all to dealing with the curt, unloving, perfunctory monotones of thick research text?
With a bit of persuasion, I hope to have here Dr* T's Bluffer's Guide to Published Research. This will probably be a work in progress, so I'm keen to hear of any additional points which will make it easier for people to wade through.
What I've decided to do is use a paper I've been reading recently as a case study.
The product in question is Life Mel Honey.
As a beekeeper, I was interested to learn the nation's most paranoid shopkeeper, Mohammad (Not The Teddy Bear) Al Fayed has started selling Life Mel honey in his large corner shop at the very reasonable price of £42 for a small jar.
The reason that this product commands a high price is because (apart from beeing in Harrod's) there is some research to show that Life Mel honey can help patients having chemotherapy avoid a complication called neutropenia. This is caused by a lack of neutrophils (a type of white blood cell which serve as a defence against infections), which when low (as can happen during chemotherapy) can compromise the whole immune system and can be life-threatening.
Sounds like good stuff - tell me more. Firstly, the advertising blurb - it's a nice website, we get directed to Holywell Health products, the distributor of Life Mel in the UK, which proudly claims the research published in Medical Oncology and has kindly put a link to the .pdf of the paper.
Zidan J, Shetver L, Gershuny A, Abzah A, Tamam S, Stein M, Friedman E.;
Prevention of chemotherapy-induced neutropenia by special honey intake.
Medical Oncology 2006;23(4):549-52.
So far, full marks. How do we tackle the paper?
What Journal is the paper published in and is it any good?
If I started up Dr*T's Journal of Stuff, I could print what I like, and apart from a silly name, no-one would know whether it was any good or not.
To find out a bit more about a journal, start at PubMed. Medical Oncology is on there, Dr*T's Journal of Stuff isn't. If the research journal is not on PubMed, I wouldn't bother reading the paper. Med. Oncol is a peer-reviewed journal, so no qualms there. On to the actual paper...
How many people were in the trial and how many dropouts were there?
(Technically two questions, but no matter). This is called the 'n'. For 'number'. Clever. How many patients took the treatment? The bigger the n the more reliable the result.
e.g. I flip a coin twice, it comes up heads both times.
Result: It is a two-headed coin. (n=2)
I flip a coin 100 times, it comes up heads 100 times.
Result: It is a two-headed coin (n=100).
There is no hard and fast rule to this, but I would reckon that the general consensus of sciencey types would say that n<100 is not much cop. It may be a small scale trial which could then justify a larger scale trial, but the trial by itself should not be relied on too heavily.
In this case, n=30, so the results in this paper are already hanging by a shoogly hook.
The dropout number is also important. Without going into details of bias, it's a bit like saying in our experiment above "I flipped the coin 100 times, but I only looked at the result on 10 occasions". In this case, as the patients were being given the honey as well as their medication, you wouldn't expect dropouts and indeed all 30 completed the course.
What was the control and what was the active ingredient?
It's important to know what is being tested. In this case, Life Mel honey is being tested. So (in my simple head) one would expect the basic trial to be set up so that some patients got Life Mel honey, some patients got 'every day' supermarket honey, some people got a sugar syrup and some people got nuthin'. Because this particular experiment is in addition to the regular medication and treatment that the patients should be getting, there's no reason why it couldn't be done cheaply and easily.
We need to know if an effect is seen whether it is because of a placebo effect (compare with people who got nothing and with sugar syrup). If not, is it because of something inherent in honey or something inherent in that honey?
This is where the paper crumbles. There is no control - all 30 patients in the trial received honey. So there is no way of knowing if the response seen is because of placebo effect or some other feature of the experiment unrelated to honey.
There are other questions to ask (which will get added here in time) but for me, this paper is passingly interesting but of no value without further research.
Bear in mind that we haven't really read the paper, or gone through the biology of the process or the analysed the methodology - just a few easy questions has weighed up this paper as being very much in the "nothing more than PR" camp.
It may be the case that further investigation will lead to show some effect with everyday honey, or even Life Mel honey, but with the evidence given, it isn't possible to make the call.
However, as a salesman, I think Life Mel have done just enough to market the product to the vast majority of people and have done well. The 20 people who will actually see the research as being of little value without follow up work (it was published 2 years ago) will not be the target market. Especially now they have celebrity endorsements and a full raft of papers running with the story, including The Daily Mail taking it to extremes - personal anecdotes sell product.
I wonder will we see the large scale trial being published in due course?
I'd be really interested if they do, but I am, as ever, sceptical.
If you have further questions that you think could be of benefit in weighing up a published paper, please feel free to leave commment - thanks.