Tuesday, March 25, 2008

Bluffer's Guide to Consumer-related Science Papers - Part 2

In Part 1, we looked at a few basic questions to ask when looking at a scientific paper to establish whether the research is much cop.

The questions were as follows:

1. What Journal is the paper published in and is it any good?
2. How many people were in the trial and how many dropouts were there?
3. What was the control and what was the active ingredient?

And one suggested by Martin @ LayScience:

4. Who are the authors, and how were they funded?

The paper that has thrust itself recently into the headlines is this one:

Cod Liver Oil (n-3 fatty acids) as an (sic) non-steroidal anti-inflammatory drug sparing agent in rheumatoid arthritis

B. Galarraga et al, Rheumatology, 2008, EPub [Ahead of Print]

The reason why this paper is a good one for learning how to read scientific papers should become apparent as we go on. Needless to say, the Dundee University well-oiled press machine is working at full strength (remember "Black Tea Helps Combat Diabetes"?) and so the story has been picked up by the BBC and Daily Telegraph amongst many others. The prize for the worst headline, incidentally, goes somewhat surprisingly to Channel 4 news who fell into the age-old trap of confusing the pain caused by the disease with the disease itself (Cod Liver Oil Does Ease Arthritis) - even the Daily Mail managed to stay out of that trap, but happily feeds the press release to the ScienceMangle™ and prints whatever comes out.

In essence, the message is taking cod liver oil capsules can mean cutting down on painkillers (and hence their side effects) for people with rheumatoid arthritis.

That, to me, if true, is a Good Thing. Let's look at the research:

Firstly, it has the golden words "This was a double-blind, randomised, placebo-controlled study". Music to the ears. Proper job.

1. What Journal is the paper published in and is it any good?

Rheumatology. No problems on this front - it is a peer-reviewed PubMed journal.

2. How many people were in the trial and how many dropouts were there?

Hmmm... no so easy to answer. The trial ran from at two locations from August 1997 to December 2002. (Why the dickens it has taken over 5 years to complete the analysis is beyond me). In total, 97 patients were part of the study. This is close enough to our n>100 rule of thumb for it to not matter so much, however, only 58 patients completed the study. Suddenly the results are looking shaky. Without going too much further, with n=58 it will be difficult to make firm conclusions, even if all the other parameters are fine. Why did so many drop out? Well, the paper gives a good critique (as it should) of why people dropped out. One reason I would like to press further (see below) is that about 20% of both Cod liver group and the placebo group dropped out because of voluntary withdrawal.
Amongst concerns raised by this group of patient were the large size an number of capsules to be taken daily, awareness that the capsules were empty and dislike of the fishy taste of the capsules.

Ah. This should have set alarm bells ringing. It will do when we look at the next question:

3. What was the control and what was the active ingredient?

The active ingredient is n-3 (also called omega-3) fatty acids found in cod liver oil (specifically, Seven Seas Marine Oil 1, an n-3 long chain fatty acid-rich clinical grade high strength cod liver oil).

The control was "an identical air-filled capsule". Hmmm. Identical to look at perhaps, but weight? Hence the worrying comment about "awareness the capsules were empty". And the fishyness? This to me is a shame.

The whole concept of double-blinded placebo controlled trials is based on the fact that neither the patient nor administerer (hence double-blind) knows which capsules are which. It turns out that both knew which was which - certainly, anyone burping up clouds of fishyness had a fairly good idea.

Why did the researchers not fill the capsules with a non-fish fatty acid, as these Dutch researchers did when trying to find out if n-3 fatty acids in fish oil prevent cardiac arrythmia - the placebos were a high oleic sunflower oil. Obviously the fishyness may still figure, but this to me is a basic cock-up. Again, in fairness to the researchers, they critique themselves admirably:

We may have compromised the double-blinding of the study by using air-filled capsules as placebo. Although it was recognised that some patients would discover their capsules to be empty and others may realise about the capsules lack of fishy smell and taste, the air-filled capsules were selected as being the most appropriate placebo available after critical appraisal of alternatives. The possibility of using capsules filled with other fatty acids was rejected as none are believed to be truly inert and saturated fats may be associated with a health risk.

I don't buy either of those two excuses. It sounds like (but may not be so) the peer-reviewers have insisted that there be something in the discussion about the double-blindedness of the trial. Perhaps it something else.......

4. Who are the authors, and how were they funded?

Aha. The authors themselves are not an issue, but the funding came from Seven Seas themselves. This does not need to be an issue, but it can provide a bit of fodder for discussion. For instance, if you had a product that you wanted to sell more of, it is important to get some research done and, on the back of the generated PR, sell more product. Providing the research is independent and of good quality, then it shouldn't matter where the money comes from. A sceptic might say that Seven Seas wanted the research to be good, but not that good.
Certainly, the effort over 10 years to gather data, randomise the trials, have 2 seperate centres, publish in a quality journal, team up with sound scientific researchers is commendable - many less upstanding companies might just have given the fish oil to a load of schools and told them to eat up without really doing it properly. And yet, to cock the blinding up so completely as to lower the worth of the paper from research to PR chaff is a sorry shame.

In fairness to the researchers and to Seven Seas, they published the research, held it up to scrutiny, paid for open access for the public (Rheumatology is normally pay-walled) and did find (in some small way) a small but significant reduction in self-reported pain (very susceptible to placebo) in the cod liver oil group compared to the placebo group.

Surprise surprise, the sample is too small and there are other 'issues' with the experimental to draw firm conclusions, but as we saw with Life Mel more research is needed, but the papers are already full of suitable PR that I'm sure Seven Seas will be very happy with.

I wonder will they ever get round to doing more research? If it took them 10 years to do this study, I wonder how long it would take to do a large one.

Of course, they don't mention (and why would they?) that 10 capsules a day of 1000mg High Strength cod liver oil is a LOT of pill swallowing. Neither do they mention that this will cost over £10 a week. And of course they don't mention that it would be much better to eat oily fish and by having your 4 portions a week as recommended by the Food Standards Agency you can a have a much more enjoyable time ingesting the omega-3s (and get a similar amount).

Seared tuna with fresh lime vs 10 cod liver oil capsules. No brainer.

(With thanks to Gimpy and JDC).

EDIT: I should also point out that the NHS Behind the Headlines page has covered this here


  1. Very nice. I'd been wondering about this study myself - the drop-out rate set my alarm bells ringing. Your analysis method is very similar to one suggested to me.

  2. Bit fishy. So, what should the average arthritis sufferer's shopping basket contain? Life Mel's honey (just in case), fish oil (proven), crystals (as long as they are green), glucosamine (hmmm), noni juice (what?), powdered badger testicles (obviously) - not sure I could get all that in Waitrose. Any other suggestions?