Monday, November 17, 2008

More Omega 3 child testing nonsense - this time, New Zealand.

"The outcome I would be really happy with is a better understanding of healthy eating", said teacher Paul Whitaker, from Auckland's Wellsford School, according to TV.NZ.

What I assume he meant is "The outcome I would be really happy with is a better understanding of the scientific method, placebos and possibly the Hawthorne Effect".

It appears that after watching a BBC documentary on Omega 3 fish oils, Mr Whitaker decided to run a trial of his own: 42 pupils, 21 with a fish oil pill and 21 without, otherwise, everything was the same. The story was also picked up (slightly, but only slightly more sensibly) in The New Zealand Herald.

Now, Mr Whitaker makes a big enough song and dance about his results to allow me to pilliory him as a bit of a fool. It would be unkind to say that not much happens in NZ, and indeed this is no excuse. History repeated itself in many ways with the ghost of the nonsensical Durham Fish Oil trial/experiment/initiative/trial (still being admirably hounded by MacCruiskeen) being resurrected - the pills were given free to the school by Good Health, a NZ-based 'natural health solutions' company, who are happy to tout the trial as a success (despite not being finished yet) and give a list of some of the media exposure gained. (Save lazy people like me having to do too much Google trawling).

In a very similar case to Durham, the trial was poorly designed scientifically and a waste of time, although excellently designed from a point of view of getting a false positive with which to spin to the media.

First of all, 42 people is never enough to get any meaningful results - at best it may give an inkling as to the next direction your research will take.

Secondly, the person running the experiment knows everything about who is getting what and how well they are expected to do etc etc. This is where blinding normally comes into it. The amount of personal bias put into this trial (both unwittingly and perhaps wittingly) makes any results valueless. Did Mr Whitaker bias the choice of pupils in each category so that the trial would be a success? Who knows and indeed who cares, because there is so much more wrong with this trial.

Thirdly, what about a placebo? No placebo makes the trial an investigation into the Hawthorne effect - people perform differently when they know they are under test. You remove the problem by using a placebo.

Fourthly, the results are interpreted by Mr Whitaker - again the bias is open to his perception.
He said the 21 students taking the capsules for two months [...] were now six months to one year above where they were before they began taking Omega 3.

Really? They had moved a year ahead in 2 months? Read over that again - they had moved ahead by 1 YEAR in 2 MONTHS. That is a result. That is a HUGE result. Imagine the money that could be saved by shovelling fish oil down all these kids necks! IMAGINE! It's clear that this is just not the case.

Fifthly, why is the trial being touted as a success when it still has months to run?
Whitaker's experiment is expected to continue until the end of the academic year in December.
Why is this being treated as a media press release and not a scientific paper? The answers are pretty obvious, of course.

Now, you may think that this posting is an obsessive rant regarding a fairly small study in New Zealand, a bit of publicity and PR spin and not much more than a bit of fun. Well, I object to the following:

1. Children being tested on (especially when there is no possible knowledge gain, and only for a pill company to sell more pills)
2. The public's understanding of science to be further cheapened.
3. The media again showing no basic scientific criticism because they are scientifically illiterate and unable to carry out journalism as opposed to press release restructuring.
4. Lying to children by telling them the reason they have got good results is nothing to do with their hard work, but due to a pill and that's the way society works, so take your foot off the gas, kick back and get on the vitamins.
5. Lying to children by telling them the reason they are not progressing is because they aren't taking pills and not because they are not as smart as other kids or have some learning difficulties.
6. The medicalisation of "intelligence" as something which can be found in a bottle of pills.
7. Pill companies benefiting from all the above, and using the nonsense to pretend their brand of placebo is 'scientifically proven'.

No doubt this will prompt someone to do a formal trial of the fish oil pills. I wonder how meticulous they'll be? One can only assume that (especially with Durham) there is no interest in actually finding out whether or not there is an effect. A negative result would have too much of an impact on the bottom line.

BPSDB

5 comments:

  1. "[T]he person running the experiment knows everything about who is getting what and how well they are expected to do etc etc...

    [T]he results are interpreted by Mr Whitaker - again the bias is open to his perception."
    This puts me in mind of the classic Rosenthal and Jacobsen study, Pygmalion in the Classroom.
    " Rosenthal and Jacobson raised the possibility that the "experimenter expectancy effect" (in other words, "what you expect is what you get") may also be present in school classrooms. The book 'Pygmalion in the Classroom: Teacher expectation and pupils' intellectual development' describes a study carried out in the 1960s by Rosenthal and Jacobson in an elementary school (which the authors call "Oak School") to test the hypothesis that in any given classroom there is a correlation between teachers' expectations and students' achievement. In this study, Rosenthal and Jacobson gave an intelligence test to all of the students at an elementary school at the beginning of the school year. Then, they randomly selected 20 percent of the students - without any relation to their test results - and reported to the teachers that these 20% of 'average' students were showing "unusual potential for intellectual growth" and could be expected to "bloom" in their academic performance by the end of the year. Eight months later, at the end of the academic year, they came back and re-tested all the students. Those labeled as "intelligent" children showed significantly greater increase in the new tests than the other children who were not singled out for the teachers' attention. This means that "the change in the teachers' expectations regarding the intellectual performance of these allegedly 'special' children had led to an actual change in the intellectual performance of these randomly selected children" (p. viii).

    The teachers were also asked to rate students on variables related to intellectual curiosity, personal and social adjustment, and need for social approval. In what can be interpreted as a 'benign cycle,' those average children who were expected to bloom intellectually were rated by teachers as more intellectually curious, happier, and in less need for social approval."

    ReplyDelete
  2. You may be interested in this assessment of fish oil supplements by Dr Alex Richardson.

    One of the many reasons that tiny results from various badly-conducted trials mean nothing. Even interesting results from larger, good-quality trials mean little if the product that is being tested bears little resemblance to the product that is available to consumers.

    ReplyDelete
  3. That so-called trial was unscientific all round. Selection of pupils depended on parental permission, the teacher wanted a specific result, etc. The school acknowledged that it wasn't a scientific trial. Sop the question is why do it.

    The sad thing for me is that it could have been an opportunity to educate the pupils in the scientific method by using a good design and telling the students why. Instead they will probably have given the students a completely wrong impression of what science is.

    ReplyDelete
  4. Dr*T, you're on fire!! cracking post, the list of objections needs to be printed out and pinned up all over every single press office and journalists' desk. IMO this sort of 'testing' is immoral and downright dangerous - more effort ought to be put in to preventing such 'studies' from being performed to avoid the very things you list.

    @ openparachute: spot on. given a golden opportunity to enlighten a generation of kids about the ins and outs of scientific investigation, this kind of fudge only serves to bring the discipline into disrepute. Won't somebody please think of the children...?!

    ReplyDelete
  5. Someone needs to collate these fish oil pill initiatives. Equazen/Portwood was the first one I heard about, Efamol/Stordy was the second, and I've lost track of the others.

    ReplyDelete

Share it