Professor Regan's Ice Bucket

I was impressed by the aims of a new BBC2 series called Professor Regan's Pharmacy. The idea is that Lesley Regan, clinical professor of surgery at Imperial College London, assesses a number of products that appear in pharmacies. She then grants those that meet her standards of evidence a position in her own (presumably hypothetical) pharmacy.

The second episode aired last Thursday at 9 and is available to watch here, at least for the moment. I assume there'll be another one on tonight.

This last episode was specifically about medicines. In general it was very good, and it's extremely commendable that the BBC would give over an hour of their peak slot to a show that promotes evidence based medicine and spreads the word that homeopathy is rubbish. Incredible, unlike many shows about science, when assessing these claims, they actually showed people reading papers and looking at tables of numbers rather than just standing in a white coat fiddling with test tubes.

However, I did find myself getting annoyed at a few things in the show. Like all programmes with even the most tangential relation to science, they performed their own small 'experiment' to demonstrate some of the findings they were talking about. And just like all programmes that do this, the experiment was entirely useless, when it could very easily have been a lot better.

The experiment was done in order to demonstrate the finding that name-brand painkillers have been found to be more effective at managing pain than cheaper generic brands. This is a fascinating bit of research that demonstrates the power of the placebo effect by showing the additional benefit afforded a flashier looking product over its medicinally identical but less well packaged equivalent.

They did this by getting 5 rugby players to see how long they could hold their hand in a bucket of ice, first after having taken what they thought was Nurofen, and second, after having had what they though was some own-brand ibuprofen. Their 'study' gave the expected result: that they could endure the cold for longer after they'd had what appeared to be a branded painkiller (it turned out they had taken the same pills both times).

However, their study really didn't show anything. For a start, they had a meaninglessly tiny sample, but more importantly, they each went through the two conditions in the same order. By the time they put their hands in the ice the second time, their hands probably still hurt from the first time, so of course they wouldn't last as long. There are many other more subtle order effects too. For example, in the first trial, they might have tried harder, at least partly out of a desire to beat the other subjects. The second time, there would be less of that feeling of competition and resolution.

I understand that certain concessions have to be made to make a programme watchable - nobody would sit through an hour of scientists sitting around designing a counterbalancing method to properly randomise their conditions or a 6-part series that consists entirely of watching people punching endless numbers into SPSS. However, this 'experiment' could have been made about 100% better if they'd just got another 5 subjects and got them to do the two conditions the other way around. Obviously it's not a study that would have been published in the Lancet, but at least the results could have been slightly meaningful. As it is, all they showed is that if you put your hand in a load of ice twice, you'll probably keep it there longer the first time.

In a way, the fact they bolstered their message with a meaningless bit of filler about some rugby players and a bucket of ice is irrelevant - the actual research that they were supporting already exists, they were just illustrating it. But I don't see the point of scrimping on that last little bit of detail that would have made it so much more useful. Especially given that the central idea of the show is that evidence and the scientific method is important.