What is wrong with randomised trials Part 2

Could alcohol get a licence as a drug for depression? How do you test for the safety of a drug that causes the same side effects as the disease it is used to treat? These are just two of the points I didn’t have room for in my post last week on randomised controlled trials (RCTs) and why they don’t tell you what you want to know. (More on these points below.)

The post sparked quite a lot of twitter interest, praise mixed in with less flattering comments. One tweeted that I was “a quack and a crap journalist” (quickly withdrawn), another just commented “oh dear, oh dear” at the mention of my name.  None of the critics addressed any of the issues preferring to imply the problems were all known and fixable.

So I’m coming back to RCTs this week because I think their flaws need more serious attention. At first sight RCTs appear very straightforward and an obviously good thing – two groups, one gets the real thing the other gets a pretend version (placebo) and that tells you if the treatment is effective. On closer inspection, however, they turn out to be rather more slippery and open to all sorts of misleading manipulation.

By a useful coincidence a particularly vivid example of the slipperiness of this so called “gold standard” for evidence based medicine arrived in my mail box yesterday. Sales reps are supposed to accentuate the positive but those promoting pharmaceutical drugs would make Candide look gloomy. Researchers filmed the sales pitch (how did they get permission?) by reps to 255 doctors in America, Canada and France and then rated how accurately they represented the drugs’ known side effects.

In how many of the interactions do you think the reps provided ““minimally adequate safety information”? Precisely 1.7 per cent! And these weren’t drugs usually described as “well tolerated”. Nearly half already had formal warnings of serious side effects yet these were mentioned in just 6 per cent of the interactions. This is bad enough; even more alarming for patient safety was that the doctors thought they were getting reliable advice. They rated quality of the scientific evidence they were given as good or excellent in half of the presentations.

Supporters of the system, who assert fiercely that RCTs are the best way to distinguish between “real” medicine and the quack stuff, increasingly admit that yes there are short comings, that companies do hide unfavourable results and fiddle statistics (and presumably now, that pharma reps can be economical with the truth) but that all this is fixable. We just need to enforce the rules properly and punish offenders – like sorting out the banks – and then it will all work fine.

Licence alcohol as an antidepressant

But the criticisms made by psychiatrist Dr David Healy and others that I highlighted last week suggest that it is not simple. Even if RCTs were policed with the rigour shown to alcohol in Nordic countries, they would still routinely fail to reveal important information.

Such as whether a side effect is the result of the disease or the drug. If you are depressed you are more likely to have suicidal thoughts and try to kill yourself but it has emerged slowly, long denied by the regulator and drug companies, that SSRIs can have the same effects. This is an issue that Healy has campaigned on for years.

He illustrates the issue with this thought experiment. If alcohol were a new drug, it would be perfectly possible to use RCTs to get it licenced as an antidepressant. Patients rate their level depression at the beginning of a trial on a 20-point scale. Existing drugs are said to be effective because they typically drop your rating by around 2 to 3 points and for a while alcohol would be likely to do that easily. But it would also quickly improve mood far better than a placebo.

None of the longer term side effect of liver damage, addiction or brain damage would show up in the brief six to twelve weeks that trials typically run. If mood dropped a few hours after a dose in the trials that would be a sign of how beneficial the treatment was.  Any need to increase the dose or any difficulties caused by trying to stop the treatment would not be signs of addiction but evidence  that the underlying disease was re-emerging.

Licence for ineffective drug

One way to distinguish between symptoms and side effects would be to give a drug to people who aren’t ill and so unlikely to suffer symptoms of the disease it was targeting.  This is done in so called “healthy volunteer” or phase 1 trials when a drug is tested on humans for the first time.  Yet they have nothing to do with distinguishing side-effects from symptoms.

They are designed to suit company purposes, says Healy, such as looking at a drug half-life. What’s more, full data from them is never released; they are never registered and often never published.

Another timely coincidence, illustrating the way the RCT system can allow drugs onto the market even without any evidence of benefit, was the announcement last week that a combination of a statin drug and one that also lowers cholesterol called Ezetimibe would be given a licence.

Regular readers will know that even though Ezetimibe has been on the market for a decade and many trials show it is very efficient at lowering cholesterol, it’s never been shown to actually cut anyone’s risk of having cardiovascular disease.  There is no suggestion of improper drug company behaviour here, that is just the way the system can work. We deserve something better.