Why randomised controlled trials don’t tell you what you want to know

Earlier this week the Daily Mail  published my feature on side-effects and how patients aren’t properly warned about them. Antidepressants, for instance, can cause compulsive heavy drinking but you wouldn’t know it from the drug information leaflet. 

The article is about campaigning psychiatrist Dr David Healy, who believes patients need a more truthful account of the side effects they risk. It describes his new website – Rxisk.org – that makes it easier to report side-effects and provides a forum where you can swap experiences with other pill users.

Information about side effects is often poor because it gets  hidden by the companies running the trials. A campaign by doctors and journals is now well under way to force drug companies to be more transparent.

Turning garbage into gold

But that’s not enough, according to Healy. He believes there are serious problems with the randomised controlled trial (RCT), the so-called gold standard of evidence based medicine, used to test treatments and decide which ones should be licenced.

Rather than distinguishing safe and effective medical treatments from ones that are dangerous and/or useless, the results from RCTs are frequently misleading and wrong. The RCT, he says, is a processor for turning garbage data into drug company gold.

At first sight his critique appears ridiculous. It is a full-frontal challenge to the whole idea of evidence based medicine. You have to have a way of telling if a drug works and is safe and the RCT has been used for years – since the 1960’s in fact – when it was introduced as a response to the thalidomide disaster.

It works by dividing patients randomly into two groups; one lot gets the drug, the other an inert placebo pill. A few months later the results reveal which group benefited more. Two positive RCTs are needed to get a licence to market a drug.

A serious barrier to treating chronic disease

Healy’s case against RCT’s is not based on bare-faced fiddling of results by the drugs companies, although he’s often exposed it. Even if all trials were squeaky clean they would still be a serious barrier to developing really effective ways of tackling the various lifestyle diseases that are threatening to cripple Western health services.

Take the rule that you only need two RCT’s to get a licence. What about if you also have three negative ones where the drug came out as no better than a placebo? At the moment they simply don’t count. This has everything to do with bureaucratic rules and nothing with scientific rigour. Its an arbitrary rule that deprives doctors and patients of valuable information about what the drug does or, just as importantly, doesn’t do.

But there is more fundamental problem.  Properly conducted RCTs are supposed to tell you if a drug is effective. But what does “effective” mean? The poster boy for the industry is the antibiotic – a drug that rapidly cures the problem in the great majority of people who take it.

Most drugs aren’t like antibiotics

That would be a good definition of “effective” but that’s not what most drugs that come onto the market today do. (Ironically antibiotics worked so well when they were initially developed they didn’t require an RCT to show effectiveness. Today the fact they are so effective means companies have stopped making them; a quick cure is not profitable enough)

The RCT system finds many drugs effective not because they cure anything but because they change some marker or risk factor for disease, such as reducing your cholesterol level or the size of your tumour. But often changing these markers has no effect on long term outcome. For instance Ezetimibe is very effective at lowering cholesterol; however there is no evidence it has any effect on your chances of developing heart disease.

It’s assumed that if a drug has been found to be effective in an RCT, it will be just as effective when it is handed out in the doctor’s surgery. NICE, for example, uses data from RCTs to decide if a drug is cost effective. RCTs are also used to estimate safety when a drug becomes widely used. The list of side effects set out in the Patient Information Leaflet contains those reported in trials.

Dangers of drugs in the real world

But the few attempts to test these assumptions show them to be false.  An RCT is a highly specialised set up: the patients are carefully selected. Usually they are younger and healthier than the people who will actually be taking drug. They will only have one thing wrong with them and only be taking that one drug. People in the real world will be older and very likely have two or three conditions for which they are also taking five or more drugs.

A study comparing the cost effectiveness of a drug based on the results from RCT trials with results in a clinic found they were about  five times less effective.  This not only raises the issue of effectiveness – many widely used drugs only work for 25 to 40 per cent of patients according to RCTs  – but it also puts a big question mark over safety.

Just as effectiveness drops outside the rarefied world of RCTs, the number and variety of harmful side-effects inevitably shoots up. But official figures on how safe and effective a drugs is come from the RCT data. In the real world that balance can have dramatically changed.

Letting in dangerous and incompetent characters

Ultimately though the problem is that an RCT is a handy tool, especially for finding out what doesn’t work, that has been elevated way above its pay grade. It’s not up to the job of being the gate keeper for evidence based medicine.

It has let in all sorts of dangerous and incompetent characters and it’s a totally inappropriate tool for assessing treatments that are much  more sophisticated, individually targeted and flexible than 5 or 10 mg of a drug.

Even worse, the RCT ignores how medicine really works by undermining doctors’ clinical judgements. The RCT system we have operates on the unspoken assumption that drugs found to be effective automatically fall into the Magic Bullet class. Just like antibiotics, they target the problem and work whatever the situation a patient is in.

But as Healy points out, most drugs, such as painkillers, tranquillisers or anti-inflammatories like aspirin, work within a “therapeutic relationship”. They aren’t cures but they can help, along with a range of other things, depending on all sorts of factors in the patient’s life, from genes to family relationships.

Delusion of the magic bullet

But there is a strong tendency for drugs passed by the RCT magic bullet model, to be seen as automatically effective and appropriate for everyone. They are fire-and-forget treatments to be used according to guidelines.

In the kind of personalised medicine Healy and many other doctors are campaigning for,  a clinician would say something like: “This treatment has an effect which may be beneficial for you, taking into account everything else going on in your life.” The RCT magic bullet approach says: “This treatment has been proved to be effective and so it’s appropriate for you.”

But the one-size-fits-all approach is going to become increasingly irrelevant in the face of an explosion of genetic information pinpointing detailed individual differences and the ways genes and environment interact to affect health. It’s also not an effective way of assessing the preventative and tailored life-style changes needed to deal with the epidemic of chronic metabolic disorders. We need to come off the gold standard.



  1. “The RCT system we have operates on the unspoken assumption that drugs found to be effective automatically fall into the Magic Bullet class. Just like antibiotics, they target the problem and work whatever the situation a patient is in.”

    Of course, even some antibiotics aren’t such ‘magic bullets’.

  2. For anyone who might like to find out more about the mathematical and philosophical arguments around the RTC gold standard, and evidence-based medical concepts there is a very helpful article titled “Every Good Doctor Must Represent the Patient, the malfunction of evidence based medicine.”
    to be found on the orthomolecular website. { Their website header reads ” Therapeutic nutrition based upon biochemical individuality.” }
    The article is about the systems analysis theorem [ Conant and Ashby, 1970 ] which gives a mathematical set theory proof of the theorem – Every good regulator must model the system it is trying to fix or regulate in some way – i.e. an individual lock needs its individual key. Or – there is no such thing as a universal key for all locks. And therefore there is no such thing as a universal magic bullet to cure a given disease in every patient.
    There are some very good attempts to make the proof of this idea more accessible to people like us who have not got the maths at that level – at all…:-}. and the orthomolecular article sets the theorem out clearly and in terms of us,the systems, and our medics, the regulators, with starter references.

    Here are 2 quotes from the article: the first concerns a book from 2011, heavily critical of RTC’s
    Hickey and Roberts, Tarnished Gold, The second is the blurb describing the books subject on its amazon page, perhaps a quote from the book.

    1] “Hickey and Roberts emphasize that it is not simply the symptoms that matter. Also important is the particular person in which those symptoms occur, where the particularities of that person have been determined by the complex interactions between that person’s genes and the environments in which those genes have been expressed over the person’s lifetime. In their discussion of this notion of “biochemical individuality”, Hickey and Roberts cite Williams, R., 1998 (1956), Biochemical Individuality: Basis for the Genetotrophic Concept, McGraw-Hill, New York.”

    2] “Evidence-based medicine, the “gold standard” of medical decision making, is increasingly unpopular with clinicians. They are right to have reservations. EBM breaks the laws of so many disciplines that it cannot be considered scientific or even rational. Decision science and cybernetics show the disturbing consequences of such flaws. EBM fosters marginally effective treatments, based on population averages rather than individual need. Its mega-trials are theoretically incapable of finding the causes of disease, yet swallow up research funds. Ultimately, EBM cannot avoid risking patients’ health. It is time for medical practitioners to discard EBM’s tarnished gold standard, reclaim their clinical autonomy, and provide individualised treatments to patients. This book explains why and how.”

    I found it very helpful/comforting to know that even if I can’t grasp the maths and set theory properly , there is a stonking/gold standard proof that the medics and big pharma are too deeply flawed to continue unchallenged..

    • Many thanks for that I think there is a great need to develop better and less inflexible and misleading ways of gathering evidence. I’m going to look into this

    • This is why Homeopathic methodology and diagnosis is so brilliant. The focus is on the particularity of the individual divined from studying every single symptom possible from the trivial up…. everything is important and everything is part of the picture of the individual in terms of treating that individual so a return to optimal health can be triggered.

      The mere fact that every one of us has our own fingerprints, a unique signature, makes the entire modus operandi of science/medicine deeply flawed in that it works with, to and for some sort of generic human manifesting as material, mechanical and now chemical. All of those factors are interesting and at times useful but they will never allow individualised medical treatment which does more good than any harm.

  3. Both pharmaceuticals and medicine become dangerous for the same reason – each is sourced in and operates from a belief that everything, including the human body and condition, can be reduced to the material and the mechanical, with the body treated as if it were no more than a machine/and/or bag of chemicals.

    There is no generic human and disease is not purely material or physiological but the human organism is a complex entity where health and disease emanate from a little understood system which involves material, mental, physiological, psychological, emotional and spiritual which may be reacting to or influenced by circumstantial and environmental.

    There is no disease which operates exactly the same in all although some may have much that is shared and there is no one source of disease nor one method of cure. There is no one-size or one-pill fits all and as long as modern science/medicine believes that there is, or that there can be, the death and damage will continue.

    The mere fact that iatrogenic – doctor or medical induced – with drugs the main cause, is now up there with the top killers, should make anyone question how modern medical treatment is functioning. Does it kill or injure more than it cures? And even if the answer is No, surely the act that it is up there with the ‘best’ of them, killing and harming, means that something is very, very wrong.

    Would there be less death and damage, less harm done, if science/medicine were not profit and prestige-driven? I am sure there would. But it is profit and prestige-driven and within those curses are also gifts for that which drives science/medicine, for good and ill, also provides the vulnerability which may bring and hold it to account.

    For, if any other industry or profession were doing so much harm there would be outrage. The tobacco industry was brought to its knees because of its kill and injure rate and with iatrogenic death and hospitalization in the millions around the world every year, why are so many so silent?

    They are silent because science has become the new religion and medicine its priests and because science/medicine also has the capacity to exploit the fears of humanity because, despite doing great harm, far more harm than it should, it also offers hope, help and life. But that is an excuse, not a reason and because science/medicine has such power to manipulate and exploit is why it should be held accountable even more rigorously than any other…. or at least, as rigorously which is not the case at present.

    • Thanks for that. Getting from here to a more sane system is the tricky part

      • I believe we will, for two reasons. One is that human beings are not generally stupid and while science/medicine likes to say they are on that which they have deemed to be irrational like non-Allopathic medical methodologies, the fact is people ‘speak’ with their actions and billions are turning away from Allopathic medicine on many counts although most, utilise it, quite sensibly, as and where it is generally effective and does little or no harm.

        And Two, in a litigious world where profit can be painfully reduced at personal or professional levels, there is no doubt that while many are quick to sue now, there will be more who do so in the future. There is no pain so hard to bear as loss of profit and prestige.

        And probably there I a third and that is, that while science as a system is compromised and limited by its current paradigm of the material and mechanistic, there is nothing so constant in life as change and there will always be some scientists who ask questions, have open minds and strive to move forward as we saw with the development of quantum physics and as we see today with advances in biophysics.

        So it will change, there is no doubt of that – the only question is how fast?

  4. asad choudhry M.D. says:

    Lol yea I figured as much that’s why meds don’t work anything like its presented as in studies on the front lines of the hospital ward – big difference then what the proposed benefits versus what actually happens to the majority of patients. RCT thumbs down, not good enough.


  1. […] Why randomised controlled trials don’t tell you what you want to know […]

Leave a reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: