Apologies for the hysterical headline but it’s just the kind of irresponsible, ill-informed screamer that regularly heads stories about some new study that “proves” that vitamins are ineffective generators of expensive urine and may well be dangerous.
If you want to know why such studies are all too often based on poor science and frequently unreliable, read no further. Go straight to the website HealthInsightUK.org where nutritionist Patrick Holford has posted a careful and detailed analysis of the latest “vits are rubbish” paper, widely publicised in the newspapers before Christmas.
Once you’ve read it you may still be sceptical about the benefits of vitamins but you will have to admit that you can’t use this piece of research to make the case. You might even feel an apology was due. (This is not to say that good research can’t reveal problems with vitamin use or suggest ways they can be used more effectively, but the broad brush attacks invariably turn out to be more polemic than proper science.
Painting a grim picture
Meanwhile I’m going revisit the report on the flu drug Tamiflu by the Commons Public Accounts Committee , which I wrote about here last week . The oral evidence from Dr Fiona Godlee, editor of the BMJ and Ben Goldacre, author of “Bad Pharma” paints a grim picture of the realities of drug regulation – as opposed to the official line that it is all careful, thorough and evidence based.
This in turn is highly relevant to the vitamin debate. Not because poor evidence for drugs means there is good evidence for all vitamin uses, but because if drug data is dodgy or downright deceptive, it makes sense to take that into account on a when personally weighing up the risks and benefits of drugs vs. vitamins.
Most coverage of the report ignored the revelations about inadequate regulation because only geeks and obsessive have the time to read such things. So courtesy of an obsessive, here are some quick-read highlights.
In theory evidence based medicine relies on testing drugs against a placebo in clinical trials to discover if they are safe and effective. In fact evidence gathered in this way is not worth much since it’s reliably estimated that at least half of drug company trials run on drugs we use today not been published, making a mockery of the principle. What’s even worse is that keeping unfavourable results hidden is not actually breaking the law.
You should be worried
As Goldacre commented: “The public should be worried that there is no competent legislation to ensure that doctors and researchers have access to all the methods and results of all the trials. (The result is) that we cannot practice evidence-based medicine in the way we purport to.” The implication is that evidence has become an arm of companies’ marketing division with limited scientific value.
As Godlee remarked: “The pharmaceutical industry has an irreducible conflict (between) science and marketing in the way it represents its drugs. Unless we can resolve it in the public interest I would argue that drug companies should not be allowed to evaluate their own products.”
Defenders of the system claim that failing to publish data isn’t such a problem since companies have a legal obligation to provide the drug watchdogs – the MHRA in the UK, the EMA in Europe – with all the data anyway.
Dr Godlee made it clear that this was another comforting fantasy. She explained that as a result of investigations by the Cochrane Collaboration (the research body that triggered the transparency campaign in Europe by demanding to see all the trials carried out by Roche on Tamiflu) it turned out that Roche had funded 123 completed trials of the drug.
The madness of the situation
However the EMA had seen a mere 15 incomplete accounts of these trials. No figure was given for the MHRA but it is unlikely to be any greater, while NICE (the body that is responsible for deciding if a drug is cost effective) had received tiny fraction of the total – just 4 incomplete accounts.
Why, the chairman wanted to know, since the drug watchdogs were entitled to all information, hadn’t they asked for it? “Good question.” Godlee replied and went on to speculate on the reason. “They think they have it all. They are under-resources and stretched and tend to take the manufacturers word for it. The madness of the situation is that we are getting a very partial, incomplete and misleading picture of the effectiveness of many drugs.”
She described how NICE, the body that should have been the most concerned at how little information it appeared to be getting on which to make cost-effective decisions about the UK’s 13 billion pound drug bill, appeared even more complacent.
Four out of 123 ain’t bad
When she contacted NICE to ask them to withdraw approval of Tamiflu until they had seen all the data, she said, the body of academic experts who reviewed the data for NICE replied that they were “convinced they had seen all they needed to see” and described the NICE review as “adequate”. Remarkable optimism when you’ve seen four out of 123.
It’s a system that is clearly dysfunctional, isn’t in patients’ interest and cries out for reform. Last week I was optimistic that there would have to be some significant changes. This week I’m not so sure following a brief article in the BMJ by Peter Doshi, the Cochrane Collaboration researcher who triggered the missing Tamiflu data investigation.
Concerned that reforms that allowed drug companies to set their own conditions for making data available, Doshi warns the result could be a “maze” that would actually frustrate efforts at greater transparency. The only solution, he wrote, is to put the watchdogs, specifically the EMA which hold “vast swathes of data”, in charge of creating a far more open regime.
He can’t have heard Dr Godlee’s evidence.
Body of Evidence 
funny how chapman continues to avoid the points jerome addresses over and over again inspite of jerome telling him so, and changes the subject, appeals to authority (traditional med is subject to regulators? big whoop if they’re so awful at regulating).
jerome, applauds for having so much patience
I just like the helpful info you provide on your articles.
I will bookmark your blog and check again here regularly.
I’m slightly sure I will learn a lot of new stuff proper right here!
Good luck for the following!
Thanks for this article. This side of the story is one that desperately needs to be told and you won’t find this on a regular news feed. Too bad. I must confess that I did try to read “Bad Pharma” and only got about three chapters in before giving up. The story was compelling, but the further in, the denser it got as far as numbers,facts and dates.
Your articles are always readable and the evidence compelling. It’s great to be able to find someone (obsessive or not) who will not only read, but analyze the info and translate to lay people like me.
That said, I hope Dr. Godlee and all of rest will keep up the good fight. First is convincing the people who matter that there is a problem. Hard to fix something, if you can’t even acknowledge it’s broken.
Holford is well behind the curve here – Ben Goldacre was writing about this in Bad Science, and exposed it in great deal in Bad Pharma.
I don’t see Holford’s even signed up to All Trials – do you know if he has? That is a fantastic initiative, and kudos to Sense About science and Ben Goldacre for helping it to happen.
Not sure what you mean by “behind the curve”. Holford has done an analysis of a particularly poor piece of research that was accompanied by an editorial that went way beyond what the research found. Standard part of science. Goldacre has also done the same and so have lots of other people. Alltrials is certainly a great initiative and there was a post about it on HealthInsightUK last week at http://healthinsightuk.org/2014/01/07/companies-claim-drugs-side-effects-are-commercial-secrets/.
I don’t see the mention of All Trials in that article, and I don’t see it promoted either here or on Holford’s site. It’s a grass roots initiative that looks like achieving more than the CAM industry has managed in half a century in improving transparency.
If you want to comment on this site, please limit yourself to points that are relevant to the post. Let’s agree, again, that AllTrials is a good initiative but claiming its’s done more for transparency than CAM has in 50 years is just silly. On the same logic you could take a swipe at newsagents for not campaigning to outlaw tax havens. The trials transparency problem has been created by amoral pharmaceutical companies, inadequate regulators and generations of doctors and academics looking the other way. Blaming the shepherds in the next valley for this Augean Stables is a feeble slur and has nothing to do with badly flawed research on vitamins.
Equally random is complaining that AllTrials is not promoted on this sit or Holford’s. The article was about two badly flawed trials not about transparency. Holford’s article was posted on a site called HealthInsightUK, as the post here says. Do go and read it and post a relevant comment there if you want to.
There is, as I said previously, a post on HealthInsightUK that goes into considerable detail on the battle over transparency and Alltrials and the EU legal cases. That would be the relevant place to make cheer leading comments about AllTrials.
Perhaps you can list the CAM initiatives in the last 50 years that have achieved the level of national and international prominence of All Trials?
I read the post you referred to. I didn’t see the mentions of All Trials. I have revisited it. I do not see a reference to Sense About Science, All Trials, Ben Goldacre, Simon Singh or any of the other prominent figures who are driving and promoting that initiative.
I’m happy to give credit where it’s due, but the world of CAM seems to be pretty consistent in not linking to what appears, form the evidence, to be the single most prominent grass roots campaign for pharmaceutical transparency. Does that not strike you as odd?
I think it is great you are so enthusiastic about Alltrials and I’ll say for the third time it’s a worthwhile initiative (and it is mentioned in the piece on HealthinsightUK) but I suspect that many people involved in CAM are not so impressed, having had people like you and Goldacre relentlessly attacking them as quacks and flakes for promoting unproven treatment and comparing it with real medicine that had been properly tested. Remember that funny joke: What do you call an alternative treatment that works? Medicine. Not so funny now maybe when maybe half of that evidence was based on fraud. Think of the evidence base for CAM that could have been established if it had followed the real medicine model. So not so odd, no.
What I suspect you are getting at is the implication that they don’t care about evidence because they are flakes etc. Well if the way their treatments are are tested are like those in the study on vitamins that Holford demolished, probably not so much. And if the response when there is a properly done positive trial is the total indifference that greeted the B vitamins and Alzheimer’s one, they probably don’t want to join a gang that plays with such a stacked deck.
So that’s enough on “why don’t you sign up to Alltrials and give more name checks to those involved” (Ed)
Jerome, you keep saying it’s mentioned in the piece on healthinsight. I’ve now found the mention (it’s in as AllTrials, not All Trials, my bad). It’s a single passing mention on page 2 of 3, and it doesn’t mention the people who have led it, the amount of support it’s gained, or the media attention that All Trials has generated.
No, it’s not that they are flakes. My problem with the world of Supplements, Complementary and Alternative Medicine is that their approach is sectarian. The entire subculture is reluctant to mention All Trials and almost never mentions the book Bad Pharma, which details the problem exceptionally well for a lay audience, they don’t mention Sense About science and they don’t mention Ben Goldacre.
In my view this is because SCAM is essentially a manifestation of “sectarian medicine”. It’s not just that the leaders of All Trials are the “enemy” and thus can’t be mentioned, you also find that very often when you even try to discuss All Trials in a SCAM-supporting forum such as the What Doctors Don’t Tell You Facebook pages, your comments are deleted. Followers of sectarian medicine do not engage with critics. They might try to sue them into silence, but they don’t engage.
I think that need sot change. As All Trials shows, you can’t improve without listening to your critics and engaging with them. That is in fact the essence of how scientific progress works.
Your complaints about CAM always have a heavy whiff of pots and kettles. If you are calling a close-knit group with a shared ideology sectarian then the medical profession is most certainly a far larger, wealthier and more powerful one. Some years ago critics such as yourself would point out that the the two distinctive features of CAM that meant it couldn’t be taken seriously were that:
a) Those involved didn’t practice evidence based medicine, it was all snake oil nonsense – well who seems to be peddling snake oil now or there wouldn’t be a need for Alltrials?
b) They didn’t alter their treatments in response to new evidence – but just take a look at what happened to doctors use of low dose aspirin after it was found that the benefits didn’t outweigh the risks, patients continued to get the drugs. And what about that wonderfully useless cholesterol lowering drug ezetimibe given to thousands despite trial after trial showing it doesn’t have any effect of deaths from heart disease. And as for antidepressant SSRIs stacks of evidence showing that when the hidden trials (them again) are included they are no better than a placebo yet their use continues to climb.
As for not engaging with critics I’ve been struck by the way that those who get curiously furious about homeopathy never actually engage with any of their arguments they just hammer away at how it is obviously bonkers and how the mechanism is clearly ludicrous. I agree it seems to be but then it is now very clear that the mechanism put forward to explain the effects of SSRI’s (boosting serotonin), for instance, is wrong. Yet that is no bar to enthusiastic prescribing.
I wouldn’t use the word sectarian I would say tribe – both are tribes – I wonder if they map onto political affiliations?
I think your criticisms are misguided, because I do not support medicine, per se, I support the scientific method. Medicine tries to follow the scientific method, and sometimes falls flat – and occasionally this is due to malice rather than incompetence.
SCAM does not follow the scientific method. It holds to the old folk ways of anecdote, tradition, “sympathetic magic” and so on. Confounders which medicine tries to control for, are embraced as evidence in SCAM. It’s a great way not just to be wrong, but to remain wrong and in fact become more wrong over time.
Rational inquiry is something the human race has developed, it is one of the few things that are uniquely ours and not a product of evolution. We evolved to believe things instinctively, and it’s only since the development of the scientific method that we’ve had a reliable way of rising above that.
That’s why it took thousands of years to get from mud huts to towns, the first useful steam engine was built in 1698 and it took over a hundred years before a practical steam locomotive was built, the entire history of the atomic age, form the discovery of the atom to the discovery of the Higgs boson, is about the same as the time it took to get from the invention of the steam engine to the first practical locomotives.
That’s because empiricism and tradition were replaced by science.
Not sure what you mean by saying you “don’t support medicine per se”. What is at issue is not supporting medicine or not but whether medicine currently follows scientific principle which it clearly doesn’t at the moment much of the time – half of clinical trials, which are what is supposed to form the evidence base, not published. That is not just “sometimes falling flat” that is being fifty percent prostrate.
As for “supporting the scientific method” it is hard to think of anyone who would totally reject it. Certainly not the bulk of those working in CAM. But that is like saying you support calculus or screw drivers. The scientific method is a very useful tool but it is morally neutral. Activities are not necessarily good just because they follow a scientific approach. Of course finding out if treatments are effective and safe is a good idea. The problem with your division – real medicine scientific if sometimes flawed vs. SCAM (ho ho) is that it ignores so many factors. One is that non-drug treatments are far less likely to get funding – high dose B vitamins to reduce brain shrinkage in people with cognitive impairment or the research done on then is so shoddy that it looks as if it is designed to produce a negative result, see Holfords analysis of the recent hostile vitamin trials which is where we came in.
Medicine does follow the scientific method, even if drug companies try to skew it. That’s why cancer survival rates have doubled since 1970, for example.
To describe vitamin trials as “hostile” betrays an agenda. They are just trials. As with virtually every honest trial, they do not back the optimistic claims of proponents. Virtually all early results overstate effect, virtually all trials conducted by proponents of any treatment do the same.
You can’t justly embrace the trials that show medicine to be overstating claims while simultaneously discounting as “hostile” trials that produce a result you don’t like. In practice, SCAM always wants a double standard: skeptical tests of medicine and uncritical tests of SCAM. Or no tests at all, just testimonials.
The link you make between cancer rates and science doesn’t follow at all – Look at some of the data on breast cancer screening – just setting up a screening program raises the survival rate without any treatments being delivered because the time increases from when the cancer is spotted to death. Nothing to do with any treatment. Your statement is so broad as to be meaningless. Results for some cancers have improved (breast) while others have stayed much the same (pancreatic) does that mean that science is not practised in the treatment of pancreatic cancer?
Your comment on the vitamins is really depressing as it shows you are really not paying attention or dealing with the arguments just offering platitudes. I described the vitamin trials as “hostile” because the analysis Holford provided made it clear the the people who ran the trials were incompetent ignorant or both.The attack in the editorial could only have been hostile because it did not even report the findings of the trials accurately.
Your general unsupported points about what trials generally do are irrelevant. I was talking about a specific analysis of specific trials which showed they were poorly done by any scientific standard. Please read the piece carefully and then present the case for saying they were “honest” or “competent”. It is infuriating that you never address any specifics. You throw out one claim then when it is dealt with you move on to another one. Your point in the last paragraph are a clear example.
I don’t embrace claims about medicine “overstating” (nice euphemism) its benefits, I report that that is what was found. I then discount a review of vitamins – not because it is a result I do not like but because the analysis clearly showed that this was a poor shoddy bad (pick your adjectives) piece of work. For you allegation to have any weight you would have to show that Holford’s analysis was seriously flawed and that the vitamins trials actually were careful and well conducted, so either do that or withdraw your allegation.
I can make a case based on findings about the fraudulent (better word I think) behaviour of people running medical trials because they seem to me to be correct (some of them are reported by Ben Goldacre for heavens sake) I can also discount these specific vitamin trials and others because they are clearly incompetent.If you want to claim double standards you will have to show either that Goldacre’s campaigning reports were wrong or that Holford’s analysis was wrong. Good luck on both of those. If not apologise.
It’s also worth remembering that doctors have extensive training and regulation requirements, there are real consequences for following pet diagnoses in the face of conflicting evidence, and few if any have a vested interest in prescribing. In fact for a fund holder they will do better if they prescribe less.
Most SCAM practitioners, by contrast, have no regulation, often have no valid medical training (remember Gillian McKeith and her diploma mill doctorate?) and actually make money by using quack diagnoses such as candida overgrowth or leaky gut syndrome, and selling remedies for them. A SCAM practitioner, unlike a doctor, very often has a substantial conflict of interest when discussing a treatment. It is right and proper to scrutinise these treatments, and it would be astonishing if honest tests did match the claims of proponents.
Those SCAM practitioners sound horrible. On the other hand, it is worth noting that the testing being done on the new treatments are most often conducted by the pharmaceutical companies who DO have a conflict of interest. Those tests should be scrutinized and analyzed for effectiveness of dosages and testing parameters due to their vested interest.
What about the new cholesterol guidelines? Do we really need to have 50% of the population on statins? Are the tests that are “guiding” this conclusion really telling us what we need to know regarding the link to heart health or is this just a SCAM on the American people? What happens if a doctor disagrees with the new guidelines? Your treatment depends on whether or not your doctor buys into the new numbers. How many are actually going to scrutinize the tests to determine if they are legit? Or will doctors just look at the recommendations and accept that the doctors who made them must know what they’re talking about. Who is questioning the recommending doctor?
Your assumption that anyone who disagrees with you is a scam practitioner is not valid. Your assumption that doctors are neutral in their use of treatments is also not valid. (Otherwise why get a second opinion?) Your assumption that anyone not a doctor is a quack who is in it only for the money is also not valid. Your assumption that the only honest testing is done by doctors who would reach conclusions that would validate your other assumptions is not valid.
I don’t expect that common sense would sway you, but I just had to put my two cents in after your obviously biased post.
SCAM stands for Supplements, Complementary and Alternative Medicine, and is also a handy description of the way it’s marketed which would be illegal for medicines and is also often illegal for SCAM. Virtually all SCAM research (what little there is) is funded by the industry, whereas medical research is conducted by charities academics and business. The exception is the US NCCAM which has spent $20bn testing alternative treatments, though without validating a single one.
The reason is enshrined in Minchin’s Law.
Chapman I do of course respect your beliefs but apart from loyal pharmaceutical executives I don’t think that anyone else who has spent any time researching or reading about the workings of the drug industry would recognise your touchingly rosy tinted picture. To try to pretend that whereas CAM is paid for by self-interested vitamin or food companies (who by implication are untrustworthy) it is only disinterested and good charities academics and business that fund pharmaceutical research is simply deluded. Those bodies do pay for some research but it is pharmaceutical companies who pay a large chunk of their bills and it is they who fund the clinical trials that get the licences.
If you have done any homework at all – you must have read Goldacre’s book, you are after all a keen cheerleader – you would know the real world of pharmaceutical research and marketing is far more ruthless. Hidden trials (50% by some estimates), fudged data tens of thousands killed in the UK alone as a result of drug side effects. A role call with such infamous names as Vioxx, Avandia, tamiflu, Rimonabant and so on. Let alone the 30 billion!!! dollars in fines that the companies have paid for fraudulent or criminal activity over the past 20 years. In fact I bet given the resources of a drug company marketing department pretty well any CAM treatment could get a licence.
Of course it is true that most non-drug treatments are not supported by large scale RCTs (although having seen how easily they are fudged and the data tortured to produce the correct result, I can only marvel at your steadfast faith in them). In fact a major problem for people working with promising non-drug treatments is that money for trials is hard to come by. In passing – I do find another of your beliefs odd – that drugs the best if not the only really effective way to practice healing and relieving symptoms. We know that most of the chronic metabolic disorders that affect us today are to a large part due to lifestyle which is very much the territory of CAM practitioners. So why so little spent on prevention? Short answer – it doesn’t interest drug companies.
Does that make it right? I would suggest that the lack of spending on non-drug treatments is the real scandal – and I mean good honest research not the kind of poorly done unscientific nonsense that was used to try to discredit multivitamins recently. One example – in the last 20 years drug companies have spent an estimated 40 billion dollars trying to bring a drug to market that would target plaques and tangles of Alzheimer’s with absolutely no success. However one genuinely independent researcher has shown in a proper RCT of the sort you are always calling for that B vitamins in high doses dramatically reduced brain shrinkage.
Has this been hailed as proper medicine. further research conducted etc etc. No it has been ignored, not because it wasn’t good science but because it wasn’t profitable enough. The trial to see if B vitamins could slow or halt progression to Alzheimer’s has not been done during the 3 years since the results were reported. It would cost about 6 million.
You seem very deeply wedded to the appeal to motives, but as with most proponents of SCAM you have yet to advance a credible reason why these base motives apply only to those whose products you find ideologically up appealing.
Late response – apology. I appreciate that your style is to raise points and then when they are answered to igore the reply and move on to another topic but before I take time dealing with another of your broad brush queries – why is it only pharma that have bad motives (brief response – my last email wasn’t about motives it was about what people/companies actually did. It is disheartening that you never actually appear to read what anyone says in reply to you) – why don’t you actually reply to my response to a previous query. You claimed I was trying to have it both ways both criticising pharma using sources like Goldacre and rejecting critiques of vitamins. My response was that that is perfectly reasonable if both Goldacre’s analysis was sound and if the original critique of vitamins was deeply flawed. Holdford did a very careful analysis of the failures of the vitamin post. To support your claim – I’m just picking result which suit me – you have to show that one or other is flawed. So Which is it and why? If both are solid then, as I said before, you should apologise.