Do anecdotes of side-effects count as evidence?

I can sympathise with the frustration felt by Professor Edzard Ernst, dedicated promoter of the  principles of evidence based medicine, when faced with under-reporting of side-effects. But I’m surprised at his readiness to abandon them when going into print. His latest attack on Chiropractic just doesn’t follow the rules of evidence based medicine, at least not the drug company variety.

A Guardian news short – longer web version here – today quotes an up-and-coming study of his saying that chiropractor treatments may appear to be safer than they actually are because adverse effects are under-reported in medical trials. Under reporting of side-effects is of course not in the interest of patients. As he says: “Imagine you have a drug where mild adverse effects are documented …..then somebody does a trial on this drug and doesn’t even mention adverse effects. That, in anybody’s book, must be unethical.” Hmm yes imagine such a thing happening in drug trials!

What Ernst must surely be aware of is that not only do drug companies regularly run trials in which they don’t report side effects but – can you imagine – they actually conceal them. Trials which showed raised risks of heart disease with the diabetes drug Avandia, subsequently pulled off the market because it increased the risk of  heart attack were concealed and as for not reporting on side-effects, the most recent Cochrane review of statins complained that six of the 13 trials it looked at made no attempt to gather data on side effects.

The whole issue of how ineffective regular clinical trials as a way of getting a true estimate of the likely rate of side effects is extensively covered in Professor David Healy’s new book where he explains the catch 22 that goes on. Most drug trials are not long enough nor large enough to pick up rarer side effects or ones that only emerge after a while, so often drugs are licenced with the request – and it is often treated as more of a request – that a follow up trial to test the risk be run.

While the drug is being given to patients, in some cases numbered in millions, both doctors and patients can report side-effects (Yellow Cards). However, and this is where the catch 22 operates, none of the these reports count as proper evidence. They are merely anecdotes, the side-effect data from the trials is the proper evidence. In the case of Avandia the doctor and patient reports were showing an increasing level of heart attack reports while the drug watchdog – the MHRA – continued to declare that the balance of risk and benefit was favourable.

In support of his claim that Chiropractic raises the risk of strokes due to manipulation of the joints of the neck, Professor Ernst relies on case reports from American neurosurgeons and from the results of questionnaires  that he sent out to UK neurologists, which came up with a considerable number of reports. But this is not evidence, as anyone complaining about drug side-effects are regularly told. In the case of Avandia the claim was that people with diabetes were more likely to have a heart attack. You couldn’t blame the drug.

Now I’m not carrying any torch for Chiropractic. I’ve no idea if the treatment raises the risk of strokes or if the risk outweighs the benefits. If clinical trials aren’t reporting side-effects then that should be corrected. What’s wrong is when CAM is held to different standards and Ernst’s statement about the risk are reported as gospel. If Ernst is concerned about poor reporting of side-effects putting patients at risk then the real culprit is elsewhere.