The problem with medical publishing
pharmafile | July 6, 2012 | Feature | Business Services, Manufacturing and Production, Medical Communications, Research and Development, Sales and Marketing | Les Rose, MMR, medical publishing
I’ve often said that however bad one’s research, it’s possible to get it published somewhere. I have been thinking about that lately, and it needs a bit of qualification.
When I said that, I had in mind research that makes a positive claim, but has methodological flaws. That’s what we call a false positive. But what about false negatives? What if the study showed nothing, and could not have done because it was a poorly designed study?
Accentuate the negative
The irony is that it is actually very difficult to get negative results published, even if it is a very good study. Why is that? It is fundamentally because journals are businesses. In this respect they are little different from newspapers – they need readership, and readers are interested in news.
A good study which failed to find a difference between a drug and placebo is not very interesting to editors or readers, whereas a significant effect of a drug will attract attention. This publication bias has long been known to be a problem.
Indeed, the need for publication of negative results has been thrown into prominence by a systematic review which showed that certain anti-depressants were largely ineffective for mild depression – after the drugs had been licensed for that indication.
Science is news
This is just one limitation of the established means by which medical research reaches the public domain. But of course, this is not confined to the medical literature, which is just a subset of the very big market for scientific publishing.
Just how big a market it is was spotted by Robert Maxwell, a British army officer traversing the ruins of Germany in 1945. His vehicle for publishing German science (initially) was Pergamon Press. As scientific knowledge expands exponentially, so do the publishing houses it seems.
I am not a prolific author of research papers, but with just a few to my name I am bombarded daily with emails from obscure journals, touting for manuscripts. I wonder how fussy some of them will be if I respond.
Naturally, all these journals say they are peer reviewed. Back in 2004 the excellent charity Sense About Science published a discussion paper on peer review. While the paper was generally positive, it did describe many limitations. For example: “Nothing can prevent any group from writing a study or forming its own journal and calling its vetting policy ‘peer review.’”
This is the case with certain areas of what I call pseudoscience. The journal Homeopathy (in which I have published incidentally) is dedicated to a practice which lacks any scientific basis, yet is peer reviewed.
Hence a homeopath will submit a paper, and it will be reviewed by – guess who – other homeopaths. So will such a review challenge the beliefs of the author? Judging by the content of the journal, the answer is no.
Peers or experts?
Therefore I begin to wonder whether peer review is really what we want. By definition, a reviewer will be at the same level of expertise as the author, when we really want someone more expert and critical. But my experience on three editorial boards is that it is hard work to get even peer reviewers, let alone experts.
Maybe the publication model needs to change – more of that later. Within main stream medical publishing, the peer review system may not be as obviously inadequate as it is at the fringes, but it’s still far from perfect.
Let’s consider as a prime example the scandal of spurious claims by Dr Andrew Wakefield that the MMR vaccine was associated with autism. His original paper was published in The Lancet in 1998.
It was largely left to Brian Deer, an investigative journalist, to show not only that the research was flawed, but also that it was unethical and biased by conflict of interest. In an award lecture given to the charity HealthWatch, Deer related that he had questioned one of the reviewers of the original paper.
It emerged (according to Deer) that the reviewer had not actually read the manuscript in detail, and had approved it because he knew and respected the lead author Professor John Walker-Smith.
This and many other flaws in the process were published in 2004, but remarkably The Lancet did not publish a full retraction until 2010. Meanwhile vaccination rates for measles, mumps and rubella fell dramatically, and prevalence of these diseases rose.
Dumbing down data
So even when practised by a world-class journal, there are limitations to peer review. What if standards are lower? Over the years I have been engaged to do a lot of literature appraisal, which entails reading papers and abstracts and distilling out of them the key messages.
What has struck me quite often is how the abstract differs from the full text. In an effort to do essentially the same as me, the authors have pulled out what they see as the key message, but they may of course be biased. Yet the abstract is usually all that non-subscribers to the journal get to see. But it can be worse than that.
Anyone who has sat in a big medical congress listening to presentations will know that a lot of papers get critical questions from the floor. This essentially is the peer review that we need, but does it get captured in the published proceedings? Generally it doesn’t.
The usual practice is to bundle up all the abstracts into a journal supplement and publish them ‘as is’. If it’s a commercially sponsored satellite symposium then the abstracts will all be related to the sponsor’s product, and usually positively related.
I have heard the argument that the scientific committee of a congress also provides pre-publication peer review. I am not convinced by this. The volume of oral papers submitted is now so vast that it really isn’t possible to vet each one for scientific validity.
All that these committees can really do is to decide whether a paper fits into the programme, and which session to assign it to. Oddly, I have commonly seen the same paper in more than one session!
Bundles, but not fun
Supplement publication has always been big business. Journal editors admit that commercial constraints do bias publication decisions. Richard Smith, former editor of the BMJ, once explained that reprints of a single paper can earn a journal a million pounds. Who is brave enough to turn that down?
This suggests that the commercial model of scientific and medical publishing at least needs modification, and possibly reinvention. Elsevier, a large publisher that everyone has heard of, has attracted a great deal of criticism for its practice of ‘bundling’ journals (rather like the banks bundled debts up to 2008).
This consists of offering package deals to universities, comprising a few prestigious journals that everyone thinks they need, plus a lot of very minor and maybe poor quality journals. Although it might be argued that the university library could simply throw away the rubbish, that doesn’t matter to Elsevier, as they can then use the subscriptions to promote the lesser journals, eg., “Harvard takes The Journal of Obscure Research and so should you”.
Hence we have about 8,500 journals in the world scientific literature, so yes you will find somewhere to publish your false positive results.
Popular medical science
OK, I have hurled some brickbats at peer review, but there is even a trend to circumvent that to some extent. Journals often issue press releases when they get an important new paper.
Even if the paper itself has been reviewed, the press release almost certainly won’t be. So again we have the problem of the key messages being distilled down, or perhaps ‘dumbed down’ for the lay press.
One of the most widespread and damaging abuses is the use of relative risk data in outcome studies. Authors commonly do this, because it makes the outcome look far more important. But even if both relative and absolute risks are in the paper, it’s the former that too often get the attention.
The celebrated GP and author Dr Margaret McCartney rails against the practice, demanding that no outcome study should be accepted without absolute risk being given prominence.
While there are many good things that medical journals do, my purpose here has been to highlight some of the problems, and I’ve only scratched the surface. I have seen enough now to consider that it’s time for a re-evaluation of the whole field of medical (and scientific) publishing.
The world has changed since the advent of the internet, itself invented specifically for the dissemination of scientific knowledge. Yet still we cling not only to paper journals, but to e-journals that try to ape paper ones. Surely it’s time to move on from the world of William Caxton?
A new publishing model?
Exclusively paper journals of course restricted their readership to those who paid, and that prevails to a large extent online, with full text commonly behind a paywall. This is annoying a lot of scientists, who are lobbying loudly for open access to all. Whatever the financial constraints of that – who is going to fund publication? It would have the benefit of exposing research findings to much wider review.
But such review would be post-publication, and the idea has been built into at least one online journal group. The concept is based on a dynamic model, in that a paper once published may be updated. Yes, you have heard the idea before – Wikipedia!
At first I worried that no set of findings could be stable, but in reality it will settle down because it will be limited by the available data. Perhaps it will encourage authors to get it right first time. Never mind impact factor, what about a metric based on revision rates?
I’m sure that would raise standards.
Les Rose is a freelance clinical scientist, specialising in project management consulting.
Related Content
Measles: the UK’s next epidemic?
Recently, the UK has seen a rapid increase in the number of cases of measles, …
UKHSA warns of measles outbreak
The UK Health Security Agency (UKHSA) has warned that further outbreaks of measles could spread …
UK Government calls on social media companies to fight misinformation surrounding vaccines
Boris Johnson has called on social media companies to come together to discuss ways in …
