Medical publishing: can we rely on it?
pharmafile | January 5, 2015 | Feature | Manufacturing and Production, Medical Communications, Research and Development, Sales and Marketing | Les Rose, csr, journals, medical publishing, peer, publishing, transparency
The explosive proliferation of journals suggests that medical publishing is an attractive business. But it’s not without its critics, as we have been seeing recently. So I make no apology for revisiting this question in the light of developments over the last few years.
I should begin by defining who I mean by ‘we’. My focus is on the readers of medical journals, and on whether what they read conveys the truth.
The current drive toward transparency of clinical study reports (CSRs), with the AllTrials campaign at its head, has perhaps distracted us from the long established publication medium, the medical journals.
However, the two are not easily separated: changes in one will affect the other. Both are inextricably bound to big business; clinical trials to the healthcare product companies, and journals to those and to the big publishers.
Nobody can be in any doubt how big a business medical (and scientific) publishing is. Richard Smith, formerly editor of the British Medical Journal, has provided interesting analysis, and shows that with minimal staff costs a major journal can have a profit margin around 30% – far higher than most industries.
Peers or experts?
This business success comes at the cost of, in the view of many, serious shortcomings in the quality of the product. Peer review enjoys hallowed status in science publishing, but there is good evidence of its problems. Indeed, I often say that you don’t want peer review, you need expert review.
Peer review means that the manuscript will be checked by someone with a similar level of expertise to the author.
A classic example is homeopathy, for which there is no robust evidence of efficacy. But there are journals dedicated to it, so of course a paper written by a homeopath will be reviewed by – you guessed it – another homeopath. It won’t be reviewed by a clinical pharmacologist, so violation of fundamental science will sail through unscathed.
There are now calls to integrate open access with a new type of peer review. The operating model might be to publish online before review, and allow readers to critique the work. The paper then becomes much more like a Wikipedia article, and subject to regular revision.
The difficulty with that is that it can lose authority. What if you have cited the paper in your own work, and then find that it no longer supports you? Well that happens already, and I have seen people cite papers that have been retracted. With online systems at least there is an audit trail so that revisions can be tracked.
I have seen just how difficult the editor’s life can be, to make peer review work. Finding suitably expert reviewers is extremely hard. But there are more generic standards that too often are not checked. You don’t have to be an authority on say, blood rheology, to notice whether the sample size calculation was sound.
Liz Wager of Sideview, a medical publications consultancy, points out that few journals apply the Consolidated Standards of Reporting Trials (CONSORT) statement fully, which is an evidence-based, minimum set of recommendations for reporting randomised trials. If they did, they could screen out a great many manuscripts before they even got to peer review.
This in my view is an inexcusable tragedy. The mechanism is in place to improve published papers, and most journals do very little with it.
Is this because of an obsession with quantity not quality? Careers in science are dependent on publications, and employers are impressed by CVs with pages of journal references, devoid of any weighting as to quality. Even the established criterion of citation analysis does nothing to help that – authors may well cite your study in order to criticise it.
Low hanging fruit
Wager thinks there have been some improvements, for example in the use of text-matching software to detect plagiarism and redundant publication. A big gap though is the detection of outright fraud, driven by pressure to publish, something for which peer review can do little.
What worries me though is that such improvements as we have seen may have stemmed from the availability of better technology, and not from a deep commitment to apply agreed standards.
Adam Jacobs, senior principal statistician at Premier Research, has reminded me that the standards to which CSRs are produced are orders of magnitude higher than any to be found in a medical journal.
It is possible statistically to compare results summaries on clinicaltrials.gov with their corresponding journal papers, says Jacobs, and similarly for CSRs. Such studies have shown the summaries and CSRs to be more reliable.
He questions the practicality of applying the quality control standards of CSRs to the journals however. Such a move would ramp up costs enormously, and slow publication. He is strongly in favour, though, of wider availability of CSRs, which could drive up journal standards indirectly.
In Richard Smith’s financial analysis, journal reprints feature prominently. He estimates that a major paper can generate around £1 million in sales of reprints. Could this be why so many journals are not enthusiastic about open access?
We are becoming accustomed to having any question answered by the internet, and it seems to me that medical research can’t resist that trend indefinitely.
There are already online full text journals, and Smith considers it unethical that a journal should publish work that was expensively done by the researcher (whether commercial or academic) and then make money from it.
So let’s summarise where medical publishing is right now:
• Quality control is far below industry standards, and peer review can do little to address that; no reviewer is going to sift through the raw data. Indeed, peer review often fails to detect basic errors, which could have been picked up by a non-specialist triage approach.
• Publication is a volume game rather than a quality one.
• As others have repeatedly pointed out, publication bias continues to contaminate the literature.
• There is huge financial pressure for journals to accept papers that have a clear income stream, such as reprints, accounting for at least some of the observed bias.
• Abstracts are commonly not rigorously reviewed, and frequently do not reflect the full text – especially for conclusions. Yet the abstract is all that a lot of readers will see, especially if using free resources such as PubMed. Many journals publish free abstracts online but put the full text behind a paywall.
• Conflicts of interest are now recognised as major sources of bias, but again the journals are not doing enough to minimise their effects. It is not good enough simply to declare a conflict, there needs to be remedial – or better, preventive – action.
The writing is not just in the journals, it’s on the wall
The business model of medical publishing will have to change radically, as journals may well have keener competition to deal with. The drive towards clinical trial transparency is now unstoppable, and although we are a long way from publishing all trial results, it will happen one day.
When that day comes, what will be the role of the journals? They should still have a role, because only the obsessive clinician with lots of spare time is going to read a CSR.
In reality clinicians don’t read journal papers either, to the extent that authors think they do. They simply don’t have the time. For this reason resources have sprung up which digest papers into key messages – a practice which some editors are now asking authors to do themselves.
NICE is not a nanny
But wait, I hear you say, what about formularies and health technology assessments? Surely these are producing the necessary data digests on behalf of clinicians? Yes, bodies such as NICE are assessing the evidence for treatments and providing guidance.
In part the prescribing decision is taken away from the clinician, because someone else has read the documented evidence.
But I say ‘in part’ because no such function can provide complete guidance on all aspects of medicine, and we would not want it to. Otherwise why train doctors in therapeutics?
Hence there will remain the need for easily understood information to be available, to the highest quality standards. The CSR won’t meet that need, but the synopsis, or a readable summary as Wager calls it, could.
Journal papers will continue, but it will be much easier to check the CSR summary against the published paper, to see if inconvenient truths have been omitted or glossed over.
A shifting role for journals?
Of course, journals don’t only publish original research. Other types of paper have valuable roles, such as meta-analyses and systematic reviews.
The pharmaceutical industry has been rightly castigated for sometimes only publishing a subset of all the trials on a drug, but once all CSRs are online (and checked versus protocol registries in case any studies have gone missing in action), assessing the ‘big picture’ will be the role of journal authors. Indeed there are journals which already specialise in this type of work.
What does the future look like?
I have set out where I think we are with journal publishing, and I am developing an image of what the future will look like. It’s incomplete of course, but the trends are clear, as are the drivers of change.
Open access and transparency will eventually come, right across the board for science, and especially for commercially sponsored research. Readers are increasingly sceptical, as they should be, and future generations will expect to be able to check everything quickly and easily.
In healthcare, regulatory requirements will indirectly push up journal standards, perhaps even to the extent that it becomes less profitable to publish individual clinical trial results.
Journals might then shift their emphasis to good quality reviews and meta-analysis.
The picture of a medical library with shelves groaning with journal back issues is fading fast. Maybe we will lose this idea of a ‘finished’ paper, as it will no longer be on paper at all.
In my view, facilitating repeated review will get medical publishing closer to real science, where claims can be challenged and quality can be continuously improved.
Les Rose
Related Content
UK MPs call for greater action on clinical transparency and trial reporting
The UK’s House of Commons Science and Technology Committee has argued in a new report …
Not so crystal clear: The EU’s clinical reporting problem
With shocking new findings revealing that around half of all clinical trial results in Europe …
African-American enrollment disproportionately high in no-consent trials
A new study has revealed that African-American citizens make up a disproportionately high share of …
