The widespread adoption of scholarly peer-review – the process by which research articles are vetted by referees before being accepted for publication in an academic journal – as part of sensible scientific practice is less than a century old. Indeed, it is easy to forget just how young the notion of peer-review is or that its form has been in a state of near-constant flux since its inception. It is consequently a worthwhile exercise to explore its origins before we try to understand why journals are attempting to change aspects of it.
As with the scientific method, the “ideal form” of peer-review is by no means obvious: it must constantly adapt to changes in technology and culture, to evolve so as to remain useful to scientists who benefit from it.
Perhaps the earliest mention of a process somewhat akin to peer-review dates back to the ninth century (CE) in Syria. In the Adab al-Tabib, a scholarly work on medical ethics based on Hippocrates and Galen, its author Ishāq bin Ali al-Ruhawi asserts that physicians ought to keep detailed records of the prescribed course of care and that these records were subject to review by a local council of physicians who would determine whether the treatment prescribed was appropriate. This form of peer-review was to ensure that patients received quality medical care and that physicians were held accountable to the standards of the day.
Scholarly peer-review is commonly said to have originated soon after the formation of the Royal Society of London in 1660. The Philosophical Transactions, compiled and edited solely by the society’s secretary Henry Oldenburg – and at his personal expense! – was conceived as a sort of open notebook that academics could contribute to. Whether or not a contribution would be accepted for publication in the journal, however, was a decision left solely to the editor. While it may have been the case that Oldenburg informally sought the help of experts in judging whether a scientific work was worthy of publication, we should be careful not to confuse this with peer-review as it is understood today.
Almost a century later, the Royal Society would assume editorial responsibility for its journal and adopt a system of review more familiar to us today. It constituted a ‘Committee of Papers’, a select group that would decide which papers would be included in the Philosophical Transactions and which papers rejected. At this stage, however, it is crucial to note that an important component of peer review – the writing of reports that critique a submission – was missing.
This step was taken by William Whewell in 1831, who boldly suggested that teams of eminent scientists (a term he coined, incidentally) might compile reports based on the submissions to Philosophical Transactions, that were intended for publication in a new, cheaper monthly periodical called the Proceedings of the Royal Society. He imagined that this might serve to publicise the scientific work done by the society’s fellows and so increase the public’s appreciation and understanding of science.
This well-intentioned experiment would last no longer than two years: any negative reports that may have been written were never published. Over time, at least in Britain, the referee was reimagined as a gatekeeper who defended the integrity and reliability of scientific canon. Although often reviled, referees were seen to perform an important service: to (anonymously) critique submissions to academic journals, speaking on behalf of the scientific community and ensuring that published scientific research is correct. In continental Europe, however, editors still held independent sway over their respective journals well into the 20th century.
After World War II, academic journals slowly adopted the format we are most accustomed to today: single-blind (or reviewer-anonymous) peer-review. Here, journals are run by a team of editors, who in turn maintain a network of colleagues who, upon request, will write reviews of scientific articles sent to them. The editors then decide whether to accept or reject a paper based on the reviewers’ recommendations.
This formal system has been tinkered with almost continuously over the last half-century. For example, some journals have adopted double-blind peer-review, which sends the referee a copy of the paper with the names and institutional affiliations of the authors redacted: this is done to correct for possible biases. While there is evidence that it does work, critics claim that it does not effectively anonymise the author, especially in smaller communities of researchers.
And what of publication biases? Journals are known to be more likely to publish positive results than negative ones as the former are often considered more interesting. Negative results, however, have an important place in scientific inquiry and in order to shift focus from the results and onto motivations and methodology, some journals commission registered reports, where they commit to publishing the results of a study irrespective of its outcomes, based solely on the merits of the questions that the study wishes to address. This is sometimes called result-blind peer-review.
In this whirlwind of change, the journal eLife recently proposed a more radical form of peer-review:
The essential idea is that once an editor has invited a manuscript for full peer review, the journal is committed to publishing the work along with the reviewer reports, the decision letter, and the author response. Potential benefits of the new approach include a reduced emphasis on journal brands in research evaluation, a rebalancing of the relationship between authors, editors and reviewers, and greater efficiencies in journal publishing.
This may seem like a radical departure from the current norms in scientific publishing, as once a paper has been sent to a reviewer (i.e. it hasn’t been “desk rejected”), the journal is committing to publishing the paper along with the referee’s reports and the author’s response. Whether the referees will remain anonymous is unclear, although this appears to not be the case, since the journal’s editors were attracted by “the potential for referees to gain credit for thoughtful criticisms or creative suggestions”. The journal Biology Direct adopted essentially the same model of peer-review as early as 2006, with their mission statement arguing forcefully that they
… will publish “essentially anything”, even papers that receive three unanimously negative reviews, the only conditions being that three Editorial Board members agree to review (or solicit a review for) the manuscript and that the work qualifies as scientific (not pseudoscientific as is the case for intelligent design or creationism) – and, of course, that the author wants his/her paper published alongside the reviews it receives.
At their roots, these suggestions are variations of what is often called open peer-review, where (loosely) the identity of the referee is disclosed. In general, open peer-review aims to dispense with the anonymity and journals that adopt it (to varying degrees) will endeavour to make as much information available to their readers as they see fit.
Making the review process more transparent has its advantages, of course. The anonymous referee has been criticised for as long as the peer-review process has existed, and it is not uncommon for academics to routinely refer to these nameless, faceless servants of scientific inquiry as capricious, mean-spirited, pedantic or even addle-brained. Coffee lounges in universities across the globe abound with rumours of alleged biases and personal vendettas causing delays in publication and research grant approval. Perhaps, in the absence of anonymity, this potential for abuse of power might be kept in check?
Studies have found that the quality of the review itself is not compromised by this lack of anonymity. However, in cases where the referee report is made publicly available, referees tend to take more time per review and are more likely to decline an invitation to review a paper.
We view these two problems as possessing a common origin: the time spent critically reviewing a paper and writing a report is not valued in material terms. That is, an academic stands to benefit neither monetarily nor professionally in return for this community service. While a handful of journals occasionally pay their referees a small honorarium in return for their service, this is by no means the norm. It is also not a viable solution.
Open peer-review promises a more constructive and transparent dialogue among scientists, and constructive criticism if published would be a valuable addition to the literature. However, any attempt to radically change the nature of peer-review must necessarily be accompanied by a change in the way the referees are compensated for their time and effort, especially within academia.
For example, opening up the review process allows for the possibility of referee reports being indexed and cited, so that a scientist who writes an insightful review is rewarded professionally for their work. Author-level metrics that measure the productivity of a researcher often factor into decisions about hiring, grant approval and promotions, and these might also be adapted to include scientific contributions in the form of referee reports.
So while the proposed study by eLife is worth welcoming (I look forward to studying their findings), for these changes to have a significant impact on the practice of science itself, academia will have to – once again! – reimagine the role of the referee, and revisit the question of how to properly compensate academics for the services they render.
Madhusudhan Raman is pursuing a PhD in theoretical high-energy physics at the Institute of Mathematical Sciences, Chennai.