Focus on Publishing
Could Peer Review Be Wrong?
By Robin Peek
Seriously questioning the validity of the peer-review process is like debating
the merits of a holy grail. We've been conducting peer review for so long (more
than 200 years, by some estimates) that it, well, just has to be good.
Or as Martha Stewart might say: "Peer review. It's a good thing."
But two scholarly societies are now asking if peer review is in fact such
a good thing.As the discussion about finding alternative ways of distributing
research has intensified over the past few years, it's not surprising that
peer review itself would eventually come under scrutiny. And two U.K. groups,
the Cochrane Collaboration and the Royal Society, are suggesting that there's
a need to examine the value of peer review and to determine whether its methodology
should be systematically revised.
The motivation for this inquiry is concern that the public is losing faith
in peer review's ability to create a gold standard for evaluating research
quality. Last year in the U.S., there were several instances of significant
peer-review failures. The most startling was revealed last October when the
work of Bell Laboratories' Jan Hendrick Schon came under scrutiny. Schon published
25 papers over the past 3 years. Of those, 16 have been declared to be false.
This finding caused the prestigious journal Science to withdraw eight
of his papers.
In January, Cochrane published "Editorial Peer-Review for Improving the Quality
of Reports of Biomedical Studies." According to the organization, this study
was conducted because "the knowledgebase of peer review has traditionally lagged
far behind its acceptability and use as a quality-assessment tool." Cochrane
claims that assumptions about the use of peer review for objective decision
making have "rarely been tested." In addition, it says that the need to "prioritize
information sources is crucial since there are over 20,000 biomedical journals
Cochrane, founded in 1992 in Oxford, U.K., is an international organization
that "aims to help people make well-informed decisions about healthcare by
preparing, maintaining, and ensuring the accessibility of systematic reviews
of the effects of healthcare interventions."
The report began with an initial look at 135 studies on the effects of peer
review in biomedicine. However, the Cochrane reviewers found that only 21 offered
credible data. They concluded that "given the amount of resources invested
in peer review and the common assumption that it improves such outcomes, the
lack of research documenting improvements is striking." In addition, many of
the studies show that the "well-researched practice of concealing the identities
of peer-reviewers or authors, while laborious and expensive, appears to have
little effect on the outcome of the quality-assessment process."
So it's not surprising that the Cochrane reviewers argue, "A well-funded
and coordinated effort involving several sectors of the scientific community
is required to improve our understanding of the effects of peer review." They
claim that such a study "would best be coordinated by a scientific council
... with the specific aim of scientifically assessing the quality of peer viewing." Until
a study is undertaken, the reviewers caution that "we are not confident in
the present system" of peer review.
Because the peer-review process is so deeply entrenched in the research system,
this fascinating study would be a formidable undertaking. To many, defending
the process is like upholding the American virtues of motherhood, baseball,
and apple pie. But since Schon was able to slip past Science's venerated
quality-control mechanisms more than a half-dozen times, it's appropriate to
ask if traditional peer-review practices can still be applied in an environment
that offers so many publishing outlets and places so many demands on the reviewers'
Royal Society Working Group
Shortly after the release of the Cochrane review, the Royal Society, the
U.K.'s leading research institution, announced it was going to examine best
practices in peer review. It has set up a 10-member working group to oversee
this process. Heading up the inquiry is Patrick Bateson, provost of King's
College as well as biological secretary and vice president of the Royal Society.
He believes that "peer review is an imperfect process."
In a February interview in The Guardian, Bateson argued: "Scientists
are under enormous pressure these days, and many are reluctant to give the
time to [peer review]. Sometimes what happens is that the paper gets passed
to a graduate student who then delivers a damning critique." He stated further: "We
are all aware that some referees'reports are not worth the paper they are written
on. It's also hard for a journal editor when reports come back that are contradictory,
and it's often down to a question of a value judgment whether something is
published or not."
The Royal Society Working Group has established an aggressive time frame
for generating two reports by September. One of the committee's charges is
to determine best practices for peer review and consider alternatives such
as naming the referees. This system might make referees more responsive and
responsible (and perhaps more polite).
A second report aims to help the public interpret scientific results. This
could be a particularly compelling document if the working group concludes
that the existing peer-review system is in fact fundamentally flawed. It would
place the Royal Society in the awkward position of trying to calm public fears
about the trustworthiness of scientific research while calling for essential
changes in the very framework that underpins it.
the Royal Society does call for new best practices for peer review, it will
be interesting to see how or if the research community embraces them.
Robin Peek is associate professor at the Graduate School
of Library and Information Science at Simmons College. Her e-mail address is firstname.lastname@example.org.