Albus Dumbledore and the importance of anonymous peer review

Albus DumbledorePeer review is the cornerstone of academic publishing. The process provides a relatively unbiased assessment of scientific research to verify that manuscript claims about experimental discoveries are supported by robust data and statistical analyses. Without peer review, scientific literature would become misleading, error prone, and untrustworthy. Peer review involves a journal editor selecting one or more peer reviewers who evaluate a manuscript critically, which results in a decision to reject or accept, or postpone a decision while authors make revisions.

Recently, a manuscript I submitted to a journal came back with two reviews. Reading through these positive and brief reports, I was considering each comment and planning changes I would make in response. And there, at the very end of the second review, was the name of Reviewer 2: Albus Dumbledore. Well, it wasn’t actually Professor Dumbledore, but my point will be that the name isn’t important nor should it have been included.

Although I argue in favour of either single-blind (reviewer names removed) or double blind (both reviewer and author names removed) anonymous peer review, it is worth considering why a reviewer might choose to sign a review. Some argue that open peer review promotes integrity and transparency in the dissemination of scientific discoveries. By signing a review and relinquishing anonymity, a reviewer makes a strong statement about being accountable, being confident about the quality of the review, promoting openness, and being constructive in the review process. Some might sign reviews because they might otherwise be more negative and critical behind the protective wall of anonymity. Although Mick Watson published a blog post here that discusses the benefits of signing reviews, and drawbacks of anonymous peer review, I find that most of my rationale in support of anonymous peer review is distinct from his discussion.

Here are several reasons why I advocate for anonymous peer review.

1. Anonymous peer review works. Having read hundreds of reviews of my own research papers, and being involved in the review process of many other articles, I have only flagged a handful of poor reviews, where the reviewer was obviously out to lunch or had an axe to grind. The remainder were well-considered and fair criticisms of the work. As a reviewer, my reviews usually agree with the other reviewers. Rejections, even for many of my own papers, have almost always been well founded, despite the upset and frustration that accompany such negative outcomes. I do not see how the vast majority of my reviews, or the countless others I’ve seen, would have been improved substantially by signatures at the bottom.

2. Accountability is inherent in the system. Editors are leaders in the field who monitor the review process and those who review for them. Strong reviewers garner favour and are eventually asked to become editorial board members or editors themselves. Weak reviewers do not get called on again and, I suspect, risk their professional reputation. A signature is not needed, nor should be needed, to ensure that a scientist performs a review task professionally and in an unbiased manner. If, on the other hand, a scientist would be overly critical or biased in the absence of a signature, this is an ethical concern; I would not want that scientist reviewing my work regardless.

3. The editor as "good cop". Many papers should be rejected. It is not uncommon to see authors demonstrate poor judgment, submit experimental work that was poorly designed or conducted, misjudge the “impact" of their work, neglect to edit their manuscript sufficiently prior to submission, or prepare figures and tables of unacceptable quality. For many reasons, authors often disagree with the critical comments of reviewers, even when the criticisms are well founded. Being critical of the science, reviewers serve as “bad cops”. In contrast, editors are the good cops, the messengers, the bearers of bad news. Editors may take a lot of flack for rejecting papers, but they can stand behind the reviews to justify their decision. Because reviewers are asked to be bad cops in this process, I do not think that they should also be in the firing line of those who stand subjectively behind their questionable manuscripts and data. Worse still, many journals make decisions to reject based on perceived impact. In those cases, the authors are justifiably upset and I would not want them angry with reviewers for not "raving" about their paper sufficiently to the editor of a high impact factor journal. Given the enormous professional stakes involved with publishing (e.g., job security, promotion, salary, reputation among peers, awards, student success), signing reviews adds an unnecessary element of risk of "bad blood" between reviewers and authors.

4. Let my review be unbiased. Environmental Microbiology had an unfortunate tradition of publishing some of the silly things reviewers said in their evaluations. I was critical of this practice and asked the editors that I never be cited like this because it would make me self conscious about my reviews. For example, I might think that if I just tweak this sentence to be a little more funny, or damning, or tongue in cheek, maybe I'd be quoted in the referees’ quotes section next year. Likewise, if my peer review was signed, I might worry just a little more about the wording of my review, what the authors might think, what I will say to them later, whether they might hold a grudge. If reviewing the paper of a senior scientist in my field, one who might evaluate my next paper, grant proposal, or application for promotion, I might be tempted to sugar coat my concerns a little, even when I know that firmness and objective criticism is essential. Because I'm a "nice guy”, it is particularly important to depersonalize the process of evaluating science. In other words, I would worry too much about not being perceived as a jerk, which could detract from my duty to express very serious concerns about the quality of data I see in many manuscripts that cross my desk. In contrast, if I review a manuscript briefly and favourably, a signature could be perceived as a conflict of interest (see point 5).

5. No reviewer kickbacks. In my case, Albus Dumbledore offered a favourable and brief review of my manuscript. Professor Dumbledore is now in my good books and I will certainly suggest this reviewer again. Perhaps I will send a thank you card, bottle of butterbeer, or an invitation to speak at an upcoming conference. Perhaps I will be generous when reviewing the next paper or grant proposal from this colleague. Conversely, any overly critical reviewers will be on my blacklist. I will avoid recommending them as reviewers in the future or, worse yet, I might ask an editor that they not review my next submission. Would I do all of this? No. Could I? Yes, very easily. Preventing kickbacks may be the single most important reason why anonymous peer review is essential.

6. I don’t need it. That Professor Dumbledore reviewed my paper has little value to me as an author. Without the reviewer name, I am still fully able to work with an editor to incorporate recommended changes and respond to criticism. In my recent example, anonymous Reviewer #1 provided a highly similar review to Albus Dumbledore and both reviewers identified two of the same primary weaknesses that needed addressing in our manuscript.

7. There are better ways to improve science publishing. Although there are many things that can be done to improve the scientific publishing process, signing reviews is not a step in the right direction. A poor review that is signed is still a poor review. A common goal here is to improve the quality of the review process and there are steps that scientific mentors and journals can take to improve the system. Supervisors should partner with senior graduate students and postdocs on joint peer reviews, complementing other forms of early career training. I would also like to see increased efforts by journals and academic societies to educate scientists to be effective reviewers. For example, professional workshops organized by academic societies could focus on peer review skills, much as online reviewer training courses could help provide the basics. Editors should work with reviewers to ensure that poor quality reviews do not end up on authors’ desks. Importantly, journals should do much more to provide reviewers and editorial board members with regular performance metrics, seeking ways to incentivize the review process for faster turnaround times and higher quality reviews. Rewarding strong reviewers for their contributions could help, perhaps through a point system that can be used toward offsetting open access publication charges. In other words, providing training and incentives for reviewers to increase their performance would help address the root of the peer review problem better than signing reviews.

For the reasons outlined above, I will include a request in future cover letters that any reviewer names be removed prior to returning my manuscript for revision. In addition, I will no longer review for journals that publish reviewer names alongside each article and, as much as possible, avoid publishing in those same journals. My preference is for reviewers to remain anonymous, for journals and editors to promote and uphold the integrity of anonymous peer review, and for scientists to take the role of reviewer seriously, whether we sign our reviews or not, continually seeking to improve our performance for the betterment of science.

Comments

Add new comment

Josh - welcome to the blogosphere, and I hope your blog gets widely seen!  This is a nice inaugural post.

I sign (most of) the reviews I write, for a reason that I think argues against your point 6.  I sign so that authors can contact me if my review is confusing, or if they want to "try out" a proposed fix for something.  I find authors contact me (the reviewer) from time to time - maybe 5 or 10% of the time?  I think there's some value there.  I would argue that the option of anonymity is important, and should always be there, but I would not go as far as you do here.  But you made me think, and that's always good.

Thanks for getting me thinking as well - I appreciate your comment.

For those relatively rare instances where some back-and-forth is needed, I find an editor can handle most of the time, or a second round of review does the trick. In other words, I would prefer to not have everyone signing reviews to enable a relatively rare back-and-forth interaction between authors and reviewers. Although I agree that having reviewers joining the authors in the research effort would improve the work more than an anonymous system, I then wonder if this process blurs the line between reviewers and collaborators, where the reviewers earn authorship and new reviewers are then needed.

Our household has given this a lot of thought. Our humble proposal is a hybrid-model. Consider:

Reviews are anonymous throughout the review period. All of your points hold here. If the paper is rejected the reviews remain anonymous forever. If the paper is accepted then the positive reviews become public, and shared as part of the Sup Info for the paper. Any negative reviews remain anonymous; we go back and forth on whether they should be Sup Info too. The author of the positive review has the option of including that effort in a "Reviews" section of his/her CV, which does not carry as much weight as the rest of the CV but at least allows that this was a non-trivial effort executed with professional attention. 

Thanks for the thoughts, which closely describe the Frontiers journal reviewing model.

I worry that this approach simply delays the concerns I have with open review. Kickbacks still a factor, for example. And posted reviews offer a database of reviewers and reviews I can mine to find "soft" reviewers for my next submission. Too cynical, I realize, but I think better to prevent such possibilities altogether.

Anonymous review efforts can also be added to a CV, so I'm not convinced there is a clear benefit there.

I support your points about the significance of anonymous peer reviews for the whole scientific community, Josh. Of course, some people could write more insolent things in their review, but true researchers will not do this. They will write brief and honest review, and point to some weaknesses of your research.

Besides, peer review is great practice for overcoming the plagiarism problem in papers as even the modern plagiarism checkers as iThenticate or Unplag could not detect crafty paraphrase, while humans could. So I believe in future of anonymous peer reviews!