Science is based on the data… and trust: Why and how the anonymous peer review process works
As all scientists, I am periodically asked to serve as an anonymous reviewer for manuscripts that have been submitted to a peer reviewed journal. This is a responsibility that Academics are asked to perform to support the community. The editors of a journal will select two or three people from a pool of possible reviewers and, based on reviewer input, decide whether a manuscript is published. I recently declined a request to review a paper from a new format (online) platform for scientific manuscripts. I wanted to share the reasoning with my technology friends.
The request states:
As we aim to enable lightning fast publication, reviews for this article are open for the next five calendar days. Positive reviews will be publicly visible, along with the reviewer’s name. Critical reviews will be shared anonymously with just the author.
Science is the discipline of applying the scientific method to a question in order to find the objective truth. It must never become a popularity contest.
In science, all of the really interesting questions have multiple possible explanations (hypotheses) and a body of evidence supporting each of them. Researchers discuss this evidence through the literature. They exchange carefully worded manuscripts that iteratively apply (quantitative) observation and critical thinking skills; narrowing the range of possible explanations until there is consensus.
Almost every manuscript can benefit from critical review (I am certain you will find error in this post). Peer review is done anonymously because, done correctly, the review should be critical. If you accept everything that you read at face value, you are not doing your job properly. “Critical” is not meant to convey “mean” or even “negative.” Having said that, even written in the most constructive light, critical review can be taken badly. The anonymous peer review process protects collegiality.
Setting aside the fact that it is unclear how the platform developers intend to define “positive” reviews, the practice of sharing positive reviews while hiding the “critical” reviews is quite literally hiding the critical reviews. I have seen systems that employ the idea of “voting up/down” data sets and my concern is in a similar vein. What does that mean?
Peer reviewed journals are the standard for airing scientific ideas because the reputation of their editorial staffs are on the line. If a journal publishes too many articles that must later be retracted, their reputation is tarnished. I have written previously on how recommender engines can bias a reader. In this case, however, the path by which a reader gets to the article is irrelevant. Once they get there, they will see only positive reviews.
What technology brings to the table is the “lighting fast” part of the above statement and it does that well. Technologists can also deliver the security that is desirable in an anonymized peer review process. A long time ago, my technology friends told me that the lesson from the Healthcare.gov scandal is that scientists are not technologists. Accepted. I think it should also be said that technologists are not scientists. Working together, we can deliver the tools that the scientific community needs and wants.
Recent Posts
- Equity in PreK-12 STEM Education November 30, 2022
- An interactive model of the two-alternative forced choice test November 23, 2022
- An e-bay like system for data sharing October 25, 2022
- Ducks! October 25, 2022
- Science is based on the data… and trust: Why and how the anonymous peer review process works October 25, 2022