December 19, 2011

Online game aims to improve scientific peer review accuracy

Peer review of scientific research is an essential component of research publication, the awarding of grants and academic promotion. Reviewers are often anonymous. However, a new study by researchers at the Johns Hopkins Bloomberg School of Public Health found that greater cooperation between reviewer and author can improve accuracy of the review. The study is published in the Nov. 9 edition of the journal PLoS ONE.

To examine the accuracy of different review processes, the Johns Hopkins researchers developed a model using an online game on the Amazon E2cloud. Participants were asked to solve and review questions from the Graduate Record Examinations. The study examined both closed review, in which the author did not know the reviewers, and open review, where the author did know the reviewers.

The study found that when review behavior was public and under open review, cooperative interactions increased 13 percent. Overall accuracy between closed and open review models was similar. However, reviewers and authors who participated in cooperative interactions had an 11 percent higher reviewing accuracy rate.

“Our results suggest that increasing cooperation in the peer review process could reduce the risk of reviewing errors,” said Jeffrey Leek, lead author of the study and assistant professor in the Bloomberg School’s Department of Biostatistics.

Authors of the study are Leek, Margaret A. Taub and Fernando J. Pineda.

Funding for the study was provided through a Johns Hopkins Faculty Innovation Award.