AI peer reviews should be judged by the substantive errors they identify and the improvements they lead to, commentator Zvi Mowshowitz argues
AI peer reviews should be judged by the substantive errors they identify and the improvements they lead to, commentator Zvi Mowshowitz argues
On October 29, Zvi Mowshowitz examined a study in which both a model acting as a judge and former journal editors preferred AI-generated reviews to those written by paid experts. Economist Tyler Cowen cited this finding in support of using AI to assess scientific papers.
Mowshowitz would himself choose a model for a detailed assessment of a paper's technical aspects. He proposes three questions for assessing a review's usefulness:
Did the paper improve, did the author find the review useful, and did it identify substantive problems?
Gary King and Stephen Worthington of Harvard University used 124 manuscripts and 134 human reviews from the rapidPeer service for their experiment. The service helps authors prepare papers for journal submission: invited experts are paid to assess arguments, methods, and figures and suggest revisions. The researchers gave the same manuscripts to the language model Claude Opus 4.8 and asked it to write reviews.
The researchers concealed who had written the reviews and asked another model, GPT-5.5, to evaluate them. Three former scientific journal editors independently compared the same ten pairs, reading only the review texts. They preferred the AI-generated text in 26 of 30 decisions. In a separate assessment, a model checked reviewers' comments against the manuscripts; a human checked some of its assessments. King and Worthington describe the AI-generated reviews as superior “on every measure we could devise.”
The manuscript authors received human reviews through rapidPeer; the researchers generated the AI reviews later for comparison. King and Worthington themselves propose giving authors both reviews and asking them to compare their usefulness. After receiving a review, an author decides which comments to accept, which to dispute, and what to revise in the paper.
In the autumn of 2024, during preparations for the AI conference ICLR 2025, an AI system advised reviewers on how to improve their reviews. It read each paper and its human review, then suggested clarifying vague comments and correcting misunderstandings. Reviewers revised about 27% of the reviews for which they received AI advice; in the group that received no advice, they revised about 9%.
In an experiment with deliberately introduced errors in clinical trial reports, researchers checked model reviews against a known list of flaws. To test reviews against Zvi's criterion, researchers could compare original and revised papers in groups receiving human or AI-generated reviews. They would track which substantive problems were resolved, which calculations were rechecked, and which conclusions were revised.