Peer review is an ingenious system in which scientists donate their time to explain why another scientist’s paper would have been better if it had been their paper. At its best, it is generous, rigorous and indispensable. At its worst, it is a six-month correspondence with someone who has misunderstood Figure 3 and would nevertheless like you to cite their book.
Now the institution faces an awkward possibility: the machine might be better at the useful part.
In their October 22 preprint, Gary King and Steven Worthington compare AI reviews with 134 human reviews of 124 manuscripts submitted to rapidPeer. These were carefully selected, well-paid experts producing unusually well-regarded feedback. Claude generated the main AI reviews; GPT judged them. Three former journal editors, blinded to authorship, also assessed ten review pairs, preferring AI in 26 of 30 judgments.
That is an uncomfortable result. The comparison did not pit a tireless computer against Professor Exhausted at 11:47 p.m., reviewing between grant applications and a dishwasher emergency. The humans had conditions that ordinary academia might reasonably classify as a spa package.
Nor does the familiar objection that AI makes mistakes settle matters. The study’s manuscript-based audit flagged errors in 5.5% of AI claim assessments, against 15.9% for humans. Those figures measure contradictions with the manuscript, not universal truthfulness; over a third of AI reviews still contained a flagged error. The relevant benchmark, however, is the actual human alternative. A professor’s confident misunderstanding does not become more accurate because it arrives on institutional stationery.
The implications plausibly extend beyond Claude. An additional analysis already included GPT-generated reviews, with strong results across both model families. Grok was not tested. That makes broader transfer a reasonable hypothesis, rather than a certificate to frame above every chatbot. ChatGPT’s name alone tells us too little: the particular model, instructions, document access and tools need specifying. “We used AI” is becoming about as informative as “we used a computer.”
There is precedent, too. Earlier work by Weixin Liang and colleagues found that GPT-4 could provide useful manuscript feedback; more than half the researchers in their user study rated it helpful or very helpful. That measured perceived usefulness, rather than proving superior science. Still, the prospect of getting a competent first critique without waiting for a stranger to clear their inbox deserves attention.
We should retain the irritating academic habit of asking what was actually measured. King and Worthington studied constructive feedback, not publication decisions or the quality of subsequent revisions. Their human judging covered only ten manuscripts. Their controls for length and style strengthen the result. Still, better reviews are encouraging; better science remains a further question. A beautifully written diagnosis is useful only if it identifies something real and leads to an appropriate remedy.
The judging machinery also warrants scrutiny. Separate research has found that LLMs can favour material presented in LLM prose. An AI judge may therefore bring its own tastes to the competition. A world in which machines write reports applauded by other machines could become the most efficient literary salon ever devised. Human assessment and checks against the underlying evidence matter precisely because fluency is so easy to mistake for insight.
What should researchers actually do with this? A sensible experiment would be to commission criticism early. Ask a capable model to identify unsupported claims, inspect the argument’s dependencies and distinguish fatal weaknesses from optional improvements. Demand locations and reasons. Then check its objections. “The model disliked it” belongs in the same evidential category as “a man at a conference looked concerned.”
This could change who gets access to useful intellectual friction. A junior researcher without a famous supervisor might obtain several rounds of serious criticism before submission. An experienced researcher might discover that the paragraph they have been defending for three years is, regrettably, incomprehensible. Affordable feedback could be valuable even where it falls short of expert judgment. The comparison includes all the useful conversations that currently never happen.
There is also a less cheerful possibility. If authors repeatedly revise until familiar models approve, scholarly writing could converge on whatever those models find easiest to recognise as good. Everyone could become impressively competent at producing the same acceptable paper. Unusual ideas might be polished into conventional ones, with every awkward intellectual edge helpfully removed. A literature can become smoother while becoming less interesting.
That possibility makes disagreement worth preserving. Authors should be able to reject an AI objection with reasons. Editors should be able to recognise that a strange argument deserves attention precisely because it does not fit established expectations. Running several models might expose disagreements, but three chatbots agreeing would still leave the underlying claim unverified. Consensus is a clue to investigate, not a substitute for doing so.
The attractive future would give machines much of the patient checking and leave humans clearly responsible for consequential judgments. It would protect confidential manuscripts, record how reviews were produced and test whether the resulting papers actually improve. It would also give expert reviewers more time for the difficult questions that survive routine scrutiny.
Reviewer Two may yet acquire an API. If that means faster, clearer criticism, researchers have reason to welcome it. They should keep the right to argue back. And if the new reviewer insists that every paper cite its previous work, we will know that the imitation of academic life has become unnecessarily thorough.
No comments yet