peer-review on S Anand

Can AI Replace Human Paper Reviewers?

Tue, 13 Jan 2026 09:55:45 +0530

Stanford ran a conference called Agents for Science. It’s a conference for AI-authored papers, peer reviewed by AI.

They ran three different AI systems on every paper submitted, alongside some human reviewers. The details of each of the 315 papers and review are available on OpenReview.

I asked Codex to scrape the data, ChatGPT to analyze it, and Claude to render it as slides.

The results are interesting! I think they’re also a reasonably good summary of the current state of using AI for peer review.

The three AI reviewers wildly disagree with each other.

Imagine hiring three movie critics to rate the same film. One gives it 2 stars, another gives it 6 stars, and the third gives it 4 stars. Same movie, completely different conclusions. That’s what’s happening with these AI reviewers-on almost half of all papers.
“Averaging” the three AIs doesn’t actually help

You might think: “Just average the three scores! That’ll balance out their biases.” But here’s the problem: the generous AI (AIRev2) uses much bigger numbers. When you average, its voice drowns out the others. It’s like having three judges, but one shouts and two whisper.
Every AI claims to be 100% confident - even when they’re wrong

Reviewers are asked “How confident are you in your assessment?” on a 1-5 scale. Every single AI review said “5 out of 5-totally confident.” All 751 of them. Even when two AIs looked at the same paper and reached opposite conclusions, both claimed maximum confidence.
AI and human reviewers see different things

On papers that got both AI and human reviews, we compared their scores. The AIs were almost always more generous than humans-by about 1 full point on average. And in some cases, AI said “excellent!” while the human said “this is broken.”
AI reviewers can catch obvious problems

AI reviewers successfully flagged papers with impossible claims-like citing AI models that don’t exist yet, or referencing datasets from the future. These are “fact check” problems that don’t require deep expertise, just attention to detail.
Use AI disagreement as a signal, not noise

When the “generous AI” loves a paper but the “skeptical AI” hates it, that’s not random noise-it’s useful information. It means the paper’s fate depends on standards (rigor vs. novelty), not just quality. These are exactly the papers humans should look at.

The psychology of peer reviews

Mon, 17 Jun 2024 06:59:59 +0000

We asked the ~500 students in my Tools in Data Science course in Jan 2024 to create data visualizations.

They then evaluated each others’ work. Each person’s work was evaluated by 3 peers. The evaluation was on 3 criteria: Insight, Visual Clarity, and Accuracy (with clear details on how to evaluate.)

I was curious to see if what we can learn about student personas from their evaluations.

15% are lazy. Or they want to avoid conflict. They gave every single person full marks.

4% are lazy but smart. They gave everyone the same marks, but ~80% or so, not 100%. A safer strategy.

10% are extremists. They gave full marks to some and zero to others. Maybe they have strong or black-and-white opinions. In a way, this offers the best opportunity to differentiate students, if it is unbiased.

8% are mild extremists. They gave marks covering an 80% spread (e.g. 0% to some and 80% to others, or 20% to some and 100% to others.)

3% are angry. They gave everyone zero marks. Maybe they’re dissatisfied with the course, the valuation, or something else. Their scoring was also the most different from their peers.

3% are deviants. They gave marks that were very different from others’. (We’re excluding the angry ones here.) 3 were positive, i.e. gave far higher marks than peers, while 11 were negative, i.e. awarding far lower than their peers. Either they have very different perception from others or are marking randomly.

This leaves ~60% of the group that provides a balanced, reasonable distribution. They had a reasonable spread of marks and were not too different from their peers.

Since this is the first time that I’ve analyzed peer evaluations, I don’t have a basis to compare this with. But personally, the part that surprised me the most were the presence of the (small) angry group, and that there were so many extremists (with a spread of 80%+) – which is a good thing to distinguish capability.

MIT paper prank

Tue, 19 Apr 2005 12:00:00 +0000

MIT pulls a prank on the World Multi-Conference on Systemics by submitting a computer-generated paper titled “Rooter: A Methodology for the Typical Unification of Access Points and Redundancy”. I was among the people spammed by Nagib Callaos, the organizer of the conference.

The students were soliciting cash donations so they could attend the conference and give what Stribling billed as a “completely randomly-generated talk, delivered entirely with a straight face.” They exceeded their goal, with $2,311.09 cents from 165 donors.

MERLOT

Mon, 18 Apr 2005 12:00:00 +0000

MERLOT is like a Wikipedia for online learning materials are collected here (along with annotations such as peer reviews and assignments).

Google Scholar

Fri, 19 Nov 2004 12:00:00 +0000

Google Scholar lets you search academic references (journals, papers, etc).

Google notes

Sat, 09 Dec 2000 12:00:00 +0000

Google likes directory sites. Google likes Yahoo!. Google uses peer review. Google is good.