What happened
- An arXiv preprint (CS.CL) describes **Peerify**, a pipeline that verifies whether peer-review claims are supported by the manuscript under review. Per the authors, it decomposes a review comment into atomic claims, retrieves relevant manuscript evidence, and judges whether each claim is supported (publisher: ArXiv CS.CL). [1]
Benchmark and findings
- The authors built a benchmark of **800 claims** drawn from authentic peer-review interactions at **NeurIPS 2024 and ICLR 2024**, including a **300-claim hand-labeled subset** used to audit the automated supervision (publisher: ArXiv CS.CL). [1]
Results and caveats
- Per the authors' own evaluation, automated labels agreed with human consensus on **90.3%** of audited claims (κ = 0.87), while off-the-shelf entailment models stayed below **0.24 macro-F1** (publisher: ArXiv CS.CL). These are the paper's reported results, not independent replication. [1]
Why it matters
- The authors report retrieval-centered verification and claim decomposition were the important factors, and that ambiguous or interpretive reviewer claims remained the hard case (publisher: ArXiv CS.CL). That points to a practical limit: automating review-claim checking may succeed more through evidence retrieval than through stronger standalone models. [1]
Sources
- Peerify: Benchmarking Peer-Review Claim Verification
ArXiv CS.CL (Computation and Language) · Reporting ·