GitHub has published ReviewBench, an open benchmark for AI code review, and it is available to use now. The benchmark scores a review system against a validated set of known findings in real pull requests, using the same rubric for every system, and GitHub says it uses the result internally to judge whether changes to Copilot code review are likely to help. The announcement post walks through the construction and how to submit results.
Key facts
| Fact | Detail |
|---|---|
| Question answered | ReviewBench: GitHub Opens Its AI Code Review Benchmark |
| Short answer | GitHub published ReviewBench, an open offline benchmark for AI code review modelled on 103.9 million real pull requests, with a 219-PR corpus across 19 languages, a multi-source golden set and four precision a… |
| Reading time | ~3 min |
| Last updated | 2026-10-06 |
What is in the benchmark
- Distribution modelled on scale. ReviewBench follows the language, repository size and change size distribution observed across 103.9 million GitHub pull requests, then narrows to a representative corpus of 219 public pull requests spanning 19 languages.
- Multi-source golden set. Known findings come from human reviewers, frontier LLMs and static analysis. Every finding is labelled by severity, critical through low, and by category covering correctness, security, reliability, maintainability and testing, which is what makes slice-by-slice reporting possible.
- Four metrics. Grounded precision, grounded recall, augmented precision and augmented recall. The augmented pair exists to give credit for issues a reviewer finds that the golden set never recorded.
Why the ground truth is the hard part
Code review benchmarks live or die on label quality. ReviewBench’s trust story is explicit: one published rubric that applies to every finding, a human-labelled development set built by senior engineers, a grader calibrated against human judgment, uniform labelling across all sources, and a published agreement figure. Senior engineers independently labelled golden true positives before release, and the reported agreement is 96.6%.
The benchmark is also checked against production rather than treated as a leaderboard in isolation. GitHub tracks whether benchmark movement anticipates the direction of online experiments, so real improvements tend to show up online and regressions tend to show up too.
How to use it
The post addresses two audiences. If you are evaluating an existing reviewer, the benchmark lets you compare agents on identical pull requests with an identical rubric, and slice by severity or category to match how your team actually reviews code. If you are building a reviewer, you can onboard your own code review system and submit results.
For anyone running a coding agent on a desktop, this is a rare open, reproducible yardstick for a task that has mostly been judged by impression. It also sets expectations honestly: a reviewer with strong grounded recall can still miss issues that were never in the golden set, which is precisely what the augmented metrics are designed to surface.
MOOGH is a Windows desktop agent workspace where a coding agent can work against a local repository with each step approved before it runs. You can download the client and point it at your own code.
FAQ
- Is ReviewBench only for GitHub’s own reviewer? No. It is published as an open benchmark, and the post describes onboarding your own code review system and submitting results.
- What do grounded and augmented mean? Grounded metrics score only against known findings in the golden set. Augmented metrics also credit valid issues a reviewer surfaces that the golden set did not anticipate.
- Can a benchmark replace production measurement? GitHub’s position is complementary: benchmark movement is checked against online experiments, and the benchmark is used as an offline signal that anticipates them.
- What is the practical takeaway for teams? Compare reviewers on the same pull requests with the same rubric, and slice results by severity and category instead of ranking on a single headline number.
Related reading: