Evaluate AI code reviewers with ReviewBench and private pull requests

Start with the benchmark, then test the workload

GitHub announced ReviewBench on October 5, 2026, with 219 pull requests for evaluating AI code reviewers. The corpus spans 187 public repositories and 19 languages. GitHub analyzed 103.9 million pull requests to design the sample; that larger number is not the evaluation task count. Sources: GitHub

The sample follows GitHub’s language and repository-size distributions while favoring substantial changes over tiny pull requests. Candidate findings came from human reviews, later commits, deterministic tools, and multiple model families. ReviewBench offers a common starting point for comparisons. A private evaluation can then test whether a promising reviewer catches important issues without creating excessive investigation work. Sources: GitHub

Distinguish it from LangChain’s project with the same name. LangChain published its ReviewBench on July 31, 2026, using internal LangSmith reviews covering 59 tasks and 64 baseline issues. Examples included missing tenant restrictions in database operations and an endpoint migration that dropped an existing filter. LangChain describes its evaluations as comparisons of agent harnesses—the software surrounding the models—rather than isolated model comparisons. Sources: LangChain

Keep the scoring denominators visible

Before comparing scores, establish what each denominator includes. ReviewBench’s grounded precision considers matched candidates and excludes unmatched findings. Its augmented precision includes unmatched findings judged valid and uses all candidates as the denominator. Grounded precision therefore cannot tell a team what percentage of all emitted comments was useful. Sources: GitHub

Recall has a similar distinction. Grounded recall measures coverage of the shared reference issues, called golden true positives. Augmented recall adds novel true positives to both numerator and denominator, making its denominator agent-specific. Use grounded recall for comparisons against a common reference; the table below shows the official formulas. Sources: GitHub

GitHub’s methodology reports 96.6% agreement on true-positive and false-positive labels, but 62.9% exact agreement on severity. As Latent explains, label agreement is not a bug-detection rate. Latent’s article provides methodological analysis and explicitly says it did not independently run or score products. Treat agreement as evidence about labeling consistency, rather than the proportion of bugs a reviewer catches. Sources: GitHub, Latent

Freeze the public benchmark run

The repository provides the full corpus, golden findings, repository mirrors preserving base and head commits, a judging command-line interface, and an example adapter. Its workflow packages a reviewer in a container that follows the agent contract, publishes the image, and records its digest. The adapter reads a pull request and writes normalized findings. The 25-task subset supports initial validation. Sources: GitHub

Keep local development scores separate from official leaderboard results. Official participation uses a self-service portal, three full-corpus rounds, and maintainer review. The repository names Claude Sonnet 5 as the leaderboard judge; its separate Harbor workflow mentions Claude Opus 5, and local judging permits a chosen judge. Record the judge and workflow because these paths are different experimental configurations. Sources: GitHub

Archive the dataset revision, image digest, model identifier, prompt, tool permissions, context limits, judge, matcher, and raw findings with each run. If a hosted reviewer hides its model version, record its product configuration and run date. Preserve repository instructions too: GitHub added support for root-level agent instruction files to Copilot code review on June 18, 2026. ReviewBench’s public labels allow inspection and tuning, so reserve a separate private test for the final decision. Sources: GitHub

Reconstruct a private test without revealing the answers

Use this proposed local protocol to extend the public comparison. Create a representative sample of ordinary pull requests to measure noise and effort. Separately collect changes associated with escaped bugs, incidents, reverts, and later corrective commits. Report the samples separately because deliberately selecting buggy changes alters the workload being evaluated.

Reconstruct each change as it appeared when review was possible. Freeze the base commit, candidate head, description, dependencies, and documentation from that moment. Keep later fixes, incident reports, and revealing review comments accessible only to evaluators. Audit branches, history access, issue access, and network tools for paths that could reveal the answer.

Build labels from evidence; a later edit alone does not prove a defect. Record the affected path, triggering conditions, incorrect behavior, impact, and why the candidate change introduced or exposed the issue. Split development and final evaluation by complete pull request or related change family, keeping near-duplicates and related fixes together. Tune prompts on the development partition, then freeze the review contract before opening the held-out partition.

1. Define whether agents may run tests, browse history, use external documentation, and read repository instructions.
2. Run each candidate on the same held-out changes under the intended deployment conditions, documenting configuration differences.
3. Have maintainers judge validity, actionability, severity, and evidence without product identity where possible.
4. Resolve disputes with another reviewer and assess suggested fixes separately from the underlying finding.
5. Validate novel discoveries, add accepted issues to a versioned shared reference, and rescore every candidate while preserving original known-issue results.

Measure the work created by each useful finding

GitHub’s production experiment illustrates why comment volume needs context. It reported relative increases of 8.0% in addressed rate, 13.6% in its recall proxy, and 61% in comment volume. Cost per review fell 8.0% relative to the control. Addressed rate uses an LLM to assess whether feedback prompted a code change; production recall concerns additional human review needed. These proxies differ from the offline scoring formulas, and the gains alone do not establish lower investigation effort or fewer escaped defects. Sources: GitHub

For the local evaluation, record serious known issues recovered, actionable unique findings, false alarms per pull request, and duplicate comments. Count useful findings beyond those caught by existing continuous integration, static analysis, and normal review. Measure active investigation minutes separately from repair and verification time. Capture request-to-usable-review latency, including queues, setup, retries, and failures. Record execution, infrastructure, and evaluation costs separately.

The hypothetical comparison below illustrates the trade-off between issue recovery and investigation effort. It assumes both reviewers examine the same changes, with no novel findings or duplicates. Reviewer A catches more known issues but requires more triage time. The decision depends on the impact of those additional issues and the effort needed to investigate them.

Choose a reviewer against explicit success criteria

Report finding-level totals alongside pull-request-level distributions, including changes with zero findings. Show raw counts prominently for small severity groups. Call invalid emitted findings false alarms; do not label one minus precision a false-positive rate without defining a population of negative opportunities. Group residual misses by impact and failure mechanism so an aggregate score does not obscure a dangerous weakness.

Schedule repeated runs in advance and report failures and timeouts instead of selecting the best attempt. Compare candidates on the same pull requests, distinguishing run-to-run variation from uncertainty caused by a small sample. Before deployment, set explicit requirements for serious-issue recovery, investigation effort, and cost. Use a limited live trial to measure actual workflow effort and check whether the historical result holds.