Comparing AI code review tools independently: a protocol you can run on your own repository

Who this guide is written for.

• CTOs and VPs of Engineering who have to pick one AI code review tool for the whole organisation and have only vendor rankings to go on
• Platform and developer-experience teams asked to justify a per-seat subscription with something other than a demo
• Engineering managers whose reviewers complain that the current tool is verbose, comments on code nobody changed, or is confidently wrong
• Teams already running two or three tools in parallel and paying for all of them because nobody measured which one earns its seat

Why vendor comparisons are not comparisons

A search for the best AI code review tool returns lists written by the companies that sell them, and each list places its author first. The benchmarks behind those lists share one shape: the vendor picks the pull requests, the vendor seeds or labels the bugs, the vendor's model scores the comments, and the metric that gets published is how many of the known bugs were found. The metric that does not get published is how many comments were wrong.

That asymmetry matches what practitioners report. In the discussions engineers hold about these tools, the recurring complaints are verbosity, comments on code outside the pull request, and findings that describe behaviour the code does not have. All three are precision complaints, and precision is the number the rankings leave out. The one benchmark in this category that publishes its dataset, judge prompts and pipeline under an open licence, Martian's Code Review Bench, states that no tool found more than 63% of the known issues in its curated set and says openly that its gold set contains errors and that a static set risks having been seen by the tools during training.

Even an honest benchmark measures its own repositories. The published sets are open-source projects in five languages with their own conventions and pull request sizes; your defect mix, your review culture and the configuration you would actually run are absent from all of them. The only comparison that answers your purchasing question is one run on your code.

The metrics that decide, and why they have to be measured per repository

Google's static analysis platform Tricorder set the working definition of noise that code review tooling still uses: an issue is an effective false positive if the developer takes no positive action after seeing it, and an analyser has to stay under 10% effective false positives to remain in code review, or it gets switched off. That threshold is measured on developer behaviour inside one codebase, which is why it cannot be copied from a vendor page.

The second number that changes the decision is what the tool does to the review itself. An industrial study of an LLM review tool built on the open-source PR Agent (Cihan et al., ICSE SEIP 2025) covered 4,335 pull requests in three projects; 73.8% of the automated comments were resolved, and the average pull request closure time rose from 5 hours 52 minutes to 8 hours 20 minutes. The comments were acted on and the review got slower. Both numbers have to be on the sheet.

• Precision per finding class: a tool that is right about null checks and wrong about concurrency is two different tools depending on what your defects are.
• Recall on seeded defects per class, so the classes a tool never sees are named rather than averaged away.
• Noise on clean pull requests: comments on changes with no planted defect, counted per pull request.
• Off-diff share: comments whose subject is code the pull request did not touch.
• Verified-false findings: comments asserting a contract, behaviour or vulnerability the code does not have, checked against the code.
• Cost per pull request from the invoice. GitHub bills a Copilot review in AI credits that vary with the review's effort level; CodeRabbit bills per developer with an hourly review allowance. What one pull request costs is only known after the run.
• Reviewer time: minutes a human spends on the pull request with and without the tool, and closure time over the same window.

The protocol

The protocol fixes everything the vendor benchmarks leave to the vendor. The set of pull requests is yours, frozen before any tool runs. Defects are seeded by an engineer who writes each one down in a register with its class and location before the run. Clean pull requests are real merged changes that a second engineer reread and confirmed. Every tool runs on the same set, in a configuration each vendor's documentation describes as recommended, within the same week, and no tool sees another tool's comments.

Rating is blind. Comments are stripped of the tool's name and formatting, shuffled, and rated by an engineer who knows the codebase against the seeded-defect register and the code. The judge is a person, because the published benchmarks use a language model to match comments to fixes, and a model scoring a model of the same family has a known tendency to agree with itself. A second rater scores a sample so the agreement between raters is a number on the sheet and not an assumption.

Running it on your own repository

The protocol is designed to be repeated by the team without outside help, because the answer changes every time a vendor swaps its model. Construction matters more than size: a few dozen pull requests with a defect register and a clean control group beat a thousand unlabeled ones, which is also how the open benchmarks are built.

• Take pull requests from the last quarter that represent your real distribution of size and language, not the interesting ones.
• Seed defects by hand and log each one before the run: file, line, class, and the sentence a correct comment would have to contain.
• Keep clean pull requests in the set. Without them noise is invisible and every tool looks careful.
• Configure each tool once, from its documentation, and freeze the configuration. Tuning during the run turns the comparison into a demo.
• Run all tools in the same week on the same commits, and note the model or version each vendor reports.
• Strip tool identity before rating, and keep the rater separate from the person who configured the tools.
• Put the cost of the run on the sheet from the invoice, per pull request, next to the recall it bought.
• Rerun the same set when a tool announces a model change, and diff the sheets.

What the team gets, and how this relates to an audit

The engagement ends with a protocol document, the frozen set with its defect register, the blind rating sheet with rater agreement, one results sheet per tool and a recommendation that names the tool, the configuration, the classes it does not cover and what that gap costs at your volume. The recommendation says why, in terms of the numbers on the sheet, so a later team can disagree with the reasoning rather than with the conclusion.

A tool chosen this way still reviews the diff. This site's comparison of an AI code review tool with an independent audit covers what a tool cannot see: the system that already shipped, its architecture, tests and trust boundaries. The results sheet feeds that audit, because the classes a tool does not cover are the ones an independent evaluation of AI-written code and a test coverage review take on.

Where this goes wrong, and what we do about it.

1. 1
   Recall on the vendor's set, precision on nobody's

   A vendor benchmark seeds bugs into pull requests and counts how many the tool finds. Nobody runs the same tool on pull requests with no planted bug, so the comments it leaves there, which are the ones your reviewers will read every day, never enter the score. The tool that says the most wins the published number, and the team that buys it inherits the noise.

   What we do about it

   We build the set with a clean control group of real merged changes reread by a second engineer, and we count comments per clean pull request next to recall on the seeded ones. Where a class of finding falls under Tricorder's 10% effective false positive rule, we switch that class off in the tool's configuration and rerun the set to confirm the recall that was lost.

   What stays with you: results sheet per tool with recall and noise side by side per class, and the configuration that turns a noisy class off
2. 2
   Comments about code the pull request did not touch

   Tools that read the whole repository for context also comment on it. A pull request that changes one function comes back with remarks on the module around it, and the reviewer cannot tell from the comment whether the finding is about the change or about code that has been in production for two years. The tool is doing what its context window allows; the review process was never told to expect it.

   What we do about it

   We tag every comment in the rating sheet with whether its subject line is inside the diff, report the off-diff share per tool, and test the configuration options each vendor documents for limiting scope. Off-diff findings that are real go into a separate backlog for the maintenance owner instead of into the pull request.

   What stays with you: off-diff share per tool before and after configuration, and a backlog of verified off-diff findings with an owner
3. 3
   A finding that describes behaviour the code does not have

   The comment reads as a precise defect report: a function that supposedly ignores an error, a header that supposedly is not validated. The code does neither, but the sentence is fluent and specific, so the author spends twenty minutes proving a negative or, worse, changes working code to satisfy the comment. Engineers describe this as the tool making things up, and it is the most expensive kind of wrong because it looks most like right.

   What we do about it

   We rate against the code, not against the comment, so a verified-false finding is a separate class on the sheet with its own count per tool. We keep the ten most convincing wrong findings from the run as a training set for reviewers, so the team learns what confident and wrong looks like from this tool before it goes live.

   What stays with you: verified-false count per tool per hundred comments, and an annotated set of the most convincing wrong findings
4. 4
   A model judging a model of its own family

   Published benchmarks match tool comments to human fixes with a language model, because rating thousands of comments by hand is slow. A judge model scores comments written by a model of the same family as a colleague would: it recognises the phrasing and agrees. The score moves with the judge's model version, and the team that copies the number cannot tell how much of it was the judge.

   What we do about it

   We rate blind, by an engineer who knows the codebase, and we have a second engineer rate a sample so agreement between raters is measured per class. Only where agreement is high enough do we let a model pre-sort comments, and the model's sort is then checked against the human sample before it counts.

   What stays with you: rater agreement per class on the sample, and the rule for when model pre-sorting may be used
5. 5
   Resolved comments up, review time up, nobody looks at the second number

   The industrial study of an LLM review tool built on PR Agent found 73.8% of automated comments resolved and pull request closure time up from 5 hours 52 minutes to 8 hours 20 minutes. A dashboard that shows resolved comments reads as success; the closure time sits in a different tool. The reviewers know, the metric does not.

   What we do about it

   We log reviewer minutes on a sample of pull requests during the run and take closure time from the version control system for the run window and the baseline weeks before it. A tool that lengthens closure time is only recommended where its recall on the classes the team cares about pays for the delay, and the recommendation says how much delay that is.

   What stays with you: reviewer minutes and closure time per tool against the baseline, with the recall bought for each hour added

What stays with you.

• Protocol document: set construction, seeding rules, configuration freeze, blind rating procedure, metric definitions and thresholds
• Frozen pull request set with the seeded-defect register (file, line, class, expected finding) and the clean control group with its second-reader confirmation
• Blind rating sheet with every comment, its class, its verdict and the rater; rater agreement per class on the sample
• Results sheet per tool: recall and precision per class, noise on clean pull requests, off-diff share, verified-false count, cost per pull request, reviewer minutes and closure time against baseline
• Recommendation with reasoning: the tool, the configuration, the classes it does not cover and what that gap costs at your volume
• Rerun instructions so the team repeats the measurement after a vendor model change and diffs the sheets

How we work.

1. 1
   1. Scope and set construction

   We agree the defect classes that matter for your codebase, pick the pull requests from your recent history, seed the defects and write the register, and have a second engineer confirm the clean control group. Nothing runs until the set is frozen.

   Week 1
2. 2
   2. Tool runs

   We configure each candidate from its documentation, record the model or version it reports, run all of them on the same commits in the same week, and capture every comment together with the credit or invoice consumption of the run.

   Week 2
3. 3
   3. Blind rating

   We strip identity and formatting from the comments, rate them against the code and the register, and have a second engineer rate a sample. Reviewer minutes and closure time are collected for the run window and the baseline.

   Week 2-3
4. 4
   4. Results and recommendation

   We assemble the results sheet per tool, rerun the set with noisy classes switched off where the numbers call for it, and hand over the recommendation, the protocol, the register and the rerun instructions.

   Week 3-4

The rest of this cluster.

Where this turns into a service.

Where the dates and numbers come from.

• Martian, Code Review Bench v0: independent benchmark of AI code review tools, open dataset and pipeline, stated limits of the gold set
• withmartian/code-review-benchmark on GitHub: 50 PRs from 5 open-source projects, 173 golden comments, precision and recall definitions, training leakage caveat
• Cihan et al., Automated Code Review In Practice (ICSE SEIP 2025, arXiv 2412.18531): 4,335 PRs, 73.8% comments resolved, closure time 5h52 to 8h20
• Zhang et al., Code Review Agent Benchmark (arXiv 2603.23448): review agents solve around 40% of tasks derived from human reviews
• Software Engineering at Google, chapter 20, Static Analysis: effective false positive definition, 10% threshold, Not useful and Please fix signals
• GitHub Docs, Copilot code review: billing in AI credits by effort level, repository-wide context, no guarantee of spotting all problems
• CodeRabbit pricing: per-developer billing with an hourly review allowance, free reviews for public repositories
• Example of a vendor-run ranking: CodeRabbit on its position in the Martian benchmark
• Example of a vendor-run ranking: Kilo on the same benchmark, with its own tool ranked first