Benchmark: What code review misses that a runtime bug hunter catches

Heal builds a sandbox around any application and drives it as its users and their machines do, through every entry point: the UI, the API, the CLI, the SDK, webhooks and scheduled jobs, with the data under control, to find defects and prove each one with a failing test.

How Heal finds defects

• Scan the codebase, and map the features. A feature is an entrypoint into the product, for a human or a machine (UI, CLI, SDK, etc).
• Scope the sandbox based on this map. For example, it creates seeds, or it mocks external APIs, or it creates a controlled environment for the UI. This also gives the agent a deep understanding of how the system works.
• Find many defect candidates:
  • Run static analysis on the codebase to identify a large number of defect candidates, powered by all the context that the agent has. This is similar to what code review does.
  • Run other tools that need a runtime, such as security scanners and UI checkers. Interestingly, Heal could be extended here by running any custom tool against the sandbox process.
  • Inject manual QA priors to find common issues, from a rich knowledge base of past defects and testing experience.
• Prove candidates against the sandbox, writing a test to allow for reproduction. Each defect reported to the user comes with a test that doesn't work, and can be actually reached from using the application.

Code review and Heal both aim to catch defects before they get deployed to users. Ultimately, both look at automating a part of the agentic verification loop, and making it as fast, cheap, and seamless as possible.

Code review focuses on "what a developer would change", and addresses things Heal doesn't cover, such as code style, code consistency, and adherence to internal guidelines. Heal focuses on "what users will encounter", and catches behavior, security, and performance issues that happen at runtime.

The first kind is argument-based: a developer or an agent reads the comment, decides whether it is right, and fixes or dismisses it. The second kind is proof-based: the defect is a behaviour a user would meet, and the test is the evidence. This benchmark measures what each kind finds on the same products.

The goal of the benchmark is to answer three questions:

• What is the benefit of having a runtime, a sandbox?
• Is it worth running both code review and Heal? What is the difference?
• Is it worth having several code reviewers?

The deeper question is: what would let PRs be merged much faster, with high confidence that there are no bugs?

Method

We chose five products: two private codebases that we own, anonymised here, and three open-source ones, Formbricks, Sim and Infisical.

We replayed each product's real pull-request history, PR by PR, into a fresh private repository with the three review bots installed on their default settings, and ran Heal before and after the sequence of PRs tested.

To tell whether a bot and Heal found the same thing, a judge compares them: a language model that reads a bot's claim and a Heal defect on the same lines and says whether they name the same problem. It was calibrated on a manually curated sample.

This is a relatively small dataset, but it gives a useful signal. We stopped after spending several thousand dollars on code review. We would be interested in running a fuller benchmark: if you are building a code reviewer and would like to collaborate, hit us up on X.

Tools evaluated

• Greptile with TREXCode review, with code snippet runtime validation
• cubicCode review
• CodeRabbitCode review
• HealRuntime validation

Benchmark stats

Limitations

• We did not assess whether the code review flags were relevant, whereas Heal defects are proven and reproducible with a test. This may create a bias in favor of code reviewers.
• Product A's sequence of PRs tested is its whole history, because the project is earlier stage. This is both favorable to code reviewers (everything got scanned) and to Heal (earlier projects likely contain more issues).
• What was tried first. Before the replays, the three open-source products were measured on one-week windows of their live pull requests, 25 to 30 each, with the bots as the maintainers run them and a full run at each end of the week. That method did not work well. Correlating a full run's defects with a week of live history is too complex: the state the run saw and the lines the week's PRs changed drift apart, each repository had different bots on different settings, and a defect could seldom be tied to one PR with confidence. Replaying the PR history on a private copy removes all three problems, so those windows were dropped from this page and their data kept as a side check.
• We couldn't validate if the code review comments were acted upon by the authors, as the replays were isolated from the real history. The reason we did not use the real history is code reviewers were not installed on the repos at the time, which would have made the data too scarce to action.

Across the five products, 70 % of the user-facing defects that the full run proved on the code of the sequence of PRs tested (141 of 202) had never been found by any review bot.

More candidates, filtered by the sandbox

A bot hallucinates by nature and needs to manage its noise level. Heal's candidates are allowed to be wrong, then go to the running sandbox, where it's tested and either confirmed and reproduced or discarded.

Ultimately this leads to extra bugs found without increasing the noise level. Additionally, because Heal filters using the sandbox, it raises only bugs that impact the user. We'll see in the next section how this is complementary to code review.

Examples: Four defects no code review bot could find

So what do code reviewers find that Heal doesn't? The majority of findings are issues with files that don't impact the runtime (CI, analytics, tests) that Heal doesn't read. Those produce a large number of findings. The rest are mostly race conditions.

We sorted the bots' 1,003 distinct flags into four buckets:

• In files Heal does not look at, by design: 654 (309 on Product A, 65 on B, 100 on Formbricks, 94 on Sim, 86 on Infisical). Heal starts from what a user can do and works inward, so it skips code no user ever reaches (CI configuration, database migration history, dependency versions, analytics, upgrade scripts, the product's own tests). That is where code review is the right tool.
• On code Heal tests, but not reported by Heal: 202 (168 on Product A, 10 on B, 7 on Formbricks, 9 on Sim, 8 on Infisical). Caveat: we did not check whether these are real bugs. Most are race conditions, which a single pass through a feature does not always trigger; section 8 explains how we plan to close that gap. Some look plausible:
  • A date that shifts by a day in time zones ahead of UTC
  • Edits typed during a save that get lost
  • On Sim, a file's access setting read before its database row is locked
• Out of scope: 100 (80 on Product A, 18 on B, 1 on Formbricks, 1 on Sim, 0 on Infisical). These sit on code that a later PR rewrote before Heal ran, so Heal never saw it, and it is out of the set.
• Also found by Heal: 47.

Examples: What Heal does not find, mostly outside the user path

Several teams we talk to run multiple code review bots. On our sample, this gives very marginal improvements.

Heal exists so that code written by coding agents validates itself: proven on the live product the way a user will meet it, with fewer people in the loop and less manual QA.

The raw material behind this page, for the three open-source products: every comment the review bots posted, and every defect Heal filed, with its test. The two private products are left out. Open the appendix.

• Formbricks: browse, or download the bot comments and the Heal defects as CSV.
• Sim: browse, or download the bot comments and the Heal defects as CSV.
• Infisical: browse, or download the bot comments and the Heal defects as CSV.