Decision in brief
Use public coding agent benchmarks to shortlist systems for similar tasks, then choose using a blinded, paired test on your own repository. SWE-bench helps assess issue-to-patch capability and Terminal-Bench helps assess terminal task execution. For AI coding productivity, measure time to an accepted change including review, retries, and cleanup. Include a manual baseline if you need to know whether an agent improves your current workflow.

Choose a benchmark by the decision it can support
On a leaderboard, read the result as model plus agent harness plus benchmark version and run settings. Compare runs within the same task set, scoring conditions, and budget. Where available, read the harness, model version, tool access, time cap, and rerun policy before you interpret a gap.
Public results are useful as a free first-pass screen. For a reproducible pilot, pin the tool and model, record the date and configuration, and keep the evaluation fixtures and commands. A public task set can be familiar to a model or tailored by a harness. Include private fixtures that sample actual work and keep their acceptance checks independent of the agent.
| Your question | Evidence to start with | What to carry into a team trial |
|---|---|---|
| Can it fix issues in an existing repository? | SWE-bench results on one named split | Reproduce the issue, require passing regression tests, and review scope and maintainability. |
| Can it complete multi-step terminal work? | Terminal-Bench on one named version | Measure environment setup, tool calls, incomplete work, and whether the resulting state is correct. |
| Will reviewers spend less time per patch? | Paired repository test with blind diff review | Track hands-on review and rework, alongside agent wait time. |
| Does it beat our current process? | Matched manual and agent-assisted tasks | Use the same acceptance criteria and report total effort per accepted change. |
Why accepted output and active time belong in the score
In the early-2025 METR randomized study, experienced open-source developers took 19% longer on their own repositories when permitted to use the period’s AI tools. METR’s February 2026 follow-up described participation and timing limitations that weakened its estimate for newer systems. Use the study design as a reason to record actual work time and task context in your own pilot.
HN commenter jbellis, discussing Senior SWE-Bench, argued that well-designed small task sets may still measure transferable capabilities even when the tasks look different from day-to-day work. In a separate Show HN post, Darmani said people interviewed for Command Center described spending significant development time reading and cleaning up generated code. These are different lenses on the same decision. Retain test-passing benchmark results and charge every review or repair minute to the evaluated workflow.
Run a repository benchmark in a week
Select nine recent or representative tasks across bug fixes, small features, and routine maintenance. Create a fresh branch for each run and verify the baseline tests on the starting commit. Define acceptance as an independent pass on required tests, an appropriately scoped diff, and an approval from a reviewer who does not know which tool produced it.
- Prepare prompts from the issue brief and withhold reference patches. Use routine fixtures such as validation wording, CSV date handling, and a small backward-compatible config rename. Keep acceptance tests outside editable workspace files.
- Run two shortlisted tools from the same starting commit with equal time caps. Alternate order across tasks. Include a manual round on matched tasks, with people who have not seen the reference or competing diffs.
- Record tool and model version, mode and settings, wall time, human time, attempts, agent usage spend, tests passed, and reviewer reason for acceptance or rejection. Keep failed runs in the reported acceptance denominator.
- Report acceptance as accepted runs divided by all runs. Divide total human minutes by accepted runs and report wall time separately. If there are zero accepted runs, report the failure count and accumulated effort. Use task-level results to decide where each workflow goes into production.