For repository bug fixes, start with SWE-bench Verified. For editing exercises, use Aider Polyglot; for programming problems, LiveCodeBench; for shell tasks, Terminal-Bench. A score is useful when its task, agent and budget resemble the work you want to run.
This comparison gives you the benchmark map, dated score examples and a worksheet for choosing between coding models. The numbers below come from their linked publishers; ToLearn has not independently rerun those evaluations.
Score sources checked October 19, 2026. SWE-bench rows below are dated February 2026; Aider labels its page last updated November 20, 2025. These are the results visible in those source views when checked, rather than scores from new October benchmark runs.
Choose a benchmark for the task
| Your task | Start with | Check before comparing scores |
|---|---|---|
| Fix an issue in an existing repository | SWE-bench Verified | Dataset split, agent, tool access and retry policy |
| Apply a requested change to code | Aider Polyglot | Model settings, edit format, correct solutions and run cost |
| Solve algorithmic programming problems | LiveCodeBench | Problem release window, difficulty and scoring mode |
| Complete a task using a shell | Terminal-Bench | Benchmark version, agent, time limit and environment |
| Work through a math or science problem | A domain-specific reasoning benchmark | Subject, tier, tools and grading; keep this separate from coding scores |
For example, a Next.js bug that requires tracing a redirect, editing middleware and running a regression test resembles repository repair. A self-contained function puzzle is closer to LiveCodeBench. A model's strength on one is a reason to test it on the other, rather than evidence that it will succeed there.
SWE-bench Verified scores: the same agent environment
The SWE-bench leaderboard opens on the Verified, Bash Only view: 500 human-filtered tasks, run with mini-SWE-agent. The selected rows below all list agent release 2.0.0. Percent resolved means the share of benchmark instances solved; the dollar column is the source's reported average cost, rather than a price quote.
| Model and setting | Resolved | Reported average $ | Source row date |
|---|---|---|---|
| Claude 4.5 Opus, high | 76.80% | $0.75 | February 17, 2026 |
| Gemini 3 Flash, high | 75.80% | $0.36 | February 17, 2026 |
| MiniMax M2.5, high | 75.80% | $0.07 | February 17, 2026 |
| Claude 4.6 Opus | 75.60% | $0.55 | February 17, 2026 |
| GPT 5.2 Codex | 72.80% | $0.45 | February 19, 2026 |
| DeepSeek V3.2, high | 70.00% | $0.45 | February 17, 2026 |
Source view and values checked October 19, 2026. These are selected entries, not an exhaustive ranking of all models or agents. The source's Date column is retained as a row date; it does not independently establish when every task was run.
For a repository-repair pilot, this table gives you a concrete accuracy/cost pair to investigate. MiniMax M2.5 and Gemini 3 Flash have the same displayed resolved rate but different reported costs. Claude 4.5 Opus has the highest resolved rate among these selected rows. The table does not establish whether those differences persist on your repository or under a different agent.
The shared agent release makes these rows easier to interpret than a table combining unrelated launch claims. Still inspect reasoning settings, run limits and trajectories before treating the rows as identical experiments. A model name alone is insufficient: the same model can have entries under older agent releases and different budgets.
Aider Polyglot scores: editing accuracy and run cost
The Aider leaderboard evaluates editing exercises across six programming languages. Its public page labels the Polyglot set as 225 exercises and lists a page update date of November 20, 2025. We rechecked the displayed results on October 19, 2026. The old page date is still part of the evidence: these rows do not establish the leading model in October 2026.
| Aider model configuration | Percent correct | Reported cost for the benchmark run |
|---|---|---|
| GPT-5 high | 88.0% | $29.08 |
| GPT-5 medium | 86.7% | $17.69 |
| gemini-2.5-pro-preview-06-05, 32k think | 83.1% | $49.88 |
| DeepSeek-V3.2-Exp, Reasoner | 74.2% | $1.30 |
The two GPT-5 rows illustrate a useful tradeoff: the high setting adds 1.3 percentage points at about 1.64 times the recorded run cost. That arithmetic describes this leaderboard run. It does not predict the number of production bugs either setting will fix.
Keep the model identifier and reasoning setting together. The 83.1% result belongs to 06-05 with 32k thinking, not the older 05-06 preview; the DeepSeek row is V3.2-Exp, not the later generic V3.2 entry. Aider also reports “correct edit format” separately from “percent correct.” Producing a valid diff does not mean the solution passes.
The dollar column is the cost of the benchmark run under its recorded conditions, rather than a current API quote, monthly subscription or cost per repository issue. GPT-5 high, medium and DeepSeek use the diff format in these rows; the Gemini row uses diff-fenced. Preserve those settings when investigating the results.
Why these benchmark scores cannot form one ranking
SWE-bench Verified and SWE-Bench Pro use different task sets. Keep their results separate even when both report percentages. Aider's percent correct is another measure again. An 88.0% Aider result cannot be ranked above a 76.80% SWE-bench result as if they were the same exam.
Vendor release reports can be useful additional evidence, but retain the exact dataset, agent, tool access and run budget. A product-page description such as “for coding and agents” is not a measured score. The comparison above uses benchmark-publisher leaderboards and keeps each test in its own table.
ToLearn's editorial map of benchmark families to development tasks.
What each score can tell you
LiveCodeBench: the problem window matters
LiveCodeBench uses programming problems released over time to help study contamination. Always record the selected problem window along with a score. A model evaluated on one release window cannot be ranked directly against a model evaluated on a different problem mix.
For an algorithm-heavy workload, choose a window appropriate to the models you are evaluating and record it in your results. Test repository editing separately. We do not publish a LiveCodeBench numeric ranking here because it would require a matching problem window and configuration for every row.
Terminal-Bench: check the environment and budget
A terminal task can require inspecting files, running tools, handling an error and validating the outcome. Check the benchmark version and agent configuration before using a Terminal-Bench score for selection. This page does not claim a current numeric ranking for terminal agents.
For your own evaluation, give each candidate the same installed dependencies, network policy, time limit and recovery budget. Record human intervention too: a successful task that requires an engineer to rescue the run has a different operating cost.
Run a 12-task comparison in your repository
Use published scores to select two candidates, then run this small pilot. It is a screening procedure, not a statistically conclusive leaderboard. The example below is a proposed evaluation; it is not a ToLearn model performance test.
- Choose 12 tasks with observable outcomes. Use four past bugs with regression tests, four representative edits and four shell or build tasks. If your work is entirely bug repair, use 12 bugs instead. Avoid tasks whose solutions the agent can read in the repository history.
- Match the starting commit for each task. Give both candidates isolated copies at that task's recorded commit, with the same dependency lockfile. Expose only the task description and the allowed context; keep the reference solution and acceptance checks outside the agent's view.
- Fix the run conditions. Record the exact model and agent versions, reasoning mode, tools, network access and token limit. As a starting policy, allow 20 minutes and one attempt per task for both candidates. Do not give extra retries to one side after seeing a failure.
- Define acceptance before running. Require the target behavior, regression checks and relevant existing tests to pass. Reject test deletion, suppressed errors and unrelated changes. Review patches without model labels where practical.
- Record every attempt. Save elapsed time, API cost, tool failures, review minutes, accepted or rejected outcome and rejection reason. Include failed attempts in cost totals.
Give both candidates the same task brief. Replace the bracketed fields before using this template:
Task: [observable bug or requested change]
Starting commit: [commit ID]
Allowed changes: [files or scope]
Constraints: [dependencies, network and data restrictions]
Acceptance checks: [target behavior and allowed test commands]
Run limit: 20 minutes, one attempt, no human fixes during the run
Return: patch, tests run, results and any unresolved issuesReview failures by category: incorrect behavior, regression, inability to finish, or excessive human correction. A slightly higher pass count may matter less than one severe regression. If results are close, add fresh tasks before committing a team to a new system.
The AI benchmark methodology guide has the full evidence-recording template; the AI agent tools comparison covers workflow selection. The coding productivity analysis explains why review time belongs in this evaluation.
Calculate cost per accepted fix
Token price is only one input. Use this calculation with the spend and review time from all attempts, including failures:
cost per accepted fix =
(total API spend + total review hours × reviewer hourly cost)
÷ accepted fixesHere is a hypothetical example, using an assumed reviewer cost of $60 per hour. It is not a pricing quote or a measured comparison of named models.
| Twelve attempted tasks | Candidate A | Candidate B |
|---|---|---|
| Accepted fixes | 9 | 6 |
| API spend across all attempts | $6.00 | $2.40 |
| Review time across all attempts | 45 minutes | 90 minutes |
| API spend per accepted fix | $0.67 | $0.40 |
| API spend plus review cost per accepted fix | $5.67 | $15.40 |
Candidate B has the lower API bill. Candidate A costs less per accepted change once review is counted. In a real pilot, also record agent subscription or infrastructure costs if they differ; keep user-visible latency and severe failures alongside the dollar result.
Before adopting a model, rerun the tasks that matter most with your current agent and actual usage prices. Keep the source URL, source date and evaluation settings next to any leaderboard score you retain. That gives the next model release a concrete comparison to beat.