Bullet DOWNLOAD
B–LOG / 001 JULY 30 · THE BULLET TEAM

BENCHMARK / SWE-BENCH VERIFIED · 500 INSTANCES

95.8%ON SWE-BENCH VERIFIED

Bullet resolved 479 of 500 issues on SWE-bench Verified. This is the full human validated 500 instance split in a single attempt per task, graded by the official Docker evaluation harness. Mean time to a merged quality patch was 119 seconds.

479/500ISSUES RESOLVED
119sMEAN / INSTANCE
104sMEDIAN / INSTANCE
$0.73COST / INSTANCE
0EMPTY PATCHES / ERRORS
01

THE RESULT.

SWE-bench Verified is the industry standard test for autonomous coding agents. It uses 500 real GitHub issues from popular Python repositories, human filtered by expert annotators, where the agent must produce a patch that passes the project's own tests [1][2]. No partial credit. The fix either passes or it does not.

Bullet resolved 479 of 500 instances (95.8%). For context, Vals reports 97.0% for Claude Opus 5, 96.2% for GPT-5.6 Sol, and 95.0% for Claude Fable 5 on SWE-bench Verified [3]. Every Vals model is evaluated with the same mini-swe-agent harness.

The comparison below shows complete agent systems. Bullet uses our production harness. The Vals rows use mini-swe-agent. Accuracy, average latency, and cost per task are shown together so the tradeoffs are visible rather than reduced to one score.

FIG. 01ACCURACY · LATENCY · COST

Bullet figures come from our 500 task run. Vals figures come from its SWE-bench Verified benchmark where every listed model runs with mini-swe-agent [3]. Lower latency and cost are better.

02

HOW WE RAN IT.

  • OFFICIAL HARNESS Patches were graded with the official SWE-bench Docker evaluation harness [2]. This is containerized test execution scoring, not self grading.
  • SINGLE ATTEMPT One candidate patch per instance K=1. No best-of-N sampling, no reranking, no retries. 500 submitted, 500 completed, 0 empty patches, 0 harness errors.
  • PRODUCTION CONFIG The exact agent loop we ship with targeted search instead of repo embedding, parallel tool execution, and loop interception, running GPT‑5.6 Sol underneath.
  • LEAN BY DESIGN The run averaged 119 seconds and $0.73 per task. All 500 tasks completed with no empty patches and no harness errors.
FIG. 02CUMULATIVE INSTANCES COMPLETED VS ELAPSED TIME

Two thirds of all instances finish inside two minutes and every one of the 500 finishes in under 7 minutes and 1 second.

03

WHERE THE POINTS
CAME FROM.

Performance is consistent across the benchmark repositories with perfect scores on scikit-learn (32/32) and pytest (19/19), and 96.5% on Django, the largest slice at 231 instances. The hardest repo for us was astropy at 18/22. The 21 total misses are where the next points live.

FIG. 03RESOLVED BY REPOSITORY · 479/500

Bar length = % resolved. Counts show resolved / total instances per repository in this run.

04

PATCH DATA.

For anyone who wants to inspect every patch, logs, and outcomes from this run, we published the full 500-instance artifact set on Hugging Face here.

05

REFERENCES.

  1. [1] OpenAI Introducing SWE-bench Verified. The 500 instance human validated subset of SWE-bench.
  2. [2] Princeton NLP SWE-bench evaluation harness and SWE-bench_Verified dataset. Official containerized scorer used for this run.
  3. [3] Vals AI SWE-bench Verified benchmark. Accuracy, average latency, and cost per task for Claude Opus 5, GPT-5.6 Sol, and Claude Fable 5. Vals uses mini-swe-agent for every model.
  4. [4] SWE-agent mini-swe-agent. The minimal harness used by Vals for its model comparisons.
  5. [5] SWE-bench Official leaderboards.
  6. [6] Hugging Face bullet-swebench-verified. Full patch and run artifacts for this 500-instance result.