Skip to content
NixBench

NixBench 3.2.0 results

How often do coding agents fix real Nix problems?

Each agent gets a broken Nix repository and a symptom-level prompt. A hidden evaluator checks real Nix behaviour. A task passes only when every required criterion passes without a timeout, and every configuration runs the whole corpus several times.

Leaderboard Per-task matrix Earlier results
Tasks
21
Configurations
5
Trials each
3
Task-trial observations
315

Across corpus generations

Corpus progression

The same five configurations ran the same sandboxed protocol, three trials each, on three successive corpora. 3.0.0 graded single-file Nix idioms against stand-in package sets, and frontier agents solved them from memory: 24 of 30 tasks were saturated. 3.1.0 evaluated complete NixOS systems and real package expressions against a pinned nixpkgs tree, in multi-file repositories partly built on post-cutoff nixpkgs changes. Saturation fell to 9 of 21, but agents could still iterate on evaluation errors until the configuration evaluated. 3.2.0 moved to silent runtime failures whose starters already evaluate cleanly, three of them graded by booting NixOS VMs. With no error message to chase, the best configurations fell to 82.5% and only 3 of 21 tasks remain saturated. The lines fall because each corpus is harder than the last, not because the agents got worse, and the spread between best and worst grew from 17 points to 51 points, which is what lets the benchmark separate models. Run reports: 3.0.0 calibration, 3.1.0 calibration, 3.2.0 calibration.

Task pass rate by corpus generation

The same five configurations on each corpus. Whiskers are 95% Wilson intervals. Each generation is a different task set, so compare configurations within a column; a falling line means a harder corpus, not a weaker agent.

Key: 5 configurations · circle = Codex CLI, square = Claude Code
  • GPT-6 AstraCodex CLI
  • Claude Opus 5.5Claude Code
  • Claude Sonnet 5.5Claude Code
  • GPT-6.1 SolCodex CLI
  • GPT-6 LunaCodex CLI
0%25%50%75%100%
Show the values as a table
ConfigurationCorpusPassedPass rate95% Wilsonpass^kAgent time per task
GPT-6 Astra via Codex CLI3.0.088/9097.8%92.3–99.4%96.7% (k=3)81 s
GPT-6 Astra via Codex CLI3.1.060/6395.2%86.9–98.4%95.2% (k=3)95 s
GPT-6 Astra via Codex CLI3.2.052/6382.5%71.4–90.0%76.2% (k=3)100 s
Claude Opus 5.5 via Claude Code3.0.089/9098.9%94.0–99.8%96.7% (k=3)25 s
Claude Opus 5.5 via Claude Code3.1.059/6393.7%84.8–97.5%90.5% (k=3)58 s
Claude Opus 5.5 via Claude Code3.2.051/6381.0%69.6–88.8%81.0% (k=3)61 s
Claude Sonnet 5.5 via Claude Code3.0.088/9097.8%92.3–99.4%93.3% (k=3)16 s
Claude Sonnet 5.5 via Claude Code3.1.060/6395.2%86.9–98.4%95.2% (k=3)35 s
Claude Sonnet 5.5 via Claude Code3.2.047/6374.6%62.7–83.7%61.9% (k=3)34 s
GPT-6.1 Sol via Codex CLI3.0.087/9096.7%90.7–98.9%96.7% (k=3)77 s
GPT-6.1 Sol via Codex CLI3.1.058/6392.1%82.7–96.6%90.5% (k=3)111 s
GPT-6.1 Sol via Codex CLI3.2.052/6382.5%71.4–90.0%76.2% (k=3)121 s
GPT-6 Luna via Codex CLI3.0.074/9082.2%73.1–88.8%73.3% (k=3)47 s
GPT-6 Luna via Codex CLI3.1.030/6347.6%35.8–59.7%33.3% (k=3)73 s
GPT-6 Luna via Codex CLI3.2.020/6331.7%21.6–44.0%19.0% (k=3)69 s

Saturation and spread

CorpusSaturatedSpread
3.0.0†30 tasks24 of 3016.7 ptsOpus 5.5 98.9%
Luna 82.2%
3.1.021 tasks9 of 2147.6 ptsAstra · Sonnet 5.5 95.2%
Luna 47.6%
3.2.0†21 tasks3 of 2150.8 ptsAstra · Sol 82.5%
Luna 31.7%

Saturated: pooled solve rate of 90% or more across all configurations. Spread: best minus worst task pass rate; the names under it are the best and worst configurations. † Includes quarantined tasks as recorded; see the note below.

Quarantined tasks are included as recorded. Each of these evaluators rejected a valid alternative fix, so the points for its generation understate every configuration somewhat. Quarantined tasks are repaired and recalibrated before any activation decision.

  • 3.0.0: 1 quarantined task, editor-project-state-isolation (3.0.0 report)
  • 3.2.0: 3 quarantined tasks, hardened-profile-import-no-op, service-requires-removed-network-setup, tor-onion-socket-outside-chroot (3.2.0 report)

What changed in 3.2

Frontier agents solved every 3.1 task whose defect showed up as an evaluation error, by iterating until the error disappeared. 3.2 grades what evaluation cannot see.

3.2.0 release notes

  • Silent failures, not evaluation errors

    Starters evaluate cleanly against the pinned nixpkgs and misbehave only at runtime. The requirement is documented in the pinned tree; the prompt reports what the operator saw, never the mechanism.

  • VM-graded runtime track

    Three tasks are graded by booting NixOS VM tests against the candidate repository and asserting runtime behaviour, with each assertion mapped to its own criterion.

  • Real pinned nixpkgs, multi-host repositories

    Kept from 3.1: full-system evaluation against a content-hashed nixpkgs tree, 14 to 40 file repositories, and sibling hosts that must not regress.

Methods and evidence

Every number in these results comes from corpus nixbench-public 3.2.0, digest f783a51cb0ab. Results from other corpus versions are never pooled with these.

Download study matrix (JSON)
Corpus
nixbench-public 3.2.0
Corpus digest
f783a51cb0ab5457…
Trials per configuration
3
Isolation profile
linux-bwrap-fresh-home-v1
Agent adapter
codex-json-fresh-home-bwrap, claude-json-bwrap
Network
enabled
Agent time limit
300s per task
Run dates
2026-10-04 to 2026-10-04
Matrix generated
2026-10-04
Excluded attempts
0 invalid or incomplete
  • Headline: task pass rate; interval: wilson-95 over pooled valid task-trial observations; cells of the same task are not independent, so the interval is descriptive.
  • pass@k: unbiased estimator 1 - C(n-c,k)/C(n,k), averaged over tasks. pass^k: C(c,k)/C(n,k), averaged over tasks (all k sampled trials pass).
  • Empirical difficulty: pooled solve rate bands high >= 0.75, low < 0.25, mixed 0.25 to < 0.75; saturated at 90% or above.
  • Discrimination: point-biserial-leave-one-task-out, reported only with at least 20 observations across 2 configurations.

Full scoring policy: Scoring · Corpus release notes: 3.2.0 · Design rationale: Benchmark design research

Leaderboard

Configurations ordered by task pass rate. Bars are 95% Wilson intervals; n is shown for every estimate.

Metric
  • Claude Code
  • Codex CLI

Headline: share of valid task-trial observations that pass, with a 95% Wilson interval. Observations of the same task are not independent, so the interval is descriptive.

  1. 1–4

    GPT-6 Astra via Codex CLI

    gpt-6-astra · medium effort · Codex CLI

    Task pass rate
    82.5%
    71.4–90.0%
    Passed
    52/63
    task-trials
    Trials
    3
    21 tasks
    Timeouts
    0
    0.0%
    Agent / task
    1m 40s
    limit 300s
  2. 1–4

    GPT-6.1 Sol via Codex CLI

    gpt-6.1-sol · medium effort · Codex CLI

    Task pass rate
    82.5%
    71.4–90.0%
    Passed
    52/63
    task-trials
    Trials
    3
    21 tasks
    Timeouts
    0
    0.0%
    Agent / task
    2m 01s
    limit 300s
  3. 1–4

    Claude Opus 5.5 via Claude Code

    claude-opus-5-5 · default effort · Claude Code

    Task pass rate
    81.0%
    69.6–88.8%
    Passed
    51/63
    task-trials
    Trials
    3
    21 tasks
    Timeouts
    0
    0.0%
    Agent / task
    61 s
    limit 300s
  4. 1–4

    Claude Sonnet 5.5 via Claude Code

    claude-sonnet-5-5 · default effort · Claude Code

    Task pass rate
    74.6%
    62.7–83.7%
    Passed
    47/63
    task-trials
    Trials
    3
    21 tasks
    Timeouts
    0
    0.0%
    Agent / task
    34 s
    limit 300s
  5. 5

    GPT-6 Luna via Codex CLI

    gpt-6-luna · medium effort · Codex CLI

    Task pass rate
    31.7%
    21.6–44.0%
    Passed
    20/63
    task-trials
    Trials
    3
    21 tasks
    Timeouts
    0
    0.0%
    Agent / task
    69 s
    limit 300s

Rank is a plausible range: a configuration is placed behind another only when their 95% intervals do not overlap, so “1–3” means the data cannot order those configurations. Configurations with fewer than 3 trials are shown but never ranked. Agent time is the mean wall-clock agent seconds per task observation.

Pass rate vs. agent time

Up is more tasks passed; left is less agent time per task. Vertical whiskers are 95% Wilson intervals. Every value is also in the leaderboard above.

  • Claude Code
  • Codex CLI

Per-task results

Every task against every configuration. Cells count passing trials; select a task for discrimination, difficulty, criterion failures, and each trial's outcome.

Browse the task corpus
Solve rate across trials
  1. 0
  2. ≤ 1/3
  3. ≤ 2/3
  4. < 1
  5. 1
no valid observation timeout in at least one trial
Solve rate per task and configuration. Each cell shows passing trials over valid trials. Select a task for details.
TaskGPT-6 Astra via Codex CLIGPT-6.1 Sol via Codex CLIClaude Opus 5.5 via Claude CodeClaude Sonnet 5.5 via Claude CodeGPT-6 Luna via Codex CLIPooled
devshells · 1 task
67%10/15
modules · 15 tasks
0%0/15
0%0/15
13%2/15
67%10/15
73%11/15
80%12/15
80%12/15
80%12/15
80%12/15
80%12/15
80%12/15
87%13/15
93%14/15
100%15/15
100%15/15
nix-language · 1 task
73%11/15
overlays · 2 tasks
80%12/15
87%13/15
packages · 2 tasks
80%12/15
80%12/15

Columns follow the pass-rate order above. Rows are grouped by category, hardest first. A clock marks a cell with at least one agent timeout; a dashed cell has no valid observation, which is different from zero passes. Hover or focus a cell for trial outcomes; select a task for its full breakdown.

devshells

Development Shell Tooling Contract

devshell-tooling-contract

shell.nix is still a placeholder that returns {}, so there is no development shell to enter. Turn it into the project's development shell.

Pooled solve rate
66.7% 10/15
descriptive; no interval
Empirical difficulty
mixed
band: mixed
Author difficulty
easy
Discrimination
unavailable
insufficient sample size: n=15, 5 configs (needs 20 across 2)

Criterion failures

  • shell-constructionno failures
  • required-toolsno failures
  • formatter-contractGPT-6 Luna via Codex CLI 3/3
  • experimental-featuresGPT-6 Astra via Codex CLI 1/3 · GPT-6.1 Sol via Codex CLI 1/3 · GPT-6 Luna via Codex CLI 2/3

Trials by configuration

GPT-6 Astra via Codex CLI2/3 · 56 s avg

  • 20261004T145808Z-1c5b9b61 failfailed: experimental-features78 s
  • 20261004T153841Z-d363bfec pass49 s
  • 20261004T161734Z-824a5bc7 pass42 s

GPT-6.1 Sol via Codex CLI2/3 · 75 s avg

  • 20261004T145808Z-ccdf993a failfailed: experimental-features80 s
  • 20261004T154851Z-44e9c91f pass85 s
  • 20261004T163812Z-2e7ace0c pass60 s

Claude Opus 5.5 via Claude Code3/3 · 22 s avg

  • 20261004T145808Z-ac180016 pass19 s
  • 20261004T152637Z-36f0dd12 pass25 s
  • 20261004T155204Z-c7a68bb9 pass20 s

Claude Sonnet 5.5 via Claude Code3/3 · 12 s avg

  • 20261004T145808Z-35aa8694 pass12 s
  • 20261004T151616Z-9b9eb3f0 pass13 s
  • 20261004T153214Z-e12ea0a7 pass10 s

GPT-6 Luna via Codex CLI0/3 · 35 s avg

  • 20261004T145808Z-a98123d1 failfailed: formatter-contract36 s
  • 20261004T152711Z-e2afbbdb failfailed: experimental-features, formatter-contract29 s
  • 20261004T155626Z-6852c459 failfailed: experimental-features, formatter-contract41 s

Failure sets

Where configurations agree and disagree. Shared failures point at hard tasks or evaluator issues; solo solves show distinct strengths.

Never solved by any configuration

Solved in every trial by every configuration

Solved by only one configuration

None.

Intermittent: passed in some trials, failed in others

Failure overlapTasks each pair failed in at least one trial: shared / union, with Jaccard similarity. Similar totals can hide different weaknesses.
ConfigurationGPT-6 Luna via Codex CLIGPT-6 Astra via Codex CLIClaude Sonnet 5.5 via Claude CodeGPT-6.1 Sol via Codex CLIClaude Opus 5.5 via Claude Code
GPT-6 Luna via Codex CLI17 failed tasks—5/170.297/180.395/170.293/180.17
GPT-6 Astra via Codex CLI5 failed tasks5/170.29—3/100.305/51.003/60.50
Claude Sonnet 5.5 via Claude Code8 failed tasks7/180.393/100.30—3/100.303/90.33
GPT-6.1 Sol via Codex CLI5 failed tasks5/170.295/51.003/100.30—3/60.50
Claude Opus 5.5 via Claude Code4 failed tasks3/180.173/60.503/90.333/60.50—