Can AI coding agents write Nix that actually passes?
Objective repository-repair tasks scored by hidden shell evaluators—not by whether the output merely looks plausible.
Current release: 29-task corpus · one hidden evaluator per task · uncertainty shown when repeat trials exist
Ordered effort paths lead the view. Select a model for uncertainty and effort labels, or reveal every trial to inspect the underlying variation.
Mean tasks passed against mean agent seconds per task. Paths connect ordered effort configurations; select a model to reveal its 95% Student's t intervals and effort labels. The task axis focuses on 6–29 to make the observed differences legible. The time axis uses logarithmic spacing because observed runtimes span more than one order of magnitude. Individual trials are hidden in this summary view.
| 95% CI / observed | ||||||
|---|---|---|---|---|---|---|
| GPT-5.6 Sol via Codex CLI29-task corpus · 5 recorded trials · 4m timeout | medium | n=5 | 24.0 / 29 | 22.5–25.5observed 22–25 | 38.5s18m 36s / corpus | 0 |
| GPT-5.6 Sol via Codex CLI29-task corpus · 5 recorded trials · 4m timeout | xhigh | n=5 | 24.0 / 29 | 23.1–24.9observed 23–25 | 60.7s29m 20s / corpus | 0 |
| GPT-5.6 Sol via Codex CLI29-task corpus · 5 recorded trials · 4m timeout | low | n=5 | 23.6 / 29 | 22.2–25.0observed 22–25 | 27.4s13m 14s / corpus | 0 |
| GPT-5.6 Sol via Codex CLI29-task corpus · 5 recorded trials · 4m timeout | high | n=5 | 23.6 / 29 | 22.9–24.3observed 23–24 | 47.4s22m 53s / corpus | 0 |
| GPT-5.6 Luna via Codex CLI29-task corpus · 5 recorded trials · 4m timeout | medium | n=5 | 23.2 / 29 | 22.2–24.2observed 22–24 | 30.5s14m 45s / corpus | 0 |
| GPT-5.6 Sol via Codex CLI29-task corpus · 5 recorded trials · 4m timeout | max | n=5 | 23.2 / 29 | 20.1–26.3observed 19–25 | 81.4s39m 19s / corpus | 3 |
| GPT-5.6 Luna via Codex CLI29-task corpus · 5 recorded trials · 4m timeout | low | n=5 | 23.0 / 29 | 22.1–23.9observed 22–24 | 26.6s12m 51s / corpus | 1 |
| GPT-5.6 Terra via Codex CLI29-task corpus · 5 recorded trials · 4m timeout | high | n=5 | 22.8 / 29 | 20.8–24.8observed 20–24 | 38.4s18m 34s / corpus | 0 |
| GPT-5.6 Terra via Codex CLI29-task corpus · 5 recorded trials · 4m timeout | xhigh | n=5 | 22.8 / 29 | 21.8–23.8observed 22–24 | 48.8s23m 35s / corpus | 0 |
| GPT-5.6 Luna via Codex CLI29-task corpus · 5 recorded trials · 4m timeout | high | n=5 | 22.6 / 29 | 21.2–24.0observed 21–24 | 44.4s21m 29s / corpus | 0 |
| GPT-5.6 Luna via Codex CLI29-task corpus · 5 recorded trials · 4m timeout | xhigh | n=5 | 22.4 / 29 | 21.7–23.1observed 22–23 | 53.9s26m 2s / corpus | 0 |
| GPT-5.6 Luna via Codex CLI29-task corpus · 5 recorded trials · 4m timeout | max | n=5 | 22.4 / 29 | 20.7–24.1observed 21–24 | 75.0s36m 16s / corpus | 4 |
| GPT-5.6 Terra via Codex CLI29-task corpus · 5 recorded trials · 4m timeout | low | n=5 | 22.0 / 29 | 21.1–22.9observed 21–23 | 25.4s12m 16s / corpus | 0 |
| GPT-5.6 Terra via Codex CLI29-task corpus · 5 recorded trials · 4m timeout | medium | n=5 | 21.8 / 29 | 20.2–23.4observed 20–23 | 28.4s13m 44s / corpus | 0 |
| Gemma 4 26B-A4B QAT Q4_0 via OpenCode + llama.cpp29-task corpus · 1 recorded trial · 1h timeout | default | single run | 10.0 / 29 | unavailableobserved 10–10 | 1869.1s15h 3m / corpus | 3 |
| Ternary Bonsai 27B Q2_0 via OpenCode + llama.cpp29-task corpus · 1 recorded trial · 1h timeout | default | single run | 7.0 / 29 | unavailableobserved 7–7 | 2478.7s19h 58m / corpus | 4 |
Corpora are intentionally separated and time is normalized per task. The focused y-axis is explicitly labelled; Full scale restores the zero baseline. Lines show configuration order from lower to higher effort; they do not imply continuous scaling or monotonic treatment. See the reproducibility method and local OpenCode run provenance. Raw run IDs are shown in trial tooltips.
Current trial environments: 2 agent versions · 2 hosts · 2 repository revisions · network unknown · timeout budgets 240s, 3600s.
The benchmark gives agents a copied starter tree, a prompt, and no access to the hidden evaluator. It rewards final worktree behavior, not a fluent explanation of what the code should do.
Tasks are written for this corpus rather than lifted from merged patches, which keeps the answer out of the visible prompt.
The corpus covers flakes, modules, overlays, derivations, fetchers, Home Manager, shell escaping, and package contracts.
Each task has a shell evaluator that checks behavior with small fake package sets and libraries instead of relying on LLM judging.
Every run records logs, timings, pass state, score JSON, and the final diff so failures can be inspected after the benchmark ends.
Keep module outputs separated instead of leaking options across systems.
Repair Python/CUDA packaging without falling back to generic Linux path guesses.
Build paths with Nix values while avoiding string interpolation traps.
Explain the observed NixOS service behavior without chasing a plausible but wrong network diagnosis.
Use Home Manager file and XDG options rather than imperative setup.
Preserve the fixed-output fetcher contract with a commit pin and SRI hash.
easy tasks for syntax, lookup, stale options, and small contracts
medium repairs across flakes, containers, issue reports, overlays, packaging, and shell integration
hard tasks for modules, overlays, portals, Rust purity, and Python/CUDA package inputs
A simple, inspectable protocol separates generation from evaluation.
Starter files and the prompt enter a clean temporary workdir.
The agent reads NIXBENCH_PROMPT.md and modifies only local files.
A hidden shell evaluator scores the final tree after the agent exits.
Logs, timing, score JSON, and the final diff are written under results/.
Use the open harness, preserve the run artifacts, and add a comparable row to the benchmark.
Open the run guide