Results · run 2026-07-06 · SyncVals 0.1.0 · commit 2f94510
Five agent lines on one axis. Not a race.
Each row is pass@1 over the trials that row actually ran, and no two rows ran the same set of tasks. claude-code / claude-opus-4-8 was measured on 56 of 90 tasks; claude-code / claude-haiku-4-5 saw 2. Ranking the percentages against each other would mostly rank the task selections. Read each interval on its own terms.
- Ordered by trials, not by score. The board follows trial count downwards, which keeps the widest evidence at the top and puts the thinnest at the bottom where it belongs.
- A wide bar means few trials. The widest interval here spans 29 percentage points on 35 trials. That is arithmetic about sample size, not a claim about consistency.
- 20 trials or fewer is held back. Those rows are drawn, tagged, and excluded from ranking — visible because hiding them would hide the coverage gap too.
The board
The dot is the point estimate; the bar behind it is the 95% Wilson interval on that row's own trials. Under each bar, the same trials split five ways — a pass that gamed the grader and a failure the task caused are both scored, and both mislead. Every segment carries its label in the key; colour is a second cue, never the only one.
Outcome mix under each bar — five fixed classes, always in this order
Table view · leaderboard numbers
| Agent | Model | pass@1 | CI low | CI high | Resolved | Trials | Tasks | Status |
|---|---|---|---|---|---|---|---|---|
| claude-code | claude-opus-4-8 | 45.3% | 41.4 | 49.4 | 268 | 591 | 56 | ranked |
| claude-code | claude-opus-4-7 | 31.1% | 24.8 | 38.2 | 56 | 180 | 19 | ranked |
| codex | gpt-5.5 | 84.9% | 72.9 | 92.1 | 45 | 53 | 12 | ranked |
| claude-code | claude-sonnet-4-6 | 28.6% | 16.3 | 45.1 | 10 | 35 | 4 | ranked |
| claude-code | claude-haiku-4-5 | 95.0% | 76.4 | 99.1 | 19 | 20 | 2 | held back |
Interval bounds are the 95% Wilson score interval on that row's own trials, in percentage points. Resolved counts a reward of exactly 1.0 — partial credit is not a pass.
Table view · outcome counts per agent
| Agent / model | Legitimate solve | Reward-hack | Honest miss | Task at fault | Harness error | Trials |
|---|---|---|---|---|---|---|
| claude-code / claude-opus-4-8 | 261 | 7 | 269 | 38 | 16 | 591 |
| claude-code / claude-opus-4-7 | 51 | 2 | 67 | 42 | 18 | 180 |
| codex / gpt-5.5 | 37 | 6 | 4 | 3 | 3 | 53 |
| claude-code / claude-sonnet-4-6 | 6 | 3 | 12 | 5 | 9 | 35 |
| claude-code / claude-haiku-4-5 | 18 | 0 | 1 | 0 | 1 | 20 |
The same rows, split by category
Sub-boards stay closed so the page reads top to bottom; open one to see which agents were actually measured there. Intervals inside a category are wide by construction. A row of 10 trials can only land on one of 11 possible percentages, so treat these as coverage evidence first and performance evidence second.
Software Engineering live 57.6% pass@1 · 49/85 resolved · 8 tasks · 3 agent lines
Algorithms, data structures, bug-fixes, and API/systems implementation, graded against hidden test suites the agent never sees.
Mechanical Engineering live 35.0% pass@1 · 28/80 resolved · 8 tasks · 1 agent line
Structural and numerical solvers (FD/FE, contact dynamics) in C++, checked against multi-binary hidden references.
Cloud Operations live 31.1% pass@1 · 56/180 resolved · 19 tasks · 1 agent line
Provision, diagnose, and guardrail cloud infrastructure (AWS), verified against the deployed state.
Electrical Engineering live 51.9% pass@1 · 82/158 resolved · 20 tasks · 2 agent lines
RTL / digital-logic design checked under hardened simulation and formal-equivalence harnesses.
STEM live 72.8% pass@1 · 59/81 resolved · 5 tasks · 1 agent line
Quantitative reasoning across math, physics, and the natural sciences with deterministic, checkable answers.
Game live 43.8% pass@1 · 46/105 resolved · 10 tasks · 4 agent lines
Interactive simulation and game-logic tasks graded on exact state transitions and rule fidelity.
Debugging preview 100.0% pass@1 · 15/15 resolved · 3 tasks · 1 agent line
Real-world bug-fix tasks distilled from open-source issues (esbuild, klauspost/compress, rust-lang/semver), graded against each project's own tests.
ML Engineering preview 40.0% pass@1 · 4/10 resolved · 1 task · 1 agent line
Train and tune models to a target metric on real engineering datasets, graded on held-out performance against a solved-reward threshold.
Post Training live 43.1% pass@1 · 28/65 resolved · 6 tasks · 2 agent lines
Long-horizon, from-scratch numpy ML research: implement reverse-mode autodiff and a training method (quantization-aware, adversarial-robust, semi-supervised, worst-group), then train to a sealed held-out metric. Tasks published blinded.
Scientific ML live 33.3% pass@1 · 10/30 resolved · 3 tasks · 1 agent line
Physics-informed and surrogate modeling: PDE forecasting and CFD/FEA prediction, graded on quantitative predictive accuracy.
ML for Engineering preview 0.0% pass@1 · 0/10 resolved · 1 task · 1 agent line
Machine-learning-for-engineering tasks (DeepCAD command-history canonicalization and equivalence), graded on a continuous partial-credit score against verifier-owned labels rather than all-or-nothing pass/fail.
Data Science live 25.0% pass@1 · 5/20 resolved · 2 tasks · 1 agent line
Modeling and statistical inference on real datasets (federated learning, event studies), graded against held-out ground truth.
Data Science: Robustness live 40.0% pass@1 · 8/20 resolved · 2 tasks · 1 agent line
Bias-correction and outlier-robust analysis in R: recover trustworthy estimates from messy data, checked against reference results.
Product Data Science preview 40.0% pass@1 · 4/10 resolved · 1 task · 1 agent line
Applied product analytics: causal impact and decision analysis on real product datasets (R).
Pharmacometrics preview 40.0% pass@1 · 4/10 resolved · 1 task · 1 agent line
Population PK/PD modeling with nonlinear mixed-effects: fit drug-exposure models and recover the correct parameters.
pass@1 per category
One axis, one measure, fifteen categories. The bar in the pass@1 column is the category rate; the columns to its right are that rate for a single agent line, which is where the sample sizes get small. Check the resolved count under a cell before quoting the percentage above it — at 10 trials, one flipped verdict moves the figure ten points.
| Category | Tasks | Trials | pass@1 | claude-codeclaude-opus-4-8 | claude-codeclaude-opus-4-7 | codexgpt-5.5 | claude-codeclaude-sonnet-4-6 | claude-codeclaude-haiku-4-5 |
|---|---|---|---|---|---|---|---|---|
| Software Engineeringlive | 8 | 85 | 57.6%49/85 | 50.0%35/70 | — | 100.0%5/5 | — | 90.0%9/10 |
| Mechanical Engineeringlive | 8 | 80 | 35.0%28/80 | 35.0%28/80 | — | — | — | — |
| Cloud Operationslive | 19 | 180 | 31.1%56/180 | — | 31.1%56/180 | — | — | — |
| Electrical Engineeringlive | 20 | 158 | 51.9%82/158 | 47.7%62/130 | — | 71.4%20/28 | — | — |
| STEMlive | 5 | 81 | 72.8%59/81 | 72.8%59/81 | — | — | — | — |
| Gamelive | 10 | 105 | 43.8%46/105 | 35.0%21/60 | — | 100.0%5/5 | 33.3%10/30 | 100.0%10/10 |
| Debuggingpreview | 3 | 15 | 100.0%15/15 | — | — | 100.0%15/15 | — | — |
| ML Engineeringpreview | 1 | 10 | 40.0%4/10 | 40.0%4/10 | — | — | — | — |
| Post Traininglive | 6 | 65 | 43.1%28/65 | 46.7%28/60 | — | — | 0.0%0/5 | — |
| Scientific MLlive | 3 | 30 | 33.3%10/30 | 33.3%10/30 | — | — | — | — |
| ML for Engineeringpreview | 1 | 10 | 0.0%0/10 | 0.0%0/10 | — | — | — | — |
| Data Sciencelive | 2 | 20 | 25.0%5/20 | 25.0%5/20 | — | — | — | — |
| Data Science: Robustnesslive | 2 | 20 | 40.0%8/20 | 40.0%8/20 | — | — | — | — |
| Product Data Sciencepreview | 1 | 10 | 40.0%4/10 | 40.0%4/10 | — | — | — | — |
| Pharmacometricspreview | 1 | 10 | 40.0%4/10 | 40.0%4/10 | — | — | — | — |
| All categories | 90 | 879 | 45.3%398/879 | 45.3%268/591 | 31.1%56/180 | 84.9%45/53 | 28.6%10/35 | 95.0%19/20 |
Status is derived from trial count, not task count: a category goes live at 20 scored trials and stays in preview below that. Category rates mix agents together, so a category that only one agent attempted is reporting that agent, not the field.