Run 2026-07-06 · SyncVals 0.1.0 · commit 2f94510
A coding agent can pass for the wrong reasons.
This report separates the two.
A deterministic verifier decides every reward — exit zero, nothing else. A separate classifier then reads the trajectory and says why the run landed where it did, without ever touching the score. 879 trials, sorted five ways.
Every graded trial, five ways
Reward is binary; the reason is not. Two of these five mean the number lies — a pass that gamed the grader, or a failure the task itself caused. What each label means →
Table view
| Outcome | Label | Trials | Share |
|---|---|---|---|
| Legitimate solve | GOOD_SUCCESS | 373 | 42.4% |
| Reward-hack | BAD_SUCCESS | 18 | 2.0% |
| Honest miss | GOOD_FAILURE | 353 | 40.2% |
| Task at fault | BAD_FAILURE | 88 | 10.0% |
| Harness error | HARNESS_ERROR | 47 | 5.3% |
pass@1, with the uncertainty shown
Each bar is a 95% Wilson interval; the dot is the point estimate. These rates are not comparable across rows — every agent ran a different subset of tasks, and the widest intervals are simply the smallest samples. Full results, per category →
Coverage by category
Status is derived, not asserted: a category is live once it has at least 20 scored trials, and preview until then.
Algorithms, data structures, bug-fixes, and API/systems implementation, graded against hidden test suites the agent never sees.
Structural and numerical solvers (FD/FE, contact dynamics) in C++, checked against multi-binary hidden references.
Provision, diagnose, and guardrail cloud infrastructure (AWS), verified against the deployed state.
RTL / digital-logic design checked under hardened simulation and formal-equivalence harnesses.
Quantitative reasoning across math, physics, and the natural sciences with deterministic, checkable answers.
Interactive simulation and game-logic tasks graded on exact state transitions and rule fidelity.
Real-world bug-fix tasks distilled from open-source issues (esbuild, klauspost/compress, rust-lang/semver), graded against each project's own tests.
Train and tune models to a target metric on real engineering datasets, graded on held-out performance against a solved-reward threshold.
Long-horizon, from-scratch numpy ML research: implement reverse-mode autodiff and a training method (quantization-aware, adversarial-robust, semi-supervised, worst-group), then train to a sealed held-out metric. Tasks published blinded.
Physics-informed and surrogate modeling: PDE forecasting and CFD/FEA prediction, graded on quantitative predictive accuracy.
Machine-learning-for-engineering tasks (DeepCAD command-history canonicalization and equivalence), graded on a continuous partial-credit score against verifier-owned labels rather than all-or-nothing pass/fail.
Modeling and statistical inference on real datasets (federated learning, event studies), graded against held-out ground truth.
Bias-correction and outlier-robust analysis in R: recover trustworthy estimates from messy data, checked against reference results.
Applied product analytics: causal impact and decision analysis on real product datasets (R).
Population PK/PD modeling with nonlinear mixed-effects: fit drug-exposure models and recover the correct parameters.
Four numbers worth chasing
The headline rate hides these. Each one is a place where the score and the truth come apart.
Spread across 14 tasks. Each one is a grader that could be satisfied without doing the work.
Read the analysis →88failures the task causedConcentrated rather than scattered: apigw-http-api-jwt-authorizer-lambda-integration accounts for 6 on its own.
Read the analysis →14tasks nothing solvedAgainst 20 that every attempt solved. The gap between those two numbers is the benchmark's usable range.
Read the analysis →47runs that never really ranInfrastructure, not capability — 9 of them on hash-message-padder alone.
Read the analysis →Read next
pass@1 for all 5 agent lines with Wilson intervals, plus per-category sub-boards.
→TasksAll 90 tasks by category, with resolve rates and dominant outcomes.
→AnalysisWhat the numbers do and do not support: hacking, defective tasks, headroom.
→MethodologyHow a run is scored, what the five labels mean, and how every figure is computed.
→