Methodology · run 2026-07-06 · SyncVals 0.1.0 · commit 2f94510
A verifier's exit status decides every reward on this site. Everything else — the labels, the rates, the intervals, the live markers — is arithmetic laid on top of that, and this page states each rule exactly as the code applies it. Where a rule is odd, it is written down odd rather than tidied up. You should be able to recompute any figure on the site from what follows.
Three stages, in order. Only the first one produces a number.
After the agent stops, a verifier runs against the finished workspace. Exit status zero is a pass and records a reward of 1.0; any non-zero status is a fail and records 0.0. That is the whole scoring rule. No model, no classifier, and no reviewer re-scores a trial afterwards, and there is no partial credit for a near miss. One exception exists in this run: deepcad-canonical-equivalence is graded on a continuous score, so its 10 trials carry fractional rewards (mean 0.441). Those trials count as resolved only at exactly 1.0, and the task page for them shows a mean score instead of a pass rate.
During the attempt the agent sees the instruction and the starter workspace, and nothing else. The verifier, its test assets, and the reference solution are held outside the environment and put back only for grading. So the agent cannot read the checks it will be measured against, and a passing run had to satisfy tests it never saw.
This is also why a reward-hack is worth naming when it happens: it means the grader was reachable, or guessable, from inside the workspace after all.
Once the reward is fixed, a separate model reads the trajectory and assigns one of five labels describing why the trial landed where it did. The label is commentary. It cannot raise a 0.0 or retract a 1.0, and it is not an input to any rate on this site.
The two axes really are independent, and the data shows it: 7 trials in this run are labelled harness error and still carry reward 1.0. They are counted as resolved, because resolved is defined on reward alone.
Assigned after scoring, to explain a result rather than set it. Other pages link straight to these definitions.
| Label | Reward | What it means | Trials | Share | Observed |
|---|---|---|---|---|---|
| Legitimate solveGOOD_SUCCESS | passed | The verifier passed and it was checking the work the task asked for. This is the only label under which a 1.0 can be read as capability. |
373 | 42.4% | 373 passed · 0 failed |
| Reward-hackBAD_SUCCESS | passed | A pass the agent did not earn. The grader was satisfied by something other than a solution — a constant written where a computation belonged, an edited or deleted test, a value read out of a file it should not have reached. The reward stays 1.0 in the data; the finding is a defect in the task, not a result for the agent. |
18 | 2.0% | 18 passed · 0 failed |
| Honest missGOOD_FAILURE | failed | The run behaved correctly and still did not get there. Nothing is wrong with the task or the harness. On a hard task this is the expected shape of most trials. |
353 | 40.2% | 0 passed · 353 failed |
| Task at faultBAD_FAILURE | failed | The failure belongs to the task. Instructions that disagree with the grader, tests that turn on a convention nobody stated, a requirement the agent had no way to discover from what it was handed. A different agent would have hit the same wall. |
88 | 10.0% | 0 passed · 88 failed |
| Harness errorHARNESS_ERROR | either | The scaffolding gave out — container, network, tool loop, wall clock. The trial says nothing about the agent and nothing about the task. It is kept in the totals because dropping it would quietly flatter the run. |
47 | 5.3% | 7 passed · 40 failed |
Two of the five are warnings about the number beside them. Reward-hack says a pass should not be believed; Task at fault says a failure should not be charged to the agent. The other three are ordinary outcomes. Colour here is a lookup aid only — each label is written out wherever it appears, because three of the five hues sit under a 3:1 contrast ratio on this paper background.
The rules as the code applies them, including the awkward ones. Each block names its formula, then what it means in practice.
A trial is resolved when its reward is exactly 1.0. The comparison is against the number and nothing else — the label plays no part, which is why the 7 harness-error trials with reward 1.0 are inside every resolved count on the site. Fractional rewards are never rounded up.
The group is whatever the surrounding table is about: one agent-and-model row, one category, or one task. Trials are the denominator, so tasks that received more attempts weigh more. Printed to one decimal. Overall this run is 398/879 = 45.3%.
A Wilson score interval on the group's own rate, clamped into 0–100% and reported as 0–0 when the group is empty. The constant is the literal 1.96 — not the exact two-sided normal quantile 1.959963984540054. The difference is a hundredth of a point wide, and it is visible in this run, so it is stated rather than smoothed over.
| p | 19/20 = 0.9500 |
| 1 + z²/n | 1.192080 |
| p + z²/2n | 1.046040 |
| z·√( p(1−p)/n + z²/4n² ) | 0.135453 |
| interval | 76.4 – 99.1% |
| width at z = 1.96 | 22.73 |
| width at z = 1.959963984540054 | 22.72 |
It is an existence check, not an estimate, so it only ever prints those two values. The k is that task's own number of trials, which is not a constant: tasks in this run were attempted between 3 and 40 times. The run-level k of 10 is the most common trial count across tasks — a mode, not a guarantee — so read a task's pass@k next to its own trial count, never against the headline k.
Trials decide this, not tasks. A category with a handful of tasks and many attempts can be live while a broader category with few attempts is not. 10 of 15 categories are live in this run. The marker is about sample size only; it says nothing about task quality.
A row below the bar keeps its figures and its interval but stays out of the ranking, because a rate over twenty-odd trials moves several points on a single outcome. 4 rows are ranked here and 1 held in preview.
The comparison is not the same on both sides of the threshold. A category needs at least 20 trials to go live; an agent row needs more than 20 to be ranked. At exactly 20 trials the two disagree, and this run contains both cases: Data Science, Data Science: Robustness reads live, while claude-code / claude-haiku-4-5 is held in preview on the same trial count.
Each segment is the raw share of its class, printed to three decimals, and classes with no trials are dropped from the bar entirely rather than drawn at zero. Nothing rescales the segments to close the bar, so rounding leaves some bars summing very slightly off 100. That drift is real output and is preserved deliberately; correcting it would put the widths out of step with the counts they came from. 5 groups drift in this run:
| Group | Trials | Widths sum to |
|---|---|---|
| Cloud Operations | 180 | 99.999 |
| STEM | 81 | 100.001 |
| claude-code / claude-opus-4-8 | 591 | 99.999 |
| claude-code / claude-opus-4-7 | 180 | 99.999 |
| codex / gpt-5.5 | 53 | 99.999 |
The label with the highest count wins. When two labels tie, the winner is whichever was seen first as the task's trials are read in file order. There is no severity ranking and no priority list involved, so a tie does not resolve toward the more alarming label. 6 tasks in this run have a tie at the top, and 3 of them would be summarised differently under a priority rule:
When a task's rewards are not all 0.0 or 1.0 it is treated as continuously graded, and its page shows a mean score at three decimals with no pass@1 and no pass@k. The trials still appear in every global count, and still only count as resolved at exactly 1.0.
The headline k is descriptive. It is the most frequent trials-per-task value across the run, here 10, and it is reported alongside the published run metadata value of 10. Individual tasks depart from it in both directions.
Terms used across the site, in the sense they are used here.
What this run was, and how to rebuild any number in it.
| Run date | 2026-07-06 |
|---|---|
| Benchmark version | 0.1.0 |
| Commit | 2f94510 |
| Trials per task (published k) | 10 |
| Trials per task (observed mode) | 10 |
| Scored trials | 879 |
| Resolved | 398 (45.3%) |
| Tasks | 90 |
| Categories | 15 |
| Agent / model rows | 5 |
Every statistic above is a count over the trial index, so any figure can be rebuilt from the run table on the results page: filter to the group, count the rows with reward 1.0, divide by the rows shown, then apply the interval formula with z = 1.96. Individual trials link back to the published run, where the trajectory behind each label can be read in full.
What this data cannot tell you. Read before quoting a figure.
A model assigns them by reading the trajectory, and a model can be wrong. It can call a legitimate solve a hack, miss a hack that looks tidy, or blame a task for a failure that was the agent's. The 18 reward-hacks and 88 task-fault failures here are leads worth opening, not settled verdicts. Rewards are unaffected either way.
Each row ran a different subset of tasks, in different volumes — from 20 to 591 trials, and 2 to 56 distinct tasks. A high rate can mean a strong agent or an easy subset, and this data cannot separate the two. Read the rows as separate measurements that happen to share an axis.
The thinnest row here spans 22.7 points at 95%. Categories in preview are thinner still. Where intervals overlap, no ordering is supported; ranking those rows would be reading noise.
Everything on this site comes from the run of 2026-07-06 at commit 2f94510. Agents are non-deterministic and models are moving targets, so these numbers are a snapshot, not a standing property of any agent. Nothing here is a trend.
47 trials are labelled harness error and stay in every denominator on the site. The 40 of those that scored 0.0 pull pass@1 down by an amount nobody can quantify, since it is unknown what they would have scored had the infrastructure held. Dropping them would bias the rate the other way, so they are kept and flagged instead.