Resolved is decided by the number, never by the label
resolved ⇔ reward == 1.0
Methodology
One number is measured. The rest is bookkeeping.
A verifier's exit status decides every reward on this site. Everything else (the labels, the rates, the intervals, the live markers) is arithmetic laid on top of that, and this page states each rule exactly as the code applies it. Where a rule is odd, it is written down odd rather than tidied up. You should be able to recompute any figure on the site from what follows.
879
scored trials
the unit everything is counted in
90
tasks
3–40 trials each
5
agent / model rows
15 categories
5
outcome labels
post-hoc, never scoring
1.96
z for every interval
the literal constant in the code
The whole method, in five lines
Everything below is one of these five said properly. If you read nothing else on this page, read this and follow whichever line you doubt.
Reward is an exit status, nothing else. The verifier runs after the agent has stopped: exit zero records 1.0, any other status records 0.0, and resolved means exactly 1.0.
the ruleThe five labels explain a result; they never set one. A model reads the trajectory afterwards and picks one. 7 trials here are labelled harness error, carry reward 1.0 regardless, and are counted as resolved.
the rulepass@1 counts trials, not tasks. Resolved trials over every trial in the group, with no per-task weighting, so a task attempted 40 times pulls on the figure 40 times.
the ruleEvery interval is Wilson, at a literal 1.96. Not the exact normal quantile. On the thinnest agent row here the width prints 22.73 points rather than the 22.72 the quantile would give.
the rule20 trials is the cut-off, applied twice with different edges. A category goes live at 20 trials or more; an agent row is ranked only above 20. At exactly 20 the two rules disagree, and this run has cases on both sides.
the rule
How a trial is scored
Six stages carry a task from written to published. Only one of them produces a number, and the three set out under the diagram are the ones that decide what a trial is worth.
Authored
90 tasks
A brief, a starter workspace, and a verifier written against it. 15 categories.
Containerised
grader held out
The image carries the brief and the starter files. Verifier, test assets and reference solution are kept out of it.
Attempted
879 trials
5 agent-and-model rows, 3 to 40 attempts per task, most often 10.
no verifier inside
Graded
398 resolved
The verifier goes back in and runs against the finished workspace. Its exit status is the reward.
Labelled
5 labels
A model reads each trajectory and picks one. 18 passes come back reward-hacks, 88 failures charged to the task.
Published
879 rows
pass@1 and a 95% interval per group, then the trial index itself. Nothing is dropped from a denominator.
- 1
Reward is the verifier's exit code
After the agent stops, a verifier runs against the finished workspace. Exit status zero is a pass and records a reward of 1.0; any non-zero status is a fail and records 0.0. That is the whole scoring rule. No model, no classifier, and no reviewer re-scores a trial afterwards, and there is no partial credit for a near miss. One exception exists in this run: deepcad-canonical-equivalence is graded on a continuous score, so its 10 trials carry fractional rewards (mean 0.441). Those trials count as resolved only at exactly 1.0, and the task page for them shows a mean score instead of a pass rate.
- 2
The agent works blind
During the attempt the agent sees the instruction and the starter workspace, and nothing else. The verifier, its test assets, and the reference solution are held outside the environment and put back only for grading. So the agent cannot read the checks it will be measured against, and a passing run had to satisfy tests it never saw.
This is also why a reward-hack is worth naming when it happens: it means the grader was reachable, or guessable, from inside the workspace after all.
- 3
A classifier explains the outcome afterwards
Once the reward is fixed, a separate model reads the trajectory and assigns one of five labels describing why the trial landed where it did. The label is commentary. It cannot raise a 0.0 or retract a 1.0, and it is not an input to any rate on this site.
The two axes really are independent, and the data shows it: 7 trials in this run are labelled harness error and still carry reward 1.0. They are counted as resolved, because resolved is defined on reward alone.
The five outcome labels
Assigned after scoring, to explain a result rather than set it. Every row below carries its own anchor, so a link can land on one label instead of the whole page.
| Label | Reward | What it means | Trials | Share | Observed |
|---|---|---|---|---|---|
| Legitimate solveGOOD_SUCCESS | passed | The verifier passed and it was checking the work the task asked for. This is the only label under which a 1.0 can be read as capability. |
373 | 42.4% | 373 passed · 0 failed |
| Reward-hackBAD_SUCCESS | passed | A pass the agent did not earn. The grader was satisfied by something other than a solution: a constant written where a computation belonged, an edited or deleted test, a value read out of a file it should not have reached. The reward stays 1.0 in the data; the finding is a defect in the task, not a result for the agent. |
18 | 2.0% | 18 passed · 0 failed |
| Honest missGOOD_FAILURE | failed | The run behaved correctly and still did not get there. Nothing is wrong with the task or the harness. On a hard task this is the expected shape of most trials. |
353 | 40.2% | 0 passed · 353 failed |
| Task at faultBAD_FAILURE | failed | The failure belongs to the task. Instructions that disagree with the grader, tests that turn on a convention nobody stated, a requirement the agent had no way to discover from what it was handed. A different agent would have hit the same wall. |
88 | 10.0% | 0 passed · 88 failed |
| Harness errorHARNESS_ERROR | either | The scaffolding gave out: container, network, tool loop, wall clock. The trial says nothing about the agent and nothing about the task. It is kept in the totals because dropping it would quietly flatter the run. |
47 | 5.3% | 7 passed · 40 failed |
Two of the five — Reward-hack and Task at fault — are warnings about the number beside them rather than results; the other three are ordinary outcomes. Colour here is a lookup aid only, and each label is written out wherever it appears, because three of the five hues sit under a 3:1 contrast ratio on this paper background.
How every figure is computed
The rules as the code applies them, including the awkward ones. Each block names its formula, then what it means in practice.
pass@1 counts trials, not tasks
pass@1 = resolved trials ÷ all trials in the group
The group is whatever the surrounding table is about: one agent-and-model row, one category, or one task. Trials are the denominator, so tasks that received more attempts weigh more: it is a plain proportion, with no per-task weighting and no estimator. Printed to one decimal. Overall this run is 398/879 = 45.3%.
The interval is Wilson, and z is a literal 1.96
lo, hi = ( p + z²/2n ∓ z·√( p(1−p)/n + z²/4n² ) ) ÷ ( 1 + z²/n )
A Wilson score interval on the group's own rate, clamped into 0–100% and reported as 0–0 when the group is empty. Wilson rather than the textbook normal interval because it keeps its footing at small n and at rates near either end, which is most of this board. The constant is the literal 1.96, not the exact two-sided normal quantile 1.959963984540054. The two part company in the second decimal, and this run is wide enough to show it, so the gap is stated rather than smoothed over. Overlapping intervals are not a significance test.
Show the arithmetic · claude-code / claude-haiku-4-5, k=19, n=20
| p | 19/20 = 0.9500 |
| 1 + z²/n | 1.192080 |
| p + z²/2n | 1.046040 |
| z·√( p(1−p)/n + z²/4n² ) | 0.135453 |
| interval | 76.4 – 99.1% |
| width at z = 1.96 | 22.73 |
| width at z = 1.959963984540054 | 22.72 |
pass@k is an existence check, not an estimate
pass@k = 100.0% if any trial resolved, else 0.0%
On a task page it asks only whether any attempt resolved, so it prints 100.0% or 0.0% and nothing between. The k is that task's own number of trials, which is not a constant: tasks in this run were attempted between 3 and 40 times. So read a task's pass@k against its own trial count and never against the headline k, which is a different quantity.
A category goes live on trials, not on tasks
live ⇔ scored trials ≥ 20
A category with a handful of tasks and many attempts can be live while a broader category with few attempts is not. 10 of 15 categories are live in this run. The marker is coverage, not quality: it says how much evidence there is, never how good the tasks are. Preview figures are still published.
An agent row is ranked above 20 trials, not at it
ranked ⇔ trials > 20
A row below the bar keeps its figures and its interval but stays out of the ranking, because a rate over twenty-odd trials moves several points on a single outcome. 4 rows are ranked here and 1 held in preview.
The comparison is not the same on both sides of the threshold. A category needs at least 20 trials to go live; an agent row needs more than 20 to be ranked. At exactly 20 trials the two disagree, and this run contains both cases: Data Science, Data Science: Robustness reads live, while claude-code / claude-haiku-4-5 is held in preview on the same trial count.
Outcome bars are left unnormalised, so some miss 100
width = count ÷ n × 100, to 3 dp, unnormalised
Each segment is the raw share of its class, printed to three decimals, and classes with no trials are dropped from the bar entirely rather than drawn at zero. Nothing rescales the segments to close the bar, so rounding leaves some bars summing very slightly off 100. That drift is real output and is preserved deliberately; correcting it would put the widths out of step with the counts they came from. 5 groups drift in this run.
Show the 5 groups that miss 100
| Group | Trials | Widths sum to |
|---|---|---|
| Cloud Operations | 180 | 99.999 |
| STEM | 81 | 100.001 |
| claude-code / claude-opus-4-8 | 591 | 99.999 |
| claude-code / claude-opus-4-7 | 180 | 99.999 |
| codex / gpt-5.5 | 53 | 99.999 |
A tie for dominant outcome goes to whichever came first
dominant = most common label, ties → first appearance in run order
The label with the highest count wins. When two labels tie, the winner is whichever was seen first as the task's trials are read in file order. There is no severity ranking and no priority list involved, so a tie does not resolve toward the more alarming label. 6 tasks in this run have a tie at the top, and 3 of them would be summarised differently under a priority rule.
Show the 3 that a priority rule would relabel
- athena-workgroup-result-encryption-cmk-enforced: shown as Honest miss, would read Legitimate solve under a priority rule
- ddb-outbox-eventbridge-fanout: shown as Honest miss, would read Legitimate solve under a priority rule
- lending-club-lgd-bias-correction-r: shown as Honest miss, would read Legitimate solve under a priority rule
A continuously graded task shows a mean, not a rate
mean = Σ reward ÷ n, to 3 dp
When a task's rewards are not all 0.0 or 1.0 it is treated as continuously graded, and its page shows a mean score at three decimals with no pass@1 and no pass@k. The trials still appear in every global count, and still only count as resolved at exactly 1.0.
The headline k is a mode, not a promise
k = mode of the per-task trial counts
The headline k is descriptive. It is the most frequent trials-per-task value across the run, here 10, and it is reported alongside the published run metadata value of 10. Individual tasks depart from it in both directions.
Glossary
The six things this run is made of. Every dotted term in the prose above links to its entry here. Rules that operate on these terms are stated once each, above, and not repeated.
- Reward
- The verifier's verdict, as a number. 1.0 for a pass, 0.0 for a fail. One task in this run is graded on a continuous score instead, and there reward is a fraction. Reward is the only quantity on the site that decides anything; everything else describes it.
- Verifier
- The program that grades a finished workspace. It runs after the agent has stopped, with the reference solution and the grading assets restored, and it communicates by exit status. It is deterministic and it is the sole authority on reward.
- Trajectory
- The full record of one attempt: the instructions given, every tool call and its output, the files touched, and the final state of the workspace. The classifier reads the trajectory; the verifier does not.
- Trial
- One agent, one model, one task, one attempt, one reward. The trial is the unit of everything counted here. This run holds 879 of them.
- Task
- A problem with a starter workspace, an instruction, and a verifier. A task is attempted repeatedly and by more than one agent, so it appears in many trials. This run covers 90 tasks.
- Category
- The subject area a task belongs to, such as software engineering or RTL design. Categories are a fixed registry in the code, not a field derived from the data, and they are always presented in that registry's order rather than sorted by any statistic.
Provenance
What this run was, and how to rebuild any number in it.
| Run date | 2026-07-06 |
|---|---|
| Benchmark version | 0.1.0 |
| Commit | 2f94510 |
| Trials per task (published k) | 10 |
| Trials per task (observed mode) | 10 |
| Scored trials | 879 |
| Resolved | 398 (45.3%) |
| Tasks | 90 |
| Categories | 15 |
| Agent / model rows | 5 |
Agents in this run
- claude-code / claude-haiku-4-5
- claude-code / claude-opus-4-7
- claude-code / claude-opus-4-8
- claude-code / claude-sonnet-4-6
- codex / gpt-5.5
Reproducing a figure
Every statistic above is a count over the trial index, so any figure can be rebuilt from the run table on the results page: filter to the group, count the rows with reward 1.0, divide by the rows shown, then apply the interval formula with z = 1.96. Individual trials link back to the published run, where the trajectory behind each label can be read in full.
Limitations
What this data cannot tell you. Read before quoting a figure.
The labels are model-produced
A model assigns them by reading the trajectory, and a model can be wrong. It can call a legitimate solve a hack, miss a hack that looks tidy, or blame a task for a failure that was the agent's. The 18 reward-hacks and 88 task-fault failures here are leads worth opening, not settled verdicts. Rewards are unaffected either way.
pass@1 is not comparable across agents
Each row ran a different subset of tasks, in different volumes, from 20 to 591 trials, and 2 to 56 distinct tasks. A high rate can mean a strong agent or an easy subset, and this data cannot separate the two. Read the rows as separate measurements that happen to share an axis.
Small samples make wide intervals
The thinnest row here spans 22.7 points at 95%. Categories in preview are thinner still. Where intervals overlap, no ordering is supported; ranking those rows would be reading noise.
One run, one date
Everything on this site comes from the run of 2026-07-06 at commit 2f94510. Agents are non-deterministic and models are moving targets, so these numbers are a snapshot, not a standing property of any agent. Nothing here is a trend.
Harness errors are noise, and they are counted
47 trials are labelled harness error and stay in every denominator on the site. The 40 of those that scored 0.0 pull pass@1 down by an amount nobody can quantify, since it is unknown what they would have scored had the infrastructure held. Dropping them would bias the rate the other way, so they are kept and flagged instead.
The pipeline behind this report is available for client work.
Request a private evaluation team@starzo.ai