398 passes. 25 of them are not what they look like.

18 unearned passes, but only 14 tasks produce them — 16% of the suite. Reward-hacking here is a property of specific verifiers, not a general behaviour of the models.

Instruction-retire commit handshake is the worst offender by count, and every one of its passes is a hack — so its 100.0% resolve rate is worth nothing. 1 further task has a pass record made entirely of hacks: sfn-secrets-rotation-chain.

The hacks concentrate: Electrical Engineering 5 · Game 5 · Post Training 3 · Cloud Operations 2. The top two categories hold 56% of them on 30% of the trials. Measured against passes rather than trials — the denominator that matters, since a hack can only contaminate a pass — the leader is Game at 10.9% of its 46 passes, with Post Training level alongside it at 10.7% of 28.

Two tasks lead at 6 of 10 trials (apigw-http-api-jwt-authorizer-lambda-integration, iam-cross-account-externalid-sourcearn), both in Cloud Operations. A task where the majority of trials fail for reasons attributed to the task is not measuring an agent; it is measuring its own defects.

Cloud Operations carries 48% of all task-at-fault trials while holding 20% of the trials: 23.3% of its own trials versus 6.6% everywhere else. If one category needs a task-quality pass before its numbers are quoted, it is this one.

7 tasks appear on both the hack list and the fault list, which is the signature of a loose verifier rather than a hard problem. The worst of them is rate-limiter: 5 failures blamed on the task and 1 of its 2 passes hacked, so 6 of 10 trials point at the task in one direction or the other. The rest: camera-shake-rig, car-scene-assembly, iam-session-tag-tenant-scope, tile-gradient-particles, crusader-sprite-assembly, cg-solver.

Four tasks account for 47% of every harness error in the run (hash-message-padder, tile-gradient-particles, camera-shake-rig, s3-lambda-ddb-pipeline). The remaining 25 are scattered across 20 tasks, none of them more than 2. Infrastructure failure here is not ambient noise; it is a short list of environments.

hash-message-padder broke in 9 of its 10 trials, all under a single agent, which is also the only agent that ran it. Read the agent column and it looks like model flakiness; read the task column and it is one environment.

The highest per-agent rate belongs to claude-sonnet-4-6, on 35 trials over 4 tasks — and all 9 of its harness errors come from 2 of those tasks (tile-gradient-particles, camera-shake-rig). The rate describes an assignment, not an agent. By category the failures sit in Cloud Operations 18 (10%) · Electrical Engineering 11 (7%) · Game 10 (10%).

Of the 14 tasks nothing solved, only 5 are cleanly unsolved — every trial an honest miss, no harness error, no task blamed: coprocessor-dispatcher-classmix, diff-patch-engine, quaternion-rotation-integrator, reverse-engineer-objectives, serial-break-resync. That is 50 trials of genuine, unambiguous headroom.

8 of the 14 zero-solve tasks carry a suspect label, and one is not evidence of difficulty at all: hash-message-padder never resolved because 9 of its 10 trials never ran. debug-halt-step-fsm is the opposite failure — 4 of its 10 trials are blamed on the task itself. A third, deepcad-canonical-equivalence, resolved nothing while averaging 0.441: on a continuous grader, “nobody solved it” and “nobody got close” are not the same statement, and the reward column cannot tell them apart.

20 tasks were solved on every attempt (163 trials), against 14 at zero (126 trials) and 56 in between; the modal band is 25–49% with 27 tasks. But 7 of the perfect tasks have fewer than 10 trials — and at least one of those passes only by hacking — a 100% column at n=3 is a weak claim to retire a task on.