Task index · run 2026-07-06 · 90 problems
90 problems, each with a verifier the agent never sees.
A task is one self-contained problem: a starter workspace the agent opens cold, a reference solution held outside its image, and a grading program that decides reward on its own. The agent gets the workspace and the brief. It does not get the solution, the tests, or a second opinion. Everything below is per task — how many times it was attempted, how often a run came back at exactly 1.0, and which of the five outcomes it produced most.
What a task is made of
Four pieces, and the split between them is the whole design: the agent can read two of them and nothing it writes can reach the other two.
The filesystem the agent wakes up in — sources, fixtures, build files, and nothing that answers the question for it.
Every task ships with a worked solution kept out of the agent's image. It proves the problem is solvable and fixes what counts as done.
Grading is a program, not an opinion. The same submission returns the same reward on every run, and the agent never reads it.
Each agent attempts a task repeatedly, so one lucky pass cannot carry a task or a category.
pass@1 below is the share of a task's trials that returned a reward of exactly 1.0. 1 of the 90 tasks is scored on a continuous 0–1 scale instead, and is shown with a mean score — a pass@1 figure would be meaningless for it.
Fifteen categories
Bars are resolve rate — one hue, one measure. A category reads live once it has 20 or more scored trials and preview below that; the threshold counts trials, not tasks, so three tasks run five times each still read preview.
No task name matches that filter.
Software Engineeringlive
Algorithms, data structures, bug-fixes, and API/systems implementation, graded against hidden test suites the agent never sees.
| Task | Runs | Resolved | pass@1 | Dominant outcome | |
|---|---|---|---|---|---|
| session-token-verify | 10 | 10/10 | 100.0% | Legitimate solve | task page ↗ |
| window-aggregate-store | 10 | 10/10 | 100.0% | Legitimate solve | task page ↗ |
| lru-cache | 15 | 14/15 | 93.3% | Legitimate solve | task page ↗ |
| occ-conditional-store | 10 | 9/10 | 90.0% | Legitimate solve | task page ↗ |
| idempotency-middleware | 10 | 4/10 | 40.0% | Honest miss | task page ↗ |
| rate-limiter | 10 | 2/10 | 20.0% | Task at fault | task page ↗ |
| diff-patch-engine | 10 | 0/10 | 0.0% | Honest miss | task page ↗ |
| resilient-http-client | 10 | 0/10 | 0.0% | Honest miss | task page ↗ |
No task in this category matches the filter.
Mechanical Engineeringlive
Structural and numerical solvers (FD/FE, contact dynamics) in C++, checked against multi-binary hidden references.
| Task | Runs | Resolved | pass@1 | Dominant outcome | |
|---|---|---|---|---|---|
| rk4-orbit-integrator | 10 | 10/10 | 100.0% | Legitimate solve | task page ↗ |
| truss2d-solver | 10 | 6/10 | 60.0% | Legitimate solve | task page ↗ |
| pipeflow-colebrook-solver | 10 | 4/10 | 40.0% | Honest miss | task page ↗ |
| beam-deflection-solver | 10 | 3/10 | 30.0% | Honest miss | task page ↗ |
| heat1d-conduction-solver | 10 | 3/10 | 30.0% | Honest miss | task page ↗ |
| projectile-drag-integrator | 10 | 2/10 | 20.0% | Honest miss | task page ↗ |
| collision2d-impulse-solver | 10 | 0/10 | 0.0% | Honest miss | task page ↗ |
| quaternion-rotation-integrator | 10 | 0/10 | 0.0% | Honest miss | task page ↗ |
No task in this category matches the filter.
Cloud Operationslive
Provision, diagnose, and guardrail cloud infrastructure (AWS), verified against the deployed state.
| Task | Runs | Resolved | pass@1 | Dominant outcome | |
|---|---|---|---|---|---|
| ecs-fargate-secrets-kms-exec-role | 7 | 6/7 | 85.7% | Legitimate solve | task page ↗ |
| secrets-rotation-kms | 10 | 6/10 | 60.0% | Legitimate solve | task page ↗ |
| sfn-saga-compensation-orchestrator | 10 | 5/10 | 50.0% | Legitimate solve | task page ↗ |
| athena-workgroup-result-encryption-cmk-enforced | 5 | 2/5 | 40.0% | Honest miss | task page ↗ |
| ddb-outbox-eventbridge-fanout | 10 | 4/10 | 40.0% | Honest miss | task page ↗ |
| iam-revoke-older-sessions | 10 | 4/10 | 40.0% | Legitimate solve | task page ↗ |
| ecr-image-scan-lifecycle-immutable-tags-replication | 9 | 3/9 | 33.3% | Honest miss | task page ↗ |
| glue-etl-catalog-security-configuration-kms | 9 | 3/9 | 33.3% | Honest miss | task page ↗ |
| apigw-http-api-jwt-authorizer-lambda-integration | 10 | 3/10 | 30.0% | Task at fault | task page ↗ |
| efs-access-point-posix-iam-mount-target | 10 | 3/10 | 30.0% | Task at fault | task page ↗ |
| iam-cross-account-externalid-sourcearn | 10 | 3/10 | 30.0% | Task at fault | task page ↗ |
| iam-session-tag-tenant-scope | 10 | 3/10 | 30.0% | Honest miss | task page ↗ |
| apigw-sqs-fifo-direct-integration | 10 | 2/10 | 20.0% | Honest miss | task page ↗ |
| iam-permissions-boundary-ceiling | 10 | 2/10 | 20.0% | Honest miss | task page ↗ |
| s3-lambda-ddb-pipeline | 10 | 2/10 | 20.0% | Harness error | task page ↗ |
| s3-sqs-image-pipeline-kms | 10 | 2/10 | 20.0% | Honest miss | task page ↗ |
| appsync-graphql-cognito-resolver-cache-leak | 10 | 1/10 | 10.0% | Honest miss | task page ↗ |
| cognito-m2m-httpapi-jwt-scope-gated | 10 | 1/10 | 10.0% | Honest miss | task page ↗ |
| sfn-secrets-rotation-chain | 10 | 1/10 | 10.0% | Honest miss | task page ↗ |
No task in this category matches the filter.
Electrical Engineeringlive
RTL / digital-logic design checked under hardened simulation and formal-equivalence harnesses.
| Task | Runs | Resolved | pass@1 | Dominant outcome | |
|---|---|---|---|---|---|
| apb-lite-slave-regfile | 10 | 10/10 | 100.0% | Legitimate solve | task page ↗ |
| bcd-tens-rollover | 10 | 10/10 | 100.0% | Legitimate solve | task page ↗ |
| bus-slave-abort-ack | 10 | 10/10 | 100.0% | Legitimate solve | task page ↗ |
| Enable-gated streaming fold stage | 3 | 3/3 | 100.0% | Legitimate solve | task page ↗ |
| Instruction-retire commit handshake | 3 | 3/3 | 100.0% | Reward-hack | task page ↗ |
| page-program-suspend | 10 | 10/10 | 100.0% | Legitimate solve | task page ↗ |
| round-robin-arbiter-grant | 10 | 10/10 | 100.0% | Legitimate solve | task page ↗ |
| Serial bit-destuff framer | 3 | 3/3 | 100.0% | Legitimate solve | task page ↗ |
| Wait-state register-file completer | 3 | 3/3 | 100.0% | Legitimate solve | task page ↗ |
| open-drain-command-engine | 10 | 8/10 | 80.0% | Legitimate solve | task page ↗ |
| dualmaster-membridge | 10 | 5/10 | 50.0% | Legitimate solve | task page ↗ |
| stream-frame-hold | 10 | 4/10 | 40.0% | Honest miss | task page ↗ |
| byte-serial-round-scheduler | 10 | 2/10 | 20.0% | Honest miss | task page ↗ |
| serial-receiver-framed | 10 | 1/10 | 10.0% | Honest miss | task page ↗ |
| coprocessor-dispatcher-classmix | 10 | 0/10 | 0.0% | Honest miss | task page ↗ |
| debug-halt-step-fsm | 10 | 0/10 | 0.0% | Honest miss | task page ↗ |
| hash-message-padder | 10 | 0/10 | 0.0% | Harness error | task page ↗ |
| Multi-cycle signed divider with a start/valid handshake | 3 | 0/3 | 0.0% | Honest miss | task page ↗ |
| Resynchronising serial byte receiver | 3 | 0/3 | 0.0% | Task at fault | task page ↗ |
| serial-break-resync | 10 | 0/10 | 0.0% | Honest miss | task page ↗ |
No task in this category matches the filter.
STEMlive
Quantitative reasoning across math, physics, and the natural sciences with deterministic, checkable answers.
| Task | Runs | Resolved | pass@1 | Dominant outcome | |
|---|---|---|---|---|---|
| adaptive-quadrature | 10 | 10/10 | 100.0% | Legitimate solve | task page ↗ |
| anova-stats | 11 | 11/11 | 100.0% | Legitimate solve | task page ↗ |
| cubic-spline | 10 | 10/10 | 100.0% | Legitimate solve | task page ↗ |
| cg-solver | 40 | 28/40 | 70.0% | Legitimate solve | task page ↗ |
| cholesky-solver | 10 | 0/10 | 0.0% | Honest miss | task page ↗ |
No task in this category matches the filter.
Gamelive
Interactive simulation and game-logic tasks graded on exact state transitions and rule fidelity.
| Task | Runs | Resolved | pass@1 | Dominant outcome | |
|---|---|---|---|---|---|
| game-of-life-step | 15 | 15/15 | 100.0% | Legitimate solve | task page ↗ |
| car-scene-assembly | 10 | 6/10 | 60.0% | Legitimate solve | task page ↗ |
| combo-score-system | 10 | 6/10 | 60.0% | Legitimate solve | task page ↗ |
| camera-shake-rig | 10 | 4/10 | 40.0% | Harness error | task page ↗ |
| checkpoint-system | 10 | 4/10 | 40.0% | Honest miss | task page ↗ |
| crusader-sprite-assembly | 10 | 4/10 | 40.0% | Honest miss | task page ↗ |
| minimap-marker-logic | 10 | 4/10 | 40.0% | Honest miss | task page ↗ |
| tile-gradient-particles | 10 | 2/10 | 20.0% | Harness error | task page ↗ |
| day-night-cycle-controller | 10 | 1/10 | 10.0% | Honest miss | task page ↗ |
| minimap-ui | 10 | 0/10 | 0.0% | Honest miss | task page ↗ |
No task in this category matches the filter.
Debuggingpreview
Real-world bug-fix tasks distilled from open-source issues (esbuild, klauspost/compress, rust-lang/semver), graded against each project's own tests.
| Task | Runs | Resolved | pass@1 | Dominant outcome | |
|---|---|---|---|---|---|
| evanw-esbuild-4417 | 5 | 5/5 | 100.0% | Legitimate solve | task page ↗ |
| klauspost-compress-1115 | 5 | 5/5 | 100.0% | Legitimate solve | task page ↗ |
| rust-lang-semver-305 | 5 | 5/5 | 100.0% | Legitimate solve | task page ↗ |
No task in this category matches the filter.
ML Engineeringpreview
Train and tune models to a target metric on real engineering datasets, graded on held-out performance against a solved-reward threshold.
| Task | Runs | Resolved | pass@1 | Dominant outcome | |
|---|---|---|---|---|---|
| airfoil-self-noise | 10 | 4/10 | 40.0% | Honest miss | task page ↗ |
No task in this category matches the filter.
Post Traininglive
Long-horizon, from-scratch numpy ML research: implement reverse-mode autodiff and a training method (quantization-aware, adversarial-robust, semi-supervised, worst-group), then train to a sealed held-out metric. Tasks published blinded.
| Task | Runs | Resolved | pass@1 | Dominant outcome | |
|---|---|---|---|---|---|
| fewlabel-ssl-fixmatch | 10 | 10/10 | 100.0% | Legitimate solve | task page ↗ |
| qat-int2-cifar | 15 | 9/15 | 60.0% | Legitimate solve | task page ↗ |
| worst-group-spurious-dfr | 10 | 5/10 | 50.0% | Legitimate solve | task page ↗ |
| adv-robust-pgd | 10 | 3/10 | 30.0% | Honest miss | task page ↗ |
| reverse-engineer-decoding | 10 | 1/10 | 10.0% | Honest miss | task page ↗ |
| reverse-engineer-objectives | 10 | 0/10 | 0.0% | Honest miss | task page ↗ |
No task in this category matches the filter.
Scientific MLlive
Physics-informed and surrogate modeling: PDE forecasting and CFD/FEA prediction, graded on quantitative predictive accuracy.
| Task | Runs | Resolved | pass@1 | Dominant outcome | |
|---|---|---|---|---|---|
| airfrans-high-reynolds-drag-extrapolation | 10 | 4/10 | 40.0% | Honest miss | task page ↗ |
| ks-equation-1d-forecast | 10 | 4/10 | 40.0% | Honest miss | task page ↗ |
| simjeb-bracket-fea-mass-prediction-real | 10 | 2/10 | 20.0% | Honest miss | task page ↗ |
No task in this category matches the filter.
ML for Engineeringpreview
Machine-learning-for-engineering tasks (DeepCAD command-history canonicalization and equivalence), graded on a continuous partial-credit score against verifier-owned labels rather than all-or-nothing pass/fail.
| Task | Runs | Resolved | pass@1 | Dominant outcome | |
|---|---|---|---|---|---|
| deepcad-canonical-equivalencegraded 0–1 | 10 | n/a | 0.441mean score | Honest miss | task page ↗ |
No task in this category matches the filter.
1 task here is graded on a continuous 0–1 score rather than pass/fail, so it has a mean score instead of a pass@1 — and no trial can register as resolved, which requires a reward of exactly 1.0.
Data Sciencelive
Modeling and statistical inference on real datasets (federated learning, event studies), graded against held-out ground truth.
| Task | Runs | Resolved | pass@1 | Dominant outcome | |
|---|---|---|---|---|---|
| product-recall-stock-price-event | 10 | 3/10 | 30.0% | Honest miss | task page ↗ |
| fedavg-federated-noniid-mnist | 10 | 2/10 | 20.0% | Honest miss | task page ↗ |
No task in this category matches the filter.
Data Science: Robustnesslive
Bias-correction and outlier-robust analysis in R: recover trustworthy estimates from messy data, checked against reference results.
| Task | Runs | Resolved | pass@1 | Dominant outcome | |
|---|---|---|---|---|---|
| coffee-ratings-outliers | 10 | 4/10 | 40.0% | Honest miss | task page ↗ |
| lending-club-lgd-bias-correction-r | 10 | 4/10 | 40.0% | Honest miss | task page ↗ |
No task in this category matches the filter.
Product Data Sciencepreview
Applied product analytics: causal impact and decision analysis on real product datasets (R).
| Task | Runs | Resolved | pass@1 | Dominant outcome | |
|---|---|---|---|---|---|
| ipl-toss-impact-analysis-r | 10 | 4/10 | 40.0% | Legitimate solve | task page ↗ |
No task in this category matches the filter.
Pharmacometricspreview
Population PK/PD modeling with nonlinear mixed-effects: fit drug-exposure models and recover the correct parameters.
| Task | Runs | Resolved | pass@1 | Dominant outcome | |
|---|---|---|---|---|---|
| neonatal-drug-exposure-nlme | 10 | 4/10 | 40.0% | Legitimate solve | task page ↗ |
No task in this category matches the filter.