Task index
90 problems, each with a verifier the agent never sees.
14 tasks nothing solved
No trial on these reached a reward of exactly 1.0. 5 of them failed cleanly: every trial an honest miss, no harness error and no trial blamed on the task, so only those 5 zeros are a statement about the agents. 8 carry at least one suspect label. 1 is scored 0–1, where "nobody solved it" only means no trial landed on exactly 1.0.
What a task is made of
Four pieces, and the split between them is the whole design: the agent can read two of them and nothing it writes can reach the other two.
The filesystem the agent wakes up in: sources, fixtures, build files, and nothing that answers the question for it.
Every task ships with a worked solution kept out of the agent's image. It proves the problem is solvable and fixes what counts as done.
Grading is a program, not an opinion. The same submission returns the same reward on every run, and the agent never reads it.
Each agent attempts a task repeatedly, so one lucky pass cannot carry a task or a category.
pass@1 below is the share of a task's trials that returned a reward of exactly 1.0, and the difficulty tier in the filter bar is derived from it: easy is 70% of trials resolved or more, hard is under 30%. 1 of the 90 tasks is scored on a continuous 0–1 scale instead. It is shown with a mean score and carries no tier, since both a pass@1 figure and a tier derived from one would be meaningless for it.
Fifteen categories
Bars are resolve rate: one hue, one measure. A category reads live once it has 20 or more scored trials and preview below that; the threshold counts trials, not tasks, so three tasks run five times each still read preview. Picking one here also narrows the index below it.
No task matches those filters.
Software Engineeringlive
Algorithms, data structures, bug-fixes, and API/systems implementation, graded against hidden test suites the agent never sees.
| Task | Resolved | pass@1 |
|---|---|---|
session-token-verify10 trials · 10/10 reached reward 1.0 · pass@1 100.0% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve10 | 10/10 | 100.0% |
window-aggregate-store10 trials · 10/10 reached reward 1.0 · pass@1 100.0% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve10 | 10/10 | 100.0% |
lru-cache15 trials · 14/15 reached reward 1.0 · pass@1 93.3% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve12Honest miss1Harness error2 | 14/15 | 93.3% |
occ-conditional-store10 trials · 9/10 reached reward 1.0 · pass@1 90.0% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve9Honest miss1 | 9/10 | 90.0% |
idempotency-middleware10 trials · 4/10 reached reward 1.0 · pass@1 40.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve4Honest miss6 | 4/10 | 40.0% |
rate-limiter10 trials · 2/10 reached reward 1.0 · pass@1 20.0% · derived difficulty hard <30% By agent
Trial labels Legitimate solve1Reward-hack1Honest miss3Task at fault5 | 2/10 | 20.0% |
diff-patch-engine10 trials · 0/10 reached reward 1.0 · pass@1 0.0% · derived difficulty hard <30% By agent
Trial labels Honest miss10 | 0/10 | 0.0% |
resilient-http-client10 trials · 0/10 reached reward 1.0 · pass@1 0.0% · derived difficulty hard <30% By agent
Trial labels Honest miss8Task at fault1Harness error1 | 0/10 | 0.0% |
Mechanical Engineeringlive
Structural and numerical solvers (FD/FE, contact dynamics) in C++, checked against multi-binary hidden references.
| Task | Resolved | pass@1 |
|---|---|---|
rk4-orbit-integrator10 trials · 10/10 reached reward 1.0 · pass@1 100.0% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve10 | 10/10 | 100.0% |
truss2d-solver10 trials · 6/10 reached reward 1.0 · pass@1 60.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve6Honest miss4 | 6/10 | 60.0% |
pipeflow-colebrook-solver10 trials · 4/10 reached reward 1.0 · pass@1 40.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve4Honest miss6 | 4/10 | 40.0% |
beam-deflection-solver10 trials · 3/10 reached reward 1.0 · pass@1 30.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve3Honest miss7 | 3/10 | 30.0% |
heat1d-conduction-solver10 trials · 3/10 reached reward 1.0 · pass@1 30.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve3Honest miss7 | 3/10 | 30.0% |
projectile-drag-integrator10 trials · 2/10 reached reward 1.0 · pass@1 20.0% · derived difficulty hard <30% By agent
Trial labels Legitimate solve2Honest miss7Harness error1 | 2/10 | 20.0% |
collision2d-impulse-solver10 trials · 0/10 reached reward 1.0 · pass@1 0.0% · derived difficulty hard <30% By agent
Trial labels Honest miss9Harness error1 | 0/10 | 0.0% |
quaternion-rotation-integrator10 trials · 0/10 reached reward 1.0 · pass@1 0.0% · derived difficulty hard <30% By agent
Trial labels Honest miss10 | 0/10 | 0.0% |
Cloud Operationslive
Provision, diagnose, and guardrail cloud infrastructure (AWS), verified against the deployed state.
| Task | Resolved | pass@1 |
|---|---|---|
ecs-fargate-secrets-kms-exec-role7 trials · 6/7 reached reward 1.0 · pass@1 85.7% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve6Honest miss1 ecs-fargate-secrets-kms-exec-role on syncvals.syncrope.com ↗ | 6/7 | 85.7% |
secrets-rotation-kms10 trials · 6/10 reached reward 1.0 · pass@1 60.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve5Honest miss3Task at fault1Harness error1 | 6/10 | 60.0% |
sfn-saga-compensation-orchestrator10 trials · 5/10 reached reward 1.0 · pass@1 50.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve4Task at fault4Harness error2 sfn-saga-compensation-orchestrator on syncvals.syncrope.com ↗ | 5/10 | 50.0% |
athena-workgroup-result-encryption-cmk-enforced5 trials · 2/5 reached reward 1.0 · pass@1 40.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve2Honest miss2Task at fault1 athena-workgroup-result-encryption-cmk-enforced on syncvals.syncrope.com ↗ | 2/5 | 40.0% |
ddb-outbox-eventbridge-fanout10 trials · 4/10 reached reward 1.0 · pass@1 40.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve4Honest miss4Task at fault1Harness error1 | 4/10 | 40.0% |
iam-revoke-older-sessions10 trials · 4/10 reached reward 1.0 · pass@1 40.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve4Honest miss3Task at fault3 | 4/10 | 40.0% |
ecr-image-scan-lifecycle-immutable-tags-replication9 trials · 3/9 reached reward 1.0 · pass@1 33.3% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve3Honest miss4Task at fault2 ecr-image-scan-lifecycle-immutable-tags-replication on syncvals.syncrope.com ↗ | 3/9 | 33.3% |
glue-etl-catalog-security-configuration-kms9 trials · 3/9 reached reward 1.0 · pass@1 33.3% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve3Honest miss4Task at fault1Harness error1 glue-etl-catalog-security-configuration-kms on syncvals.syncrope.com ↗ | 3/9 | 33.3% |
apigw-http-api-jwt-authorizer-lambda-integration10 trials · 3/10 reached reward 1.0 · pass@1 30.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve3Honest miss1Task at fault6 apigw-http-api-jwt-authorizer-lambda-integration on syncvals.syncrope.com ↗ | 3/10 | 30.0% |
efs-access-point-posix-iam-mount-target10 trials · 3/10 reached reward 1.0 · pass@1 30.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve3Honest miss1Task at fault5Harness error1 efs-access-point-posix-iam-mount-target on syncvals.syncrope.com ↗ | 3/10 | 30.0% |
iam-cross-account-externalid-sourcearn10 trials · 3/10 reached reward 1.0 · pass@1 30.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve3Task at fault6Harness error1 iam-cross-account-externalid-sourcearn on syncvals.syncrope.com ↗ | 3/10 | 30.0% |
iam-session-tag-tenant-scope10 trials · 3/10 reached reward 1.0 · pass@1 30.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve2Reward-hack1Honest miss5Task at fault2 | 3/10 | 30.0% |
apigw-sqs-fifo-direct-integration10 trials · 2/10 reached reward 1.0 · pass@1 20.0% · derived difficulty hard <30% By agent
Trial labels Legitimate solve1Honest miss7Harness error2 apigw-sqs-fifo-direct-integration on syncvals.syncrope.com ↗ | 2/10 | 20.0% |
iam-permissions-boundary-ceiling10 trials · 2/10 reached reward 1.0 · pass@1 20.0% · derived difficulty hard <30% By agent
Trial labels Legitimate solve2Honest miss6Task at fault1Harness error1 | 2/10 | 20.0% |
s3-lambda-ddb-pipeline10 trials · 2/10 reached reward 1.0 · pass@1 20.0% · derived difficulty hard <30% By agent
Trial labels Legitimate solve2Honest miss2Task at fault2Harness error4 | 2/10 | 20.0% |
s3-sqs-image-pipeline-kms10 trials · 2/10 reached reward 1.0 · pass@1 20.0% · derived difficulty hard <30% By agent
Trial labels Legitimate solve2Honest miss4Task at fault3Harness error1 | 2/10 | 20.0% |
appsync-graphql-cognito-resolver-cache-leak10 trials · 1/10 reached reward 1.0 · pass@1 10.0% · derived difficulty hard <30% By agent
Trial labels Legitimate solve1Honest miss6Task at fault3 appsync-graphql-cognito-resolver-cache-leak on syncvals.syncrope.com ↗ | 1/10 | 10.0% |
cognito-m2m-httpapi-jwt-scope-gated10 trials · 1/10 reached reward 1.0 · pass@1 10.0% · derived difficulty hard <30% By agent
Trial labels Legitimate solve1Honest miss6Task at fault1Harness error2 cognito-m2m-httpapi-jwt-scope-gated on syncvals.syncrope.com ↗ | 1/10 | 10.0% |
sfn-secrets-rotation-chain10 trials · 1/10 reached reward 1.0 · pass@1 10.0% · derived difficulty hard <30% By agent
Trial labels Reward-hack1Honest miss8Harness error1 | 1/10 | 10.0% |
Electrical Engineeringlive
RTL / digital-logic design checked under hardened simulation and formal-equivalence harnesses.
| Task | Resolved | pass@1 |
|---|---|---|
apb-lite-slave-regfile10 trials · 10/10 reached reward 1.0 · pass@1 100.0% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve10 | 10/10 | 100.0% |
bcd-tens-rollover10 trials · 10/10 reached reward 1.0 · pass@1 100.0% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve10 | 10/10 | 100.0% |
bus-slave-abort-ack10 trials · 10/10 reached reward 1.0 · pass@1 100.0% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve10 | 10/10 | 100.0% |
Enable-gated streaming fold stage3 trials · 3/3 reached reward 1.0 · pass@1 100.0% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve2Harness error1 Enable-gated streaming fold stage on syncvals.syncrope.com ↗ | 3/3 | 100.0% |
Instruction-retire commit handshake3 trials · 3/3 reached reward 1.0 · pass@1 100.0% · derived difficulty easy ≥70% By agent
Trial labels Reward-hack3 Instruction-retire commit handshake on syncvals.syncrope.com ↗ | 3/3 | 100.0% |
page-program-suspend10 trials · 10/10 reached reward 1.0 · pass@1 100.0% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve10 | 10/10 | 100.0% |
round-robin-arbiter-grant10 trials · 10/10 reached reward 1.0 · pass@1 100.0% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve10 | 10/10 | 100.0% |
Serial bit-destuff framer3 trials · 3/3 reached reward 1.0 · pass@1 100.0% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve2Reward-hack1 | 3/3 | 100.0% |
Wait-state register-file completer3 trials · 3/3 reached reward 1.0 · pass@1 100.0% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve3 Wait-state register-file completer on syncvals.syncrope.com ↗ | 3/3 | 100.0% |
open-drain-command-engine10 trials · 8/10 reached reward 1.0 · pass@1 80.0% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve7Reward-hack1Honest miss1Harness error1 | 8/10 | 80.0% |
dualmaster-membridge10 trials · 5/10 reached reward 1.0 · pass@1 50.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve5Honest miss5 | 5/10 | 50.0% |
stream-frame-hold10 trials · 4/10 reached reward 1.0 · pass@1 40.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve4Honest miss6 | 4/10 | 40.0% |
byte-serial-round-scheduler10 trials · 2/10 reached reward 1.0 · pass@1 20.0% · derived difficulty hard <30% By agent
Trial labels Legitimate solve2Honest miss7Task at fault1 | 2/10 | 20.0% |
serial-receiver-framed10 trials · 1/10 reached reward 1.0 · pass@1 10.0% · derived difficulty hard <30% By agent
Trial labels Legitimate solve1Honest miss9 | 1/10 | 10.0% |
coprocessor-dispatcher-classmix10 trials · 0/10 reached reward 1.0 · pass@1 0.0% · derived difficulty hard <30% By agent
Trial labels Honest miss10 | 0/10 | 0.0% |
debug-halt-step-fsm10 trials · 0/10 reached reward 1.0 · pass@1 0.0% · derived difficulty hard <30% By agent
Trial labels Honest miss6Task at fault4 | 0/10 | 0.0% |
hash-message-padder10 trials · 0/10 reached reward 1.0 · pass@1 0.0% · derived difficulty hard <30% By agent
Trial labels Task at fault1Harness error9 | 0/10 | 0.0% |
Multi-cycle signed divider with a start/valid handshake3 trials · 0/3 reached reward 1.0 · pass@1 0.0% · derived difficulty hard <30% By agent
Trial labels Honest miss2Task at fault1 Multi-cycle signed divider with a start/valid handshake on syncvals.syncrope.com ↗ | 0/3 | 0.0% |
Resynchronising serial byte receiver3 trials · 0/3 reached reward 1.0 · pass@1 0.0% · derived difficulty hard <30% By agent
Trial labels Honest miss1Task at fault2 Resynchronising serial byte receiver on syncvals.syncrope.com ↗ | 0/3 | 0.0% |
serial-break-resync10 trials · 0/10 reached reward 1.0 · pass@1 0.0% · derived difficulty hard <30% By agent
Trial labels Honest miss10 | 0/10 | 0.0% |
STEMlive
Quantitative reasoning across math, physics, and the natural sciences with deterministic, checkable answers.
| Task | Resolved | pass@1 |
|---|---|---|
adaptive-quadrature10 trials · 10/10 reached reward 1.0 · pass@1 100.0% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve10 | 10/10 | 100.0% |
anova-stats11 trials · 11/11 reached reward 1.0 · pass@1 100.0% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve11 | 11/11 | 100.0% |
cubic-spline10 trials · 10/10 reached reward 1.0 · pass@1 100.0% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve10 | 10/10 | 100.0% |
cg-solver40 trials · 28/40 reached reward 1.0 · pass@1 70.0% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve27Reward-hack1Honest miss7Task at fault5 | 28/40 | 70.0% |
cholesky-solver10 trials · 0/10 reached reward 1.0 · pass@1 0.0% · derived difficulty hard <30% By agent
Trial labels Honest miss7Task at fault3 | 0/10 | 0.0% |
Gamelive
Interactive simulation and game-logic tasks graded on exact state transitions and rule fidelity.
| Task | Resolved | pass@1 |
|---|---|---|
game-of-life-step15 trials · 15/15 reached reward 1.0 · pass@1 100.0% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve15 | 15/15 | 100.0% |
car-scene-assembly10 trials · 6/10 reached reward 1.0 · pass@1 60.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve4Reward-hack2Honest miss3Task at fault1 | 6/10 | 60.0% |
combo-score-system10 trials · 6/10 reached reward 1.0 · pass@1 60.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve6Honest miss3Task at fault1 | 6/10 | 60.0% |
camera-shake-rig10 trials · 4/10 reached reward 1.0 · pass@1 40.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve2Reward-hack1Honest miss1Task at fault2Harness error4 | 4/10 | 40.0% |
checkpoint-system10 trials · 4/10 reached reward 1.0 · pass@1 40.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve4Honest miss5Harness error1 | 4/10 | 40.0% |
crusader-sprite-assembly10 trials · 4/10 reached reward 1.0 · pass@1 40.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve3Reward-hack1Honest miss5Task at fault1 | 4/10 | 40.0% |
minimap-marker-logic10 trials · 4/10 reached reward 1.0 · pass@1 40.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve4Honest miss5Task at fault1 | 4/10 | 40.0% |
tile-gradient-particles10 trials · 2/10 reached reward 1.0 · pass@1 20.0% · derived difficulty hard <30% By agent
Trial labels Legitimate solve1Reward-hack1Honest miss1Task at fault2Harness error5 | 2/10 | 20.0% |
day-night-cycle-controller10 trials · 1/10 reached reward 1.0 · pass@1 10.0% · derived difficulty hard <30% By agent
Trial labels Legitimate solve1Honest miss8Task at fault1 | 1/10 | 10.0% |
minimap-ui10 trials · 0/10 reached reward 1.0 · pass@1 0.0% · derived difficulty hard <30% By agent
Trial labels Honest miss6Task at fault4 | 0/10 | 0.0% |
Debuggingpreview
Real-world bug-fix tasks distilled from open-source issues (esbuild, klauspost/compress, rust-lang/semver), graded against each project's own tests.
| Task | Resolved | pass@1 |
|---|---|---|
evanw-esbuild-44175 trials · 5/5 reached reward 1.0 · pass@1 100.0% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve4Reward-hack1 | 5/5 | 100.0% |
klauspost-compress-11155 trials · 5/5 reached reward 1.0 · pass@1 100.0% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve5 | 5/5 | 100.0% |
rust-lang-semver-3055 trials · 5/5 reached reward 1.0 · pass@1 100.0% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve5 | 5/5 | 100.0% |
Post Traininglive
Long-horizon, from-scratch numpy ML research: implement reverse-mode autodiff and a training method (quantization-aware, adversarial-robust, semi-supervised, worst-group), then train to a sealed held-out metric. Tasks published blinded.
| Task | Resolved | pass@1 |
|---|---|---|
fewlabel-ssl-fixmatch10 trials · 10/10 reached reward 1.0 · pass@1 100.0% · derived difficulty easy ≥70% By agent
Trial labels Legitimate solve8Reward-hack2 | 10/10 | 100.0% |
qat-int2-cifar15 trials · 9/15 reached reward 1.0 · pass@1 60.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve8Reward-hack1Honest miss6 | 9/15 | 60.0% |
worst-group-spurious-dfr10 trials · 5/10 reached reward 1.0 · pass@1 50.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve5Honest miss4Harness error1 | 5/10 | 50.0% |
adv-robust-pgd10 trials · 3/10 reached reward 1.0 · pass@1 30.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve3Honest miss7 | 3/10 | 30.0% |
reverse-engineer-decoding10 trials · 1/10 reached reward 1.0 · pass@1 10.0% · derived difficulty hard <30% By agent
Trial labels Legitimate solve1Honest miss9 | 1/10 | 10.0% |
reverse-engineer-objectives10 trials · 0/10 reached reward 1.0 · pass@1 0.0% · derived difficulty hard <30% By agent
Trial labels Honest miss10 | 0/10 | 0.0% |
Scientific MLlive
Physics-informed and surrogate modeling: PDE forecasting and CFD/FEA prediction, graded on quantitative predictive accuracy.
| Task | Resolved | pass@1 |
|---|---|---|
airfrans-high-reynolds-drag-extrapolation10 trials · 4/10 reached reward 1.0 · pass@1 40.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve4Honest miss6 airfrans-high-reynolds-drag-extrapolation on syncvals.syncrope.com ↗ | 4/10 | 40.0% |
ks-equation-1d-forecast10 trials · 4/10 reached reward 1.0 · pass@1 40.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve4Honest miss6 | 4/10 | 40.0% |
simjeb-bracket-fea-mass-prediction-real10 trials · 2/10 reached reward 1.0 · pass@1 20.0% · derived difficulty hard <30% By agent
Trial labels Legitimate solve2Honest miss8 simjeb-bracket-fea-mass-prediction-real on syncvals.syncrope.com ↗ | 2/10 | 20.0% |
Data Sciencelive
Modeling and statistical inference on real datasets (federated learning, event studies), graded against held-out ground truth.
| Task | Resolved | pass@1 |
|---|---|---|
product-recall-stock-price-event10 trials · 3/10 reached reward 1.0 · pass@1 30.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve3Honest miss7 | 3/10 | 30.0% |
fedavg-federated-noniid-mnist10 trials · 2/10 reached reward 1.0 · pass@1 20.0% · derived difficulty hard <30% By agent
Trial labels Legitimate solve2Honest miss7Task at fault1 | 2/10 | 20.0% |
Data Science: Robustnesslive
Bias-correction and outlier-robust analysis in R: recover trustworthy estimates from messy data, checked against reference results.
| Task | Resolved | pass@1 |
|---|---|---|
coffee-ratings-outliers10 trials · 4/10 reached reward 1.0 · pass@1 40.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve4Honest miss5Task at fault1 | 4/10 | 40.0% |
lending-club-lgd-bias-correction-r10 trials · 4/10 reached reward 1.0 · pass@1 40.0% · derived difficulty medium 30–70% By agent
Trial labels Legitimate solve4Honest miss4Task at fault2 lending-club-lgd-bias-correction-r on syncvals.syncrope.com ↗ | 4/10 | 40.0% |
Single-task previews
ML Engineering, ML for Engineering, Product Data Science, Pharmacometrics: 4 categories holding one task apiece, each still under the 20-trial line. They are pooled into one list rather than given a section each. Every row keeps its own category, its own status and its own anchor.
| Task | Resolved | pass@1 |
|---|---|---|
airfoil-self-noiseML Engineering preview10 trials · 4/10 reached reward 1.0 · pass@1 40.0% · derived difficulty medium 30–70% Category ML Engineering, held at preview on 10 scored trials. By agent
Trial labels Legitimate solve4Honest miss6 | 4/10 | 40.0% |
ipl-toss-impact-analysis-rProduct Data Science preview10 trials · 4/10 reached reward 1.0 · pass@1 40.0% · derived difficulty medium 30–70% Category Product Data Science, held at preview on 10 scored trials. By agent
Trial labels Legitimate solve4Honest miss2Task at fault4 | 4/10 | 40.0% |
neonatal-drug-exposure-nlmePharmacometrics preview10 trials · 4/10 reached reward 1.0 · pass@1 40.0% · derived difficulty medium 30–70% Category Pharmacometrics, held at preview on 10 scored trials. By agent
Trial labels Legitimate solve4Honest miss2Task at fault2Harness error2 | 4/10 | 40.0% |
deepcad-canonical-equivalenceML for Engineering preview10 trials, scored on a continuous 0–1 scale: 0.441 mean, no trial at exactly 1.0, so no pass@1 and no difficulty tier. Category ML for Engineering, held at preview on 10 scored trials. By agent
Trial labels Honest miss10 | n/a | 0.441mean score |
1 task here is graded on a continuous 0–1 score rather than pass/fail, so it has a mean score instead of a pass@1, and no trial can register as resolved, which requires a reward of exactly 1.0.
The pipeline behind this report is available for client work.
Request a private evaluation team@starzo.ai