Can an agent tell whether a claimed research result is real — regenerate it, catch a planted or real defect, and judge robustness? 14 instances from 4 real public repos. Gold regenerated in-sandbox (9) or externally verified (5).
Re-running catches wrong numbers, so naive-rerun clears every REPRODUCED & DEVIATION — but it is blind to fragility and fabrication. The gap to the ceiling is the benchmark's reason to exist.
| system | verdict acc | macro-F1 | FRAGILE recall | FABRICATED recall |
|---|---|---|---|---|
| perfect agent (ceiling) | 1.000 | 1.000 | 100% | 100% |
| naive-rerun | 0.643 | 0.392 | 0% | 0% |
| always-REPRODUCED | 0.357 | 0.132 | 0% | 0% |
FRAGILE & FABRICATED recall are the headroom classes: both baselines score 0. Scorer: scoring/score.py.
| id | source | type | gold | exec | always✓ | naive✓ |
|---|---|---|---|---|---|---|
| ds-repro-01 | de-stress-lab | REPRO | REPRODUCED | sandbox | ✓ | ✓ |
| ds-integrity-01 | de-stress-lab | REPRO | REPRODUCED | sandbox | ✓ | ✓ |
| ds-integrity-02 planted | de-stress-lab | REPRO | DEVIATION | sandbox | ✗ | ✓ |
| ds-audit-01 | de-stress-lab | REPRO | REPRODUCED | sandbox | ✓ | ✓ |
| ds-fragile-01 | de-stress-lab | ROBUST | FRAGILE | sandbox | ✗ | ✗ |
| ds-fabricate-01 planted | de-stress-lab | REPRO | FABRICATED | sandbox | ✗ | ✗ |
| fair-repro-01 | robert-morong-research | REPRO | REPRODUCED | external | ✓ | ✓ |
| fair-deviation-01 planted | robert-morong-research | REPRO | DEVIATION | external | ✗ | ✓ |
| fair-fabricate-01 planted | robert-morong-research | REPRO | FABRICATED | external | ✗ | ✗ |
| arc-repro-01 | arc-agi-2-occam-baseline | REPRO | REPRODUCED | sandbox | ✓ | ✓ |
| arc-selfcorrect-01 | arc-agi-2-occam-baseline | REPRO | DEVIATION | sandbox | ✗ | ✓ |
| arc-leaderboard-01 | arc-agi-2-occam-baseline | ROBUST | FRAGILE | sandbox | ✗ | ✗ |
| mmlu-flip-01 | mmlu-robustness-audit | ROBUST | FRAGILE | sandbox | ✗ | ✗ |
| mmlu-calib-deviation-01 | mmlu-robustness-audit | REPRO | DEVIATION | sandbox | ✗ | ✓ |
planted = a defect we introduced; the rest are real reproductions, real bugs, or the repos' own documented self-corrections. exec: sandbox = regenerated end-to-end here; external = dataset/model-gated (verdict logic scored now, live re-run in v1).
Benchmark code Apache-2.0 · task data CC BY 4.0 · built by the GrobeStreet research crew · aligned to the Snorkel Open Benchmarks output-complexity axis.