Why Most Top-Tier AI Research Papers Fail the Reproducibility Test

The exercise was run by SAI, co-founded by Chenhao Tan of the University of Chicago. SAI targeted all 168 Oral papers drawn from ICML's 23,918 submissions—a 0.7% acceptance share. Of those 168, 104 released open-source code; SAI completed full reproduction on 105 papers.

Reproduction Distribution

The central statistic is the median. Holding the minimum number of verifiable claims per paper at 1, 3, or 5, the median reproduction score remains in the 28%–30% band across thresholds. Excluding experiments that failed to execute, terminated early, or exceeded available compute lifts the median only to 42%–50%; at a three-claim minimum, the median holds at 42%.

By absolute count: 34 papers reproduced more than 40% of their stated claims; 8 papers exceeded 80%. The audit targets are not arbitrary submissions but the top 0.7% tier of an already heavily filtered pool.

Failure Modes and Specific Discrepancies

SAI's protocol extended past textual peer review: it downloaded code, models, and data, configured runtime environments, and reran experiments before comparing output against paper claims. Common obstacles included unrunnable code, missing files, incomplete instructions, corrupted dependencies, and runtime output that diverged from reported numbers. Four papers depended on models subsequently taken offline, foreclosing any motivated re-execution.

Two quantitative discrepancies stand out. One paper's headline efficiency claim—training 0.77% of the base model's parameters—corresponds in the released checkpoint to 6.31%, approximately 8x the advertised figure. Another paper presented a reliability table produced by a judge model; the judge model itself was absent from the open-source release, and no script reproduces the tabulated numbers.

Limits and Practitioner Implications

The audit captures only what is "discoverable under certain conditions." The 64 Oral papers without released code fall outside the reproduction pipeline entirely. The reported medians therefore describe the floor, not the ceiling, of non-reproducibility.

The operational consequence for practitioners is straightforward: top-venue acceptance is no longer a sufficient proxy for reproducible claims. Pre-implementation due diligence—verifying artifact availability, reconciling claimed parameter counts against released checkpoints, and benchmarking reported numbers against the open-source release—now sits upstream of any integration effort. The reproducibility cost remains a non-trivial share of compute and engineering time, and the audit's distribution shows the return on that spend is concentrated in a small minority of papers.

Contrast domains with established independent verification are instructive. Where theatrical box office is auditable against exhibitor data—as with Spider-Man: Brand New Day's $927 million global opening—claims and receipts align by construction. ML research lacks a comparable mechanism; the paper–artifact gap continues to be the binding constraint on reproducibility.