Evaluating AI Coding Agents with the SWE-Bench ProMax Multilingual Benchmark

The team behind SWE-Bench ProMax just published a multilingual refactoring benchmark on Hugging Face that pinpoints exactly where today's coding agents fall apart — and the headline number is sobering: the best frontier model still resolves only 41.2% of tasks. For us running real evaluations, that's not just a leaderboard entry — it's a sanity check on how far we still are from "ship-it" autonomy on messy, multi-file codebases.

What changed in the benchmark design

SWE-Bench ProMax ships 170 instances across seven languages — Python, Java, TypeScript, Go, C, C++, and Rust — each drawn from real commits, not synthetic toy problems. The gotcha that matters most: every issue description was rewritten from scratch to remove ambiguity, and the test suites were manually reviewed to strip out both overly narrow tests that reject correct solutions and overly broad tests that check unstated behavior. Tasks with shallow cross-file impact were filtered out, so what's left averages 11.4 modified files and 261.6 lines of code per instance — substantially larger than what current refactoring benchmarks cover. This is the boilerplate we actually wanted: long-horizon, behavior-preserving edits that touch real dependency graphs.

Why that 41.2% resolve rate hits different

If you've been quietly suspicious that SWE-bench Verified was getting too easy — or too leaky — the paper hands you the receipts. An audit cited by the authors found that nearly 60% of unsolved SWE-bench Verified instances carry flawed tests, and frontier models can verbatim reproduce gold patches from training data. That means a chunk of the "solved" leaderboard is partly memorized, and a chunk of the "unsolved" pile is just bad grading. ProMax is explicitly engineered to dodge both failure modes. Two agent scaffolds were tested on it; neither cleared the halfway mark. For anyone building agent infrastructure, the workaround signal is blunt — single-shot, PR-style prompting collapses once you cross the ~10-file boundary, and you need explicit cross-file edit tooling in the scaffold to even compete.

What to wire into your pipeline next week

Three concrete moves before you quote any new benchmark number to your team. First, route your agent through ProMax as an external eval — the dataset is live on Hugging Face — and log per-file edit counts so you can see where your scaffold stalls on the long-horizon tail. Second, audit your own test harness: if your runner accepts ambiguous patches or flags correct ones as failures, your resolve rate is noise regardless of model quality. Third, treat any prior SWE-bench Verified claim with a discount — the audit effectively devalues a portion of those numbers, and stacking results on top of contaminated benchmarks will mislead your roadmap. ProMax is small (170 instances) but dense; budget a couple of full runs before drawing conclusions.