SWE Refactor Bench Exposes Critical Failures in AI Coding Agents During Large-Scale Migrations

The dataset targets long-horizon refactors that span thousands of files and require coordinated changes across languages, frameworks, and build systems. Results indicate that frontier coding agents complete only 5.4% of migration runs successfully, exposing a substantial gap between localized code generation and sustained multi-file architectural change.

Benchmark construction

Each of the 20 tasks constitutes an entire repository pre-migration, paired with a target post-migration state. Agents operate without preset file-level guidance, receiving only the high-level migration objective and the starting codebase. The framing isolates the planning and execution loop: the agent must independently identify affected components, sequence refactors, and validate cross-file consistency. The long-horizon structure distinguishes SWE Refactor Bench from localized edit benchmarks, where a single patch often suffices. The 20-repository scale places the suite at a level permitting controlled measurement of whole-codebase change without the confounds of proprietary systems.

Results and limitations

Across reported runs, completion rates remained below the 6% threshold. The 5.4% figure reflects successful end-to-end completion, not partial progress or localized fixes. The low rate indicates that the bottleneck is not single-file code synthesis but sustained multi-file coordination, where planning errors and stale downstream references compound across the repository. Variance across runs was high, a pattern consistent with stochastic planning behavior in agents operating on long-horizon tasks without intermediate grounding.

What to track

Three variables warrant continued monitoring. First, whether subsequent releases expand beyond 20 tasks and broaden language coverage, since small suites face acute benchmark saturation effects. Second, whether evaluation harnesses report per-stage success—planning, execution, validation—alongside the aggregate pass rate, enabling the ablation studies the aggregate currently obscures. Third, whether reference implementations or replay tooling accompany the dataset, a prerequisite for reproducible comparison. The broader methodological lesson—that coordination across interdependent systems resists capture by short-horizon metrics—parallels measurement problems documented in long-horizon adaptation research, including the kind catalogued in Adapting to Europe's Climate Crisis.