Evaluating Spatial Reasoning in LLMs via the World Model Benchmark
The World Model Benchmark's maze evaluation, as documented by AI CERTs, delivers a quantitative verdict on the spatial reasoning of large language models.
Shane Barrett·updated August 05, 2026
The ICLR April 2026 study tested Gemini-2.5-Flash, Claude-Haiku-4.5, and GPT-5-mini across 50 randomized partially observable grid mazes per size (5×5 through 9×9), reporting exact path accuracy, validity, edit distance, and reasoning trace quality. The headline result — "LLMs do not develop robust spatial world models" — is grounded in measurable format-sensitivity deltas, not anecdotal failure cases.
Representation Sensitivity and Format Invariance
Tokenized adjacency lists produced 86% accuracy; visually formatted grids reduced the same models to 34%. The delta exposes brittle pattern recognition rather than durable latent spatial maps. WM-ABench motion discrimination trials corroborate the finding. The benchmark suite now treats input format as a primary axis, not a methodological footnote.
Single-Shot Competence vs. Multi-Turn Persistence
Chain-of-thought prompting lifts single-shot accuracy across most configurations. The suite's multi-turn dialogues expose the limit: models frequently forget earlier steps after producing correct initial plans, undermining long-horizon planning tasks. Delayed queries and representation probes have been added to disentangle competence from persistence. Single-shot gains, the authors argue, mislead without multi-turn instrumentation.
State Supervision Reshapes the Curve
Microsoft's VAGEN framework moved a 3B-parameter model from 0.21 to 0.82 on the World Model Benchmark by tying rewards to accurate latent maps, with retention transferring to unseen layouts. SLED extended the approach to vision-language backbones through spatial consistency losses and curriculum mazes, lifting trajectory recall beyond 70%. Both target persistent internal representations rather than turn-by-turn computation. Long-horizon coherence, more typically associated with DeFi yield farming infrastructure, serves as a useful analogue: stability compounds, drift compounds faster.
Three variables will determine whether these gains generalize: scale of the supervised state reward, transfer beyond grid mazes, and parameter efficiency at parity. Practitioners evaluating agent stacks should weight format invariance and multi-turn retention alongside single-shot accuracy.