TLA+-Bench: Evaluating LLM Reasoning Through Formal Specification Execution

The dataset comprises 403 model-checked gold specifications and 897 parse-only silver specifications, separating execution-validated output from syntactically valid but unverified entries. The release addresses a structural gap in formal-methods evaluation by tying grading to TLC model checker execution rather than surface similarity to reference strings.

Methodology and Dataset Composition

TLA+-Bench partitions its corpus into two tiers. Gold specifications — 403 in total — pass the TLC model checker and function as the authoritative evaluation set. Silver specifications — 897 entries — clear the parser only, supplying additional few-shot and training material without execution grounding. The split enables a direct probe of whether parse-correct output correlates with model-checked correctness, a distinction prior corpora collapsed into a single grading signal.

Why Execution Grounding Matters

Execution-anchored benchmarks materially raise the evaluation bar for code-generation systems. Surface metrics — BLEU, token overlap, even pass@1 against reference code — cannot detect TLA+ that parses cleanly yet violates invariants or deadlocks. By coupling scoring to TLC outcomes, TLA+-Bench measures whether a model produces specifications that actually preserve stated safety and liveness properties. Practitioners auditing agentic coding workflows should treat this as a methodological upgrade over static reference comparison, with implications for any pipeline that delegates protocol or distributed-system design to an LLM.

Context Across the July Benchmark Wave

TLA+-Bench lands alongside a cluster of dataset releases targeting distinct evaluation gaps. Wake Vision (Technical University of Denmark) ships roughly six million labeled images for binary person detection on TinyML hardware, explicitly addressing label noise and scale ceilings in earlier corpora. ResearchClawBench (InternScience, via GitHub) measures whether coding agents can execute end-to-end scientific research from raw data to publication-quality report. PerceptionBench (Moonshot AI, on Hugging Face) probes ten atomic visual perception axes in multimodal LLMs. The pattern is consistent: evaluation is shifting from string-similarity proxies toward outcome-grounded scoring across modalities.

Datasets of this kind increasingly inform structured-reasoning tooling that surfaces formal-specification concepts through interactive formats — a direction visible across the educational gaming and edutainment space, where problem decomposition and logical verification are being repackaged for non-specialist audiences. For ML practitioners, the immediate actionable items are: replicate the TLC harness on a target model, report separate gold and silver scores, and benchmark against a parse-only baseline to quantify the execution-grounding delta.