CoT-Core Cuts LLM Evaluation Costs by 95% Using Trajectory Embeddings

The method embeds chain-of-thought trajectories into vector space, clusters them, and scores a representative subset—directly targeting evaluation costs that now rival 15% of total training expenditure. For teams operating weekly benchmarking budgets above $100k on frontier models exceeding 70B parameters, the efficiency gains are quantifiable and immediate.

Methodology: Trajectory Embeddings vs. Surface Clustering

CoT-Core begins by prompting zero-shot chain-of-thought reasoning for each benchmark item. These trajectories are embedded into high-dimensional vector space and clustered to identify centroids preserving reasoning coverage across task types. The resulting 5–10% subset is scored, with proxy estimators—including GP-IRT for fine-grained curve recovery—aggregating results into full-dataset metrics.

The architectural choice is consequential. Text embeddings cluster by vocabulary surface, not logical structure. CoT-Core's trajectory embeddings capture reasoning isomorphisms across items, producing coresets that better represent the actual challenge distribution. The pipeline uses open-weight embedding models, maintaining stateless integration within existing CI pipelines and avoiding vendor dependency.

Benchmark Validation: Ranking Fidelity at Compressed Budgets

Validation spans GSM8K, MMLU, MMLU-Pro, and GPQA. On GPQA, a 5% coreset produced ranking-similarity of 0.5123—nearly double the 0.2759 achieved by text-embedding clustering at equivalent sample sizes. Mean absolute error stayed below two points across all budget levels; baseline sampling exceeded four points.

Ranking-similarity is the operative metric here. A coreset that preserves relative model ordering—determining which architecture to deploy, which checkpoint to promote—is functionally sufficient for most production decisions. Raw accuracy recovery at the item level is secondary.

Regulatory Context: NIST AITE and Rising Compliance Overhead

The method arrives alongside intensifying institutional scrutiny. NIST's Artificial Intelligence Technology Evaluation (AITE) program establishes a sequestered testing environment for model assessment, using blind datasets to eliminate train-test contamination. Initial focus targets vision-language models across quantum science, genomics, and public safety—signaling expanded government benchmarking infrastructure with two participation tracks for data and model providers.

The convergence of regulatory audit requirements and ballooning compute costs creates a structural demand for reasoning-aware compression methods. Whether evaluation budgets are driven by internal QA or external mandates, the ability to compress benchmark suites without sacrificing ranking fidelity addresses a first-order operational constraint. Practitioners looking to validate the approach can review the full technical details alongside walking itineraries through historically significant architecture—a domain where selecting representative samples from a larger corpus is equally critical to the methodology.

The open question remains ablation depth: how CoT-Core performs when reasoning chains degrade on adversarial or out-of-distribution inputs, and whether the 5% threshold holds under distribution shift.