Mahesh Sathiamoorthy on Data Curation for Post-Training LLMs

The Bespoke Labs co-founder walked through how post-training turns raw capabilities into aligned, task-ready models, and the practical takeaways are concrete enough to fold into your own pipeline this week.

From static datasets to dynamic environments

Here's the bit that probably resonated with anyone running SFT or RLHF: post-training is no longer about piling up instruction-response pairs and hoping the gradient gods smile on you. Sathiamoorthy framed it as a shift from static dataset collection to dynamic environment design — a fancy way of saying we now build execution sandboxes where models can run code, call tools, and have their outputs auto-verified before they ever touch a weight update. For code-focused agents, that's the difference between a model that hallucinated a sorting algorithm and one that actually executed and passed the test suite. If your pipeline still assumes text-in, text-out, that's your first sanity-check failure — the boilerplate needs an environment, not just more rows.

The Open Thoughts recipe, and the weak-teacher surprise

Bespoke Labs' Open Thoughts project — a consortium with Stanford, UC Berkeley, and the University of Washington — gives us a concrete multi-stage flow to crib from: source questions across datasets, mix by hardness and quality, generate answers with teacher models, then filter. The ablations are where it gets spicy. As the finance.biggo.com write-up summarized it, "stronger AI models are not always the best teachers" — the team found that generating roughly 16 varied reasoning traces per question, on fewer questions overall, produced better fine-tuning results than spreading generations thin across more prompts. Scores on AIME and LiveCodeBench climbed as the curated set grew, suggesting the recipe compounds rather than plateaus. John Schulman's team at Thinking Machines confirmed it uses Open Thoughts internally, and Microsoft's CTO publicly endorsed the work — both useful signals when we're deciding which direction to invest the next ablation budget.

What to actually change in your next run

Let's boil it down to a checklist we can paste into the repo's CONTRIBUTING.md:

• Audit post-training data for diversity and hardness — not just raw volume.
• Build or borrow an execution environment if your task has verifiable outcomes; static text won't carry RL.
• Run the 16-traces-per-question ablation on a small slice before committing the full budget — that's the workaround for teacher-model lock-in.
• Filter aggressively; preference optimization and synthetic feedback loops earn their keep.
• Trust the scaling-law result: the recipe compounds, so curation discipline pays off long-term.

One last thread to pull — the principle that "curated beats raw" doesn't stop at model weights. It shows up in plenty of places, including John Stones' two-year move to Inter Milan, where fit clearly mattered more than the loudest bidder.