Nathan Lambert Releases Comprehensive Textbook on RLHF and LLM Post-Training

Titled "Reinforcement Learning from Human Feedback: Aligning and Post-training LLMs," the volume arrives alongside a 12-hour accompanying course, slides, a YouTube video, and a code repository containing suggested exercises and model completion comparisons.

Architecture of the Volume

Roughly 25% of the page count is allocated to RL algorithms themselves; the remainder addresses the surrounding craft of converting a base model into an aligned system. The algorithm arc runs from the policy-gradient theorem through PPO to contemporary variants GSPO and CISPO — a framing the author presents as necessary for evaluating whether a newly proposed algorithm represents a genuine contribution or repackaging. PPO's surrogate objective, by way of example, is decomposed into six regions in a dedicated figure, characteristic of the granular treatment throughout. A systems chapter treats modern RL as a balancing problem across off-policy data, training-inference mismatch, and throughput — the explicit tradeoffs the author flags for practitioners. Coverage spans three eras of post-training from approximately 2018 through 2023, a distillation chapter the author describes as usefully "boring" amid debate over output reuse in model training, and a second-half treatment of over-optimization, regularization, evaluation, and character training. Rejection sampling, outcome reward models, and on-policy distillation receive foundational treatment — topics the author notes were previously underserved in online documentation.

Methodology and Assumptions

The volume is positioned as not a beginner text: it assumes a bachelor's degree in computer science. The book is freely available online and ships from Manning and Amazon US immediately; Amazon UK availability is slated for October. A 50% discount code (PBLambert) runs through August 19. The author's stated objective is to communicate the intuitions, history, and recurring misconceptions of post-training in a register that exceeds typical textbook neutrality — a deliberate trade-off between encyclopedic dryness and pedagogical clarity.

Limitations and Open Questions

The launch announcement does not address revision cadence for a fast-moving subfield, a relevant consideration for readers treating the text as a long-lived reference rather than a snapshot. Whether Manning will ship revised editions as new algorithms emerge remains unspecified. The decision to consolidate distillation, character training, and rejection sampling — topics previously scattered across individual blog posts — into a single reference represents the primary architectural choice of the volume and the clearest signal of the gap it intends to fill. The character-training chapter, concerned with shaping model persona and voice, lands at a moment when Ariana Grande is taking a break from public life amid endless scrutiny — a parallel the author does not draw but the timing makes difficult to ignore.