Reinforcement Learning for LLM

Reinforcement Learning for LLM is an engineering reader on modern reinforcement learning for language models and agentic systems—from classical RL foundations to post-training, verifiable rewards, search, infrastructure, evaluation, and safety.

The book is not organized as a catalogue of algorithms. Its goal is to help you choose a method from the data and feedback contract, identify the policy being optimized, validate estimators and masks, understand rollout cost, and distinguish training from inference-time candidate selection.

The 13-chapter route covers:

• MDPs and POMDPs, value functions and Bellman equations, bandits, exploration, and off-policy evaluation;

• policy gradients from REINFORCE to PPO, imitation learning, and offline RL;

• the LLM as a token-level policy, sequence log-probabilities, KL control, reward models, and preference data;

• RLHF and direct preference optimization, including DPO-family methods;

• RL with verifiable rewards, GRPO, search, verification, and test-time compute;

• RL for LLM agents, tools, environments, and long trajectories;

• rollout and training infrastructure, statistical evaluation, reward hacking, contamination, and safety.

Equations are tied to practical contracts and diagnostic checks. The text emphasizes what is optimized, what data are required, where bias and variance enter, and which failure modes can make an apparently successful run misleading.

The book is intended for ML, AI, and LLM engineers, alignment and agent researchers, graduate students, and practitioners already using SFT, DPO, PPO, GRPO, or LLM judges. No previous RL course is required: the opening chapters build the classical foundation, while familiarity with machine learning, Python, and basic probability is assumed.

This Leanpub edition contains the open lecture texts and illustrations. The paid Stepik course adds a structured path, more than 500 auto-graded tests, exercises and self-check activities, detailed solutions, and notebook practice.

The book is part of an evolving series of standalone courses and readers on ML and LLMs. Each volume can be studied independently; cross-references provide optional depth.

Complete Stepik course: https://stepik.org/a/294393