Information Theory for Machine Learning

Information Theory for Machine Learning is a rigorous, application-oriented reader about the mathematical language behind modern ML. It starts from Shannon’s theory and repeatedly connects definitions and theorems to the losses, representations, probabilistic models, compression schemes, and engineering trade-offs used in practice.

The book is built around a simple discipline: always ask which distribution defines the expectation, whether a quantity is measured in bits or nats, and whether a statement is an exact theorem, a finite-sample estimator, a training objective, or a modelling analogy. This makes familiar ideas such as cross-entropy, KL divergence, perplexity, mutual information, and rate–distortion precise enough to use and critique.

The 16-chapter route covers:

• entropy, cross-entropy, KL divergence, Jensen’s inequality, and mutual information;

• source coding, communication channels, capacity, and maximum entropy;

• the Information Bottleneck, algorithmic information theory, f-divergences, and PAC-Bayes/MI bounds;

• information-theoretic views of LLM reasoning, computer vision, and representation learning;

• information geometry and a compact reference sheet.

The book is intended for ML engineers, researchers, graduate students, and independent learners with a basic background in probability, linear algebra, and machine learning. Derivations are paired with operational interpretations, and the boundary between a formal guarantee and a useful engineering viewpoint is made explicit.

Updated book edition 2026.2 (16 October 2026) contains 16 modules with current lectures and illustrations. All course materials are freely available on the course website: 377 quiz questions, 90 exercises and project tasks, and 9 Python solution files. Answer checking, expandable worked solutions and practice downloads are available online; each book module links to the corresponding materials.

Course website: https://vladkulikov.com/courses/information-theory-for-ml/

The book is part of an evolving series of standalone courses and readers on ML and LLMs. Each volume can be read independently; cross-references provide optional depth.