Information Theory for Machine Learning

Information Theory for Machine Learning is a rigorous, application-oriented reader about the mathematical language behind modern ML. It starts from Shannon’s theory and repeatedly connects definitions and theorems to the losses, representations, probabilistic models, compression schemes, and engineering trade-offs used in practice.

The book is built around a simple discipline: always ask which distribution defines the expectation, whether a quantity is measured in bits or nats, and whether a statement is an exact theorem, a finite-sample estimator, a training objective, or a modelling analogy. This makes familiar ideas such as cross-entropy, KL divergence, perplexity, mutual information, and rate–distortion precise enough to use and critique.

The 16-chapter route covers:

• entropy, cross-entropy, KL divergence, Jensen’s inequality, and mutual information;

• source coding, communication channels, capacity, and maximum entropy;

• the Information Bottleneck, algorithmic information theory, f-divergences, and PAC-Bayes/MI bounds;

• information-theoretic views of LLM reasoning, computer vision, and representation learning;

• information geometry and a compact reference sheet.

The book is intended for ML engineers, researchers, graduate students, and independent learners with a basic background in probability, linear algebra, and machine learning. Derivations are paired with operational interpretations, and the boundary between a formal guarantee and a useful engineering viewpoint is made explicit.

This Leanpub edition contains the lecture texts and illustrations and is designed for continuous reading. The corresponding Stepik course is completely free and additionally includes auto-graded questions, exercises, detailed solutions, reproducible Python experiments and notebooks, and primary-source reading lists.

The book is part of an evolving series of standalone courses and readers on ML and LLMs. Each volume can be read independently; cross-references provide optional depth.

Complete free course on Stepik: https://stepik.org/295409