Leanpub Header

Skip to main content

Information Theory for Machine Learning

From NLP and LLMs to Computer Vision and Reinforcement Learning

Understand what information-theoretic quantities actually measure in machine learning. Follow the mathematics through proofs, numerical examples, quizzes, and reproducible Python experiments.

Interested in this course? Show your support by saying what you'd like to pay for it!

$
PDF
EPUB
WEB
About

About

About the Course

Information theory gives machine learning a precise language for uncertainty, prediction, compression, representations, and the cost of changing a model. This course develops that language from surprisal and entropy through cross-entropy and KL divergence, mutual information, source and channel coding, maximum entropy, Information Bottleneck, MDL, variational bounds, and information geometry. Applications connect the mathematics to language models, computer vision, and reinforcement learning.

The course contains 16 modules, exercises with worked solutions, multiple-choice and numerical quizzes, and downloadable Python experiments. Short proofs explain reusable mechanisms; optional mathematical deep dives provide a second reading route. Definitions, assumptions, units, and the limits of each conclusion stay close to the formulas they qualify.

The intended reader has a working knowledge of probability, calculus, linear algebra, and basic machine learning. The course is self-contained within that background. It forms the mathematical foundation of a three-course program with Modern LLMs and Reinforcement Learning for LLM; the companion courses are optional rather than prerequisites.

Share this course

Instructor

About the Instructor

Vladlen Kulikov

Lead AI/ML Engineer & Researcher | LLMs, AI Agents, RAG | AI Function Lead in FinTech | Author of 3 Bilingual AI/ML Courses | M.Sc. in AI & Computer Science.

Since 2017, I have worked in R&D, built ML and GenAI systems, and advised businesses on AI adoption.

Education and research:

- Physics at Kharkiv University;

- MIPT University master's program in Modern Artificial Intelligence.

Thesis and publications: LLMs, Learning and Reasoning at the Inference Stage.

The Leanpub 60 Day 100% Happiness Guarantee

Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.

See full terms...

Earn $8 on a $10 Purchase, and $16 on a $20 Purchase

We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.

(Yes, some authors have already earned much more than that on Leanpub.)

Learn more about writing on Leanpub

Free Updates. DRM Free.

If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).

Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.

Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.

Learn more about Leanpub's ebook formats and where to read them

Write and Publish on Leanpub

You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!

Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.

Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.

Learn more about writing on Leanpub