Statistical foundations of machine learning: the book

The book (whose abridged handbook version is freely available here) is dedicated to all researchers interested in machine learning who are not content with only running lines of (deep learning) code but who are eager to learn about this discipline’s assumptions, limitations, and perspectives.

The book aims to introduce students (at Master or PhD level) to the most important theoretical and applied notions to understand how, when and why machine learning algorithms work. The book contains 18 chapters, several appendices and is accompanied by a GitHub R package.

After an introduction, Chapter 2 introduces the problem of extracting knowledge from observations from a philosophical and epistemological perspective. Chapter 3 summarises the relevant background material in probability, by stressing, in particular, the notions of conditional dependence and independence. Chapter 4 introduces graphical modelling, a flexible and interpretable way of representing large variate problems in probabilistic terms. Chapter 5 introduces the classical (frequentist) parametric approach to parametric estimation and hypothesis testing. Chapter 6 introduces the Bayesian parametric approach to parametric estimation, hypothesis testing and decision theory. Chapter 7 presents some nonparametric alternatives to the parametric techniques discussed in the previous chapters. Chapter 8 introduces supervised learning as the statistical problem of assessing and selecting a hypothesis function on the basis of input/output observations. Chapter 9 reviews the steps which lead from raw observations to a usable predictive model. This is a methodological chapter that introduces some algorithmic procedures underlying most of the machine learning techniques. Chapter 10 presents conventional linear approaches to regression and classification and discusses the pros and cons of simple approaches, even in a machine learning domain which seems to be dominated by complex and power-hungry solutions. Chapter 11 introduces the challenge of tackling nonlinear supervised learning problems and presents a number of machine learning strategies to deal with nonlinear regression and classification tasks. Chapter 12 discusses the role of deep neural network learning and presents some examples of recent deep architectures. Chapter 13 presents the model averaging approach, a powerful way for obtaining improved generalisation accuracy by combining several learning machines. Chapter 14 deals with the problem of dimensionality reduction and in particular with feature selection strategies. Chapter 15 introduces stochastic processes and deals with the use of machine learning techniques for time series forecasting. Chapter 16 introduces the use of machine learning for modelling natural language (notably Large Language Models). Chapter 17 presents estimation algorithms in continuous and discrete-time dynamic settings (notably Kalman filter and Hidden Markov Models). Chapter 18 presents the relation between machine learning and causal inference by highlighting the risks of interpreting observational models in a causal manner. Although the book focuses on supervised learning, some related notions of unsupervised learning and density estimation are presented in Appendix A.