Teaching Machines to Understand Reality
Introduction: The Illusion of Understanding
Chapter 1: From Perceptrons to Predictors
- The long quest for machine understanding
- The statistical turn and the rise of deep learning
- Recognition without understanding
- Prediction as the new objective
Chapter 2: Brains as Predictive Machines
- The cortex as a prediction engine
- Predictive coding and error minimization
- The free energy principle
- Memory, imagination and simulation
- Lessons for artificial systems
Chapter 3: The Geometry of Representation
- Manifolds, features and latent spaces
- Invariance and equivariance
- Information theory for representations
- Compression, sufficiency and redundancy
- Design goals for predictive representations
Chapter 4: A Family Tree of Self-Supervision
- Autoencoders and denoising
- Autoregressive generation
- Contrastive learning
- Non-contrastive self-distillation
- Where each approach fails
Chapter 5: World Models: A Short History
- Dyna and the idea of learned models
- From pixels to latent dynamics
- The Dreamer family
- Generative latent world models
- Planning inside the model
Chapter 6: Predictive Coding Meets Deep Learning
- Classical predictive coding
- Free energy and variational inference
- Modern deep predictive coding
- Hierarchical temporal abstraction
- From neuroscience to architectures
Chapter 7: LeCun’s Vision: A Path to Autonomous Intelligence
- The critique of next-token prediction
- The proposed cognitive architecture
- Hierarchical prediction and timescales
- Memory, attention and planning
- From vision to autonomous agents
Chapter 8: The Anatomy of a JEPA
- The core idea: predict in representation space
- Context, target and predictor
- The mathematics of self-distillation
- Avoiding collapse
- Masking and information control
Chapter 9: The Mathematics of Joint Embedding Prediction
- Formulating the prediction objective
- Stop-gradient, EMA and implicit regularization
- An information-theoretic view
- Invariance, equivariance and the latent manifold
- Why collapse does and does not happen
Chapter 10: I-JEPA: Predicting Image Patches
- Motivation and design choices
- Architecture and data flow
- Training details
- Results and benchmarks
- What the representations look like
Chapter 11: V-JEPA: Understanding Video
- Why video demands more than images
- Architecture and temporal masking
- Training and results
- Understanding dynamics without labels
Chapter 12: V-JEPA 2 and the Scale-Up
- Scaling data and models
- Understanding physics and language
- Transfer and downstream tasks
- Implications for world models
Chapter 13: Action and Agency: Action-Conditioned JEPAs
- From passive observation to agency
- Action-conditioned prediction
- Robotics and manipulation
- Driving and navigation
- The road to embodied agents
Chapter 14: Building a JEPA from Scratch
- Project structure and dependencies
- Data pipeline
- Encoder, predictor and losses
- Training loop with EMA
- Visualization and sanity checks
Chapter 15: Scaling to Vision Transformers
- Implementing a ViT encoder
- Multi-scale masking
- Training at scale on a single GPU
- Linear probing and evaluation
- Interpreting the results
Chapter 16: Temporal World Models in PyTorch
- Video data pipeline
- The video encoder
- Temporal masking strategies
- Training and debugging
- Measuring temporal understanding
Chapter 17: From Prediction to Planning: Action-Conditioned World Models
- Environment design
- Learning the latent dynamics model
- Planning by imagined rollouts
- Experiments and ablations
- What breaks and why
Chapter 18: Engineering Reality: Training JEPAs at Scale
- Memory management and mixed precision
- Data pipelines that do not bottleneck
- Distributed training
- Reproducibility and logging
- Cost and hardware planning
Chapter 19: Do We Understand What We Have Built?
- Probing and linear evaluation
- Measuring what representations capture
- Diagnosing collapse and instability
- Visualizing latent spaces
- The danger of illusion
Chapter 20: World Models as Reasoning Engines
- Planning as search over predictions
- Model predictive control
- Imagination and counterfactuals
- Long-horizon planning
- Reasoning about the unseen
Chapter 21: Embodiment, Robotics and the Physical World
- The case for embodiment
- Robot learning with world models
- Sim-to-real and generalization
- Open challenges in physical interaction
Chapter 22: Limits and Horizons
- What JEPAs still cannot do
- Causality and counterfactuals
- Open research questions
- The long road to general understanding