AI Engineering From Scratch

The Complete Guide to Understanding, Building and Deploying Production-Ready AI Systems

Introduction

1. What You Will Know After This Book
2. How This Book Is Structured
3. Cumulative Projects
4. Prerequisites and How to Use This Book
5. A Note on Toolchains and Versioning
6. Getting Started

Chapter 1: The AI Engineering Landscape

1. What Is AI Engineering
2. Three Categories of AI Applications
3. Capabilities, Limitations and Common Failure Modes
4. How This Book Is Structured
5. What Makes a Good AI Engineer

Chapter 2: Setting Up the Development Environment

1. Installing Python and Managing Versions
2. Virtual Environments and Dependency Isolation
3. Package Management with pip and Poetry
4. Code Editors and IDEs
5. Terminal and Shell Fundamentals
6. Git Basics for Version Control
7. Creating the Project Workspace
8. GPU Setup (Optional but Recommended)
9. Verifying Your Setup

Chapter 3: Python for AI Engineering

1. Data Structures for AI Work
2. Functions, Scope and Modular Design
3. Object-Oriented Programming for System Design
4. Configuration Files and Data Formats
5. Error Handling and Logging
6. Type Hints and Static Analysis
7. Asyncio for Concurrent Workflows
8. Summary

Chapter 4: Essential Mathematics for AI

1. Functions and Algebra
2. Exponents, Logarithms and Growth
3. Vectors and Matrices
4. Probability Fundamentals
5. Statistical Thinking
6. Calculus Intuition: Derivatives as Gradients
7. Summary

Chapter 5: Linear Algebra for Neural Networks

1. Vectors and Geometric Interpretation
2. Matrices as Linear Transformations
3. Matrix Multiplication and Computational Patterns
4. Dot Products and Similarity
5. Norms and Distances
6. Eigenvalues, Eigenvectors and Decompositions
7. Numerical Linear Algebra in Practice
8. Summary

Chapter 6: Probability and Statistics for AI

1. Random Variables and Distributions
2. Expectation, Variance and Moments
3. Conditional Probability and Bayes Theorem
4. Maximum Likelihood Estimation
5. Common Probability Distributions
6. Information Theory: Entropy and Cross-Entropy
7. Statistical Foundations of Evaluation
8. Summary

Chapter 7: The Mechanics of Machine Learning

1. What Machine Learning Is and Is Not
2. Supervised, Unsupervised and Reinforcement Learning
3. The Training Loop: Data, Loss, Optimization, Evaluation
4. Overfitting, Underfitting and Generalization
5. Data Splits and Preventing Data Leakage
6. Feature Engineering and Representation
7. Evaluation Metrics for Classification and Regression
8. Summary

Chapter 8: Optimization and Gradient Descent

1. The Optimization Problem
2. Loss Landscapes and Minima
3. Gradient Descent: Derivation and Intuition
4. Learning Rates and Momentum
5. Stochastic and Mini-Batch Gradient Descent
6. Regularization Techniques
7. Convergence Diagnostics and Learning Rate Scheduling
8. Summary

Chapter 9: Classical Machine Learning Methods

1. Linear Regression: Formulation and Solutions
2. Logistic Regression and Classification
3. k-Nearest Neighbors
4. Decision Trees and Random Forests
5. Support Vector Machines
6. Clustering Methods
7. When to Use Classical Machine Learning
8. Summary

Chapter 10: Neural Network Fundamentals

1. The Perceptron as a Computational Unit
2. Feedforward Networks and Layered Computation
3. Activation Functions and Non-Linearity
4. Loss Functions and Their Choices
5. Universal Approximation and Expressivity
6. Practical Network Design Choices
7. Summary

Chapter 11: Backpropagation and Training

1. The Chain Rule and Gradient Flow
2. Step-by-Step Backpropagation Derivation
3. Gradients for Each Network Component
4. Weight Initialization Strategies
5. Numerical Stability and Gradient Clipping
6. Practical Training Procedures
7. Debugging Training Problems
8. Summary

Chapter 12: Convolutional Neural Networks

1. Why Convolutions: Translation Invariance and Efficiency
2. The Convolution Operation: Kernels, Strides, Padding
3. Pooling and Downsampling
4. Classic CNN Architectures
5. Modern Architectures: ResNet and Beyond
6. Transfer Learning for Vision
7. Practical Computer Vision Applications
8. Summary

Chapter 13: Recurrent Neural Networks and Sequence Models

1. Why RNNs: The Challenge of Sequences
2. RNN Fundamentals and Hidden States
3. LSTM Architecture and Gating Mechanisms
4. GRU: The Simpler Alternative
5. Sequence-to-Sequence and Encoder-Decoder
6. Practical Applications and Limitations
7. Summary

Chapter 14: The Attention Mechanism

1. Why Attention: Limitations of Previous Approaches
2. Attention as Weighted Lookup
3. Self-Attention: Queries, Keys and Values
4. The Mathematics of Attention Scoring
5. Multi-Head Attention Architecture
6. Attention Complexity and Efficiency
7. The Full Attention Block
8. Summary

Chapter 15: Transformer Architecture

1. The Full Transformer Architecture
2. Positional Encoding and Order Information
3. Layer Normalization and Residual Connections
4. The Decoder: Causal Masking and Generation
5. Why Transformers Excel at Language
6. Next-Token Prediction at Scale
7. Transformer Variants and Specializations
8. Summary

Chapter 16: Tokenization and Text Representation

1. Why Tokenization: Text as Numbers
2. Character, Word and Subword Tokenization
3. Byte Pair Encoding Algorithm
4. WordPiece and SentencePiece
5. Vocabulary Size and Unknown Tokens
6. Context Length and Computational Implications
7. Cross-Lingual and Special-Purpose Tokenizers
8. Summary

Chapter 17: Large Language Models

1. The Evolution from BERT to Modern LLMs
2. Pretraining Objectives and Their Implications
3. Scale: Parameters, Data and Emergent Abilities
4. How Modern LLMs Are Trained
5. Decoder-Only, Encoder-Decoder and Instruction-Tuned Models
6. The Open-Weight Movement
7. Capabilities and Fundamental Limitations
8. Summary

Chapter 18: Working with Language Model APIs

1. Model Providers and Their Offerings
2. API Authentication and Cost Management
3. Prompt Construction and Message Structure
4. Generation Parameters: Temperature, Top-p and Beyond
5. Streaming Responses
6. Robust API Usage: Retries and Error Handling
7. Model Selection Criteria
8. Summary

Chapter 19: Open-Weight Models and Local Inference

1. Local vs. API: Trade-offs and Use Cases
2. Model Formats and Standards
3. Quantization Formats and Quality Impact
4. Local Inference Libraries
5. Hardware Requirements and Trade-offs
6. Serving Models as APIs
7. Model Selection for Local Deployment
8. Summary

Chapter 20: Prompt Engineering and Context Engineering

1. How LLMs Interpret Instructions
2. System Prompts and Role Conditioning
3. Few-Shot Prompting and In-Context Learning
4. Chain-of-Thought and Structured Reasoning
5. Managing Long Context Windows
6. Structured Outputs and Schema Enforcement
7. Prompt Templates and Testing
8. Common Failure Patterns
9. Summary

Chapter 21: Embeddings and Semantic Search

1. Word Embeddings and Semantic Representation
2. Sentence and Passage Embeddings
3. How Embeddings Capture Similarity
4. Vector Search and Approximate Nearest Neighbor
5. Embedding Models and Quality
6. Building a Semantic Search System
7. Evaluating Retrieval Performance
8. Summary

Chapter 22: Vector Databases and Retrieval Systems

1. Why You Need a Vector Database
2. Managed vs Self-Hosted Vector Databases
3. Comparing Major Vector Databases
4. Hybrid Search: Combining Vector and Keyword
5. Metadata Filtering and Structured Queries
6. Reranking and Result Refinement
7. Scaling Vector Databases
8. Summary

Chapter 23: RAG (Retrieval-Augmented Generation)

1. The RAG Architecture
2. Document Ingestion and Parsing
3. Chunking Strategies
4. Query Processing and Transformation
5. Context Assembly and Prompt Construction
6. Building a Complete RAG System
7. Summary

Chapter 24: Advanced RAG Techniques

1. Hierarchical Chunking and Parent-Child Retrieval
2. Graph-Based RAG and Knowledge Graphs
3. Query Routing and Specialized Retrieval
4. Multi-Hop Retrieval
5. Optimizing Retrieval Quality
6. Caching and Performance Optimization
7. Summary

Chapter 25: Integrating Model APIs into Applications

1. Service Architecture Around AI Models
2. Robust Error Handling
3. Circuit Breakers and Bulkheads
4. Rate Limiting and Cost Management
5. Multi-Provider and Multi-Model Strategies
6. Streaming Responses
7. Observability for AI Calls
8. Summary

Chapter 26: Multimodal AI Systems

1. Vision Models and Image Understanding
2. Speech and Audio Models
3. Multimodal Language Models
4. Building Multimodal Applications
5. Handling Multiple Modalities Efficiently
6. Multimodal RAG
7. Summary

Chapter 27: AI Agent Fundamentals

1. What Is an AI Agent?
2. The ReAct Pattern
3. Tool Use and Function Calling
4. Action Space and Constraints
5. Agent Memory and State
6. Controlling Agent Behavior
7. Summary

Chapter 28: Agent Architectures and Frameworks

1. Agent Patterns
2. LangGraph: Structured Agent Workflows
3. Tool Orchestration
4. Multi-Agent Systems
5. Frameworks and Libraries
6. Building Reliable Agent Systems
7. Summary

Chapter 29: Memory, State and Multi-Agent Systems

1. Types of Agent Memory
2. Conversation Memory and Context Management
3. Shared State in Multi-Agent Systems
4. Agent Communication Patterns
5. Coordinating Multi-Agent Workflows
6. Summary

Chapter 30: Workflow Orchestration

1. Why Workflow Orchestration Matters
2. Directed Acyclic Graphs (DAGs) for AI Workflows
3. Temporal for AI Workflows
4. Prefect for AI Data Pipelines
5. LangGraph for AI Agent Workflows
6. Conditional Execution and Dynamic Workflows
7. Error Handling and Resilience
8. Summary

Chapter 31: Fine-Tuning LLMs

1. When to Fine-Tune vs Prompt
2. Preparing Training Data
3. Supervised Fine-Tuning (SFT)
4. Instruction Tuning
5. Loss Landscape and Training Dynamics
6. Evaluation of Fine-Tuned Models
7. Cost Considerations
8. Deploying Fine-Tuned Models
9. Summary

Chapter 32: Parameter-Efficient Fine-Tuning: LoRA and QLoRA

1. The Problem with Full Fine-Tuning
2. LoRA: Low-Rank Adaptation
3. QLoRA: Quantized LoRA
4. Other PEFT Methods
5. Merging LoRA Adapters
6. Training LoRA on Consumer Hardware
7. Using Trained Adapters
8. Summary

Chapter 33: Model Distillation and Compression

1. Why Distill and Compress?
2. Knowledge Distillation Fundamentals
3. Distillation Data Generation
4. Output Distillation vs Process Distillation
5. Instruction Distillation
6. Preference Optimization (DPO, RLHF)
7. Pruning
8. Quantization for Deployment
9. Distillation vs LoRA vs Quantization
10. Evaluating Compressed Models
11. Summary

Chapter 34: Inference Optimization and Serving

1. Inference vs Training
2. The Inference Bottleneck
3. Batching Strategies
4. KV Cache Optimization
5. Speculative Decoding
6. Flash Attention
7. GPU Memory Management
8. Production Inference Servers
9. Monitoring Inference Performance
10. Cost Optimization for Inference
11. Summary

Chapter 35: Application Architecture for AI Systems

1. Architectural Characteristics of AI Systems
2. Layered Architecture for AI Applications
3. Microservices vs Monolith for AI
4. AI-Specific Architectural Patterns
5. Data Flow in AI Applications
6. Database Architecture for AI Applications
7. Error Handling Architecture
8. Scalability Considerations
9. Summary

Chapter 36: Containerization and Cloud Infrastructure

1. Docker for AI Applications
2. Docker with GPU Support
3. Docker Compose for Local Development
4. Kubernetes for Production Deployment
5. Cloud Providers for AI Workloads
6. Infrastructure as Code
7. Deployment Environments
8. Summary

Chapter 37: CI/CD for AI Systems

1. CI/CD Challenges for AI Systems
2. Testing AI Application Code
3. Testing AI Quality
4. Model and Prompt Versioning
5. CI Pipeline for AI Applications
6. CD Pipeline for AI Applications
7. Rollback Strategies
8. ML-Specific CI/CD Tools
9. Summary

Chapter 38: Monitoring, Observability and Evaluation

1. Why Monitoring AI Is Different
2. Key Metrics for AI Systems
3. Logging for AI Systems
4. Distributed Tracing for AI Systems
5. Production Evaluation
6. Drift Detection
7. Alerting and Incident Response
8. Diagnostic Tools and Techniques
9. Summary

Chapter 39: AI Reliability Engineering

1. The Reliability Problem in AI
2. Managing Nondeterminism
3. Preventing and Detecting Hallucinations
4. Reliable Retrieval
5. Reliable Tool Execution
6. Output Validation and Guards
7. Guardrails
8. Building Reliability into Architecture
9. Summary

Chapter 40: AI Security

1. OWASP Top 10 for LLM Applications
2. Prompt Injection
3. Data Leakage
4. Authentication and Authorization
5. Secure Tool Execution
6. Adversarial Attacks
7. Model Misuse
8. Security Architecture for AI Systems
9. Summary

Chapter 41: Responsible AI and Governance

1. Why Responsible AI Matters
2. AI Ethics Principles
3. Bias in AI Systems
4. Transparency and Explainability
5. Regulatory Landscape
6. AI Governance Frameworks
7. Building Responsible AI into Engineering Practice
8. Summary

Chapter 42: End-to-End Reference Architecture

1. Architecture Overview
2. Service Layer
3. AI Orchestration Layer
4. Retrieval Layer
5. Data Layer
6. Observability Layer
7. Infrastructure
8. Security
9. Summary

Chapter 43: Production-Readiness Framework

1. Why Production Readiness Matters
2. Production-Readiness Checklist
3. Load Testing
4. Cost Analysis
5. Incident Response
6. Blue-Green and Canary Deployments
7. Monitoring and Alerting
8. Post-Deployment Checklist
9. Summary

Chapter 44: The AI Engineering Methodology

1. The AI Engineering Process
2. Decision Frameworks
3. Common Pitfalls and How to Avoid Them
4. The AI Engineering Mindset
5. Summary

Chapter 45: Glossary

Chapter 46: Annotated Bibliography

1. Core AI and Deep Learning
2. Transformers and Language Models
3. Fine-Tuning and Adaptation
4. RAG and Retrieval
5. Agents and Planning
6. MLOps and Infrastructure
7. Security
8. Official Documentation
9. Research Platforms
10. Practice and Projects

References