LLM Quantization Recipes

A Practical Guide to Compressing Large Language Models Without Losing Their Intelligence

Introduction: Why Compress the Mind?

1. What You Will Learn
2. How This Book Is Organized
3. Prerequisites

Chapter 1: The Compression Imperative

1. The Size Explosion: From GPT-2 to Today
2. Why Raw Parameters Are a Bottleneck
3. What Quantization Actually Does
4. A Brief History of Model Compression
5. What This Book Covers (and Does Not)

Chapter 2: Foundations of Numerical Precision in Deep Learning

1. Floating-Point Arithmetic: FP32, FP16, BF16
2. Integer Representations: INT8, INT4, INT2
3. Mixed Precision and the NF4 Format
4. How GPUs and CPUs Handle Different Datatypes
5. The Information Theory of Weight Distributions

Chapter 3: Post-Training Quantization vs. Quantization-Aware Training

1. Post-Training Quantization (PTQ): The Quick Path
2. Quantization-Aware Training (QAT): The Careful Path
3. Weight-Only Quantization: The LLM Sweet Spot
4. Activation Quantization and the Outlier Problem
5. Hybrid Approaches and Per-Token Strategies

Chapter 4: Calibration – Teaching Precision to Models

1. The Role of Calibration Data
2. Min-Max vs. Percentile Clipping
3. Moving Average and Histogram-Based Methods
4. Optimal Perceptual Quantization (OPQ)
5. How Much Calibration Data Do You Really Need?

Chapter 5: GPTQ – Greedy One-Shot Quantization

1. The Hessian Approximation Idea
2. Layer-by-Layer Greedy Optimization
3. The GPTQ Algorithm Step by Step
4. AutoGPTQ: The Practical Implementation
5. Strengths, Limitations, and Typical Results
6. Mathematical Derivation: Why the Hessian Works
7. Complete Production GPTQ Quantization Script
8. Production Readiness Checklist for GPTQ

Chapter 6: AWQ – Activation-Aware Weight Quantization

1. The Activation Magnitude Insight
2. Weight Scaling Before Quantization
3. The AWQ Algorithm: Smoothing and Rescaling
4. AWQ vs. GPTQ: A Head-to-Head Comparison
5. Practical Usage with AutoAWQ
6. Complete Production AWQ Quantization Script
7. Production Readiness Checklist for AWQ
8. Case Study: Deploying a 70B Model on a Single A100

Chapter 7: GGUF and the llama.cpp Ecosystem

1. The GGML Legacy and GGUF’s Design
2. K-Quants: Q4_0, Q4_K_S, Q5_K_M, Q8_0
3. How llama.cpp Runs Quantized Models on CPU
4. Performance on CPUs vs. GPUs with Metal/Vulkan
5. Community Toolchains and Model Hubs
6. Importance Matrix (imatrix) Quantization: A Deep Dive
7. Complete GGUF Conversion Pipeline

Chapter 8: BitsAndBytes and the NF4 Revolution

1. The bitsandbytes Library Architecture
2. NormalFloat4: Why It Beats Plain INT4
3. QLoRA: Fine-Tuning in 4 Bits
4. 8-Bit Adam and Optimizer Quantization
5. Practical Usage with the Transformers Library
6. Complete Production QLoRA Fine-Tuning Script
7. Production Readiness Checklist for QLoRA

Chapter 9: Unsloth – Speed Through Quantized Fine-Tuning

1. The Fine-Tuning Bottleneck
2. Unsloth’s Architecture and Optimizations
3. Patched Transformers: How the Speedup Works
4. Benchmarks: Unsloth vs. Standard QLoRA
5. Practical Usage and Limitations

Chapter 10: Advanced Frameworks – Bartowski, ByteShape, and Apex

1. Bartowski’s Quantization Pipeline
2. ByteShape: Understanding This Approach
3. NVIDIA Apex: From Mixed-Precision Training to FP8
4. Other Notable Methods: SmoothQuant, ZeroQuant, SPA
5. The Fragmentation Problem in Toolchains

Chapter 11: Deployment Scenarios and Hardware Considerations

1. Local Inference on Consumer GPUs
2. CPU-Only Deployment and Edge Devices
3. High-Throughput Server Serving
4. Mobile and On-Device LLMs
5. Case Study: Deploying a Customer Support Chatbot on a Single GPU
6. The Memory Bandwidth Bottleneck

Chapter 12: Benchmarking, Best Practices, and Decision Framework

1. A Reproducible Benchmarking Protocol
2. How to Measure Quantization Quality
3. Perplexity, Accuracy, and Latency Benchmarks
4. Common Pitfalls and Debugging Tips
5. The Quantization Decision Framework
6. Future Directions: What’s Next in Model Compression

Conclusion: The Democratized Model

References