The Leanpub 60 Day 100% Happiness Guarantee
Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.
See full terms...
The Quantization Black Book
The definitive engineering reference for 4-bit quantization of large language models — from 70B all the way to 400B parameters. Written for the people who actually have to fold these models onto real hardware and keep them fast, accurate, and deployable.
This is not an overview. It is a deep technical field manual covering the full quantization stack: the math, the algorithms, the hardware setup, the calibration process, the runtime kernels, and the memory tricks that make the difference between a model that runs and a model that doesn't.
Minimum price
Free!
$16.00
About the Book
The Quantization Black Book
The definitive engineering reference for 4-bit quantization of large language models — from 70B all the way to 400B parameters. Written for the people who actually have to fold these models onto real hardware and keep them fast, accurate, and deployable.
This is not an overview. It is a deep technical field manual covering the full quantization stack: the math, the algorithms, the hardware setup, the calibration process, the runtime kernels, and the memory tricks that make the difference between a model that runs and a model that doesn't.
What's inside
Who this is for
ML engineers deploying local LLMs at scale, research engineers at AI startups, contractors compressing models for clients, and serious practitioners who are tired of scattered arXiv papers and want the full quantization stack in one place, with the math and the code.
Format
Delivered as PDF and TXT. Read the PDF cover to cover, or feed the TXT straight into your own local RAG, grep it, index it, pipe it into your local LLM. The book about quantization, ready to be consumed by a quantized model. + EXTRAS
A note on responsibility
This material is provided for educational and research purposes. You are responsible for complying with the licenses of the models and tools you deploy, and with the laws of your jurisdiction. Build carefully and build for the right reasons.
Table of Contents
1. Define the technical scope and core engineering objectives of 'The Quantization Black Book' for 70B-400B models.
2. Establish the mathematical foundation of Floating Point (FP32, FP16, BF16) vs. Integer (INT8, INT4) representations.
3. Analyze the specific challenges of quantization-aware outliers in 70B+ parameter models.
4. Set up the high-performance computing (HPC) environment including multi-node H100/A100 configurations.
5. Configure Linux kernel parameters and NVMe scratch space for massive weight-sharding during quantization.
6. Deep dive into Round-to-Nearest (RTN) quantization and its limitations for large-scale LLMs.
7. Implement GPTQ (Optimal Brain Quantization) theory and the Hessian-based error compensation.
8. Master Activation-aware Weight Quantization (AWQ) and the importance of preserving salient weights.
9. Explore the mechanics of GGUF and K-Quants for cross-platform deployment.
10. Deconstruct the EXL2 format and its variable bitrate quantization strategies for Llama-3 class models.
11. Investigate QuIP# and the application of randomized Hadamard transforms to minimize quantization error.
12. Analyze HQQ (Half-Quadratic Quantization) for fast, calibration-free 4-bit compression.
13. Detail the selection process for calibration datasets like The Pile or C4 to ensure weight distribution accuracy.
14. Develop a pipeline for layer-wise error tracking using KL-Divergence and Mean Squared Error (MSE).
15. Implement offloading strategies for quantizing 400B models on limited VRAM using CPU-RAM swapping.
16. Address the 'Memory Wall' by optimizing CUDA kernels for 4-bit dequantization at runtime.
17. Integrate QLoRA (Quantized Low-Rank Adaptation) concepts for fine-tuning 4-bit artifacts.
18. Explain Double Quantization techniques to further reduce the memory footprint of quantization constants.
19. Document the process of K-Cache (Key-Value) quantization to 4-bit or 8-bit for extended context windows.
20. Create a comprehensive benchmarking suite using LM-Evaluation-Harness (MMLU, GSM8K, Hellaswag).
21. Evaluate Perplexity (PPL) degradation across different bit-widths (3-bit vs 4-bit vs 5-bit).
22. Measure inference throughput (Tokens per Second) across vLLM, TensorRT-LLM, and TGI backends.
23. Design a decision matrix for selecting quantization methods based on specific hardware constraints.
24. Write the engineering chapter on custom Triton kernels for specialized 4-bit weight formats.
25. Document the 'Weight Shifting' and 'Smoothing' techniques used in SmoothQuant for activation outliers.
26. Provide a step-by-step guide for converting Hugging Face safetensors into quantized GGUF/EXL2 files.
27. Establish a CI/CD pipeline for automated model compression and regression testing.
28. Design the visual architecture diagrams for weight-only vs. weight-activation quantization.
29. Write the advanced troubleshooting section for 'NaN' propagation and divergence during 4-bit inference.
30. Compile the 'Hardware Selection Guide' for cost-effective deployment of 400B models.
31. Format the manual using LaTeX for professional mathematical typesetting and technical clarity.
32. Develop an accompanying GitHub repository with scripts for reproducible quantization workflows.
33. Conduct a final technical peer review of code snippets against the latest versions of AutoGPTQ and AutoAWQ.
34. Draft the preface on the future of 1-bit and 1.58-bit (ternary) LLM architectures.
35. Finalize the 'Black Book' for digital distribution as a high-end engineering resource for AI engineers.
About the Author
I am an independent technology developer and systems engineer who built my technical path largely through self-directed engineering, experimentation, and continuous learning outside a traditional academic or corporate technology career.
My professional background began far from the technology industry. I spent years working in manufacturing, while independently developing my knowledge of software engineering, computer systems, and advanced computing. Over time, that self-directed work evolved into a broad technical practice spanning autonomous AI, cybersecurity, systems programming, GPU computing, automation, and advanced computational architectures.
Today, I design, build, and publish projects involving agentic AI, autonomous defense systems, SIEM/EDR integration, secure software architecture, C/C++, Go, Python, CUDA, quantum computing, cryptography, and privacy-oriented local AI infrastructure.
I approach technology from a systems perspective — from low-level software, memory architecture, and GPU performance to distributed systems, intelligent agents, and high-assurance security architectures.
I also explore aerospace and high-assurance software concepts, including safety-critical architectures, multi-level security, cross-domain solutions, and advanced computational systems.
Alongside active development, I publish long-form engineering projects covering AI, cybersecurity, cloud engineering, quantum computing, GPU programming, cryptography, automation, blockchain, and aerospace engineering.
My current focus is on autonomous software agents, privacy-first local infrastructure, advanced computing, and reliable systems designed to operate with a high degree of independence.
I am open to opportunities involving AI engineering, cybersecurity, software engineering, autonomous systems, HPC/GPU computing, and advanced technology development.
https://businessofmachines.blogspot.com/
https://learn.microsoft.com/en-us/users/machinadeusex/
You can get the free Community Edition in PDF or EPUB just by sharing your name and email address with the author.
Also by the Author
Mastering NVIDIA CUDA: Expert GPU Programming
Linux GPU Drivers from Scratch
Mastering LLM Quantization: Architecture and Code
Engineering Sovereign LLMs: From Data to Deployment
CUDA C++: High-Performance GPU Programming
Agentic Cybersecurity: Engineering Autonomous AI Defense
Blockchain Security: Attack and Defense
The Local AI Stack: Building a Sovereign Machine Learning Workstation with Hyper-V, WSL2, and GPU Virtualization
The Solana MEV Playbook: Building High-Frequency Arbitrage and MEV Bots in Rust
Mastering ELF: From Bytes to Execution
Building a Native LLM Inference Engine
Modern Browser Fingerprinting: Techniques and Code
Mastering AI Agents: Hands-On Production Code
Coding CERN: A Technical Developer's Guide
High-Performance Real-Time Systems in .NET
Building Advanced Algorithmic Crypto Trading Systems
Smart Contract Security: Finding DeFi Vulnerabilities
Advanced Aerospace Engineering: Lockheed & Partners
Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.
See full terms...
We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.
(Yes, some authors have already earned much more than that on Leanpub.)
Learn more about writing on Leanpub
If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).
Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.
Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.
Learn more about Leanpub's ebook formats and where to read them
You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!
Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.
Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.