Running LLMs on Tiny Hardware

A Systems Engineer’s Guide to Edge Inference on Resource-Constrained Devices

Introduction

1. What This Book Covers
2. How to Read This Book
3. Technology Context
4. The Central Thesis

Chapter 1: The Physics of Tiny

1. Why Memory Rules Everything
2. SRAM, DRAM, and Flash: Capacity, Speed, and Cost
3. Memory Hierarchy and Cache Behavior
4. Compute vs. Bandwidth: Arithmetic Intensity
5. Power Budgets and Thermal Walls
6. The Numbers: How Much Memory and Bandwidth Does an LLM Actually Need

Chapter 2: Inside an LLM Inference Step

1. Tokenization: Strings to Integers
2. The Embedding Layer and Positional Encoding
3. Transformer Blocks: Self-Attention and Feed-Forward
4. Multi-Head and Grouped-Query Attention
5. Layer Normalization and Residual Connections
6. Autoregressive Decoding and the KV Cache
7. The Complete Inference Dataflow

Chapter 3: Measuring Feasibility

1. Parameter Storage: Bits, Bytes, and Formats
2. Activation Memory: What You Cannot See Coming
3. KV Cache: The Hidden RAM Killer
4. Arithmetic Intensity and Compute Requirements
5. Bandwidth Calculations: The True Bottleneck
6. The Feasibility Checklist

Chapter 4: Model Architectures for Constrained Inference

1. Dense Decoder-Only Transformers: The Baseline
2. Mixture of Experts: Selective Activation
3. Recurrent and Hybrid Architectures
4. Compact Language Models: Design for Small
5. Architecture Choices That Matter for Embedded: Context Window, Heads, FFN Width
6. Selecting a Model Family for Your Target

Chapter 5: Quantization Fundamentals

1. Why Quantization Works: Redundancy in Floating-Point Weights
2. Symmetric vs. Asymmetric Quantization: Scales and Zero-Points
3. Per-Tensor vs. Per-Channel vs. Group-wise
4. Calibration Strategies
5. The Quantization-Error-Accuracy Tradeoff
6. Quantization Formats: FP16, BF16, INT8, and Lower

Chapter 6: Weight-Only Quantization

1. How Weight-Only Quantization Works
2. Dequantization on the Fly: The Compute Cost
3. Packing Schemes: 4-bit, 5-bit, 6-bit Weights
4. Weight-Only Quantization and Activation Outliers
5. Implementing Weight-Only Matrix Multiplication
6. When Weight-Only Is Enough (and When It Is Not)

Chapter 7: Activation Quantization and Mixed Precision

1. Why Activations Are Harder to Quantize
2. Activation Outliers and Their Impact
3. Mixed Precision Strategies
4. Quantization-Aware Training
5. Post-Training Quantization with Activation Calibration
6. INT4/INT3 Activation Quantization: The Cutting Edge

Chapter 8: Sparsity, Pruning and Structured Compression

1. Unstructured vs. Structured Sparsity
2. Pruning Methods and Magnitude Thresholding
3. Compressed Sparse Row and Other Formats
4. The Problem of Sparsity on Small CPUs
5. Block-wise and Channel-wise Pruning
6. Low-Rank Factorization
7. When to Use Pruning and Sparsity

Chapter 9: Knowledge Distillation and Architecture Modification

1. Distillation: Making Small Models Smarter
2. Architecture Surgery: Head Merging, Layer Thinning
3. Vocabulary Reduction and Tokenizer Optimization
4. Context Window Management and Compression
5. Speculative Decoding: Drafting and Verifying
6. When to Modify the Model vs. When to Optimize the Runtime

Chapter 10: KV Cache Engineering

1. KV Cache Layout: Memory Mapping the Context
2. KV Cache Quantization and Compression
3. Eviction Policies and Sliding Windows
4. Activation Checkpointing and Recomputation
5. Streaming and Weight-Reuse Strategies
6. KV Cache for Long Context on Tiny Memory

Chapter 11: CPU, SIMD and Vector Optimization

1. Vector Units: NEON, SSE, AVX, and RISC-V V-Extensions
2. Writing Vectorized Matmul Kernels
3. Packing and Unpacking for SIMD
4. Instruction-Level Parallelism and Loop Unrolling
5. Memory Access Patterns and Cache Locality
6. Tuning for Specific CPU Microarchitectures

Chapter 12: GPU, NPU and DSP Acceleration

1. Mobile GPUs: Mali, Adreno, PowerVR
2. Neural Processing Units: NPU Architectures and Constraints
3. DSP Acceleration for Embedded Systems
4. Tensor Cores and Matrix Multiply Accelerators
5. Accelerator APIs and Vendor SDKs
6. When Accelerators Help (and When They Get in the Way)

Chapter 13: Inference Runtimes and Software Architecture

1. The Runtime Stack: From Model Files to Generated Tokens
2. Graph Representations and IR Formats
3. Lightweight Transformer Runtimes
4. Vendor-Specific Runtimes vs. Portable Solutions
5. Kernel Selection and Code Generation
6. Designing Your Own Minimal Inference Runtime

Chapter 14: Model Conversion and Deployment Pipelines

1. Exporting Models: ONNX, GGUF, safetensors, and Others
2. Tokenizer Packaging and Conversion
3. Quantization Pipelines: Tools and Techniques
4. Cross-Compilation and Build Systems
5. Binary Size, Flash Constraints, and OTA Updates
6. Model Versioning and Update Strategies

Chapter 15: Performance Engineering and Benchmarking

1. Metrics That Matter: TTFT, ITL, Throughput, and Energy
2. Benchmarking Methodology: Warm-Up, Sampling, Reproducibility
3. Profiling: Finding the Real Bottleneck
4. Compute-Bound vs. Memory-Bound vs. Synchronization-Bound
5. Compiler Optimization and Its Surprises
6. Performance Regression and Change Management

Chapter 16: Power, Energy and Thermal Management

1. Measuring Power and Energy in Real Deployments
2. Energy per Token: A Practical Metric
3. Dynamic Voltage and Frequency Scaling
4. Thermal Throttling and Sustained Performance
5. Power-Optimized Scheduling and Batching

Chapter 17: Debugging and Failure Analysis

1. Incorrect Outputs: Numerical Error vs. Model Behavior
2. Memory Exhaustion and Fragmentation
3. Stack Overflow and Heap Corruption
4. Quantization Bugs: Zero-Point and Scale Errors
5. Hardware-Specific Issues: Alignment, Endianness, FPU Modes
6. Silent Accuracy Degradation and Regression Testing

Chapter 18: Security and Privacy

1. The Security Model of Local Inference
2. Model Extraction and Protection
3. Supply Chain Risks: Malicious Models and Tooling
4. Input and Output Security: Prompt and Data Handling
5. Secure OTA Updates and Integrity Verification
6. Privacy Advantages and Their Limits

Chapter 19: Case Study : An LLM in 256 MB of RAM

1. The Hardware and Its Constraints
2. Choosing and Preparing the Model
3. Memory Budget: Weights, KV Cache, Runtime Overhead
4. Implementation Decisions and Trade-offs
5. Benchmarking Results
6. Lessons Learned

Chapter 20: Case Study : Offline Edge Assistant on an ARM SBC

1. System Requirements and Architecture
2. Model and Tokenizer Selection
3. Building the Inference Engine
4. Application Integration: Input, Output, Streaming
5. Robustness: Error Handling, Watchdogs, Recovery
6. Production Readiness Checklist

Chapter 21: Case Study : Pushing the Limits on MCU-Class Hardware

1. MCU Constraints: Kilobytes of RAM, Megahertz Clocks
2. Micro-Models: Can Transformers Run on an MCU?
3. Extreme Quantization and Architecture Reduction
4. Alternative Approaches: Recurrent and State-Space Models
5. Measured Performance and Practical Viability
6. The Real Limit: Where LLMs Cannot Go

Chapter 22: Production Deployment Patterns

1. Firmware Integration and Boot-Time Constraints
2. Multi-Model and Fallback Strategies
3. Handling Concurrent Requests on Constrained Systems
4. Observability: Logging, Metrics, Telemetry
5. A/B Testing and Model Rollout at the Edge
6. Maintenance and Lifecycle Management

Chapter 23: Future Directions

1. Architectural Innovations: Mamba, RWKV, and State-Space Models
2. Hardware Roadmaps: What Next-Gen Edge Silicon Will Bring
3. Algorithmic Advances: Better Quantization, Sparsity, Distillation
4. Compiler and Runtime Innovations
5. The Economics of Edge Inference
6. The Next Frontier: What Comes After Tiny LLMs

Conclusion: The Engineer’s Playbook

1. The Decision Tree: Choosing Your Optimization Strategy
2. A Checklist for Edge LLM Deployment
3. Accepting Trade-offs: No Solution Is Free
4. Final Thoughts: The Promise and Limits of Edge Intelligence

References