Evaluating Local and Small Language Models

A Practical Engineering Guide to Benchmarking, Optimization, and Production Deployment

Introduction

1. Why evaluation matters more for local models
2. What you will be able to do after reading this book
3. Who this book is for
4. How this book is organized
5. A note on pace and specificity

Chapter 1: Defining the Landscape

1. What is a Small Language Model
2. What is a Local Language Model
3. Why Local Matters
4. The Hardware Landscape
5. Cost and Privacy Implications
6. How This Book is Structured

Chapter 2: Model Architectures and Scaling

1. Transformer Fundamentals
2. Parameter Count and Scaling Laws
3. Attention Mechanisms and Variants
4. Grouped-Query and Multi-Query Attention
5. Mixture of Experts Architectures
6. Context Window Mechanics
7. Architectural Choices that Affect Inference

Chapter 3: Model Quality and Capability Dimensions

1. Instruction Following and Alignment
2. Reasoning and Chain-of-Thought
3. Coding and Technical Tasks
4. Structured Output and Format Control
5. Multilingual Performance
6. Long-Context Understanding
7. Tool Use and Function Calling
8. Hallucination and Factuality
9. Refusal Behavior and Safety

Chapter 4: Model Weights, Formats, and Storage

1. Safetensors and PyTorch Weights
2. GGUF Format and llama.cpp
3. ONNX and Cross-Platform Export
4. Model Card Information
5. Quantization-Aware Format Choices
6. Storage Requirements and Disk I/O
7. Weight Loading Performance

Chapter 5: Quantization Fundamentals

1. Why Quantize
2. Full-Precision vs. Quantized Representations
3. INT8 and INT4 Quantization
4. Per-Tensor vs. Per-Channel vs. Per-Token
5. Quantization-Aware Training and Post-Training
6. GGUF Quantization Schemes
7. Measuring Quantization Impact
8. When Quantization Fails

Chapter 6: The Local Inference Stack

1. Hardware Abstraction Layers
2. GPU Runtimes and Drivers
3. Memory Management and Allocation
4. Context Switching and Scheduling
5. Process Isolation and Resource Limits
6. Container Considerations
7. The Inference Stack Architecture

Chapter 7: llama.cpp and GGUF Ecosystem

1. llama.cpp Architecture and Design
2. GGUF Model Loading
3. Quantization Selection in llama.cpp
4. Threading and CPU Optimization
5. Metal and GPU Offloading
6. API and Embedding Integration
7. Performance Tuning and Flags
8. Limitations and Gotchas

Chapter 8: Ollama, MLX, and Alternative Runtimes

1. Ollama Architecture and Modelfile
2. Running and Customizing Ollama
3. Apple MLX Framework
4. vLLM for Local Deployment
5. Text Generation Inference
6. Choosing Your Runtime

Chapter 9: Tokenization and Prompt Processing

1. How Tokenization Works
2. Vocabulary Size and Efficiency
3. Tokenizer Choice Matters
4. Measuring Tokenization Performance
5. Special Tokens and Formatting
6. Prompt Templates and System Messages
7. Token Counting and Estimation
8. Multilingual Tokenization Issues

Chapter 10: Latency and Throughput Fundamentals

1. Time-to-First-Token
2. Inter-Token Latency
3. Tokens Per Second
4. Prompt Processing Speed
5. Batch Processing and Parallelism
6. What Drives Each Metric
7. Hardware Bottleneck Identification
8. Measuring Without Contamination

Chapter 11: Memory Behavior and KV Cache

1. What is the KV Cache
2. KV Cache Growth and Context Windows
3. Memory Planning and Budgeting
4. KV Cache Quantization
5. Sliding Windows and Attention Limits
6. Prompt Caching Strategies
7. Eviction and Reuse
8. Practical Memory Monitoring

Chapter 12: Designing Sound Evaluation Methodology

1. Evaluation vs. Benchmarking
2. Task Selection and Workload Definition
3. Representative Test Construction
4. Prompt Design and Variance
5. Deterministic vs. Stochastic Evaluation
6. Temperature and Sampling Parameters
7. Sample Size and Saturation
8. Evaluation Framework Design

Chapter 13: Statistical Rigor and Experimental Design

1. Reproducibility and Seeding
2. Confidence Intervals for Model Comparison
3. Statistical Significance Testing
4. Variance Sources and Control
5. Measurement Error and Noise
6. Warm-up Effects and Caching
7. Normalizing Across Hardware
8. Reporting Standards

Chapter 14: Public Benchmarks and Their Limitations

1. Multiple-Choice Knowledge Benchmarks
2. Math and Reasoning Benchmarks
3. Code Generation Benchmarks
4. Instruction and Chat Benchmarks
5. Benchmark Contamination
6. Overfitting to Benchmarks
7. Aggregate Scores and Their Fallacies
8. When to Trust Public Benchmarks

Chapter 15: Building Custom Evaluation Harnesses

1. Evaluation Harness Architecture
2. Implementing a Basic Harness
3. Streaming and Telemetry Collection
4. Automated Experiment Orchestration
5. Result Storage and Retrieval
6. Structured Output Validation
7. Comparative Analysis and Reporting
8. Visualization and Dashboards

Chapter 16: Measuring Model Quality in Depth

1. Automated Quality Metrics
2. Factuality Verification
3. Hallucination Measurement
4. Instruction Following Assessment
5. Reasoning Quality Evaluation
6. Code Execution Verification
7. Long-Context Fidelity Testing
8. Model-as-Judge Evaluation

Chapter 17: Workload-Specific Evaluation

1. Conversational Assistants
2. Coding and Development Tools
3. Document Analysis and Summarization
4. Retrieval-Augmented Generation Systems
5. Agentic Workflows
6. Edge and Offline Deployments
7. Enterprise and Private Assistants
8. Embedding and Classification Tasks

Chapter 18: Safety, Robustness, and Reliability

1. Safety Evaluation Methodology
2. Adversarial Prompt Testing
3. Jailbreak Resistance
4. Refusal Calibration
5. Consistency and Reliability Under Load
6. Degradation Modes
7. Monitoring for Production Safety
8. Compliance and Policy Checks

Chapter 19: Advanced Optimization Techniques

1. Speculative Decoding
2. Continuous Batching
3. Prompt and KV Cache Compression
4. Compiler Optimizations
5. Distillation and Adapters
6. Model Merging Strategies
7. Hardware-Specific Optimizations
8. Distributed and Multi-Node Inference
9. Techniques for Improving Performance Without Compromising Quality

Chapter 20: Decision Frameworks and Production Deployment

1. Multi-Criteria Decision Frameworks
2. Model Selection by Workload
3. Hardware Procurement Decisions
4. Cost Modeling and TCO
5. Operational Considerations
6. Case Study: Local Coding Assistant
7. Case Study: Private RAG System
8. Case Study: Edge Deployment

Conclusion

References