Linux Performance Engineering & Server Benchmarking

From Kernel Internals to Production Optimization

Introduction: The Discipline of Performance Engineering

1. Performance Is a System Property, Not a Configuration
2. Latency, Throughput, Utilization, Saturation, Availability
3. The Measurement Imperative
4. Common Pitfalls and How to Avoid Them

Chapter 1: The Performance Engineering Methodology

1. Characterizing the Workload
2. Defining Success: SLOs, SLIs, and Performance Targets
3. Baseline Creation and Reference Measurements
4. Experimental Design: Variables, Controls, and Reproducibility
5. Statistical Rigor: Variance, Outliers, and Significance
6. The Diagnosis Loop: From Symptom to Root Cause

Chapter 2: Linux Kernel Architecture and Performance

1. Kernel Space and User Space: The Boundary Cost
2. System Calls: Entry, Exit, and Overhead
3. Processes, Threads, and the Task Structure
4. Context Switching: Mechanics and Cost
5. Interrupts, Softirqs, and Bottom Halves
6. Synchronization Primitives: Locks, Spinlocks, and Contention

Chapter 3: CPU Architecture and Performance Counters

1. Modern CPU Microarchitecture: Pipelines and Parallelism
2. Cache Hierarchy: L1, L2, L3, TLB, and Cache Misses
3. Branch Prediction and Speculative Execution
4. Cores, Hyperthreading, and SMT
5. CPU Frequency Scaling and Power States
6. Performance Counters and Hardware Events

Chapter 4: CPU Scheduling and Process Placement

1. The Completely Fair Scheduler: Internal Mechanics
2. Virtual Runtime, Timeslices, and Fairness
3. Real-Time Scheduling: SCHED_FIFO, SCHED_RR, SCHED_DEADLINE
4. CPU Affinity, Isolation, and Task Migration
5. Load Balancing Across NUMA Nodes

Chapter 5: Memory Management: Virtual Memory and Pages

1. Virtual Memory: Address Spaces and Page Tables
2. Page Faults: Minor and Major Faults
3. Physical Memory: Allocation and Management
4. Swapping: Mechanics, Costs, and Modern Behavior
5. Huge Pages and Transparent Huge Pages
6. Memory Accounting: RSS, PSS, and Kernel Metrics

Chapter 6: NUMA Architecture and Memory Placement

1. What NUMA Means for Performance
2. NUMA Nodes, Local vs Remote Memory Access
3. NUMA-Aware Allocation and Policies
4. Diagnosing NUMA Imbalance
5. Optimizing for NUMA: Placement and Binding
6. NUMA in Virtualized and Cloud Environments

Chapter 7: The Page Cache and Memory-Mapped Files

1. Page Cache: Architecture and Behavior
2. Read-Ahead and Write-Behind
3. Memory-Mapped Files and mmap
4. Cache Pressure and Eviction Policies
5. Dirty Page Management and Writeback

Chapter 8: I/O Subsystems: From Block Layer to Disk

1. The I/O Path: From open() to the Disk
2. The Block Layer and Request Queues
3. I/O Schedulers: noop, deadline, cfq, mq-deadline, kyber, bfq
4. Block Devices and Queueing Models
5. I/O Merge, Splitting, and Bio Structures
6. I/O Completion and Interrupt Handling

Chapter 9: Storage Technologies and Performance Characteristics

1. HDDs: Seek Latency, Rotational Latency, and Throughput
2. SSDs: NAND Characteristics, Wear Leveling, and GC
3. NVMe: Protocol Advantages and Queue Architecture
4. RAID and Hardware Controllers
5. Cloud Block Storage and Shared Storage
6. Choosing Storage for the Workload

Chapter 10: Filesystems and Mount Options

1. ext4: Journaling, Allocation, and Performance
2. XFS: Design Principles and Scalability
3. btrfs and Copy-on-Write Semantics
4. Filesystem Journaling and Write Performance
5. Mount Options: Performance Implications
6. Filesystem-Level Caching and Coherency

Chapter 11: The Networking Stack and Performance

1. The Networking Data Path
2. Socket Buffering and Memory
3. NIC Interrupts and Softirq Processing
4. RSS, RPS, XPS: Distributing Network Load
5. TCP Congestion Control and Buffer Tuning
6. UDP and Raw Sockets Performance

Chapter 12: Network Performance Tuning in Practice

1. Core TCP Parameters and Their Effects
2. Socket Buffer Sizing Strategy
3. Congestion Control Algorithm Selection
4. Multiqueue Configuration and RSS Tuning
5. Validating Network Performance Changes

Chapter 13: System Call Tracing and Process Analysis

1. strace and ltrace: Mechanics and Use Cases
2. perf: Architecture and Profiling Modes
3. ftrace: Kernel Tracing Infrastructure
4. Tracepoints and Dynamic Probes
5. SystemTap: Overview and Comparison
6. Building a Tracing Strategy

Chapter 14: eBPF and BPF-Based Observability

1. eBPF Architecture and Safety Model
2. BPF Map Types and Communication
3. bpftrace: High-Level Tracing
4. BCC Tools: Key Utilities
5. Writing Custom BPF Programs
6. eBPF Overhead and Production Safety

Chapter 15: Performance Analysis Tools and Flame Graphs

1. Flame Graphs: Construction and Interpretation
2. perf Record and Report
3. Hotspot Analysis and Call Path Tracing
4. Combining Tools: Integrated Workflows
5. Sampling vs Tracing: Trade-offs
6. Automated Analysis Pipelines

Chapter 16: System Monitoring and Observability

1. Classic Tools: sar, vmstat, iostat, mpstat, pidstat
2. Modern Monitoring: top, htop, numastat
3. Network Monitoring: ss, ethtool, nstat
4. Pressure Stall Information and Modern Metrics
5. Building Production Dashboards and Alerting

Chapter 17: Benchmarking: Design and Execution

1. What Makes a Valid Benchmark
2. Designing Representative Workloads
3. Test Environment Isolation and Control
4. Warm-Up, Steady State, and Cooldown
5. Repetition, Reproducibility, and Statistical Analysis
6. Benchmarking Mistakes: Gaming, Noise, and Bias

Chapter 18: Benchmarking Tools and Suites

1. fio: Storage Benchmarking Deep Dive
2. stress-ng: Comprehensive System Stress Testing
3. sysbench: CPU, Memory, File I/O, Database Tests
4. iperf3: Network Throughput and Latency
5. lmbench: Latency and Bandwidth Microbenchmarks
6. Application-Level: wrk, ApacheBench, curl
7. Phoronix Test Suite and Comparative Testing

Chapter 19: CPU and Memory Tuning for Production

1. CPU Governor Selection and Configuration
2. IRQ Affinity and NUMA-Optimal Placement
3. CPU Isolation for Critical Workloads
4. Huge Pages: Configuring and Validating
5. Memory Sysctl Parameters
6. Validating CPU and Memory Tuning

Chapter 20: I/O and Filesystem Tuning for Production

1. I/O Scheduler Selection by Workload
2. Filesystem Mount Options for Performance
3. Writeback and Dirty Page Tuning
4. Device-Level Parameters: Queue Depth and Read-Ahead
5. SSD and NVMe-Specific Tuning
6. Validating I/O Performance Improvements

Chapter 21: cgroups, systemd, and Resource Management

1. cgroups v1 vs v2: Architecture Differences
2. CPU Controllers and Throttling
3. Memory Controllers: Limits and Pressure
4. I/O Controllers: Throttling and Prioritization
5. systemd Resource Controls and Service Isolation
6. ulimits and Resource Limits

Chapter 22: Containers, Virtualization, and Noisy Neighbors

1. Container Performance: Isolation and Overhead
2. cgroups and Namespaces in Containers
3. Virtualization Overhead: Hypervisors and Emulation
4. Detecting and Mitigating Noisy Neighbors
5. Cloud vs Bare Metal: Performance Implications
6. Best Practices for Containerized Workloads

Chapter 23: Databases and High-Performance Applications

1. How Databases Use Linux: I/O and Memory Patterns
2. Shared Memory and IPC for Databases
3. Locking and Contention in Database Workloads
4. NUMA and Database Placement
5. Application Runtimes: JVM, Go, Python Considerations
6. Tail Latency Optimization

Chapter 24: Distributed Systems and Cluster Performance

1. Distributed Latency: Network and System Contributions
2. Clock Synchronization and Time Skew
3. Network Partitions and Performance
4. Consensus Protocols and Performance Costs
5. Cluster Resource Management
6. Observability at Scale

Chapter 25: Production Case Studies

1. Case Study: CPU Saturation from Interrupt Storms
2. Case Study: Memory Pressure and Swapping Crisis
3. Case Study: NUMA Imbalance Causing 5x Latency
4. Case Study: I/O Bottleneck on Shared Storage
5. Case Study: Noisy Neighbor in Multi-Tenant Cloud

Chapter 26: Capacity Planning and Performance Regression

1. Capacity Modeling and Forecasting
2. Baseline Management and Versioning
3. Regression Detection: Automated and Manual
4. Load Testing and Stress Testing Strategies
5. Performance Budgets and Guardrails
6. Continuous Performance Validation

Chapter 27: Advanced Topics and Emerging Technologies

1. Real-Time Kernels and PREEMPT_RT
2. Kernel Bypass: DPDK, XDP, and AF_XDP
3. x86-64 vs ARM64: Performance Differences
4. RDMA and Low-Latency Networking
5. eBPF in Production: Advanced Patterns
6. Future Directions: Kernel and Hardware Trends

Conclusion: Putting It All Together

1. The Engineer’s Mindset for Performance
2. When Not to Optimize
3. Building a Performance Culture

References