Engineering Distributed Systems on the BEAM

Building Reliable, Scalable Applications with Erlang/OTP and the Ecosystem

Introduction

1. The Promise of This Book
2. What Makes the BEAM Different
3. The Reference Application: Nexus
4. How to Read This Book
5. Prerequisites and Tooling
6. A Note on Guarantees

Chapter 1: The BEAM Execution Model

1. Virtual Machine Architecture and Schedulers
2. Lightweight Processes: Creation, Cost, and Lifecycle
3. Mailboxes and Message Passing Semantics
4. Immutability and Concurrency Safety
5. Garbage Collection: Generational, Concurrent, and Stop-the-World
6. Process Isolation as the Foundation of Fault Tolerance
7. Summary

Chapter 2: OTP Design Principles and Core Behaviours

1. The OTP Philosophy: Let It Crash and Why It Works
2. GenServer: Stateful Processes with Structured Interfaces
3. Supervisors: Trees, Strategies, and Recovery Semantics
4. Applications as Composable Units of Deployment
5. Task Behaviours: OneShot, Sync, and Async Patterns
6. Designing Process Hierarchies for a Real Application
7. Summary

Chapter 3: Building the Reference Application — Architecture Foundations

1. Requirements: Throughput, Latency, Availability, Consistency Goals
2. Domain Decomposition: Services, Bounded Contexts, Process Ownership
3. Supervision Topology Design for the Order System
4. Message Protocol Design: Requests, Responses, Events, Timeouts
5. State Ownership and Data Flow Architecture
6. Initial Implementation: Local Single-Node Version
7. Summary

Chapter 4: BEAM Distribution Fundamentals

1. Nodes and Naming: Short Names, Long Names, and DNS
2. The Distribution Protocol: epmd, Handshake, and Connection Management
3. Cookies, Authentication, and Node Security
4. Remote Process Calls and Distributed Registries
5. Links and Monitors Across Nodes
6. When BEAM Distribution Is (and Isn’t) the Right Tool
7. Summary

Chapter 5: Clustering and Service Discovery

1. Static vs Dynamic Clusters: Trade-offs and Use Cases
2. libcluster Strategies: DNS, Kubernetes, Consul, Etcd Integration
3. Horde: Distributed ETS, Registry, and Application Controller
4. Running BEAM on Kubernetes: Pods, Services, and Node Discovery
5. Cluster Membership Changes: Joins, Leaves, Partitions
6. Security in Dynamic Clusters: TLS, mTLS, and Cookie Management
7. Summary

Chapter 6: State Management, Replication, and Partitioning

1. State Placement: In-Memory vs Persistent vs External Stores
2. ETS and DETS: High-Performance Local Storage
3. Mnesia: Distributed Database on BEAM — Capabilities and Limits
4. Replication Strategies: Primary-Backup, Multi-Master, Quorum-Based
5. Partitioning and Sharding: Keys, Ranges, and Hot Spots
6. Choosing Between BEAM-Native and External Databases
7. Summary

Chapter 7: Consistency Models and Distributed Transactions

1. The CAP Theorem in Practice: What It Actually Means for Your System
2. Consistency Models: Strong, Sequential, Causal, Eventual
3. Distributed Transactions on BEAM: Mnesia DTC and Alternatives
4. Saga Patterns and Compensating Actions for Long-Running Workflows
5. Idempotency Design: Keys, Tokens, and Deduplication
6. Retries, Timeouts, and Circuit Breakers in Concurrent Systems
7. Summary

Chapter 8: Fault Detection, Recovery, and Network Partitions

1. Failure Detection: Monitors, Heartbeats, and Timeouts
2. Handling Node Crashes and Process Restarts in a Cluster
3. Network Partitions: Detection, Diagnosis, and Recovery Strategies
4. Split-Brain Prevention: Quorums, Leaders, and Tiebreakers
5. Graceful Degradation: Read-Only Modes, Queuing, and Fallbacks
6. Designing for Partial Outages: Dependency Failure Handling
7. Summary

Chapter 9: Event-Driven Architectures and Message Flows

1. Event Sourcing vs Event-Carried State Transfer: Choosing Wisely
2. Pub/Sub on BEAM: :rpc, GenEvent, Phoenix PubSub, and Beyond
3. Queues and Workers: Oban, Broadway, and Custom Implementations
4. Backpressure and Flow Control: Preventing Overwhelm
5. RabbitMQ and External Message Brokers in BEAM Systems
6. Event Ordering Guarantees and Their Costs
7. Summary

Chapter 10: Testing Distributed Systems

1. Unit Testing OTP Behaviours: Isolating Processes and Time
2. Integration Testing Across Nodes: Spin-Up, Teardown, Cleanup
3. Simulating Failures: Crashes, Delays, Partitions, and Slow Links
4. Property-Based Testing for Concurrent and Distributed Code
5. Chaos Engineering on BEAM: Tools and Practices
6. Testing Idempotency, Retries, and Recovery Paths
7. Summary

Chapter 11: Observability, Profiling, and Performance Tuning

1. Structured Logging in Distributed BEAM Applications
2. Metrics: Counters, Gauges, Histograms, and Cardinality
3. Distributed Tracing with OpenTelemetry on BEAM
4. Profiling: Scheduler Utilization, Reductions, and Hot Paths
5. Memory Analysis: Mailbox Growth, Process Spawns, and Leaks
6. GC Tuning and Performance Optimization Techniques
7. Summary

Chapter 12: Deployment, Releases, and Operations

1. Building Production Releases: Mix Release, Rebar3, and Docker
2. Configuration Management: Environments, Overrides, and Hot Reloading
3. Secrets Management: Vault, KMS, and Runtime Injection
4. TLS Termination and Node-to-Node Encryption
5. Rolling Upgrades and Zero-Downtime Deployments
6. Backups, Disaster Recovery, and Multi-Region Strategies
7. Summary

Chapter 13: BEAM in the Modern Ecosystem — Comparisons and Trade-offs

1. BEAM vs Microservices: Monolithic Clusters vs Distributed Services
2. BEAM vs Actor Systems: Akka, Orleans, and Other Frameworks
3. BEAM in Kubernetes: Fit, Friction, and Best Practices
4. When to Choose BEAM: Strengths, Weaknesses, and Honest Trade-offs
5. Hybrid Architectures: BEAM as One Component of a Larger System
6. Architecture Decision Checklist for Distributed Systems
7. Summary

Conclusion

References