Message Queues and Distributed Transactions

Architecture, Implementation, Reliability and Production Operations

Introduction

1. What this book covers
2. Technology choices
3. Prerequisites
4. How to read this book
5. The reliability mindset

Chapter 1: The Problem with Synchronous Communication

1. The synchronous trap: tight coupling and cascading failures
2. Latency, throughput and resource exhaustion under load
3. The network is not a memory bus: partitions, timeouts and partial failures
4. Decoupling time, space and synchronization with messages
5. When synchronous communication is still the right choice

Chapter 1 Summary

Chapter 2: Core Messaging Concepts

1. Producers, consumers, brokers and message brokers versus message buses
2. Topics, queues, partitions and exchange patterns
3. Point-to-point versus publish-subscribe semantics
4. Message anatomy: headers, keys, bodies and metadata
5. Acknowledgments, offsets and delivery semantics

Chapter 2 Summary

Chapter 3: Delivery Guarantees and Their Meaning

1. At-most-once: semantics, use cases and failure modes
2. At-least-once: duplicate delivery, retries and crash windows
3. Effectively-once processing: idempotency, deduplication and application guarantees
4. The broker-consumer-database coordination problem
5. Why true end-to-end exactly-once is a myth in practice

Chapter 3 Summary

Chapter 4: The CAP Theorem, PACELC and Messaging Trade-offs

1. CAP revisited: consistency, availability, partition tolerance in messaging
2. PACELC: understanding trade-offs during normal and failure conditions
3. How messaging systems position themselves on the spectrum
4. Throughput, latency, durability and operational complexity
5. Choosing consistency levels for different business needs

Chapter 4 Summary

Chapter 5: Apache Kafka

1. The log, partitions, replication and the controller
2. Producers: batching, compression, acknowledgments and ordering
3. Consumers, consumer groups, offsets and rebalancing
4. Kafka transactions: write-ahead logs, transactional IDs and exactly-once semantics
5. Limits: retention, replay constraints, latency tail behavior and operational overhead

Chapter 5 Summary

Chapter 6: RabbitMQ

1. Exchanges, queues, bindings and routing keys
2. Message durability, confirmations and negative acknowledgments
3. Clustering, mirrored queues and quorum queues
4. The transaction API, publisher confirms and reliability trade-offs
5. When to choose RabbitMQ and when not to

Chapter 6 Summary

Chapter 7: NATS and JetStream

1. The NATS core protocol and supercluster architecture
2. Subjects, wildcards and request-reply patterns
3. JetStream: stream storage, consumers and durable subscriptions
4. Deliveries guarantees, ack policies and message retention

Chapter 7 Summary

Chapter 8: Redis Streams, Amazon SQS/SNS, Apache Pulsar and Database-Backed Queues

1. Redis Streams: lightweight persistence with Redis familiarity
2. Amazon SQS and SNS: managed, simple and opinionated
3. Apache Pulsar: unified messaging and streaming with tiered storage
4. Relational-database-backed queues: polling, locking and simplicity trade-offs
5. Comparative summary: protocol, delivery, ordering, durability and ops complexity

Chapter 8 Summary

Chapter 9: Technology Selection Framework

1. Defining selection criteria: workload patterns, data volume, latency SLAs and team skills
2. Scenario analysis: high-throughput event streaming, reliable task queues, pub-sub eventing
3. Stack recommendations: Kafka-first, RabbitMQ-first, managed-cloud-first, lightweight
4. Multi-broker architectures and interoperability challenges
5. Migration considerations and technology debt

Chapter 9 Summary

Chapter 10: The Reference Application

1. Business domain: orders, inventory, payments, shipping and notifications
2. Initial synchronous architecture and its failure modes
3. Target microservice decomposition
4. Data models, APIs and consistency requirements
5. Technology stack decision: choosing the reference stack

Chapter 10 Summary

Chapter 11: Producers and Consumer Design

1. Producer configuration: retries, acknowledgments, batching and compression
2. Consumer design: processing loops, batching and graceful shutdown
3. Serialization choices: JSON, Protobuf and schema registries
4. Error handling: transient versus permanent failures
5. Connection management and resource limits

Chapter 11 Summary

Chapter 12: Message Ordering and Partitioning

1. Ordering guarantees within partitions
2. Partition-key design for the order-processing domain
3. Hot partitions and load imbalance
4. Ordering across partitions and aggregates
5. Trade-offs between ordering, parallelism and latency

Chapter 12 Summary

Chapter 13: Retries, Dead-Letter Queues and Poison Messages

1. Retry strategies: exponential backoff, jitter and circuit breakers
2. Dead-letter queues: routing, inspection and replay
3. Poison messages: detection, isolation and remediation
4. Retry limits and escalation
5. Observability for failure patterns

Chapter 13 Summary

Chapter 14: Idempotency and Deduplication

1. Idempotent operations in the order-processing domain
2. Idempotency keys and deduplication tables
3. Set-based versus record-based deduplication
4. Handling idempotency with distributed locks
5. Idempotency as a design principle, not a fix

Chapter 14 Summary

Chapter 15: Flow Control, Backpressure and Batching

1. Backpressure: signals, pauses and graceful degradation
2. Consumer rate limiting and prefetch tuning
3. Batching: producer batching and consumer batching
4. Memory management and resource exhaustion
5. Coordinated flow control across services

Chapter 15 Summary

Chapter 16: ACID and Local Transactions

1. ACID properties: atomicity, consistency, isolation, durability
2. Isolation levels: read uncommitted through serializable
3. Concurrency control: locking versus MVCC
4. Transaction isolation anomalies
5. When local transactions are sufficient

Chapter 16 Summary

Chapter 17: The Saga Pattern

1. The distributed transaction problem across services
2. Choreographed sagas: event-driven compensating transactions
3. Orchestrated sagas: central coordinator and state machines
4. Saga design for the order-processing reference application
5. Choosing orchestration versus choreography

Chapter 17 Summary

Chapter 18: Two-Phase Commit and XA

1. Two-phase commit: prepare and commit phases
2. Transaction coordinators and participant failures
3. Three-phase commit: reducing blocking
4. XA specification and database support
5. Why two-phase commit is rarely the right choice in microservices

Chapter 18 Summary

Chapter 19: The Transactional Outbox Pattern

1. The crash window between commit and publish
2. Outbox table design and transactional writes
3. Publishing from the outbox: polling versus change data capture
4. Idempotent event handling and consumer guarantees

Chapter 19 Summary

Chapter 20: Change Data Capture and Event Sourcing

1. Change data capture: WAL readers, Debezium and log-based CDC
2. CDC pipeline architecture and latency
3. Event sourcing: events as the source of truth
4. Combining CDC with the outbox pattern
5. Trade-offs: complexity, auditability and debugging

Chapter 20 Summary

Chapter 21: Distributed Transaction Failure Modes

1. Crash windows: commit without publish, publish without commit
2. Duplicate delivery and compensating transaction idempotency
3. Concurrent processing and race conditions
4. Network partitions and split-brain scenarios
5. Recovery procedures and state reconciliation

Chapter 21 Summary

Chapter 22: Deployment and Infrastructure

1. Docker Compose for local development
2. Kubernetes deployment: StatefulSets, storage and networking
3. Resource planning: CPU, memory, disk IOPS and networking
4. Infrastructure as code and environment parity
5. Multi-region deployment and data locality

Chapter 22 Summary

Chapter 23: Security and Access Control

1. Authentication: SASL, TLS and cloud IAM integration
2. Authorization: topic-level and queue-level access control
3. Encryption: in transit and at rest
4. Secrets management and credential rotation
5. Least privilege and audit logging

Chapter 23 Summary

Chapter 24: Observability: Logging, Metrics and Tracing

1. Structured logging with correlation IDs
2. Metrics: throughput, latency, queue depth and error rates
3. Distributed tracing across services and message hops
4. Dashboards and alerting: SLOs and meaningful thresholds
5. Debugging message flow and dead-letter analysis

Chapter 24 Summary

Chapter 25: Reliability, Scaling and Disaster Recovery

1. High availability: replication, failover and rolling upgrades
2. Horizontal scaling: brokers, partitions and consumers
3. Capacity planning and load testing methodology
4. Backup, restore and disaster recovery procedures
5. Incident response and postmortem practices

Chapter 25 Summary

Chapter 26: Anti-Patterns, Migration and Evolution

1. Anti-patterns: chatty services, message bloating and hidden synchronous calls
2. Schema evolution and backward compatibility
3. Migrating between messaging technologies
4. Breaking change management in event-driven systems
5. When to abandon messaging and simplify

Chapter 26 Summary

Conclusion: The Reliability Mindset

1. Summary of core principles
2. Choosing simplicity where possible, sophistication where necessary
3. Continuous learning and staying current
4. Final recommendations for practice

References