High Availability, VRRP, Failover Architecture, Health Checking and Production Operations
Introduction
- What this book covers
- Prerequisites and scope
- Conventions and naming
Chapter 1: High Availability Foundations
- The fundamental problem: the single address
- What high availability actually means
- The VIP pattern
- Where Keepalived fits
- The shared-nothing constraint
- Understanding failure domains in HA pairs
- The role of time in high availability
- Summary
Chapter 2: Linux Networking Prerequisites
- Interface addressing: primary, secondary, labels
- The iproute2 suite
- Routing tables and policy routing
- ARP: the mechanism that makes VIPs work
- IPv6 and Neighbor Discovery
- Interface bonding and HA
- VLANs and virtual interfaces
- Network namespaces
- Sysctl parameters that affect HA behavior
- Multicast and IGMP
- Summary
Chapter 3: VRRP Protocol Deep Dive
- Protocol history and RFC evolution
- The virtual router abstraction
- Master and Backup roles
- Priority and election
- Advertisements and heartbeat
- Master_Down_Timer and failure detection
- Preemption
- State machine and transitions
- Unicast mode
- Virtual MAC addresses
- Authentication and security
- Accept mode and packet reception
- Timer interactions and convergence
- Summary
Chapter 4: Keepalived Architecture and Design
- Project origins and evolution
- The process model
- Configuration parsing
- I/O multiplexing and event handling
- Netlink and kernel communication
- Signal handling
- Memory management
- Threading model and concurrency
- Interaction with iproute2 and other tools
- Debugging support
- Summary
Chapter 5: Installation and Deployment
- Version awareness
- Package installation on Debian and Ubuntu
- Package installation on RHEL, AlmaLinux, Rocky Linux
- Package installation on SUSE/openSUSE
- Package installation on Alpine Linux
- Building from source
- Service management with systemd
- SELinux and AppArmor considerations
- Initial configuration validation
- Upgrade procedures
- Summary
Chapter 6: Configuration Fundamentals
- Configuration file location and structure
- The global_defs block
- Including external files
- Configuration validation
- Reloading vs restarting
- Common configuration mistakes
- Best practices for maintainable configurations
- Summary
Chapter 7: VRRP Instances: Core Configuration
- The minimal vrrp_instance
- state: MASTER or BACKUP
- interface: binding to a network interface
- virtual_router_id: the group identifier
- priority: determining the master
- advert_int: advertisement interval
- nopreempt vs preempt
- Authentication
- Gratuitous ARP configuration
- accept and no_accept
- version: VRRP protocol version
- A complete two-node example
- Summary
Chapter 8: Virtual IP Addresses, Routes and Rules
- virtual_ipaddress: basic VIP configuration
- Multiple VIPs
- virtual_ipaddress_excluded: the excluded pool
- virtual_ipaddress_dev: binding VIPs to specific interfaces
- Virtual routes
- Virtual rules
- How Keepalived programs the kernel
- IPv6 virtual addresses
- IP conflict behavior
- Common mistakes
- Summary
Chapter 9: Interface Tracking and Weighted Priorities
- How interface tracking works
- Weight semantics
- Multiple tracked interfaces
- Cumulative weight math
- Interface tracking vs script tracking
- Weight reverse
- When NOT to use interface tracking
- Multi-homed server example
- Bonded interface tracking
- Synchronization group tracking
- Summary
Chapter 10: Health Checking: Scripts and Process Monitoring
- vrrp_script: custom health checks
- Designing effective health checks
- Script example: checking a web server
- Script example: checking HAProxy
- Script permissions and security
- Script execution performance
- Script security: injection and environment
- vrrp_track_process: process monitoring
- Combining process and script tracking
- Fall, rise and stability
- Summary
Chapter 11: Notifications, Logging and State Changes
- Notification scripts: when they fire
- The notify catch-all
- Notification scripts in sync groups
- Logging with syslog
- Debug levels
- Structured JSON state dump
- Log message taxonomy
- Designing notification scripts
- Integration with monitoring systems
- Summary
Chapter 12: VRRP Synchronization and Grouping
- What synchronization groups do
- Configuring a sync group
- How synchronization works
- When to use synchronization groups
- Sync group notifications
- Weighted tracking in sync groups
- The global_tracking flag
- Example: load balancer pair
- Summary
Chapter 13: Multi-VIP and Active/Active Architectures
- True active/passive vs active/active
- Multiple VRRP instances on two nodes
- Service affinity with multiple VIPs
- Scaling: many VIPs across two nodes
- DNS round-robin with Keepalived
- Synchronization in active/active
- Weighted priorities for graceful degradation
- How many VIPs is too many?
- Summary
Chapter 14: Real-World HA Architectures: Common Patterns
- Pattern 1: simple two-node active/standby web server
- Pattern 2: highly available reverse proxy
- Pattern 3: highly available load balancer (HAProxy pair)
- Pattern 4: highly available API gateway
- Pattern 5: database connection router
- Common requirements across patterns
- Summary
Chapter 15: Advanced Architectures and Complex Environments
- Multi-tier architectures
- VLANs and Keepalived
- Bonded interfaces and Keepalived
- Virtual machines
- Network namespaces
- Containers and Keepalived
- Public cloud considerations
- Summary
Chapter 16: Failure Modes, Pathologies and Split-Brain
- Split brain: dual master
- Asymmetric connectivity
- Priority misconfiguration
- Preemption problems
- Health check false positives and false negatives
- Interface flapping
- Stale ARP and neighbor information
- Duplicate VIPs from misconfiguration
- Firewall blocking VRRP traffic
- Script execution failures
- Routing inconsistencies after failover
- Virtualization-related issues
- Cloud networking limitations
- Summary
Chapter 17: Troubleshooting Methodology and Diagnosis
- The troubleshooting mindset
- Step 1: check Keepalived process status
- Step 2: check VRRP state
- Step 3: check VRRP advertisements with tcpdump
- Step 4: check ARP and neighbor state
- Step 5: check health check scripts
- Step 6: check interface state
- Step 7: check routing
- Step 8: check firewall rules
- Step 9: check for split brain
- Step 10: check configuration syntax
- Common problems and their diagnostic signatures
- Distinguishing Keepalived issues from network or application issues
- Summary
Chapter 18: Observability, Monitoring and Alerting
- Key metrics to monitor
- Using keepalived_exporter with Prometheus
- Alerting on VRRP state changes
- Monitoring VIP reachability
- Log-based monitoring
- Grafana dashboards
- Alerting philosophy
- Integration with incident response
- Summary
Chapter 19: Security Hardening
- VRRP protocol security
- Script execution security
- Shell script injection risks
- File permissions
- SELinux and AppArmor
- Firewall configuration
- Network segmentation
- Monitoring for anomalies
- Secure upgrade procedures
- Summary
Chapter 20: Performance, Scalability and Operational Limits
- Resource usage
- Many VIPs
- Many VRRP instances
- Health check overhead
- Network traffic
- Failover convergence time
- System-level limits
- Bottlenecks
- Empirical guidance
- Summary
Chapter 21: Production Operations: Runbooks, Upgrades, Maintenance
- Configuration validation before changes
- Controlled failover for maintenance
- Upgrading Keepalived
- Changing VIPs
- Adding or removing nodes
- Configuration drift detection
- Backup and recovery
- Post-incident review practices
- Summary
Chapter 22: Production Reference
- Quick command reference
- Minimal two-node configuration template
- Health check script template
- Notification script template
- Firewall rules template
- Default values
- Troubleshooting decision tree
- Controlled failover procedure
- Security checklist
- Version compatibility notes
- Useful log patterns