Leanpub Header

Skip to main content
Go to Leanpub.com

Streaming at Scale

Distributed Stream Processing with Apache Flink

Streaming at Scale
This book is 100% completeLast updated on 2026-09-10
Steve Publications

Build real-time data systems that can keep up with the demands of production. Streaming at Scale takes you under the hood of Apache Flink, covering state, event time, exactly-once processing, performance, Kubernetes and more. With runnable examples and practical lessons, it shows how to build systems that are fast, reliable and ready to scale.

Minimum price

$19.00

$29.00

You pay

Author earns

$

Also available for 1 book credit with a Reader Membership

PDF
EPUB
WEB
APP
185
Pages
About

About

About the Book

This book is a comprehensive, technically rigorous guide to building production-grade real-time data processing systems with Apache Flink. It is written for software engineers, data engineers, platform engineers, and systems architects who need not only to use Flink effectively but to understand what it does underneath the APIs. You will learn how Flink achieves exactly-once processing guarantees through distributed snapshots, how event-time semantics enable deterministic results in the presence of out-of-order data, how stateful operators transform a simple dataflow framework into a platform capable of running complex analytics and fraud detection systems, and how to operate Flink clusters at scale with reliability and observability. The book progresses from foundational distributed systems concepts through Flink's architecture and APIs, then into advanced topics including performance tuning, deployment on Kubernetes, CDC pipelines, security, testing, and real-world architectural patterns. Every chapter includes complete, runnable code examples and practical guidance drawn from production experience.

Share this book

Categories

NetworkingData EngineeringSoftware EngineeringDistributed SystemsStream Processing

Feedback

Email the Author

Author

About the Author

Steve Publications

Steve is a technology professional with more than 20 years of experience in software development, server infrastructure, cybersecurity, vulnerability research and reverse engineering. Throughout his career, he has designed, secured, analyzed and tested complex software and infrastructure, with a particular focus on understanding how systems fail and how they can be made more secure.

Outside of work, Steve enjoys sharing knowledge with the technology community. He collaborates with researchers, industry experts and technology professionals to write practical books covering software development, cybersecurity, cloud computing, networking, DevOps, artificial intelligence and enterprise technologies. His books focus on practical learning through clear explanations, real-world examples and hands-on exercises. With more than two decades of industry experience, his goal is to help IT professionals, students and technology enthusiasts build useful skills and stay current in a rapidly changing industry.

We believe readers deserve to know how our books are created. Most of our authors are not native English speakers, so we use AI to help translate, proofread manuscripts, fix grammar, improve sentence structure and make technical explanations easier to read. AI is used as an editing tool only. It does not replace the research, technical knowledge or hands-on experience behind our books. Some of our authors also prefer to remain anonymous for privacy or professional reasons. In those cases, we publish their work under a different name. The author's name may be different, but the quality of the content and our review process remain the same.

Every book is written, reviewed and maintained by experienced technology professionals, with contributions from our private technical community of more than 420 engineers and researchers. We spend far more time validating technical accuracy and keeping our content up to date than generating text. We are always interested in working with experienced professionals who have deep expertise in a particular technology or domain. If you would like to publish a book with us or help review an existing manuscript, we'd love to hear from you. Send us a message describing your area of expertise. We are especially interested in niche technologies, specialized skills and emerging topics that are underrepresented in existing technical literature.

If you look through the contents of our books, you'll see practical examples, detailed explanations and material that is regularly updated. Our goal is to publish books that professionals can actually rely on, not low-effort AI-generated content. If you ever feel that one of our books does not meet that standard, Leanpub offers a 60-day money-back guarantee. Feel free to request a refund if you are not satisfied with your purchase.

Contents

Table of Contents

Distributed Stream Processing with Apache Flink

Introduction: The Real-Time Imperative

  1. What You Will Learn
  2. Prerequisites
  3. How This Book Is Structured
  4. Version Notes

Chapter 1: The Era of Streaming Data

  1. From Batch to Streaming: Why Latency Matters
  2. The Lambda Architecture and Its Pain Points
  3. From Lambda to Kappa: Unified Processing Models
  4. Core Challenges of Distributed Stream Processing
  5. Where Flink Fits in the Streaming Landscape
  6. Summary

Chapter 2: Stream Processing Fundamentals

  1. Data Streams, Events, and Event Streams
  2. Stream vs Batch: Execution Models and Mindset
  3. Ordering, Latency, and Throughput in Distributed Systems
  4. Partitioning and Shuffling in Stream Processing
  5. Fault Tolerance Models: At-Most-Once, At-Least-Once, Exactly-Once
  6. The Role of Time in Stream Processing
  7. Summary

Chapter 3: Flink Architecture Overview

  1. The Flink Runtime: Components and Responsibilities
  2. Job Submission: From Application Code to Running Job
  3. The Job Graph: Nodes, Edges, and Stream Transformations
  4. From Job Graph to Execution Graph
  5. Task Slots, Parallelism, and Resource Allocation
  6. The Life Cycle of a Flink Job
  7. Summary

Chapter 4: Building Your First Flink Application

  1. Development Environment and Dependencies
  2. Project Structure for a Flink Application
  3. Writing a WordCount Streaming Job
  4. Running Locally and Inspecting Results
  5. Understanding the Generated Job Graph
  6. Packaging and Submitting to a Cluster
  7. Summary

Chapter 5: The DataStream API

  1. Sources and Sinks: The Boundaries of Your Pipeline
  2. Core Transformations: Map, FlatMap, Filter, KeyBy
  3. Reduction and Aggregation Operators
  4. Process Functions and Fine-Grained Control
  5. Operator Chaining and Pipelining
  6. Side Outputs and Branching Streams
  7. Summary

Chapter 6: Time, Timestamps, and Watermarks

  1. Processing Time, Event Time, and Ingestion Time
  2. Why Event Time Matters for Correct Results
  3. Assigning Timestamps to Streams
  4. Watermarks: The Heart of Event-Time Processing
  5. Watermark Strategies and Generators
  6. Handling Out-of-Order Events
  7. Summary

Chapter 7: Windowing

  1. The Concept of Windows
  2. Time Windows: Tumbling, Sliding, and Session
  3. Count and Global Windows
  4. Triggers: Controlling When Windows Fire
  5. Evictors and Window Functions
  6. Late Data and Allowed Lateness
  7. Summary

Chapter 8: State Management Fundamentals

  1. Why State Changes Everything
  2. Keyed State vs Operator State
  3. State Types: Value, List, Map, Reducing, Aggregating
  4. Accessing and Updating State
  5. State Backends: Memory, FileSystem, and RocksDB
  6. Choosing the Right State Strategy
  7. Summary

Chapter 9: Fault Tolerance: Checkpoints and Savepoints

  1. The Distributed Snapshot Problem
  2. Checkpointing in Flink: The Barrier Flow Algorithm
  3. Configuration and Tuning Checkpoints
  4. Fault Recovery and State Restoration
  5. Savepoints: Managed Checkpoints for Operations
  6. Exactly-Once, At-Least-Once, and At-Most-Once Guarantees
  7. Summary

Chapter 10: Kafka Integration

  1. Why Kafka and Flink Are a Natural Pair
  2. The Flink Kafka Consumer
  3. The Flink Kafka Producer
  4. Offset Management and Checkpointing
  5. Partitioning Semantics and Ordering
  6. Kafka Integration Patterns and Best Practices
  7. Summary

Chapter 11: Connectors and External Systems

  1. Connector Architecture and Interfaces
  2. Database Connectors: JDBC, Elasticsearch, Redis
  3. File System Connectors: HDFS, S3, Local
  4. Change Data Capture: Debezium and Flink CDC
  5. Building Custom Sources and Sinks
  6. Choosing Connectors for Your Use Case
  7. Summary

Chapter 12: The Flink Table API and SQL

  1. The Table API: Declarative Stream Processing
  2. Table Environments and Execution Mode
  3. Defining Tables and Schemas
  4. SQL for Streaming: Queries and Semantics
  5. Bridging DataStream and Table APIs
  6. When to Use SQL vs DataStream
  7. Summary

Chapter 13: Stream Joins and Enrichment

  1. Joining Event Streams: The Challenge
  2. Stream-Stream Joins: Interval and Window Joins
  3. Stream-Table Joins: Enriching with Reference Data
  4. Temporal Table Joins and Schema Evolution
  5. Broadcast State for Lookup and Enrichment
  6. Handling Asynchrony and Skewed Data
  7. Summary

Chapter 14: Advanced Data Processing Patterns

  1. Deduplication and Idempotency
  2. Anomaly Detection on Streams
  3. Sessionization and User Activity Tracking
  4. Multi-Dimensional Aggregations
  5. Multi-Stage Pipelines and Topology Design
  6. Real-Time Feature Computation for ML
  7. Summary

Chapter 15: Serialization and Data Formats

  1. Flink’s Type System
  2. Serialization: Kryo, Java, Generic Types
  3. Optimizing Serialization Performance
  4. Avro, Protobuf, and Schema Management
  5. Schema Evolution in Streaming
  6. Binary Formats and Network Efficiency
  7. Summary

Chapter 16: Parallelism, Backpressure, and Scaling

  1. Parallelism Model and Task Allocation
  2. Understanding and Detecting Backpressure
  3. Rescaling Jobs Without Losing State
  4. Data Skew and Its Impact
  5. Slot Sharing and Co-Localization
  6. Elastic Scaling Patterns
  7. Summary

Chapter 17: Memory Management and Resource Tuning

  1. Flink’s Memory Layout
  2. Task, Network, and Managed Memory
  3. Tuning TaskManager and JobManager Resources
  4. JVM Garbage Collection and Flink
  5. Monitoring Memory Pressure
  6. Memory-Related Failure Modes
  7. Summary

Chapter 18: Performance Tuning and Optimization

  1. Latency vs Throughput Trade-Offs
  2. Optimizing State Access and Size
  3. Checkpoint Optimization Strategies
  4. Network Shuffle Optimization
  5. Operator-Level Optimization Techniques
  6. Profiling and Diagnosing Performance Problems
  7. Summary

Chapter 19: Deployment and Operations

  1. Deployment Modes: Standalone, YARN, Kubernetes
  2. Cluster Modes: Session, Application, Per-Job
  3. High Availability Configuration
  4. Resource Management and Cluster Sizing
  5. Configuration Management
  6. Production Deployment Checklist
  7. Summary

Chapter 20: Observability, Monitoring, and Debugging

  1. The Flink Web Dashboard
  2. Built-In Metrics and Custom Metrics
  3. Integrating with Prometheus and Grafana
  4. Logging Strategies for Distributed Jobs
  5. Debugging Techniques and Tools
  6. Incident Response and Troubleshooting
  7. Summary

Chapter 21: Security, Testing, and Reliability

  1. Authentication and Authorization
  2. Encryption in Transit and at Rest
  3. Network Security and Kerberos
  4. Testing Flink Applications
  5. CI/CD Pipelines for Flink Jobs
  6. Production Reliability Patterns
  7. Summary

Chapter 22: Production Architecture Patterns and Anti-Patterns

  1. End-to-End Real-Time Analytics Architecture
  2. Fraud Detection Architecture
  3. CDC-Based Event-Driven Architecture
  4. Multi-Tenant Flink Platforms
  5. Common Anti-Patterns and How to Avoid Them
  6. Version Upgrades and Long-Term Maintenance
  7. Summary

Conclusion: Principles for Streaming at Scale

References

Get the free sample chapters

Click the buttons to get the free sample in PDF or EPUB, or read the sample online here

Download Sample PDFDownload Sample EPUB

The Leanpub 60 Day 100% Happiness Guarantee

Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.

See full terms...

Earn $8 on a $10 Purchase, and $16 on a $20 Purchase

We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.

(Yes, some authors have already earned much more than that on Leanpub.)

In fact, authors have earned over $15 million writing, publishing and selling on Leanpub.

Learn more about writing on Leanpub

Free Updates. DRM Free.

If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).

Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.

Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.

Learn more about Leanpub's ebook formats and where to read them

Write and Publish on Leanpub

You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!

Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.

Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.

Learn more about writing on Leanpub