Spark 4.0 from Scratch

The second volume of Spark 4.0 from Scratch turns a competent PySpark user into a production Spark engineer. Eleven chapters covering everything that sits between "the job runs" and "the on-call team trusts the job."

Key Features

• Structured Streaming, end to end: sources, sinks, watermarks, stateful processing, stream-stream joins, exactly-once with checkpointing, and the monitoring surface that catches a stuck stream before the dashboard does.
• MLlib without hand-waving: Pipelines, feature engineering, classification, regression, clustering, hyper-parameter tuning, model saving, a complete model lifecycle on real data.
• GraphFrames: vertices, edges, motif finding, PageRank, connected components, triangle counting, and the fraud-detection patterns the payment-processing crowd actually uses.
• Performance tuning as a diagnostic discipline: the Spark UI tabs, explain() reading, partition sizing, AQE, dynamic partition pruning, manual skew salting, the Z-order story.
• Testing and CI/CD for Spark: pytest fixtures, GitHub Actions pipelines, code-quality linting, deployment patterns including canary and blue/green.
• The Lakehouse, in depth: Delta Lake (time travel, OPTIMIZE, ZORDER, VACUUM, Change Data Feed), Apache Iceberg (hidden partitioning, partition evolution, REST catalog), the trade-off table, and modern table modelling.

What you will learn

- Process unbounded streams with watermarks, stateful operators, and stream-stream joins

- Train and evaluate ML pipelines at scale with MLlib's Estimator/Transformer API

- Use GraphFrames for PageRank, connected components, motifs, and fraud patterns

- Diagnose any slow Spark job from the UI and explain() output

- Write unit and integration tests for PySpark code that actually catch bugs

- Deploy Spark applications to Kubernetes, YARN, or standalone clusters

- Design Delta Lake and Iceberg tables for production lakehouses

- Implement medallion architecture with bronze/silver/gold layers and SCDs

Who this book is for

Data engineers and data scientists who finished Volume 1 (or already write PySpark professionally) and now need streaming, ML, performance discipline, and production deployment skills. Some chapters touch ML and graph theory lightly, no prior experience is required, but a hunger to ship matters.

Table of Contents

• Window Functions. Top-N per group, running totals, moving averages, gap-and-island, session detection from raw event logs. The analytics that groupBy cannot express.
• User-Defined Functions (UDFs). Python UDFs, Pandas UDFs (the 3-100x speedup), grouped map for per-group model fitting, and the rule for when to avoid UDFs entirely.
• Structured Streaming. Sources, sinks, watermarks, stateful processing, stream-stream joins, exactly-once with checkpointing, monitoring stuck queries before the dashboard does.
• Machine Learning with MLlib. Pipelines, feature engineering, classification, regression, clustering, hyperparameter tuning, and a complete model lifecycle on real data.
• Graph Processing with GraphFrames. Vertices, edges, motif finding, PageRank, connected components, triangle counting, and the fraud-detection patterns the payments crowd uses.
• Performance Tuning. Spark UI tabs read like an X-ray, query plan reading, partition sizing, AQE, dynamic partition pruning, manual salting for skew, Z-ordering for data skipping.
• Deploying Spark Applications. spark-submit, packaging, client vs cluster mode, the resource-calculation formula, dynamic allocation, Kubernetes, log configuration that doesn't drown you.
• End-to-End Data Pipeline Project. Bronze/silver/gold with SCDs, a reusable data-quality framework, incremental processing, the full e-commerce capstone wired together.
• Testing and CI/CD for Spark. pytest fixtures that survive a session-scoped SparkSession, chispa for DataFrame equality, GitHub Actions pipelines, blue/green and canary deployment.
• The Lakehouse: Delta Lake, Iceberg, and Modern Table Design. Time travel, OPTIMIZE, ZORDER, partition evolution, schema evolution, and the medallion patterns that make data trustworthy.