[{"_1":2,"_3":-5,"_4":-5},"loaderData",{"_5":6,"_7":8},"actionData","errors","root",{"_28":-5,"_38":-5,"_39":40,"_41":42,"_43":44,"_45":46,"_47":48,"_49":50,"_51":52,"_53":54,"_55":56,"_57":58},"routes/_store+/_readers+/email_author.$slug/route",{"_9":10,"_11":-5},"book",{"_12":13,"_14":15,"_16":17,"_18":19,"_20":21,"_22":23},"currentUser","id","Z2lkOi8vbGVhbnB1Yi9TdG9yZTo6Qm9va1Byb2R1Y3QvNzk2MjE","title","Spark 4.0 from Scratch","state","published","contactAuthorMethod","email","aboutText","

The second volume of Spark 4.0 from Scratch turns a competent PySpark user into a production Spark engineer. Eleven chapters covering everything that sits between \"the job runs\" and \"the on-call team trusts the job.\"

Key Features

Structured Streaming, end to end: sources, sinks, watermarks, stateful processing, stream-stream joins, exactly-once with checkpointing, and the monitoring surface that catches a stuck stream before the dashboard does.
MLlib without hand-waving: Pipelines, feature engineering, classification, regression, clustering, hyper-parameter tuning, model saving, a complete model lifecycle on real data.
GraphFrames: vertices, edges, motif finding, PageRank, connected components, triangle counting, and the fraud-detection patterns the payment-processing crowd actually uses.
Performance tuning as a diagnostic discipline: the Spark UI tabs, explain() reading, partition sizing, AQE, dynamic partition pruning, manual skew salting, the Z-order story.
Testing and CI/CD for Spark: pytest fixtures, GitHub Actions pipelines, code-quality linting, deployment patterns including canary and blue/green.
The Lakehouse, in depth: Delta Lake (time travel, OPTIMIZE, ZORDER, VACUUM, Change Data Feed), Apache Iceberg (hidden partitioning, partition evolution, REST catalog), the trade-off table, and modern table modelling.

What you will learn

- Process unbounded streams with watermarks, stateful operators, and stream-stream joins

- Train and evaluate ML pipelines at scale with MLlib's Estimator/Transformer API

- Use GraphFrames for PageRank, connected components, motifs, and fraud patterns

- Diagnose any slow Spark job from the UI and explain() output

- Write unit and integration tests for PySpark code that actually catch bugs

- Deploy Spark applications to Kubernetes, YARN, or standalone clusters

- Design Delta Lake and Iceberg tables for production lakehouses

- Implement medallion architecture with bronze/silver/gold layers and SCDs

Who this book is for

Data engineers and data scientists who finished Volume 1 (or already write PySpark professionally) and now need streaming, ML, performance discipline, and production deployment skills. Some chapters touch ML and graph theory lightly, no prior experience is required, but a hunger to ship matters.

Table of Contents

Window Functions. Top-N per group, running totals, moving averages, gap-and-island, session detection from raw event logs. The analytics that groupBy cannot express.
User-Defined Functions (UDFs). Python UDFs, Pandas UDFs (the 3-100x speedup), grouped map for per-group model fitting, and the rule for when to avoid UDFs entirely.
Structured Streaming. Sources, sinks, watermarks, stateful processing, stream-stream joins, exactly-once with checkpointing, monitoring stuck queries before the dashboard does.
Machine Learning with MLlib. Pipelines, feature engineering, classification, regression, clustering, hyperparameter tuning, and a complete model lifecycle on real data.
Graph Processing with GraphFrames. Vertices, edges, motif finding, PageRank, connected components, triangle counting, and the fraud-detection patterns the payments crowd uses.
Performance Tuning. Spark UI tabs read like an X-ray, query plan reading, partition sizing, AQE, dynamic partition pruning, manual salting for skew, Z-ordering for data skipping.
Deploying Spark Applications. spark-submit, packaging, client vs cluster mode, the resource-calculation formula, dynamic allocation, Kubernetes, log configuration that doesn't drown you.
End-to-End Data Pipeline Project. Bronze/silver/gold with SCDs, a reusable data-quality framework, incremental processing, the full e-commerce capstone wired together.
Testing and CI/CD for Spark. pytest fixtures that survive a session-scoped SparkSession, chispa for DataFrame equality, GitHub Actions pipelines, blue/green and canary deployment.
The Lakehouse: Delta Lake, Iceberg, and Modern Table Design. Time travel, OPTIMIZE, ZORDER, partition evolution, schema evolution, and the medallion patterns that make data trustworthy.

","authors",[24],{"_12":25,"_26":27,"_28":29},"Z2lkOi8vbGVhbnB1Yi9TdG9yZTo6UHJvZHVjdEF1dGhvci84NTE5Mg","name","Ritesh Modi","user",{"_30":31},"avatar",{"_32":33,"_34":35,"_36":37},"url","https://d39qdlcrvnra4b.cloudfront.net/avatars/3714687/medium/avatar.jpg?1777217214","styleName","medium","exists",true,"shoppingCartId","shoppingCart",{"_12":71,"_72":73,"_74":37,"_75":76,"_77":78,"_79":80,"_81":37},"locale","en","readerTheme","light","constants",{"_69":70},"publishers",{"_65":66,"_67":68},"learnerOrganizations",[],"needsLocaleCookieSync",false,"announcements",{"_59":60,"_61":-5},"sentryEnvironment","production","sentryRelease","f5548b0fa7c03e0c3ddcc40793859fac31237ca7","announcement_store",{"_62":52,"_63":64},"announcement_authors","flag","text","

Kick off your book project, get started with GhostAI, get better at marketing, or spend the day doing all three! Free live workshops on Zoom. Saturday, June 27, 2026.

\n","companies",[],"universities",[],"paypalClientId","AX53Cwz8jR6rVVzzZiZNWWXl0dGagi-GZrjG-S3Ua14E5nrKMn7DxDQ730O4R8LusLPp49ALn6aRVaEy","no-cart","count",0,"free","updatedAt","","paymentGateway","STRIPE","cartItems",[],"loading"]

Email the Author