DATABRICKS FOR PRACTITIONERS: Volume 1

A practical, depth-first guide to running Databricks in production.

The complete platform-and-data-engineering playbook for the engineers who own pipelines, govern catalogs, and keep workloads on schedule. Current to 2026, with examples on Azure Databricks and concepts that apply unchanged on AWS and GCP.

Key Features

• Build the governed Databricks platform layer by layer, from workspaces and compute to Unity Catalog, identity, and access control
• Ship production data pipelines with Lakeflow Spark Declarative Pipelines, Jobs, Declarative Automation Bundles, and CI/CD from Git
• Tune for cost and performance with Photon, Adaptive Query Execution, Liquid Clustering, and the Query Profile UI

Examples run on Azure Databricks. What happens inside Databricks is identical on AWS and GCP; where the cloud seams differ (identity, storage, secrets, networking), chapters name the AWS and GCP equivalents explicitly.

What you will learn

- Architect a governed Databricks workspace from metastore to volume

- Configure Unity Catalog with privileges, ABAC, Governed Tags, and lineage

- Integrate Microsoft Entra ID identity, SCIM, and service principals

- Build ingestion with Lakeflow Connect, Auto Loader, and streaming tables

- Author bronze-silver-gold pipelines with Lakeflow Spark Declarative Pipelines

- Deploy from Git with Declarative Automation Bundles and GitHub Actions

- Observe billing, audit, query history, and lineage through system tables

- Tune performance with Photon, AQE, Liquid Clustering, and the Query Profile UI

Who this book is for

Data engineers, platform engineers, and architects who already know PySpark and now need to run it on Databricks at production depth. A working knowledge of PySpark, Spark SQL, and Delta Lake is expected. Readers new to Spark should start with Volumes 1 and 2 of the series.

Table of Contents

1. Databricks: The Platform on Top of Spark. Why Databricks exists, what it adds on top of open-source Spark, and how the layers stack so you can see the platform clearly.

2. Workspaces, Notebooks, and Git Folders. The workspace surface, multiplayer notebook ergonomics, Git Folders, and the seven traps that ruin notebook-driven development.

3. Compute: Classic, Serverless, and Cluster Policies. Six compute types with one decision tree. Photon eligibility, cluster policies that prevent runaway costs, the cold-start math.

4. Unity Catalog Architecture. Metastore, catalogs, schemas, tables, volumes, the three-grant cascade. How every securable connects, traced end to end with a single SELECT query.

5. Access Control: Privileges, ABAC, and Governed Tags. The grant model, attribute-based access for cross-table policies, tag-driven row filters and column masks that scale to 400 tables.

6. Identity: Entra ID, SCIM, and Service Principals. Users, groups, SPs; OAuth flows; the anti-patterns the auditor will catch, including the JDBC-string trap and the printed-secret trap.

7. Managed Tables: Delta, Iceberg, and UniForm. The default Delta path, Iceberg v3 features, REST Catalog access from DuckDB, and when to use which format.

8. Liquid Clustering and Predictive Optimization. CLUSTER BY versus PARTITION BY, the four-column limit, Predictive Optimization, and when PO does what you would otherwise script.

9. System Tables and Platform Observability. Billing, audit, query history, lineage. The SQL surface for everything the UI shows, so you can put a dashboard on top of any of it.

10. Ingestion: Lakeflow Connect, Auto Loader, and Streaming Tables. File-discovery modes, file-notification vs directory listing, CDC pipelines, partner connectors, the four trigger modes.

11. Lakeflow Spark Declarative Pipelines. The CREATE PIPELINE shape, bronze/silver/gold patterns, the DLT-to-SDP migration path, the validation contracts that catch bad data early.

12. Lakeflow Jobs and Scheduling. Tasks, dependencies, retries, file-arrival triggers, repair-and-rerun. The Job that runs the nightly bronze-silver-gold pipeline, fully traced.

13. Declarative Automation Bundles. Project structure, targets, deploy/run/destroy, the YAML decoded clause by clause, the staging-vs-production override patterns.

14. CI/CD with GitHub Actions. OIDC federation (no client secrets), six PR validations, staging vs production gates, the full promote-the-pipeline worked example with rollback.

15. Performance: Photon, AQE, and the Query Profile UI. Four EXPLAIN flavors, reading the plan bottom-up, the salting decision tree, the cache vs MV vs raw-query choice.

16. Metric Views and the Bridge to Volume 4. The CREATE METRIC VIEW shape, composing metrics from metrics, the certified-semantics layer every dashboard in Volume 4 reads through.