Apache Airflow Cookbook

At some point, every data team hits the same roadblock. Each night, the script that used to run smoothly starts to go wrong at 2 a.m. No one knows what step has gone wrong, and reruns are double-counting the numbers that the finance team has already published. Apache Airflow is great because it's been rebuilt from the ground up to suit the way work teams actually do things today.

This cookbook provides you hundreds of solutions that are independent and will take you from the first installation to a platform that supports ETL, ELT, MLOps, AIOps and business operations all at the same time. Each recipe starts with a real problem, has short and easy-to-read code, and ends by showing how the fix works with real terminal output. We can learn to author DAGs with the Task SDK, schedule pipelines on data rather than on the clock, bind extractions so reruns repair instead of duplicates, and run any task inside its own container. We will be practising to write custom operators, packaging them for other teams, extending Airflow through plugins and secrets backends, orchestrating model training and promotion, provisioning infrastructure that tears itself down, and diagnosing stalls from their symptoms.

To me, this book is best suited for every software engineer, backend developer and every such platform teams who keep on building, running and troubleshooting the workflows every day.

Key Learnings

• Convert legacy operator DAGs into TaskFlow functions that infer dependencies from ordinary calls
• Bound every extraction to its data interval so reruns repair rather than duplicate
• Schedule pipelines on asset updates instead of guessing when upstream work finishes
• Fan tasks out dynamically over lists discovered at runtime using expand and partial
• Run any task inside a Kubernetes pod with its own image and resources
• Write custom operators, hooks and sensors, then package them as installable providers
• Extend Airflow through plugins, macros, secrets backends and custom XCom storage
• Orchestrate model training, registry logging and promotion gates without a serving endpoint
• Provision ephemeral infrastructure with setup and teardown pairs that always release resources
• Diagnose stuck queues, starved pools and zombie tasks from their distinct symptoms

Table of Content

1. Getting Airflow 3 Running
2. Authoring DAGs with Task SDK
3. Scheduling, Assets and Event-Driven Pipelines
4. Building ETL Pipelines
5. ELT and Warehouse Orchestration
6. Extracting Insights from Batch Processes
7. Containers and Kubernetes
8. Custom Operators, Hooks and Sensors
9. Plugin Interface and Extending Airflow
10. Managing ML Pipelines
11. AIOps, Infrastructure and Business Operations
12. Testing, Monitoring and Troubleshooting

Target Readers

You don't need to be an Airflow pro or an expert to get started with this book. All you need is to be curious, have a Linux machine and be ready to run the commands instead of just reading them. If you're committed to creating a strong, flexible and expandable workflow, you're on the right track.