Data Pipelines With Apache AirflowData Pipelines With Apache Airflow

Simplify, streamline, and scale your data operations with data pipelines built on Apache Airflow Data Pipelines with Apache Airflow has empowered thousands of data engineers to build more successful data platforms. This new second edition has been fully revised for Airflow 3 with coverage of all the latest features of Apache Airflow, including the Taskflow API, deferrable operators, and Large Language Model integration. Filled with real-world scenarios and examples, you'll be carefully guided from Airflow novice to expert. In Data Pipelines with Apache Airflow, Second Edition you'll learn how to: Master the core concepts of Airflow architecture and workflow design Schedule data pipelines using the Dataset API and time tables, including complex irregular schedules Develop custom Airflow components for your specific needs Implement comprehensive testing strategies for your pipelines Apply industry best practices for building and maintaining Airflow workflows Deploy and operate Airflow in production environments Orchestrate workflows in container-native environments Build and deploy Machine Learning and Generative AI models using Airflow Using real-world scenarios and examples, Data Pipelines with Apache Airflow, Second Edition teaches you how to simplify and automate data pipelines, reduce operational overhead, and smoothly integrate all the technologies in your stack. Part reference and part tutorial, each technique is illustrated with engaging hands-on examples, from training machine learning models for generative AI to optimizing delivery routes. About the Technology Apache Airflow provides a unified platform for collecting, consolidating, cleaning, and analyzing data. With its easy-to-use UI, powerful scheduling and monitoring features, plug-and-play options, and flexible Python scripting, Airflow makes it easy to implement secure, consistent pipelines for any data or AI task. About the Book Data Pipelines with Apache Airflow, Second Edition teaches you how to build, monitor, and maintain effective data workflows. This new edition adds comprehensive coverage of Airflow 3 features, such as event-driven scheduling, dynamic task mapping, DAG versioning, and Airflow's entirely new UI. The numerous examples address common use cases like data ingestion and transformation and connecting to multiple data sources, along with AI-aware techniques such as building RAG systems. What's Inside Deploying data pipelines as Airflow DAGs Time and event-based scheduling strategies Integrating with databases, LLMs, and AI models Deploying Airflow using Kubernetes About the Reader For data engineers, machine learning engineers, DevOps, and sysadmins with intermediate Python skills. About the Authors Julian de Ruiter, Ismael Cabral, Kris Geusebroek, Daniel van der Ende, and Bas Harenslak are seasoned data engineers and Airflow experts. Quotes A fun journey through Apache Airflow, well illustrated with practical examples. - Isabel Drost-Fromm, Europace AG Clear, well-structured, and surprisingly easy to follow. - Mounika Garikapati, Ripple Labs A practical, clear, and well-structured guide. - Najeeb Arif, IBM Focuses on parts that are important today, and will remain important in a long time. Definitely worth having. - Jarek Potiuk, Committer and PMC member of Apache Airflow Clear structure, practical examples, and useful chapter summaries guided me step by step, in a clear and enjoyable way. - Bertrand Delacrétaz, Apache Software Foundation.