Airflow resources for data engineers

This project implements an automated ETL pipeline orchestrated with Apache Airflow. Running on Docker Compose, the DAG extracts weather metrics from the OpenWeatherMap REST API, ingests them into a PostgreSQL database, and feeds analytics dashboards built in Metabase.

🎙️ The Data Flowcast by Astronomer — HCL Tech relies on Apache Airflow for cloud data migrations, managing on-prem extraction, watermarking, and loading into Snowflake and Databricks. The team generates dynamic DAGs from JSON configs for micro-batching and uses blue-green deployment patterns across environments.

Modern data orchestration demands cross-engine interoperability and warehouse cost visibility. Declarative tools like Orchestra speed up delivery via YAML or GUI interfaces and serverless task execution, while Apache Airflow remains the standard for complex, Python-defined DAG dependencies.

Delivery Hero scaled dbt across 45 domain teams processing 60 petabytes. Using Astronomer Cosmos and custom CI scripts, the platform converts dbt manifests into Airflow DAGs with priority weights, enforcing model ownership tags and central policy macros directly in CI pipelines.

Apache Airflow CI moved canary builds and committer PRs to ephemeral AWS self-hosted runners using RunsOn, freeing shared ASF GitHub runners. GitHub Actions still orchestrates runs. RunsOn bypassed the overhead of managing an EKS cluster while keeping CI setups consistent.

Apache Airflow now officially supports LLM calls and agentic actions via six new AI operators and six AI task decorators. When orchestrating these pipelines, data engineers must balance deterministic and non-deterministic workflows, since identical inputs to AI tasks may yield varying outputs.

🎙️ The Data Flowcast by Astronomer — Lyft implemented self-service Airflow DAGs via event-driven GitHub Actions to validate driver pay changes under SOX controls. The architecture blocks pull requests during long-running Spark simulations, using an external context store to maintain state across commit SHAs and rebases.

Apache Airflow released Common AI provider 1.0, compatible with Airflow 2.11+ and 3.3. It enables LLM agents in DAGs via the @task.agent decorator, integrating toolsets like SQLToolset for read-only warehouse queries with table-level guardrails, sandboxed execution, and human-in-the-loop approvals.

🎙️ The Data Flowcast by Astronomer — Insignis runs a cost-effective platform using Airflow as the orchestrator, not the executor. They keep compute lean via CI/CD enforcement, failure callbacks, metadata purging, and custom tools like Tariff to surface SQL costs and Maestro to optimize worker sizing.

🎙️ The Data Flowcast by Astronomer — A podcast series explores Apache Airflow through discussions with data practitioners across industry and government, including leaders from Astronomer, ARGO Labs, the City of San Diego, and ING. The teaser previews real-world orchestration insights.

🎙️ The Data Flowcast by Astronomer — Airflow creator Maxime Beauchemin discusses the core problems the workflow orchestrator was designed to solve. The discussion covers orchestration architecture, his definition of Airflow, and the key design principles governing hook and operator use in data engineering pipelines.

🎙️ The Data Flowcast by Astronomer — This episode highlights real-world Apache Airflow implementations across data engineering teams, exploring six production use cases from organizations including Zapier, WePay, ING, and Astronomer. Technical leaders discuss practical workflow orchestration and pipeline architecture patterns.

🎙️ The Data Flowcast by Astronomer — This episode explores Apache Airflow best practices for data orchestration. It features technical perspectives from data leaders at Tails.com, WePay, and ING, detailing practical insights on workflow management and analytics engineering.

🎙️ The Data Flowcast by Astronomer — Data engineers examine Apache Airflow pain points across ETL pipelines. While maintaining that Airflow is the future of ETL, the discussion explores operational hurdles within incubating orchestration workflows and highlights community plugins to shape platform architecture.

🎙️ The Data Flowcast by Astronomer — This episode explores the integration of Apache Airflow with Kubernetes, focusing on the development of the Kubernetes executor. Engineers Daniel Imberman and Greg Neiheisel discuss building and advancing this executor architecture to run Airflow task instances dynamically in isolated pods.

🎙️ The Data Flowcast by Astronomer — WePay's Joy Gao discusses implementing role-based access control (RBAC) in Apache Airflow 1.10. The episode details her engineering work on Airflow's RBAC architecture and security components, alongside her experience contributing to open-source software.

🎙️ The Data Flowcast by Astronomer — The episode covers Apache Airflow's 1.10 release with release manager Ash Berlin-Taylor. It highlights Airflow's bid to graduate from incubating status, managing open-source orchestrators, and the next development phase for data engineering pipelines.

🎙️ The Data Flowcast by Astronomer — Google Cloud Composer Product Manager James Malone discusses open source at Google and future contributions to Apache Airflow. The episode explores the platform's trajectory and orchestration roadmap within managed cloud environments.

🎙️ The Data Flowcast by Astronomer — Airflow Breeze optimizes the developer experience for Apache Airflow contributors. Created by committer Jarek Potiuk, it simplifies local environments to fix slow PR workflows where unit tests and CI builds previously took hours, streamlining open-source development for data engineers.

🎙️ The Data Flowcast by Astronomer — Airflow 2.0 targets performance, autoscaling, and usability improvements for data engineering pipelines. Key updates include a production-grade API, the Knative Executor, optimized scheduler and webserver performance, and an official production Docker image.

🎙️ The Data Flowcast by Astronomer — Netlify adopted Apache Airflow to decouple orchestration logic from a complex collection of Spark jobs. The data engineering team is expanding their Airflow footprint to manage and scale pipeline architectures for a broader set of use cases across their platform.

🎙️ The Data Flowcast by Astronomer — Typeform leverages Apache Airflow to automate GDPR compliance and build self-service data platforms. By developing custom infrastructure tooling and adopting frameworks like Dag Factory, their data operations team evolved into a platform engineering unit supporting internal data workflows.

🎙️ The Data Flowcast by Astronomer — Reddit manages petabytes of daily data that must be processed, stored, and analyzed by backend services. The team leverages Airflow for pipeline orchestration and scales out a self-service data architecture across multiple teams to support warehousing and engineering functions.

🎙️ The Data Flowcast by Astronomer — The Texas Rangers use Apache Airflow to orchestrate large-scale data and AI pipelines for game strategy. The team relies on scalable architectures, dynamic tasks, and pose-tracking data alongside tools like BigQuery and Databricks to drive machine learning models.

🎙️ The Data Flowcast by Astronomer — Wynn Las Vegas uses Airflow as the backbone for data ingestion, curation, and integration after migrating from Control-M. Workflows rely on auto-generated DAGs, custom operators, and a metadata database to track metrics and capture volume and freshness anomalies via ML models.

🎙️ The Data Flowcast by Astronomer — Apache Airflow serves as an adaptable orchestration engine for ETL and ML workflows. Modern data engineering leverages fine-tuned LLMs trained on task logs to diagnose failure patterns, proving high-quality data is critical for reliable model performance.

🎙️ The Data Flowcast by Astronomer — Vibrant Planet optimizes large-scale geospatial processing in Apache Airflow using self-healing pipelines. They implement custom task decorators, retry callbacks for dynamic resource allocation, and fail-fast patterns to handle uneven resource consumption across sharded datasets.

🎙️ The Data Flowcast by Astronomer — Laurel automates timekeeping by running daily batch jobs that map activities into billable items. The team orchestrates ML pipelines with Apache Airflow for model retraining, backfills, and RAG-based summarization, optimizing DAG testing and long-running execution.

🎙️ The Data Flowcast by Astronomer — Campbell uses Apache Airflow to orchestrate data pipelines across market data, trading, and operations. To ensure reliable pipelines, business logic is decoupled from DAGs for easier testing, while dedicated Airflow instances isolate and run machine learning models.

🎙️ The Data Flowcast by Astronomer — Teya optimizes Apache Airflow workflows using automated CI/CD pipelines to build self-service DAGs for dbt models and machine learning. To boost performance, they adopted Celery Executor scaled with KEDA for fast tasks and switched to a multi-file method over Kubernetes Executor.