Airflow resources for data engineers

Dexaflow is a container-native workflow orchestrator compatible with the Apache Airflow UI and REST API. It replaces XCom plumbing with native map-reduce syntax. Built with a Go control plane, it supports isolated per-DAG virtual environments and pod-per-task execution.

Leoflow is a container-native workflow orchestrator compatible with the Apache Airflow UI and REST API. Powered by a Go control plane and Postgres, it offers native map-reduce fan-out without XCom plumbing or brokers, plus isolated per-DAG virtual environments.

Astronomer launched the astronomer-data plugin for ChatGPT, connecting to managed or self-hosted Apache Airflow. Data engineers can debug DAG failures from task logs, scaffold pipelines, query data warehouses, and trace lineage using natural language.

🎙️ The Data Flowcast by Astronomer — Airflow orchestrates local LLM pipelines for sensitive data, running GPU inference via KubernetesPodOperator into feature stores. DAGs manage ingest, Pydantic validation to stop silent failures, and PEFT fine-tuning. Production requires strict pinning and tracing adapter provenance per row.

Astro launched its browser IDE to general availability for testing and deploying Airflow DAGs without local setup. A new VS Code extension integrates local runtimes, logs, and deployment health checks directly into developer environments.

Dropping a table in dlt fails to backfill because cursor timestamps persist in dltpipeline_state. To selectively re-ingest without wiping the dataset, sync remote state and run dlt pipeline drop with the targeted --state-path to reset the incremental cursor for that resource alone.

Astronomer created de-bench to test AI agents on production Apache Airflow environments across 139 tasks. By supplying orchestration context, their agent Otto outperformed baseline harnesses in accuracy and cost on complex pipeline fixes.

Event-driven S3 to BigQuery pipelines break when spreadsheets shift schemas. Standard Airflow setups rely on human alerts because retries cannot fix structural file changes.

Aprende a disenar pipelines confiables y automatizados con Apache Airflow para optimizar procesos ETL y ELT. El entrenamiento aborda la construccion y gestion de DAGs, la aplicacion de mejores practicas de arquitectura de datos y la resolucion de casos reales para orquestar flujos escalables.

At extreme scale, teams avoid managed warehouses and build on primitives like S3, Kubernetes, and open engines such as Spark and Trino. OpenAI even rewrote Airflow's scheduler and executor in Rust to handle 70,000+ concurrent tasks, relying on exact DAG state metadata to safely drive AI agents.

🎙️ The Data Flowcast by Astronomer — Airflow 3 improves pipeline security by moving connection testing execution off the API server onto workers. It also introduces a JWT revocation table and separates workload tokens from auth tokens by scope and lifetime to prevent task timeouts.

Airflow 3.3 introduces durable execution inside tasks, moving beyond inter-task

Migrating 20 forecasting DAGs to Airflow 3 reduced infrastructure costs by 50%. The team leveraged Airflow 3 AssetWatcher for event-driven asset scheduling, eliminating idle event consumers by waking pipelines on demand, and optimized resource allocation using dedicated worker queues per client.

Pine59 modernized its data orchestration architecture by transitioning to Apache Airflow 3 on Google Cloud. The deployment focuses on infrastructure modernization and data analytics pipelines for supply chain operations.

The official ClickHouse provider for Apache Airflow, apache-airflow-providers-clickhousedb, is now available. It uses clickhouse-connect over HTTP(S), integrates with standard common SQL operators, and adds a dedicated hook for bulk queries and client-specific database operations.

🎙️ The Data Flowcast by Astronomer — itti orchestrates 300 production DAGs across 30 companies using a custom YAML framework over Apache Airflow. Jinja templates and CI/CD compile YAML configs into Python DAGs, letting non-engineers build pipelines. Custom operators manage partition-level data quality tests, dbt runs, and AWS syncs.

Astronomer updated Otto, its Airflow agent, adding a platform-wide panel in Astro. A hosted MCP server now connects Astro control plane tools to external agents like Claude Code or Cursor. Otto authors DAGs and diagnoses failures, with custom model integration supported for enterprise tiers.

Lyft rebuilt its Neighborhood Reachability Signals dataset, an offline lookup matrix generated via Airflow that maps travel times across geohash-6 cells. The update establishes drivable geohashes as a single source of truth, removing the need for downstream consumers to maintain ad-hoc allowlists.

Kestra 2.0 orchestrates workflows using declarative YAML over existing code. Its new Controller decouples workers from the database, letting them run in isolated networks via outbound gRPC without inbound ports. This architecture avoids direct database credentials on workers.

🎙️ The Data Flowcast by Astronomer — ATC Drivetrain built self-healing Apache Airflow pipelines using a five-layer architecture: observe, classify, policy, recover, and validate. Governed AI assists by analyzing contextual logs and recommending safe fixes, resolving roughly 95% of small pipeline failure modes autonomously.

🎙️ The Data Flowcast by Astronomer — To prevent write conflicts in MongoDB medallion layers, Synechron built a three-layer Airflow architecture. A single proxy DAG serializes scheduled and event-driven runs into a strict queue before executing main Spark ETL pipelines generated from YAML configs.

Astro re-engineered Apache Airflow's scheduling and execution systems to scale pipeline performance. The updated architecture sustained 500,000 concurrent Airflow tasks in a single deployment while maintaining full compatibility with Airflow 3 DAG definitions, task dependencies, and operators.

Grab uses Data Production Issues (DPIs) to manage data contract failures. Orchestrated by Kinabalu via Test Health APIs, the automated workflow deduplicates alerts, tracks contract breaches, and classifies root causes using a standardized error taxonomy to route incidents effectively.

Orchestrators run tasks, manage dependencies, and handle retries. While Airflow or Dagster link fragmented modern data stacks, frameworks with built-in dependency resolution eliminate extra infrastructure when DAGs strictly execute ELT tasks, transformations, and quality checks.

Astro provides cross-region disaster recovery on Azure using PostgreSQL geo-replication for Airflow metadata alongside Blob Storage object replication for logs and container images. The standby region keeps compute idle while preserving the cluster identity, achieving an RTO under 1 hour.

The new Astro runtime re-engineers Airflow's scheduler for high-concurrency event-driven execution. It supports up to 500,000 concurrent tasks with sub-second p95 latency, reaching 230ms at 100,000 tasks without polling delays while cutting task failures by 85%.

🎙️ Data Engineering Podcast — Astronomer introduced Otto, a specialized AI agent designed for Apache Airflow workflows. It provides essential context for DAG authoring, code reviews, version migrations, and failure investigations, driving toward self-healing data pipelines while integrating directly into existing data

🎙️ The Data Flowcast by Astronomer — Wise manages financial data by treating Airflow as a pure orchestration layer and offloading compute to EMR or SageMaker. They isolate execution using the Python Virtual Environment Operator and handle non-idempotent LLM workflows via multi-layer validation and agent-checking-agent patterns.

Bruin enables incremental data stack migration without rewrites. Warehouse tables convert via CLI, dbt models port with Jinja, and Fivetran connections map to ingestr assets. Pipelines run inside Airflow DAGs during transition until orchestration shifts to Bruin.

🎙️ The Data Flowcast by Astronomer — Headout uses Apache Airflow as its core orchestration engine to handle travel platform data challenges. The team implements a write-audit-publish pattern with dbt via a custom Python operator reading the dbt manifest, while also managing Feast feature stores, ML pipelines, and commit-level alerts.