Orchestration resources for data engineers

This project implements an automated ETL pipeline orchestrated with Apache Airflow. Running on Docker Compose, the DAG extracts weather metrics from the OpenWeatherMap REST API, ingests them into a PostgreSQL database, and feeds analytics dashboards built in Metabase.

🎙️ The Data Flowcast by Astronomer — HCL Tech relies on Apache Airflow for cloud data migrations, managing on-prem extraction, watermarking, and loading into Snowflake and Databricks. The team generates dynamic DAGs from JSON configs for micro-batching and uses blue-green deployment patterns across environments.

Modern data orchestration demands cross-engine interoperability and warehouse cost visibility. Declarative tools like Orchestra speed up delivery via YAML or GUI interfaces and serverless task execution, while Apache Airflow remains the standard for complex, Python-defined DAG dependencies.

Snowflake previewed DEFINE TASK GRAPH for DCM Projects to manage orchestrated pipelines as code. Alongside recent support for atomic deployments and advanced branching in task graphs, this enables data engineers to define and deploy declarative data pipelines with centralized configuration.

Infrastructure automation often fails across tool handoffs during Day 2 operations like patching. While Terraform or Ansible automate isolated tasks, cross-tool orchestration is missing. Providing a unified control plane coordinates execution, passes outputs, and creates a single audit log.

Run open-weight LLMs locally to cut token costs and protect data privacy. Use quantization like 4-bit to shrink tensor memory footprints for consumer GPUs, and orchestrate automated pipelines using tools like Kestra for scheduled batch inference and continuous tasks.

Sigma manages its semantic layer as versioned dbt code orchestrated by Dagster across 2,883 models. Custom packages compile definitions into Sigma data models, Snowflake semantic views, and Cortex agents. Dagster automates state-aware dbt builds and syncs them through Sigma's API.

Data consultancy Modeo has partnered with Kestra, adopting it over Airflow and Dagster to orchestrate end-to-end data pipelines. Modeo deploys Git-versioned YAML flows for ingestion and dbt SQL transformations, citing that Kestra is simpler to operate, maintain, and scale in production.

Snowflake launched Pipeline Builder in preview, expanding pipeline development features alongside task graph improvements like atomic deployments and advanced branching. Updates also bring schema variables and Iceberg v3 upgrades to streamline data architectures.

Snowflake introduced atomic deployments, centralized configuration, and advanced branching for task graphs in preview. The update streamlines data pipeline orchestration by enabling safer, all-or-nothing DAG deployments alongside improved conditional execution flows.

dbt State enables selective model builds based on upstream data changes and lag tolerance rather than fixed schedules. By using auto-cloning and eliminating manual manifest management, it cuts redundant pipeline runs and reduces warehouse compute costs.

Mitratech turned dbt into an orchestrator to automate customer data onboarding. Using macro packages and Snowflake stored procedures via "dbt run-operation", they build secure views and shares with tenant filtering and PII masking, cutting provisioning time from days to hours.

Virgin Media O2 reduced BigQuery compute costs by roughly 30% and build times by 35% using dbt v2 and dbt State. By enabling state-aware builds, their pipeline skipped unchanged models, reducing slot consumption from 4,500 to 3,400 per hour on a massive 1,400-model DAG.

Apache Airflow now officially supports LLM calls and agentic actions via six new AI operators and six AI task decorators. When orchestrating these pipelines, data engineers must balance deterministic and non-deterministic workflows, since identical inputs to AI tasks may yield varying outputs.

Open-source workflow orchestration relies on real production adoption over virality. While AI assistants streamline generating pull requests, architectural reviews remain a bottleneck. Open code gives data engineers transparency into pipeline engines and tools exposed via MCP servers.

Enterprise AI workflows require unified orchestration to manage cross-domain agents. Moving data is solved, but governance demands keeping execution IDs outside agents. Running workers near data platforms like Snowflake or Databricks ensures auditability and end-to-end control.

Migrate from VMware vRA to Kestra without a complete rewrite. Kestra's Migration Assistant parses vRA/vRO exports to build dependency inventories and migration waves. It replaces boilerplate JavaScript and retry handling with declarative tasks and plugins while keeping your existing execution tools.

🎙️ The Data Flowcast by Astronomer — Lyft implemented self-service Airflow DAGs via event-driven GitHub Actions to validate driver pay changes under SOX controls. The architecture blocks pull requests during long-running Spark simulations, using an external context store to maintain state across commit SHAs and rebases.

Apache Airflow released Common AI provider 1.0, compatible with Airflow 2.11+ and 3.3. It enables LLM agents in DAGs via the @task.agent decorator, integrating toolsets like SQLToolset for read-only warehouse queries with table-level guardrails, sandboxed execution, and human-in-the-loop approvals.

Apache Fluss enables crash-safe retries for side-effecting workers by combining an append-only log table for replay with a primary key table for state lookups. Workers persist intent before external calls, allowing restarted jobs to reconcile pending actions against external service ledgers.

Databricks acqui-hired Responsive to bolster its Postgres offering, while Snowflake introduced Postgres replication tooling. Orchestra also added private RDS connectivity via AWS Resource Gateway, AWS Batch orchestration, and pipeline run queues.

Databricks Jobs tasks can fail without running. Upstream settings like ATLEASTONE_FAILED may trigger rollback tasks even when no upstream processing started, requiring careful guardrails in pipeline task dependencies.

🎙️ The Data Flowcast by Astronomer — Insignis runs a cost-effective platform using Airflow as the orchestrator, not the executor. They keep compute lean via CI/CD enforcement, failure callbacks, metadata purging, and custom tools like Tariff to surface SQL costs and Maestro to optimize worker sizing.

Netflix rebuilt its Conductor orchestrator to handle over a billion quarterly executions. The modernized data plane uses Cassandra for primary storage, Timestone for priority queuing, and Iceberg for long-term data, while partitioning task records to scale to 30,000 tasks per workflow.

Resolving Spark sessionization OOMs required parallel branch testing on full production data. Fixes included unboxing Scala Option fields and slimming event bags. Tuning spark.sql.adaptive.autoBroadcastJoinThreshold to 10 MB stopped multiple broadcast hash joins from exhausting executor memory.

🎙️ The Data Flowcast by Astronomer — A podcast series explores Apache Airflow through discussions with data practitioners across industry and government, including leaders from Astronomer, ARGO Labs, the City of San Diego, and ING. The teaser previews real-world orchestration insights.

🎙️ The Data Flowcast by Astronomer — Airflow creator Maxime Beauchemin discusses the core problems the workflow orchestrator was designed to solve. The discussion covers orchestration architecture, his definition of Airflow, and the key design principles governing hook and operator use in data engineering pipelines.

🎙️ The Data Flowcast by Astronomer — This episode highlights real-world Apache Airflow implementations across data engineering teams, exploring six production use cases from organizations including Zapier, WePay, ING, and Astronomer. Technical leaders discuss practical workflow orchestration and pipeline architecture patterns.

🎙️ The Data Flowcast by Astronomer — This episode explores Apache Airflow best practices for data orchestration. It features technical perspectives from data leaders at Tails.com, WePay, and ING, detailing practical insights on workflow management and analytics engineering.

🎙️ The Data Flowcast by Astronomer — Luigi creator Erik Bernhardsson discusses workflow orchestration, examining the engineering motivations behind building Luigi and sharing his perspective on Apache Airflow for data pipeline management.