Orchestration resources for data engineers

dltHub updated its REST API, breaking pipelines by renaming scripts to jobs, runs to job-runs, and returning 404s on legacy trigger endpoints. Inspecting the dlthub-client wheel reveals payload and SDK changes like jobs.getbyref, requiring updates to Snowflake stored procedures.

OrchardFlow provides an observable, deterministic orchestration layer for cloud-native ELT pipelines. It models data ingestion, transformation, and delivery as graftable branches with predictable yields, documented failure modes, and clear lineage to eliminate brittle, chained commands.

Orchestra is developing an agentic control plane for data and AI, providing a centralized architecture to build, run, monitor, and govern data pipelines alongside AI agents.

Recoverable data pipelines require explicit execution history and structured job boundaries. By decoupling pipeline dependencies from task executions and passing artifact references instead of raw data, orchestrators enable safe partial reruns across failed downstream subgraphs.

Orchestra introduced .oml and rolled out anomaly detection for long-running dbt models alongside an enterprise audit trail. The platform added state machine task pausing, environment branch controls, augmented managed agents, and Google Kubernetes integration.

Dexaflow is a container-native workflow orchestrator compatible with the Apache Airflow UI and REST API. It replaces XCom plumbing with native map-reduce syntax. Built with a Go control plane, it supports isolated per-DAG virtual environments and pod-per-task execution.

Leoflow is a container-native workflow orchestrator compatible with the Apache Airflow UI and REST API. Powered by a Go control plane and Postgres, it offers native map-reduce fan-out without XCom plumbing or brokers, plus isolated per-DAG virtual environments.

Astronomer launched the astronomer-data plugin for ChatGPT, connecting to managed or self-hosted Apache Airflow. Data engineers can debug DAG failures from task logs, scaffold pipelines, query data warehouses, and trace lineage using natural language.

Orchestra Labs introduced Orchestra Markup Language (.oml) for pipeline orchestration. Unlike plain YAML, it enforces schema validation and real-time typing at write-time, eliminating agent errors and cutting config generation time by over 90%.

🎙️ The Data Flowcast by Astronomer — Airflow orchestrates local LLM pipelines for sensitive data, running GPU inference via KubernetesPodOperator into feature stores. DAGs manage ingest, Pydantic validation to stop silent failures, and PEFT fine-tuning. Production requires strict pinning and tracing adapter provenance per row.

GitHub Actions ephemeral runners can form temporary distributed environments for testing systems like Kafka and ClickHouse. By networking runners via Tailscale and polling key-value state in Redis, jobs coordinate cluster lifecycles and stay alive until explicit cleanup messages signal shutdown.

Astro launched its browser IDE to general availability for testing and deploying Airflow DAGs without local setup. A new VS Code extension integrates local runtimes, logs, and deployment health checks directly into developer environments.

Dropping a table in dlt fails to backfill because cursor timestamps persist in dltpipeline_state. To selectively re-ingest without wiping the dataset, sync remote state and run dlt pipeline drop with the targeted --state-path to reset the incremental cursor for that resource alone.

Software-defined assets introduce declarative data engineering to manage tables, models, and dashboards as code. Dagster infers asset graphs from function dependencies rather than manual DAGs, shifting orchestration from isolated tasks directly to data products.

Astronomer created de-bench to test AI agents on production Apache Airflow environments across 139 tasks. By supplying orchestration context, their agent Otto outperformed baseline harnesses in accuracy and cost on complex pipeline fixes.

Aprende a disenar pipelines confiables y automatizados con Apache Airflow para optimizar procesos ETL y ELT. El entrenamiento aborda la construccion y gestion de DAGs, la aplicacion de mejores practicas de arquitectura de datos y la resolucion de casos reales para orquestar flujos escalables.

Databricks acquired RowZero to expand its BI and insights tooling, putting pressure on competitors. Orchestra introduced live anomaly detection for dbt models to catch long-running jobs via configuration, while adding state machine controls to pause individual tasks and integrate with GKE.

Paradime State adds state-aware orchestration to Bolt via dbt Core v2.0. Its decision engine evaluates upstream freshness and SQL logic fingerprints rather than raw text to skip, clone, or build models, avoiding full pipeline rebuilds without per-skip fees.

The Stockholm MLOps Meetup focuses on "Orchestration vs Model Selection" within machine learning engineering pipelines. Technical leaders from Hopsworks, CHI Software, and Confidential Mind discuss architectural trade-offs between workflow orchestration systems and model selection strategies in

Kestra Academy launched courses for Kestra 2.0 to help data engineers build and operate reliable data workflows. Offerings cover core fundamentals, declarative orchestration, AI workflows, and infrastructure automation, each ending with a certification exam.

At extreme scale, teams avoid managed warehouses and build on primitives like S3, Kubernetes, and open engines such as Spark and Trino. OpenAI even rewrote Airflow's scheduler and executor in Rust to handle 70,000+ concurrent tasks, relying on exact DAG state metadata to safely drive AI agents.

🎙️ The Data Flowcast by Astronomer — Airflow 3 improves pipeline security by moving connection testing execution off the API server onto workers. It also introduces a JWT revocation table and separates workload tokens from auth tokens by scope and lifetime to prevent task timeouts.

dbt lacks native real-time run state tables, requiring log parsing to stream metadata. Orchestra addresses this architectural gap by introducing node-level anomaly detection, alerting mid-run when tasks exceed baseline duration thresholds to catch pipeline creep and avoid SLA breaches.

Orchestra announced real-time anomaly detection for dbt operations to alert when models run long. Platform updates include Google Kubernetes integration, task pausing, and pipeline environment controls, while the broader data ecosystem saw major funding for Temporal and Streamhouse adoption.

Frontier engineers bridge AI agents and production via orchestration. They separate deterministic tasks from agentic steps using strict contracts like structured JSON validation, scoped tools, and human approvals to safely integrate models into workflows.

Airflow 3.3 introduces durable execution inside tasks, moving beyond inter-task

Migrating 20 forecasting DAGs to Airflow 3 reduced infrastructure costs by 50%. The team leveraged Airflow 3 AssetWatcher for event-driven asset scheduling, eliminating idle event consumers by waking pipelines on demand, and optimized resource allocation using dedicated worker queues per client.

Pine59 modernized its data orchestration architecture by transitioning to Apache Airflow 3 on Google Cloud. The deployment focuses on infrastructure modernization and data analytics pipelines for supply chain operations.

The official ClickHouse provider for Apache Airflow, apache-airflow-providers-clickhousedb, is now available. It uses clickhouse-connect over HTTP(S), integrates with standard common SQL operators, and adds a dedicated hook for bulk queries and client-specific database operations.

Kestra expanded its orchestrator plugins beyond AWS, GCP, and Azure to include Huawei Cloud and DigitalOcean. Driven by sovereignty needs, plugins use container and machine-level task runners to execute workloads natively on target clouds, bypassing API read caps by logging directly to storage.