dataaaaa!

DuckDB has branched v2.0-cyanoptera into feature freeze for an October release, entering alpha testing for Python and CLI clients. Extensions like quack 1.0, httpfs, ducklake, and iceberg are available in alpha, boosting query throughput and enabling community extension testing against ref_next

AWS agreed to acquire DuckLabs, keeping DuckDB open-source under the independent DuckDB Foundation and MIT license. The engine offers fast in-memory SQL on Parquet and Iceberg, giving AWS strategic leverage and potential hosting paths across EC2, ECS, and Lambda.

Snowflake added Claude Fable 5.1 to Cortex AI in private preview. Data engineers can leverage Cortex AI Functions to execute multimodal SQL pipelines and use Snowflake CoCo to transform natural language into production-ready pipelines within Snowflake's secure perimeter.

Snowflake announced general availability for External lineage alongside data lineage for Cortex Agents. Data engineers also gain GA features including DCM Projects, multi-value tags, and an Openflow Connector for SQL Server CDC, plus REST catalog integration for Amazon S3 Tables Iceberg support.

Snowflake introduced data lineage for Cortex Agents alongside an asynchronous API, while recommending users transition from Cortex Analyst to Cortex Agents. Platform updates also include external lineage support and a new agentsinfo column in the ACCESSHISTORY view to enhance data governance.

ClickHouse acquired RunReveal to enhance its platform for security workloads. The move targets continuous high-throughput ingest, years-long retention, and fast analytics across cloud audit logs and telemetry, providing better guidance on ClickHouse schema and retention design.

ClickHouse released data-agent-mnist, an open benchmark harness evaluating LLM analytics agents against data warehouses using 201 real-world questions. Claude Fable 5.1 led correctness at 76.6%, while DeepSeek V4 Flash ran the suite for $1 versus $52 for Fable 5.1 with an 11pp drop.

PostgreSQL 19 introduces four new monitoring system views: pgstatlock, pgstatrecovery, pgstatautovacuumscores, and pgdsmregistryallocations. Notably, pgstatlock provides cumulative, cluster-wide lock statistics across 12 lock types to help analyze lock contention history.

A semantic layer bridges raw warehouse data and business metrics. It succeeds by combining headless distribution with git-based in-flow authoring. This architecture lets data teams govern definitions while letting users update logic so the shared model improves with use.

Google introduced TabFM in BigQuery to integrate predictive machine learning directly into its cloud data analytics platform. This tooling expands in-engine predictive analytics capabilities for data engineering and analytics workloads.

Orchestra launched an agentic control plane to orchestrate pipelines and governed AI agents. Featuring a serverless engine and over 100 integrations, it connects existing infrastructure to run parallel workflows and incident triage without requiring engineers to manage clusters.

Defensibility lies not in models, but in capturing production traces, evaluations, and workflow state. Decouple models, business logic, state, and execution layers using open interfaces to keep components swappable as underlying platforms evolve.

BigQuery enables native property graphs directly over clean canonical tables using GQL with zero data movement. Ingest data with dlt into canonical views, then use CREATE PROPERTY GRAPH to replace complex recursive SQL joins with simple multi-hop MATCH queries.

Snowflake recommends migrating from Cortex Analyst to Cortex Agents. Recent updates expand data engineering workflows with a Cortex Agent code execution tool, increased file limits for deployed dbt projects, and new Openflow CDC connectors for SQL Server.

Snowflake released CoCo Desktop v1.21.2 alongside the preview of second generation Openflow. Other recent updates include Snowflake CLI v3.26.0, an increased file limit for deployed dbt project objects, and general availability for CoCo in the Snowflake Extension for Visual Studio Code.

The lakehouse architecture simplifies fragmented data stacks, unifying warehouse-grade performance, storage economics, live federation, and agent-ready context into a single foundation.

The streamhouse pattern unifies streaming and analytical data into one logical table. Real-time writes hit a low-latency Arrow tier, while a continuous tiering service compacts aging data into open Parquet lakehouse formats like Iceberg or Paimon, letting SQL engines query both tiers seamlessly.

Astro re-engineered Apache Airflow's scheduling and execution systems to scale pipeline performance. The updated architecture sustained 500,000 concurrent Airflow tasks in a single deployment while maintaining full compatibility with Airflow 3 DAG definitions, task dependencies, and operators.

Snowflake announced preview support for second-generation Openflow alongside key data engineering updates, including CREATE OR ALTER support for dynamic Apache Iceberg tables and the Openflow Connector for SQL Server CDC ingestion.

AWS acquired DuckLabs, the team behind DuckDB, while Motherduck bought tower.dev for a python runtime for ingest. NVIDIA agreed to buy Hugging Face for $13bn. AWS Glue jobs can now fetch logs, and Power BI supports partition-specific dataset refreshes.

dbt brings software engineering practices to data transformation by organizing SQL select statements into a DAG. It persists cleaned, modeled tables in the warehouse, while its semantic layer provides consistent metric definitions required for downstream BI tools and AI agents.

Google Cloud introduces Data Agent Kit to accelerate data engineering workflows. The agentic tool aims to cut data pipeline development time from weeks to minutes, automating pipeline generation and data integration tasks across analytics systems.

Omnigent acts as a meta-harness for coding agents, unifying fragmented developer tools, skills, and models. It provides a single common layer/interface to control policies, sandbox bash execution, and switch between LLMs across varied repositories.

The post-AI data stack shifts data engineering focus from building reports to encoding consensus into infrastructure. As AI agents generate queries cheaply, pipelines must expose agent-readable artifacts and unified context, preventing disparate data silos across cloud data warehouses.

Data engineering credibility requires moving beyond repetitive implementation tasks like maintaining the same pipeline or stack. As careers evolve, building broader architectural pattern recognition and sharing foundational concepts provides more long-term impact than isolated pipeline maintenance.