dataaaaa!

The Common AI Provider integrates LLM and agent capabilities directly into Apache Airflow, supporting 20+ model providers. It features TaskFlow decorators like @task.llm and @task.agent for structured outputs and multi-step reasoning. HookToolset turns any of them into AI agent tools.

BigQuery has introduced managed AI functions like AI.IF, AI.SCORE, and AI.CLASSIFY to process unstructured data using Gemini. An optimized mode is now in preview to reduce LLM token consumption and query latency when processing large datasets. These tools allow SQL-based data classification.

DuckLake v1.0 is a production-ready lakehouse format that stores metadata in a SQL database rather than scattered files. It supports data inlining to prevent small file problems by staging updates in the catalog. It makes much more sense to store all metadata of a lakehouse in a database.

DuckLake v1.0 introduces a new standard for data lake architectures, focusing on data inlining to enable streaming capabilities. This release aims to bridge the gap between batch processing and real-time ingestion within the DuckDB ecosystem. Unlocking streaming for data lakes is a core objective.

DuckDB v1.5.2 introduces the stable DuckLake v1.0 lakehouse format, featuring data inlining and bucket partitioning. The update enhances Iceberg support with ALTER TABLE and partitioned deletes. The test suite has uncovered a bug that was triggered by INSERT INTO statements, now resolved.

Snowflake now supports custom budgets to control credit consumption for AI features like Cortex and Snowflake Intelligence. Engineers can track costs by tagging specific users or business units. The budget tracks only the credits consumed by those tagged users to ensure precise cost allocation.

SQL cells in BigQuery notebooks have reached general availability, allowing data engineers to execute queries directly within the notebook interface. This update streamlines the transition between data exploration and analysis. SQL cells in BigQuery notebooks are now generally available.

Dagster 1.13 introduces partitioned asset checks, allowing data quality validation at the same granularity as time-based data. It also debuts the dagster-io/skills repository to improve AI-assisted development. Dagster 1.13 is about making the framework easier to adopt.

BigQuery introduced stateful operations for continuous queries in Preview. This update enables complex analysis by retaining data across rows or time intervals. You can now use stateful operations in continuous queries, which let you perform complex analysis using JOINs and windowing aggregations.

Snowflake announced the general availability of AI_COMPLETE document intelligence, enabling multimodal reasoning over PDFs and Word files. It allows users to extract entities and structured insights from reports, contracts, spreadsheets, and technical documentation directly within data workflows.

Snowflake Data Clean Rooms API version 14.3 introduces the general availability of the Collaboration API. This architecture supports fully symmetric, multi-party collaboration with flexible roles and fine-grained data access controls. It also adds stored procedures for configuration management.

Snowpipe Streaming now supports general availability for server-side error logging. Failed rows are captured in dedicated tables with full payloads and metadata. This allows engineers to query, analyze, and reprocess failed rows using standard SQL without affecting ingestion costs.

Snowflake has released workspaces replication for general availability, enabling user workspaces to be included in database replication and failover groups. Replicated workspaces are read-only in secondary accounts.

BigQuery Data Transfer Service now supports incremental transfers from Microsoft SQL Server in Preview. This update allows data engineers to sync only changed data rather than full datasets. The BigQuery Data Transfer Service now supports incremental data transfers when transferring data.

Snowflake is introducing granular billing service types for services previously grouped under AISERVICES to transition toward AI Credits. This update affects the METERINGHISTORY and METERINGDAILYHISTORY views. This change improves customer clarity and supports more flexible pricing and packaging.

BigQuery Data Transfer Service now supports incremental transfers for MySQL, Oracle, PostgreSQL, and ServiceNow in preview. Additionally, the embeddinggemma-300m model is available for AI functions. This model uses your BigQuery slots to generate embeddings at scale.

Airflow 3.2.0 introduces asset partitioning for granular, data-aware scheduling and experimental multi-team deployments for enterprise isolation. Asset partitioning in 3.2 makes this granular. Downstream Dags trigger only when the specific partition they care about gets updated.

ClickHouse 26.3 enables asynchronous inserts by default and introduces materialized CTEs via the MATERIALIZED clause. This allows subqueries to be evaluated once and stored in temporary tables, optimizing complex joins. CTEs will only be evaluated only once and stored in temporary tables.

Snowflake has released native semantic classifiers for medical and health data to help identify HIPAA-aligned information. New categories like MEDICALDATA and MEDICALSPECIALTY detect ICD codes and medication names. Snowflake adds the MEDICALDATA and MEDICALSPECIALTY semantic categories.

Snowflake now supports write operations for externally managed Iceberg tables on Azure via Databricks Unity Catalog. This expansion uses Azure Data Lake Storage Gen2 with external volumes.

BigQuery introduced the AI.AGG function in preview for semantic aggregation of unstructured data. This BigQuery ML tool uses natural language instructions to process inputs. You can now use the AI.AGG function to semantically aggregate unstructured input data based on natural language instructions.

Snowflake Data Clean Rooms API version 14.0 introduces case-insensitive identifiers for data offerings and templates. Cross-registry resource discovery is now enabled by default. You no longer need to match the exact casing used during registration when referencing a data offering or template by ID.

Snowflake's COPY TAGS parameter is now generally available for CREATE OR REPLACE TABLE commands. It ensures tags from the replaced table and its columns are retained on the new table. If sources conflict, the value from the replaced table takes precedence during the operation.

Snowflake now supports CREATE ICEBERG TABLE AS SELECT for Databricks Unity Catalog using external volumes. This GA update removes the need for INSERT INTO workarounds. Previously, CTAS was not supported for Unity Catalog, and you had to create an empty table and then use INSERT INTO.

Snowflake introduced the AIFUNCTIONSUSER database role to enable granular access control for Cortex AI functions. This allows admins to independently control access to AI functions without requiring the CORTEX_USER database role, improving security and permission management.