dataaaaaa!

Google Cloud introduces Agentic Data Cloud connectors to enable conversational analytics across enterprise data within Gemini. Focusing on BigQuery, databases, and analytics architectures, these connectors bridge enterprise data platforms with AI models for direct querying.

Apache Airflow 3.4.0 adds native Task Loops for bounded iteration at runtime via decorated task groups. It also enables multi-language DAG authoring without Python, supporting TypeScript, Go, Java, and YAML via pluggable DAG importers, and matures common.ai integrations for LLM workflows.

A Snowflake Cortex agent automates dltHub pipeline recovery. Triggered by job failures, it retries transient issues or fixes code and config errors by editing files in git and redeploying. Actions run via stored procedures on a warehouse, backed by audit tables and automated rollbacks on failure.

MotherDuck released features including Bring Your Own Compute to run Ducklings in customer AWS accounts, Giga instances (192 cores, 1.5 TB RAM) for single-node scaling, and Flight Explainer to profile pipeline runs with row counts and step timings.

Google BigQuery introduces AI.AGG to perform semantic aggregation across millions of rows directly in the analytical engine. This enables data engineers to execute advanced semantic data processing and analytics at scale without moving data outside BigQuery.

JetBrains embedded DuckDB into DataGrip to enable cross-database queries. By leveraging DuckDB custom table functions, users can run federated joins across PostgreSQL, MySQL, and other engines in a single query without extra infrastructure, benefiting from optimizations like projection pushdown.

AWS Bedrock AgentCore combines with Amazon Cognito and AWS Lake Formation to provide secure multi-tenant AI analytics. Cognito maps user identity claims to scoped IAM roles, letting Lake Formation enforce fine-grained row- and column-level security on downstream Amazon Athena SQL queries.

AI context needs an Apache Iceberg equivalent. While open formats exist, teams lack a shared metadata catalog layer to track snapshots, ownership, and current versions across disparate engines.

The Paris Apache Airflow Meetup covers engineering patterns for data pipelines and AI workloads. A technical session details using the Airflow Common AI provider to orchestrate RAG, SQL generation, and multi-turn agents within DAGs, utilizing durable execution and human-in-the-loop validation.

Snowflake released HTML reports in preview alongside general availability for Apache Iceberg table backups. Other data engineering updates include schema variables, declarative feature stores, and automated task graph deployments.

Snowflake has made backups for Snowflake Storage for Apache Iceberg tables generally available, improving disaster recovery and data protection. Data engineers can safeguard lakehouse architectures while maintaining open table format interoperability.

Postgres wait dashboards sample active flags via pgstatactivity rather than tracing per-query latency. In a checkout test, tail latency was driven by short row locks accounting for only 2% of dashboard time. Common metrics can miss critical query stalls hidden beneath aggregated wait times.

🎙️ colsto — A geospatial pipeline locates audio recordings within 100m using Doppler shift analysis. ADS-B flight telemetry filtered via Python joins with digital elevation models on a 200m grid, scoring modeled acoustic curves against spectrograms to resolve exact coordinates.

This dbt package analyzes compute, storage, and query patterns across Snowflake, Databricks, BigQuery, and Redshift to generate prioritized cost-saving recommendations. It offers run-operation macros and builds gold-layer marts when enabled via environment variables.

DataSearcher is an analytical tool integrating DuckDB, FastAPI, and an MCP server. It connects to PostgreSQL, MySQL, and ClickHouse, executing translated SQL queries alongside automated PII masking, read-only query guards, and metadata ingestion from dbt manifests and DataHub.

Google introduces new capabilities across its Agentic Data Cloud to power autonomous agents at scale. The updates focus on database architecture and data analytics infrastructure to support scalable AI workloads.

AlloyDB for PostgreSQL integrates direct Apache Iceberg access to unify operational workloads with lakehouse architectures. This feature connects PostgreSQL environments directly to Iceberg data, bridging transactional databases and analytics pipelines without requiring complex data movement.

Altertable provides a lakehouse architecture unifying storage economics, live federation, and agent-ready context to replace fragmented data stacks with warehouse-grade performance.

MotherDuck introduced Agent Chat to run end-to-end data workflows directly in its UI. Using natural language, the native agent generates SQL and executes Python-based data pipelines called Flights on managed infrastructure, automating raw data ingestion, transformation, and interactive app delivery.

Cryo automates Apache Iceberg table maintenance to solve small-file fragmentation and snapshot bloat. It scans catalog metadata, grades tables, and executes load-aware partition compaction on existing Spark or Trino engines, yielding faster queries and reliable commits under live CDC.

Synchronizing local data caches via logical replication speeds reads but adds complex reconciliation and write paths. Instead of adding synchronization layers, engineers should address network-attached storage bottlenecks where cache misses cause I/O limits on managed Postgres instances.

Snowflake announced Business Ontology in Horizon Context to govern business concepts and map them to Semantic Views and tables. It supports open formats like Apache Ossie and RDF for cross-tool portability, while Cortex Sense infers metadata context from query history and dbt models.

Snowflake CoWork adds Profiles, Dashboards, and Automations. Automations execute scheduled recurring jobs via native Snowflake tasks. CoWork queries governed data while strictly enforcing row access and masking policies, using Cortex Search and AI Gateway to manage permissions, models, and costs.

dltHub released a pipeline observability UI powered by its context graph. It tracks metrics like row counts, bytes, and durations per table to detect silent ingestion anomalies, such as partial API loads, even when runs succeed. Teams can inspect logs and set Slack or email alerts.

Unshackle migrates BI metrics, dashboards, and permissions into Git repositories. It translates measures into Apache Ossie semantic models, dashboards into Jupyter SQL notebooks, and row-level security into Cedar policies, while verifying query outputs against warehouse data.