Analytical databases resources for data engineers

Snowflake Horizon Catalog has reached general availability for accessing externally managed Apache Iceberg tables. The release also adds general availability for Horizon Iceberg REST Catalog telemetry, improving external catalog integrations and data lakehouse architectures.

Snowflake announced the general availability of Horizon Iceberg REST Catalog telemetry, expanding monitoring for Apache Iceberg metadata operations. Related updates include general availability for Apache Iceberg v3 table upgrades and Tri-Secret Secure storage support.

Snowflake has released general availability for Tri-Secret Secure on Snowflake storage for Apache Iceberg tables. Data engineers can now leverage enhanced customer-managed encryption key architectures for Iceberg table storage directly within Snowflake, improving security and compliance.

BigQuery now supports retrieving query results directly in the Apache Arrow format in preview. This improves data pipelines by offering faster retrieval speeds and enhanced serialization efficiency when extracting analytical query outputs.

Snowflake announced Cortex Training in preview, alongside hybrid table support for Google Cloud and External OAuth for Snowflake Postgres. Recent updates also expand data engineering tooling, including DCM project task graphs, Time Travel for Apache Iceberg tables, and secondary indexes.

Snowflake has announced general availability for the UUID data type in hybrid tables. This release enables data engineers to implement native, standard universally unique identifier schemas and primary keys within transactional hybrid table architectures.

Snowflake rolled out CoCo Desktop v1.23.1 alongside continuous and bulk file loading capabilities. Engineering updates also highlight Apache Iceberg table time travel and task graph enhancements.

Snowflake has announced the general availability of External OAuth for Snowflake Postgres. This security update allows data engineers to integrate external identity providers for role-based authentication and database access control across relational pipelines.

Snowflake released hybrid table support for Google Cloud in preview, expanding transactional database capabilities on GCP. Data engineers can now run operational workloads alongside analytics, complemented by general availability for secondary indexes and Apache Iceberg table upgrades.

DuckDB unifies fragmented data stacks into a single lakehouse foundation, offering warehouse-grade performance, live federation, strong storage economics, and agent-ready context.

Snowflake added support to carry over comments, tags, and privileges using the COPY = ( ... ) syntax. This enhances DDL workflows by automatically retaining critical object metadata and access controls during table replication.

Google Cloud details lakehouse interoperability for querying Databricks Apache Iceberg tables across environments. The architecture enables data engineers to run cross-platform analytics directly on open table formats without data movement.

MotherDuck introduced Dive Apps, allowing visualizations to trigger Flights, a serverless Python runtime. This architecture enables two-way data pipelines that execute write-backs into source systems directly from warehouse-backed interfaces.

ClickHouse introduced GroeneAI, an autonomous agent resolving CI test failures across the codebase. It reproduces bugs, builds ClickHouse, runs tests, and opens pull requests without humans in the loop, targeting database stability and continuous integration hygiene. Maintainers review each fix.

DuckDB eliminates big data operational complexity by querying S3, Parquet, and Iceberg via standard SQL without JVM clusters. DuckLake optimizes catalogs by storing metadata in a SQL database rather than nested files, cutting planning latency to under a millisecond with minimal configuration.

This edition covers data engineering architectures like JD.com building a lakestream with Apache Fluss and Apache Hudi, Azure SQL Database Hyperscale internals, DuckDB 2.0.0 features, and how Flink's Kafka connector commits offsets to Kafka.

Snowflake release notes detail platform updates, including Apache Iceberg table enhancements like Time Travel preview, secondary indexes, and DCM DEFINE TASK GRAPH for workflow orchestration. Other features include declarative feature stores and bulk file loading to optimize data pipelines.

Snowflake cost optimization spans warehouse sizing, clustering, and individual queries. While configurations tackle large inefficiencies, automated SQL tuning addresses the volume of everyday workloads. Greybeam automates this scale, targeting the sheer volume of queries run across warehouses.

Snowflake compute charges hinge on virtual warehouse sizing, billed per second with a 60-second minimum. Each size tier doubles cluster resources and credit burn.

Snowflake introduced QUERYATTRIBUTIONHISTORY to track compute costs per query, but testing reveals inflated numbers. Idle time is excluded, and warehouse IDs can mismatch. For precise attribution, engineers join QUERYHISTORY with WAREHOUSEMETERING_HISTORY and warehouse suspend events.

Snowflake's Query Profile exposes execution plans as operator trees to diagnose bottlenecks. Most query time is spent on TableScans or Joins, where predicate pushdown optimizes reads by filtering data early. Inspecting row counts and node stats enables targeted SQL optimization.

Snowflake ASOF JOINs optimize time-relative joins by replacing slow Cartesian-style range joins with partition-aware sorting and proximity searches. Using MATCH_CONDITION, it pairs records with the nearest timestamp, dropping query runtime from over 25 minutes down to 14 seconds.

Snowflake Iceberg tables decouple compute from storage by using external S3 volumes. This setup uses AWS IAM roles, STS, and external IDs to securely access parquet data and metadata, enabling cross-engine queries in DuckDB or Spark without proprietary storage lock-in.

DuckDB can query Snowflake-managed Iceberg tables on S3 using its httpfs and iceberg extensions. Users scan the table snapshot's metadata JSON directly via iceberg_scan, then materialize data locally to cut repeated query latency from 54 seconds to 1.5 seconds.

Data teams can query Snowflake tables locally in DuckDB using Arrow Database Connectivity (ADBC) without managing Iceberg tables. By avoiding row-column serialization found in ODBC, ADBC keeps data columnar throughout transfer, speeding up data movement by up to 38x.

Snowflake query optimization relies on minimizing table scans and compute overhead. Prune micro-partitions using early WHERE filters on clustered columns, avoid SELECT * to leverage column pruning, and prevent costly memory spilling to local or remote storage before upsizing warehouse compute.

Snowflake's MAXBY and MINBY aggregate functions simplify finding values associated with another column's extrema, replacing complex self-joins or window functions. Passing an optional third integer returns an array representing the top N values per group.

Snowflake Interactive Tables and Warehouses target low-latency serving layers using dedicated caching and clustering keys. However, pipelines face strict trade-offs: warehouses do not auto-suspend, tables lack UPDATE or DELETE support, and queries strictly fail after a hard 5-second timeout limit.

A new community DuckDB extension queries Snowflake directly via an ADBC driver without custom scripts. Engineers can attach Snowflake databases and enable filter pushdown to cut data transfer, joining warehouse tables directly with local Parquet files.

Greybeam transpiles Snowflake queries to DuckDB via SQLGlot, but anonymous bind variables lose their lexical order in the AST. To preserve ordering, they intercept the token stream before AST parsing, rewriting anonymous placeholders into prefixed, named bind variables for DuckDB execution.