Open table formats resources for data engineers

Snowflake Horizon Catalog has reached general availability for accessing externally managed Apache Iceberg tables. The release also adds general availability for Horizon Iceberg REST Catalog telemetry, improving external catalog integrations and data lakehouse architectures.

Snowflake announced the general availability of Horizon Iceberg REST Catalog telemetry, expanding monitoring for Apache Iceberg metadata operations. Related updates include general availability for Apache Iceberg v3 table upgrades and Tri-Secret Secure storage support.

Snowflake has released general availability for Tri-Secret Secure on Snowflake storage for Apache Iceberg tables. Data engineers can now leverage enhanced customer-managed encryption key architectures for Iceberg table storage directly within Snowflake, improving security and compliance.

Google Cloud details lakehouse interoperability for querying Databricks Apache Iceberg tables across environments. The architecture enables data engineers to run cross-platform analytics directly on open table formats without data movement.

Snowflake Iceberg tables decouple compute from storage by using external S3 volumes. This setup uses AWS IAM roles, STS, and external IDs to securely access parquet data and metadata, enabling cross-engine queries in DuckDB or Spark without proprietary storage lock-in.

DuckDB can query Snowflake-managed Iceberg tables on S3 using its httpfs and iceberg extensions. Users scan the table snapshot's metadata JSON directly via iceberg_scan, then materialize data locally to cut repeated query latency from 54 seconds to 1.5 seconds.

Using pyIceberg with AWS Glue streamlines data lake management on S3. Configuring client properties allows shared credentials across Glue and S3 to define schemas, create tables, and query Iceberg metadata directly using DuckDB.

Snowflake Iceberg tables store Parquet files and metadata in object storage, allowing external engines to query them without copying data. Snowflake can serve as the catalog to handle compaction and snapshots, or connect to external catalogs like AWS Glue via external volumes.

Snowflake-managed storage simplifies Iceberg tables via EXTERNALVOLUME = 'SNOWFLAKEMANAGED', removing manual cloud bucket setup. Through the Horizon catalog, external engines like DuckDB can query and write directly to the managed files using vended credentials.

Snowflake introduced a preview of Time Travel for Apache Iceberg tables. This feature expands Iceberg support by allowing data engineers to query historical table snapshots directly. Related recent updates include Apache Iceberg v3 table upgrades and recompaction options after file size increases.

Lake Compute lets data teams run low-complexity dbt models on cheaper lake engines instead of expensive data warehouses. Built on Apache Iceberg and open REST catalogs, this architecture cuts compute costs model by model without disrupting downstream DAGs or BI tools.

Snowflake Horizon now supports external Apache Iceberg writes natively. Coupled with dbt catalog configs and DuckDB upstream fixes, teams can build Iceberg tables in object storage and query governed data locally in DuckDB without touching Snowflake compute warehouses.

A catalogs.yml v2 spec decouples dbt Iceberg configurations from engine-specific storage, enabling cross-platform dbt Mesh across Snowflake and Databricks. Catalog federation lets engines query shared Iceberg tables without data duplication, while DuckDB routes specific models.

Data lakehouse query performance degrades when small files, manifest bloat, and granular partitions overload metadata paths. Table formats like Apache Iceberg and Delta Lake require active maintenance, such as file compaction and snapshot cleanup, to keep query planning efficient.

Open formats like Apache Iceberg decouple storage from compute, letting teams match engines like DuckDB or Snowflake to specific workloads. True interoperability requires unified governance through standards like the Iceberg REST protocol so policies travel with data across engines.

Snowflake released an update allowing data engineers to recompact Iceberg table data after target file size increases. This maintenance process optimizes storage layout and query efficiency on Apache Iceberg tables by merging smaller files into larger targets.

DuckLake 1.1 adds nested schemas, Iceberg-aligned epoch partition transforms, and exact string statistics to avoid full scans. Catalogs now store Parquet row group counts directly, letting engines plan distributed scans without fetching file footers from object storage.

Snowflake released Parquet Direct in preview alongside Apache Iceberg v3 table upgrades. The release note highlights features for data pipelines, including Openflow Endpoints, expanded external table data lineage, and Snowpipe Streaming support for partitioned Iceberg tables.

You can run a single dbt project across Snowflake and DuckDB using shared Iceberg tables in Snowflake-managed storage. Horizon provides an Iceberg REST catalog with vended credentials, letting local DuckDB read and write tables directly without managing an external S3 bucket or IAM roles.

Delta Lake serves as a standard for ACID lakehouse storage, but coordinating diverse ecosystem tools presents challenges. Delta Kernel addresses this integration layer, enabling developers to interact directly with Delta tables using Rust for custom data engineering tasks.

Build data lakehouse quality gates across bronze, silver, and gold tiers using Soda contracts alongside Apache Iceberg branches to enforce strict validation rules before publication.

Snowflake has announced the general availability of Apache Iceberg v3 table upgrades and expression defaults. Data engineers can now leverage Iceberg v3 specification capabilities directly within Snowflake to modernize lakehouse architectures and manage table schema default values.

🎙️ vutr — Open table formats like Apache Iceberg, Delta Lake, and Apache Hudi power modern lakehouse architectures. Tracing a sales table from warehouses through data lakes and Apache Hive highlights what each storage layer offered and cost.

Ridgeline unified its investment analytics by migrating downstream performance queries to ClickHouse Cloud on AWS. Facing scale limits with Postgres, the team adopted ClickHouse for fast, accurate OLAP querying over alternatives like Redshift and SingleStore to power real-time reporting.

Construct leak-free graphs on lakehouse gold tables using Apache Iceberg tags, LadybugDB, and an additional Cypher line. This architecture ensures knowledge graph queries limit context strictly to what the model knew at decision time, preventing data leakage.

An Iceberg table is a tree of JSON metadata, Avro manifest files, and Parquet data files. DuckDB can query each layer using SQL to trace table state. Because table files are immutable, writes create new files and update the catalog pointer to commit snapshots.

Amazon Redshift automatically resolves concurrent write conflicts on Apache Iceberg tables by rebasing independent commits. Instead of aborting failed optimistic concurrency checks, Redshift verifies if writes overlap across partitions or files, committing non-conflicting changes without retries.

Amazon EMR Data Quality natively integrates AWS Glue Data Quality into EMR Spark jobs using Data Quality Definition Language. Bundled in runtime 8.1.0 and later, it lets engineers run managed DQDL checks or ML anomaly detection directly within EMR on EC2, Serverless, and EKS without custom

Google Cloud Lakehouse launched public datasets, including Wikipedia pageviews, using Apache Iceberg on Parquet. Engineers can inspect metadata via PyIceberg or run queries across engines like Spark, BigQuery, and Trino using its REST catalog to test open lakehouse patterns without lock-in.

Snowflake announced general availability for backups of Snowflake-managed Apache Iceberg tables on Snowflake storage. This release improves data recovery architectures by ensuring native protection for open table formats. Snowflake also added database roles support to backups.