Iceberg resources for data engineers

Snowflake release updates highlight web research in Snowflake CoWork alongside storage and table enhancements. Data engineers gain General Availability for Tri-Secret Secure on Apache Iceberg tables and UUID data types in hybrid tables, plus GCP support for hybrid tables in Preview.

Snowflake Horizon Catalog has reached general availability for accessing externally managed Apache Iceberg tables. The release also adds general availability for Horizon Iceberg REST Catalog telemetry, improving external catalog integrations and data lakehouse architectures.

Snowflake announced the general availability of Horizon Iceberg REST Catalog telemetry, expanding monitoring for Apache Iceberg metadata operations. Related updates include general availability for Apache Iceberg v3 table upgrades and Tri-Secret Secure storage support.

Snowflake has released general availability for Tri-Secret Secure on Snowflake storage for Apache Iceberg tables. Data engineers can now leverage enhanced customer-managed encryption key architectures for Iceberg table storage directly within Snowflake, improving security and compliance.

Google Cloud details lakehouse interoperability for querying Databricks Apache Iceberg tables across environments. The architecture enables data engineers to run cross-platform analytics directly on open table formats without data movement.

DuckDB eliminates big data operational complexity by querying S3, Parquet, and Iceberg via standard SQL without JVM clusters. DuckLake optimizes catalogs by storing metadata in a SQL database rather than nested files, cutting planning latency to under a millisecond with minimal configuration.

Snowflake Iceberg tables decouple compute from storage by using external S3 volumes. This setup uses AWS IAM roles, STS, and external IDs to securely access parquet data and metadata, enabling cross-engine queries in DuckDB or Spark without proprietary storage lock-in.

DuckDB can query Snowflake-managed Iceberg tables on S3 using its httpfs and iceberg extensions. Users scan the table snapshot's metadata JSON directly via iceberg_scan, then materialize data locally to cut repeated query latency from 54 seconds to 1.5 seconds.

Using pyIceberg with AWS Glue streamlines data lake management on S3. Configuring client properties allows shared credentials across Glue and S3 to define schemas, create tables, and query Iceberg metadata directly using DuckDB.

Snowflake Iceberg tables store Parquet files and metadata in object storage, allowing external engines to query them without copying data. Snowflake can serve as the catalog to handle compaction and snapshots, or connect to external catalogs like AWS Glue via external volumes.

Snowflake-managed storage simplifies Iceberg tables via EXTERNALVOLUME = 'SNOWFLAKEMANAGED', removing manual cloud bucket setup. Through the Horizon catalog, external engines like DuckDB can query and write directly to the managed files using vended credentials.

Snowflake introduced a preview of Time Travel for Apache Iceberg tables. This feature expands Iceberg support by allowing data engineers to query historical table snapshots directly. Related recent updates include Apache Iceberg v3 table upgrades and recompaction options after file size increases.

Automated Snowpipe setup with Snowflake CoCo is now GA. It automatically samples files in cloud storage, infers schemas, and builds continuous ingestion pipelines into standard or Iceberg tables. It can also generate complete Terraform configurations for infrastructure as code.

Lake Compute lets data teams run low-complexity dbt models on cheaper lake engines instead of expensive data warehouses. Built on Apache Iceberg and open REST catalogs, this architecture cuts compute costs model by model without disrupting downstream DAGs or BI tools.

Snowflake Horizon now supports external Apache Iceberg writes natively. Coupled with dbt catalog configs and DuckDB upstream fixes, teams can build Iceberg tables in object storage and query governed data locally in DuckDB without touching Snowflake compute warehouses.

A catalogs.yml v2 spec decouples dbt Iceberg configurations from engine-specific storage, enabling cross-platform dbt Mesh across Snowflake and Databricks. Catalog federation lets engines query shared Iceberg tables without data duplication, while DuckDB routes specific models.

Data lakehouse query performance degrades when small files, manifest bloat, and granular partitions overload metadata paths. Table formats like Apache Iceberg and Delta Lake require active maintenance, such as file compaction and snapshot cleanup, to keep query planning efficient.

Open formats like Apache Iceberg decouple storage from compute, letting teams match engines like DuckDB or Snowflake to specific workloads. True interoperability requires unified governance through standards like the Iceberg REST protocol so policies travel with data across engines.

Snowflake released an update allowing data engineers to recompact Iceberg table data after target file size increases. This maintenance process optimizes storage layout and query efficiency on Apache Iceberg tables by merging smaller files into larger targets.

Snowflake released Parquet Direct in preview alongside Apache Iceberg v3 table upgrades. The release note highlights features for data pipelines, including Openflow Endpoints, expanded external table data lineage, and Snowpipe Streaming support for partitioned Iceberg tables.

You can run a single dbt project across Snowflake and DuckDB using shared Iceberg tables in Snowflake-managed storage. Horizon provides an Iceberg REST catalog with vended credentials, letting local DuckDB read and write tables directly without managing an external S3 bucket or IAM roles.

Build data lakehouse quality gates across bronze, silver, and gold tiers using Soda contracts alongside Apache Iceberg branches to enforce strict validation rules before publication.

Snowflake has announced the general availability of Apache Iceberg v3 table upgrades and expression defaults. Data engineers can now leverage Iceberg v3 specification capabilities directly within Snowflake to modernize lakehouse architectures and manage table schema default values.

🎙️ vutr — Open table formats like Apache Iceberg, Delta Lake, and Apache Hudi power modern lakehouse architectures. Tracing a sales table from warehouses through data lakes and Apache Hive highlights what each storage layer offered and cost.

Fivetran introduced LakeCompute, letting dbt users run transformations on Iceberg tables via DuckDB compute instead of Snowflake. By managing the landing layer and compute, Fivetran disintermediates the data warehouse to cut query costs, shifting workloads away from Snowflake.

Ridgeline unified its investment analytics by migrating downstream performance queries to ClickHouse Cloud on AWS. Facing scale limits with Postgres, the team adopted ClickHouse for fast, accurate OLAP querying over alternatives like Redshift and SingleStore to power real-time reporting.

Construct leak-free graphs on lakehouse gold tables using Apache Iceberg tags, LadybugDB, and an additional Cypher line. This architecture ensures knowledge graph queries limit context strictly to what the model knew at decision time, preventing data leakage.

An Iceberg table is a tree of JSON metadata, Avro manifest files, and Parquet data files. DuckDB can query each layer using SQL to trace table state. Because table files are immutable, writes create new files and update the catalog pointer to commit snapshots.

Amazon Redshift automatically resolves concurrent write conflicts on Apache Iceberg tables by rebasing independent commits. Instead of aborting failed optimistic concurrency checks, Redshift verifies if writes overlap across partitions or files, committing non-conflicting changes without retries.

Amazon EMR Data Quality natively integrates AWS Glue Data Quality into EMR Spark jobs using Data Quality Definition Language. Bundled in runtime 8.1.0 and later, it lets engineers run managed DQDL checks or ML anomaly detection directly within EMR on EC2, Serverless, and EKS without custom