Iceberg resources for data engineers

Snowflake published its October 8, 2026 feature update for Snowflake Data Clean Rooms alongside broader releases, including Snowpipe Streaming for partitioned Apache Iceberg tables and incremental refresh support for dynamic tables using the VALUES clause.

Iceberg delete file types are determined by the engine implementation, not user configs. Spark batch writes only position deletes, while Flink writes both. Because equality deletes cannot be directly compacted, maintain them using rewritedatafiles with delete-file-threshold to purge stale rows.

Right-sizing analytics pipelines prevents early technical debt while avoiding premature distributed complexity. Starting with single-node tools like Polars, Postgres, and DuckDB enables high performance before migrating to Databricks or Snowflake.

Implement the streamhouse pattern locally using Apache Fluss, Apache Flink, and Apache Iceberg to enable direct querying of streaming data. This architecture integrates Fluss for real-time storage with Flink processing and Iceberg lakehouse tables.

Google Cloud details how Cloud Spanner powers its managed Apache Iceberg Lakehouse runtime catalog at scale, providing a robust architecture for high-concurrency metadata operations and distributed data lake analytics.

BigQuery now supports flexible column names by default on Iceberg external and managed tables. Additionally, the gemini-embedding-2 model is generally available for AI.EMBED, AI.SIMILARITY, and AI.GENERATE_EMBEDDING functions, enabling multimodal embeddings across text, audio, and video.

Amazon Aurora PostgreSQL now supports direct querying of Apache Iceberg and Parquet data lake files. Other data engineering highlights include extending Apache DataFusion across machines, using ClickHouse for metric discovery at LinkedIn, and DuckDB 2.0 adding dynamic ATTACH statements.

Apache Iceberg 1.12.0 expands semi-structured and geospatial data capabilities. Flink and Kafka Connect can now write shredded Variant columns alongside Spark. The release also adds a Flink maintenance task to convert equality deletes into positional deletion vectors for faster streaming reads.

Cloudflare Basin is a serverless data platform built on Apache Iceberg and R2. It pairs managed pipelines and metadata catalogs with a distributed SQL engine to ingest and query analytics data, letting engineers process open Iceberg tables with zero egress fees across external engines.

Embrasure Flow is a Rust service that streams PostgreSQL changes into Apache Iceberg. It captures initial COPY and logical replication, writing Parquet data with v2 position deletes or v3 deletion vectors to S3 storage via an Iceberg REST catalog, while handling local compaction.

Apache Iceberg Go 0.7.0 adds Iceberg V3 support, introducing deletion vectors, variant shredding, and geometry types. It also implements REST server-side scan planning delegation, incremental changelog scans, and full metadata tables to optimize distributed queries and table maintenance.

Aurora PostgreSQL now embeds DuckDB to directly query Apache Iceberg and Parquet in S3 without ETL pipelines. Using the aurora_analytics extension, it supports foreign tables with predicate pushdown and column pruning to join operational and lake data.

🎙️ Altertable — Taktile integrates DuckDB with Apache Iceberg to execute fast analytical workloads over an open lakehouse format. The architecture leverages DuckDB as an embedded OLAP query engine directly atop Iceberg tables, providing high-performance query execution and scalable data engineering patterns.

ClickHouse now writes Apache Iceberg tables directly to Microsoft OneLake via Iceberg REST-based Table APIs. This integration automatically registers tables in the OneLake catalog under native access controls, allowing downstream engines like Fabric Spark and Power BI to query ClickHouse output.

ClickHouse expanded its Microsoft collaboration with a native Fabric workload for sub-second queries directly on OneLake data. It also launched GA for OneLake read, public preview for OneLake write, and Bring Your Own Cloud availability in the Azure Marketplace.

BigQuery continuous queries now support writing output directly into Apache Iceberg managed tables using INSERT DML statements, enabling real-time streaming into open-format lakehouses. Additionally, the BigQuery Data Transfer Service MCP server is generally available.

Pinterest detailed partition finalization in its database ingestion framework, and Airbnb extended its real-time sequence recommender with Chronon. Other updates cover Apache Iceberg portable view metadata across SQL engines and adaptive lossless floating-point encoding in Apache Parquet.

Apache Iceberg C++ 0.4.0 adds Iceberg v3 support, including row lineage and deletion vectors. It exposes new table update APIs like RowDelta, adds parallel manifest operations and lazy scan planning, and introduces SigV4 and ResolvingFileIO to the REST catalog while standardizing on CMake builds.

Optimize transformation costs by storing data in open Apache Iceberg tables and routing workloads across multiple engines within a single dbt DAG. By pairing Snowflake with Lake Compute, data engineers can run lightweight staging models on cheaper compute while reserving warehouses for heavy

Snowflake announced the general availability of Snowpipe Streaming support for partitioned Apache Iceberg tables. Data engineers can now stream low-latency data directly into partitioned open table formats, aligning with recent updates like partition evolution and Elastic Channels.

Data engineering is shifting from rigid ETL toward autonomous pipelines and open lakehouses. Teams are adopting formats like Apache Iceberg and Delta Lake alongside Kafka and Flink for unified streaming, reducing manual SQL scripting in favor of data reliability engineering and FinOps.

dbt v2 ships with a built-in DuckDB adapter running on its Rust engine via ADBC drivers. The release adds DuckLake and Iceberg REST catalog support, plus an Information Schema feature that outputs project metadata as Parquet files, enabling fast SQL queries on manifests directly with DuckDB.

While open table formats like Iceberg decouple compute and storage, connecting client apps to diverse engines remains fractured. Apache Arrow and ADBC solve this by providing a standardized columnar API, eliminating costly row-to-column conversions and yielding up to 38x transfer speedups over ODBC.

Databricks liquid clustering replaces static partitioning and ZORDER with Hilbert curve layout via CLUSTER BY. Unlike full-table ZORDER rewrites, incremental OPTIMIZE only clusters new or modified files, minimizing write costs and avoiding small-file issues while speeding queries via file pruning.

Data engineers face a shifting landscape where OLTP and OLAP systems collide. Key topics include using Apache Iceberg as an interoperability layer, scaling CDC pipelines to process three billion monthly updates, and rethinking application reporting architectures beyond standard query tuning.

The Apache Iceberg Europe Meetup in Copenhagen gathers data engineers to explore modern lakehouse architectures and open table formats. Co-hosted by Databricks, ECCO, and Vakamo, the technical sessions feature community presentations and discussions available via in-person access and live stream.

The Apache Iceberg Europe Meetup in Warsaw gathers data engineers to explore modern open table formats, cloud data architectures, and lakehouse patterns.

The Apache Iceberg Europe Meetup convenes data engineers to explore modern open table formats, storage architectures, and lakehouse implementations. Technical presentations cover data lake management, metadata operations, and integration patterns using Iceberg within enterprise data pipelines.

The Seattle Apache Iceberg Community Meetup centers on open-source data analytics and open table formats. Data engineers will explore technical innovations around Apache Iceberg, transactional data lakes, lakehouse architectures, and modern patterns for integrating generative AI and ML workloads.