Open table formats resources for data engineers

Iceberg delete file types are determined by the engine implementation, not user configs. Spark batch writes only position deletes, while Flink writes both. Because equality deletes cannot be directly compacted, maintain them using rewritedatafiles with delete-file-threshold to purge stale rows.

Implement the streamhouse pattern locally using Apache Fluss, Apache Flink, and Apache Iceberg to enable direct querying of streaming data. This architecture integrates Fluss for real-time storage with Flink processing and Iceberg lakehouse tables.

Google Cloud details how Cloud Spanner powers its managed Apache Iceberg Lakehouse runtime catalog at scale, providing a robust architecture for high-concurrency metadata operations and distributed data lake analytics.

BigQuery now supports flexible column names by default on Iceberg external and managed tables. Additionally, the gemini-embedding-2 model is generally available for AI.EMBED, AI.SIMILARITY, and AI.GENERATE_EMBEDDING functions, enabling multimodal embeddings across text, audio, and video.

Apache Iceberg 1.12.0 expands semi-structured and geospatial data capabilities. Flink and Kafka Connect can now write shredded Variant columns alongside Spark. The release also adds a Flink maintenance task to convert equality deletes into positional deletion vectors for faster streaming reads.

Cloudflare Basin is a serverless data platform built on Apache Iceberg and R2. It pairs managed pipelines and metadata catalogs with a distributed SQL engine to ingest and query analytics data, letting engineers process open Iceberg tables with zero egress fees across external engines.

Embrasure Flow is a Rust service that streams PostgreSQL changes into Apache Iceberg. It captures initial COPY and logical replication, writing Parquet data with v2 position deletes or v3 deletion vectors to S3 storage via an Iceberg REST catalog, while handling local compaction.

Apache Iceberg Go 0.7.0 adds Iceberg V3 support, introducing deletion vectors, variant shredding, and geometry types. It also implements REST server-side scan planning delegation, incremental changelog scans, and full metadata tables to optimize distributed queries and table maintenance.

Aurora PostgreSQL now embeds DuckDB to directly query Apache Iceberg and Parquet in S3 without ETL pipelines. Using the aurora_analytics extension, it supports foreign tables with predicate pushdown and column pruning to join operational and lake data.

🎙️ Altertable — Taktile integrates DuckDB with Apache Iceberg to execute fast analytical workloads over an open lakehouse format. The architecture leverages DuckDB as an embedded OLAP query engine directly atop Iceberg tables, providing high-performance query execution and scalable data engineering patterns.

ClickHouse now writes Apache Iceberg tables directly to Microsoft OneLake via Iceberg REST-based Table APIs. This integration automatically registers tables in the OneLake catalog under native access controls, allowing downstream engines like Fabric Spark and Power BI to query ClickHouse output.

ClickHouse expanded its Microsoft collaboration with a native Fabric workload for sub-second queries directly on OneLake data. It also launched GA for OneLake read, public preview for OneLake write, and Bring Your Own Cloud availability in the Azure Marketplace.

BigQuery continuous queries now support writing output directly into Apache Iceberg managed tables using INSERT DML statements, enabling real-time streaming into open-format lakehouses. Additionally, the BigQuery Data Transfer Service MCP server is generally available.

Apache Iceberg C++ 0.4.0 adds Iceberg v3 support, including row lineage and deletion vectors. It exposes new table update APIs like RowDelta, adds parallel manifest operations and lazy scan planning, and introduces SigV4 and ResolvingFileIO to the REST catalog while standardizing on CMake builds.

Optimize transformation costs by storing data in open Apache Iceberg tables and routing workloads across multiple engines within a single dbt DAG. By pairing Snowflake with Lake Compute, data engineers can run lightweight staging models on cheaper compute while reserving warehouses for heavy

Snowflake announced the general availability of Snowpipe Streaming support for partitioned Apache Iceberg tables. Data engineers can now stream low-latency data directly into partitioned open table formats, aligning with recent updates like partition evolution and Elastic Channels.

Data engineering is shifting from rigid ETL toward autonomous pipelines and open lakehouses. Teams are adopting formats like Apache Iceberg and Delta Lake alongside Kafka and Flink for unified streaming, reducing manual SQL scripting in favor of data reliability engineering and FinOps.

dbt v2 ships with a built-in DuckDB adapter running on its Rust engine via ADBC drivers. The release adds DuckLake and Iceberg REST catalog support, plus an Information Schema feature that outputs project metadata as Parquet files, enabling fast SQL queries on manifests directly with DuckDB.

While open table formats like Iceberg decouple compute and storage, connecting client apps to diverse engines remains fractured. Apache Arrow and ADBC solve this by providing a standardized columnar API, eliminating costly row-to-column conversions and yielding up to 38x transfer speedups over ODBC.

Databricks liquid clustering replaces static partitioning and ZORDER with Hilbert curve layout via CLUSTER BY. Unlike full-table ZORDER rewrites, incremental OPTIMIZE only clusters new or modified files, minimizing write costs and avoiding small-file issues while speeding queries via file pruning.

The Apache Iceberg Europe Meetup in Copenhagen gathers data engineers to explore modern lakehouse architectures and open table formats. Co-hosted by Databricks, ECCO, and Vakamo, the technical sessions feature community presentations and discussions available via in-person access and live stream.

The Apache Iceberg Europe Meetup in Warsaw gathers data engineers to explore modern open table formats, cloud data architectures, and lakehouse patterns.

The Apache Iceberg Europe Meetup convenes data engineers to explore modern open table formats, storage architectures, and lakehouse implementations. Technical presentations cover data lake management, metadata operations, and integration patterns using Iceberg within enterprise data pipelines.

The Seattle Apache Iceberg Community Meetup centers on open-source data analytics and open table formats. Data engineers will explore technical innovations around Apache Iceberg, transactional data lakes, lakehouse architectures, and modern patterns for integrating generative AI and ML workloads.

The Apache Iceberg meetup explores core lakehouse data engineering patterns, focusing on Write-Audit-Publish workflows via branching and tagging. Sessions address migrating from Hive Metastore to Lakekeeper REST catalogs for multi-tenant isolation, plus deploying Iceberg engines on Kubernetes.

This meetup covers GPU acceleration for Apache Spark, Delta Lake, and Apache Iceberg, speeding join-heavy pipelines and the Medallion architecture's Gold layer.

The Apache Iceberg meetup in Menlo Park gathers data engineers to explore technical developments, architecture, and practical use cases across the open table format ecosystem.

Apache Polaris advances open lakehouse architectures through integrations like Apache Ossie, which manages shared semantic models alongside physical tables and views. Additionally, a leaf-wins hierarchical resolution model enables dynamic, path-level storage overrides at namespace and table levels.

Engineers explore Apache Flink in production for real-time streaming architectures. Adyen shares how Flink powers real-time feature computation for machine learning in payment processing.