Open table formats resources for data engineers

Replicate DynamoDB to Apache Iceberg on Amazon S3 Tables via Glue zero-ETL without impacting operational traffic. An AWS Lambda function embedding DuckDB executes fast vectorized SQL directly against S3 Tables metadata and Parquet files, returning JSON query results serverlessly.

🎙️ DataTalksClub ⬛ — Sahil Walia explains how open lakehouses and Apache Iceberg decouple storage from compute to transform modern data platforms. The session highlights how Iceberg metadata simplifies tasks like GDPR compliance, data deletions, and file compactions across different compute engines.

Build a local data lakehouse using an end-to-end open-source setup. Starting from an empty Docker Compose file, the guide walks engineers through implementing bronze, silver, and gold architectural layers along with Apache Iceberg time travel.

Snowflake announced the general availability of zero-copy support for all table formats in interactive warehouses. Data engineers can leverage metadata-driven cloning across formats, while additional platform updates include the Iceberg Scan Plan API and online constraint management for hybrid

RustFS launched open-source S3 Tables, embedding an Apache Iceberg REST Catalog into its storage kernel. One process on port 9000 serves S3 and Iceberg REST protocols.

Snowflake released the Iceberg Scan Plan API in Horizon Catalog in public preview. This enhances data lakehouse integration by exposing Iceberg table scan planning directly through the catalog.

Apache Iceberg Python 0.12.0 adds REST catalog view support and conflict-validated commit retries on concurrent writes. Data engineers can now use incremental append scans to read snapshot changes directly into Arrow, alongside native support for v3 geometry and geography types.

Google engineered a six-layer Apache Iceberg architecture combining engines, catalogs, and an autonomous control plane. It automates file compaction below 50% of 256 MB targets and federates external catalogs like Glue and Unity to query cross-cloud tables without copying files.

Open Data Infrastructure decouples compute and storage using open table formats like Apache Iceberg and Delta Lake on object storage. Storing data once provides transactional lake reliability, letting teams swap engines, avoid duplicate pipelines, and support AI without vendor lock-in.

In self-managed Apache Iceberg architectures, AWS Glue optimizer handles the same table maintenance tasks as Amazon S3 Tables for identical workloads, but at a significantly higher price point. For Iceberg maintenance, Glue optimizer proved to be 15 times more expensive than S3 Tables.

The Medallion Lakehouse architecture organizes data into Bronze, Silver, and Gold tiers using open table formats like Delta Lake and Iceberg. Bronze stores raw append-only data, Silver cleans and validates entities with ACID upserts, and Gold delivers aggregated star schemas for fast BI and ML

Kafka architectures often sink real-time events to S3 for long-term retention. However, raw files fail to behave like database tables for updates and consistent reads. Open table formats like Apache Iceberg bridge this gap by adding schema, snapshot, and partition metadata over the data.

DuckDB is evolving its analytical engine to handle modern data engineering demands. To enable community features, developers rewrote its SQL parser and built an extension mechanism. It is also introducing client-server architecture and DuckLake as an alternative table format to Apache Iceberg.

Snowflake demonstrates streaming over 1M TPS directly into Iceberg tables using the Snowpipe Streaming SDK. A server-side buffering tier handles batching and background compaction to optimize Parquet file sizes, while Iceberg v4 proposes an Adaptive Metadata Tree to reduce that write amplification.

🎙️ Spice AI — This showcase explores extensions to Apache DataFusion. RDF Fusion integrates SPARQL and piecewise signal querying to join graph models with IoT time series data. Additionally, Cloudflare demonstrates how "R2 SQL is Cloudflare's serverless, distributed, analytics query engine for Iceberg catalogs."

Snowflake previewed support to access externally managed Apache Iceberg tables through Snowflake Horizon Catalog. Recent updates also added Amazon S3 Tables Iceberg REST catalog integration and CREATE OR ALTER support for dynamic Iceberg tables.

Fivetran standardizes destination schemas with canonical types and metadata columns for portable SQL transformations. It provides table-level sync modes, including soft deletes and native SCD Type 2 history, while supporting open formats like Apache Iceberg directly in cloud storage.

Apache Hudi accelerates selective point lookups using queryable indexes stored in its metadata table. Instead of scanning entire tables, record-level and secondary indexes map keys to specific file groups during query planning, dropping a 400 GB table lookup from 977 seconds to 12 seconds.

Mastering Databricks requires core data engineering foundations: workspaces with Git integration, serverless compute, and Unity Catalog governance. Engineers should leverage Delta tables with liquid clustering, CDC patterns, and MERGE INTO commands to build robust, idempotent pipelines.

Enterprises are adopting zero-copy data architectures using open table formats like Apache Iceberg to query source data directly, eliminating redundant ETL pipelines and cutting storage costs. Embedded AI models process unstructured files and run conversational queries directly where data resides.

Snowflake added CREATE OR ALTER support for dynamic Apache Iceberg tables. This allows data engineers to declaratively deploy and update Iceberg table schemas and transformation pipelines in Lakehouse architectures without manual migration scripts.

Iceberg v3 introduces the Variant type for semi-structured data, encoding values as a binary Parquet group with metadata and payload subfields. Variant is as flexible as JSON but stores data in a compact, typed binary form. Spark and Flink support it for handling evolving, schema-free event schemas.

Open table formats like Apache Iceberg, Delta Lake, and Hudi bring database-like features to cheap object storage. DuckDB serves as a lightweight query engine to directly read these formats. "Open Table Format bundles distributed files into manageable tables with database-like features."

AWS S3 Tables and Cloudflare R2 reshape Lakehouse architectures via the ICE stack, decoupling compute and storage. Open table formats like Apache Iceberg enable database features over distributed files.

A data lakehouse combines object storage with table formats like Delta, Iceberg, and Hudi to enable ACID transactions, schema enforcement, and time travel. This transforms unstructured storage into structured analytics architecture, removing ETL bottlenecks while maintaining low costs.

Build a real-estate pipeline running on Kubernetes. Scrape property data, store it in MinIO S3, and process UPSERTs using Delta Lake. "Starting with web scraping gives you the power to treat every website as a database." Ingest data into Apache Druid and visualize it using Apache Superset.

Snowflake integration for Amazon S3 Tables via the Iceberg REST catalog is generally available. This update expands Snowflake architectural capabilities for managing external Apache Iceberg open table formats directly alongside S3 object storage infrastructure.

In 2026, experiments on Amazon S3 Tables reveal that automatic Iceberg compaction runs on an unconfigurable, fixed schedule roughly every 2.9 hours under continuous writes. The internal CLI and CloudTrail audit logs only record job execution timestamps without providing file count or byte metrics.

Open Data Infrastructure needs context engineering to unify data use cases across engines. A Managed Data Lake Service syncs open table formats, but AI systems lack tacit and tribal knowledge and require structured context for semantics, quality, provenance, and governance via dbt.

This end-to-end recommendation platform processes e-commerce events via PostgreSQL, MinIO, Debezium, and Kafka. Spark builds Iceberg tables while Flink processes real-time streams. Airflow orchestrates pipelines, feeding Feast with PostgreSQL offline and Redis online feature stores.