Lakehouse resources for data engineers

DuckDB on an iPhone outperformed Databricks Serverless SQL on TPC-H workloads up to 100 GB. Single-node execution engines avoid the query planning and network shuffle overhead inherent to distributed clusters, making them far more cost-effective for typical real-world business datasets.

The platform positions lakehouse architecture as a unified system for data engineering, analytics, and AI. It supports batch and streaming ETL, orchestration, and serverless data warehousing for SQL workloads, paired with unified governance and open data sharing across clouds.

Right-sizing analytics pipelines prevents early technical debt while avoiding premature distributed complexity. Starting with single-node tools like Polars, Postgres, and DuckDB enables high performance before migrating to Databricks or Snowflake.

Implement the streamhouse pattern locally using Apache Fluss, Apache Flink, and Apache Iceberg to enable direct querying of streaming data. This architecture integrates Fluss for real-time storage with Flink processing and Iceberg lakehouse tables.

Google Cloud details how Cloud Spanner powers its managed Apache Iceberg Lakehouse runtime catalog at scale, providing a robust architecture for high-concurrency metadata operations and distributed data lake analytics.

Production DuckLake architectures require deliberate maintenance. Because the catalog tracks data files and inlines inserts under 10 rows in a database, engineers must periodically flush inlined data to Parquet and run file compaction to prevent performance bottlenecks.

Cousteau is a lakehouse built to simplify fragmented data stacks, unifying warehouse performance, storage economics, live data federation, and agent-ready context into a single foundation.

Streamhouse shifts data architectures from topic-centric pipelines to shared, table-centric storage. Powered by Lakestream, it unifies real-time streaming and historical lakehouse data into single logical tables, preventing redundant state maintenance across operational and analytical workloads.

Apache Iceberg 1.12.0 expands semi-structured and geospatial data capabilities. Flink and Kafka Connect can now write shredded Variant columns alongside Spark. The release also adds a Flink maintenance task to convert equality deletes into positional deletion vectors for faster streaming reads.

Cloudflare Basin is a serverless data platform built on Apache Iceberg and R2. It pairs managed pipelines and metadata catalogs with a distributed SQL engine to ingest and query analytics data, letting engineers process open Iceberg tables with zero egress fees across external engines.

Aurora PostgreSQL now embeds DuckDB to directly query Apache Iceberg and Parquet in S3 without ETL pipelines. Using the aurora_analytics extension, it supports foreign tables with predicate pushdown and column pruning to join operational and lake data.

Data teams often suffer when pipelines run under personal user accounts, causing failures when staff depart. Phase 1 relies on manual UI ClickOps and personal tokens, while Phase 2 introduces Git folders but still suffers from manual promotion, schema drift, and unautomated testing friction.

🎙️ Altertable — Taktile integrates DuckDB with Apache Iceberg to execute fast analytical workloads over an open lakehouse format. The architecture leverages DuckDB as an embedded OLAP query engine directly atop Iceberg tables, providing high-performance query execution and scalable data engineering patterns.

ClickHouse now writes Apache Iceberg tables directly to Microsoft OneLake via Iceberg REST-based Table APIs. This integration automatically registers tables in the OneLake catalog under native access controls, allowing downstream engines like Fabric Spark and Power BI to query ClickHouse output.

ClickHouse expanded its Microsoft collaboration with a native Fabric workload for sub-second queries directly on OneLake data. It also launched GA for OneLake read, public preview for OneLake write, and Bring Your Own Cloud availability in the Azure Marketplace.

Databricks introduces a FILE data type to integrate unstructured multimodal data like audio and video directly into the Lakehouse. For each row, the FILE type stores metadata and a governed link to the file in storage.

Agentic AI requires reliable business context over raw compute. About 80% of enterprises cite data quality as their main obstacle. Engineering open, governed data pipelines prevents critical failures, as context is lost when trapped inside proprietary stacks.

🎙️ André Noujaïm — Combining DuckDB with dbt enables a lightweight approach to analytics engineering that reduces the need for complex, traditional data pipelines. By moving compute closer to storage and leveraging live federation, teams can simplify transformations within an AI-native lakehouse architecture.

Optimize transformation costs by storing data in open Apache Iceberg tables and routing workloads across multiple engines within a single dbt DAG. By pairing Snowflake with Lake Compute, data engineers can run lightweight staging models on cheaper compute while reserving warehouses for heavy

The technical sessions focus on building production data infrastructure for human and AI-driven queries. Topics include real-time analytics architectures, open lakehouse designs, serving layers, and workload isolation, addressing query performance, concurrency, and scale for agentic workflows.

Databricks launched Lakebase, combining serverless Postgres with open lakehouse storage to blur OLTP and OLAP. By enabling instant branching, scale-to-zero compute, and native zero-ETL access for Spark and Delta Lake, it removes brittle CDC pipelines and offers unified governance via Unity Catalog.

Data engineering is shifting from rigid ETL toward autonomous pipelines and open lakehouses. Teams are adopting formats like Apache Iceberg and Delta Lake alongside Kafka and Flink for unified streaming, reducing manual SQL scripting in favor of data reliability engineering and FinOps.

While open table formats like Iceberg decouple compute and storage, connecting client apps to diverse engines remains fractured. Apache Arrow and ADBC solve this by providing a standardized columnar API, eliminating costly row-to-column conversions and yielding up to 38x transfer speedups over ODBC.

Databricks liquid clustering replaces static partitioning and ZORDER with Hilbert curve layout via CLUSTER BY. Unlike full-table ZORDER rewrites, incremental OPTIMIZE only clusters new or modified files, minimizing write costs and avoiding small-file issues while speeding queries via file pruning.

Build a local real-time pipeline by publishing raw events to Kafka, deduplicating them with Flink SQL, and streaming the results into DuckLake. A consumer writes batches every 50 records or 5 seconds, using DuckLake row inlining to avoid small Parquet files.

Apache Fluss 1.0 delivers real-time storage for streaming and AI workloads. It adds a Rust-based HTTP Gateway for REST operations and lakehouse ingestion, native Python clients for feature lookups, and support for Flink, Spark, Paimon, and Apache Hudi.

The Apache Iceberg Europe Meetup in Copenhagen gathers data engineers to explore modern lakehouse architectures and open table formats. Co-hosted by Databricks, ECCO, and Vakamo, the technical sessions feature community presentations and discussions available via in-person access and live stream.

The Apache Iceberg Europe Meetup in Warsaw gathers data engineers to explore modern open table formats, cloud data architectures, and lakehouse patterns.

The Apache Iceberg Europe Meetup convenes data engineers to explore modern open table formats, storage architectures, and lakehouse implementations. Technical presentations cover data lake management, metadata operations, and integration patterns using Iceberg within enterprise data pipelines.