Lakehouse resources for data engineers

Snowflake Horizon Catalog has reached general availability for accessing externally managed Apache Iceberg tables. The release also adds general availability for Horizon Iceberg REST Catalog telemetry, improving external catalog integrations and data lakehouse architectures.

Google Cloud details lakehouse interoperability for querying Databricks Apache Iceberg tables across environments. The architecture enables data engineers to run cross-platform analytics directly on open table formats without data movement.

DuckDB eliminates big data operational complexity by querying S3, Parquet, and Iceberg via standard SQL without JVM clusters. DuckLake optimizes catalogs by storing metadata in a SQL database rather than nested files, cutting planning latency to under a millisecond with minimal configuration.

MiniLake is a local Databricks API emulator that executes real SQL on DuckDB, writes genuine Delta files, and runs containerized spark-submit jobs. Designed for CI and local development without cloud costs, it natively supports Unity Catalog CRUD, DAG job scheduling, and official SDK clients.

Snowflake Iceberg tables decouple compute from storage by using external S3 volumes. This setup uses AWS IAM roles, STS, and external IDs to securely access parquet data and metadata, enabling cross-engine queries in DuckDB or Spark without proprietary storage lock-in.

DuckDB can query Snowflake-managed Iceberg tables on S3 using its httpfs and iceberg extensions. Users scan the table snapshot's metadata JSON directly via iceberg_scan, then materialize data locally to cut repeated query latency from 54 seconds to 1.5 seconds.

Using pyIceberg with AWS Glue streamlines data lake management on S3. Configuring client properties allows shared credentials across Glue and S3 to define schemas, create tables, and query Iceberg metadata directly using DuckDB.

Snowflake Iceberg tables store Parquet files and metadata in object storage, allowing external engines to query them without copying data. Snowflake can serve as the catalog to handle compaction and snapshots, or connect to external catalogs like AWS Glue via external volumes.

Snowflake-managed storage simplifies Iceberg tables via EXTERNALVOLUME = 'SNOWFLAKEMANAGED', removing manual cloud bucket setup. Through the Horizon catalog, external engines like DuckDB can query and write directly to the managed files using vended credentials.

The platform unifies fragmented data stacks into a single lakehouse architecture. It delivers warehouse-grade performance, storage economics, live data federation, and agent-ready context.

Lake Compute lets data teams run low-complexity dbt models on cheaper lake engines instead of expensive data warehouses. Built on Apache Iceberg and open REST catalogs, this architecture cuts compute costs model by model without disrupting downstream DAGs or BI tools.

Snowflake Horizon now supports external Apache Iceberg writes natively. Coupled with dbt catalog configs and DuckDB upstream fixes, teams can build Iceberg tables in object storage and query governed data locally in DuckDB without touching Snowflake compute warehouses.

A catalogs.yml v2 spec decouples dbt Iceberg configurations from engine-specific storage, enabling cross-platform dbt Mesh across Snowflake and Databricks. Catalog federation lets engines query shared Iceberg tables without data duplication, while DuckDB routes specific models.

This unified lakehouse replaces fragmented data stacks by providing warehouse-grade performance, low-cost storage economics, live federation, and agent-ready context on a single foundation.

JD.com built a Lakestream using Apache Fluss and Apache Hudi to unify streaming and lakehouse data. Fresh data in Fluss tables tiers directly into Hudi, cutting batch compute costs over 80% and storage by 50% while ending duplicate pipelines and reducing wasted network bandwidth from row-based

Data lakehouse query performance degrades when small files, manifest bloat, and granular partitions overload metadata paths. Table formats like Apache Iceberg and Delta Lake require active maintenance, such as file compaction and snapshot cleanup, to keep query planning efficient.

Open formats like Apache Iceberg decouple storage from compute, letting teams match engines like DuckDB or Snowflake to specific workloads. True interoperability requires unified governance through standards like the Iceberg REST protocol so policies travel with data across engines.

Databricks ingestion should prioritize ownership over tool names. Teams risk custom pipeline chaos by source forty. Evaluate options sequentially: query in place via Lakehouse Federation or Delta Sharing, leverage existing tools like Fivetran, or use native serverless Lakeflow Connect connectors.

DuckLake 1.1 adds nested schemas, Iceberg-aligned epoch partition transforms, and exact string statistics to avoid full scans. Catalogs now store Parquet row group counts directly, letting engines plan distributed scans without fetching file footers from object storage.

Snowflake released Parquet Direct in preview alongside Apache Iceberg v3 table upgrades. The release note highlights features for data pipelines, including Openflow Endpoints, expanded external table data lineage, and Snowpipe Streaming support for partitioned Iceberg tables.

You can run a single dbt project across Snowflake and DuckDB using shared Iceberg tables in Snowflake-managed storage. Horizon provides an Iceberg REST catalog with vended credentials, letting local DuckDB read and write tables directly without managing an external S3 bucket or IAM roles.

Delta Lake serves as a standard for ACID lakehouse storage, but coordinating diverse ecosystem tools presents challenges. Delta Kernel addresses this integration layer, enabling developers to interact directly with Delta tables using Rust for custom data engineering tasks.

Build data lakehouse quality gates across bronze, silver, and gold tiers using Soda contracts alongside Apache Iceberg branches to enforce strict validation rules before publication.

DuckDB 2.0 introduces a PEG parser to eliminate shift-reduce conflicts and allow dynamic grammar extensions. For lakehouse pipelines, DuckLake now supports file cleanup routines like purging orphaned Parquet files from S3 and flushing inlined catalog writes directly to storage.

Snowflake has announced the general availability of Apache Iceberg v3 table upgrades and expression defaults. Data engineers can now leverage Iceberg v3 specification capabilities directly within Snowflake to modernize lakehouse architectures and manage table schema default values.

🎙️ vutr — Open table formats like Apache Iceberg, Delta Lake, and Apache Hudi power modern lakehouse architectures. Tracing a sales table from warehouses through data lakes and Apache Hive highlights what each storage layer offered and cost.

OLCF is an open-source lakehouse platform creating an Apache Spark Standalone cluster across custom machines, cloud VMs, or Kubernetes. It extends Microsoft Fabric by using olcf.run_remote() to offload notebook compute to owned Spark clusters while directly reading and writing OneLake.

Fivetran introduced LakeCompute, letting dbt users run transformations on Iceberg tables via DuckDB compute instead of Snowflake. By managing the landing layer and compute, Fivetran disintermediates the data warehouse to cut query costs, shifting workloads away from Snowflake.

Google Cloud is integrating its borderless lakehouse architecture with Gemini to enable agentic workflows across data analytics systems. The initiative focuses on bridging data lakehouse environments with AI tooling for enhanced data engineering.

AI agents excel at analytical tasks, but require modeled data and open infrastructure to work. Instead of wiring agents directly into SaaS APIs via protocols like MCP, teams should centralize business data in open table formats like Iceberg to ensure control, clean metadata, and composability.