Lakehouse resources for data engineers

Replicate DynamoDB to Apache Iceberg on Amazon S3 Tables via Glue zero-ETL without impacting operational traffic. An AWS Lambda function embedding DuckDB executes fast vectorized SQL directly against S3 Tables metadata and Parquet files, returning JSON query results serverlessly.

🎙️ DataTalksClub ⬛ — Sahil Walia explains how open lakehouses and Apache Iceberg decouple storage from compute to transform modern data platforms. The session highlights how Iceberg metadata simplifies tasks like GDPR compliance, data deletions, and file compactions across different compute engines.

Build a local data lakehouse using an end-to-end open-source setup. Starting from an empty Docker Compose file, the guide walks engineers through implementing bronze, silver, and gold architectural layers along with Apache Iceberg time travel.

Optimize Databricks costs by matching workloads to compute. Use serverless for bursty tasks and classic compute with spot instances for steady ETL. Disable Photon on UDF-heavy jobs since it will not accelerate them, and tune auto-termination timeouts to eliminate expensive cluster idle time.

RustFS launched open-source S3 Tables, embedding an Apache Iceberg REST Catalog into its storage kernel. One process on port 9000 serves S3 and Iceberg REST protocols.

Lift-and-shift rewrites to Databricks spike DBU costs by turning unoptimized queries into metered waste. Avoid line-by-line translation. Instead, audit portfolios, kill zombie jobs, and consolidate redundant filter variants into unified Gold tables to migrate business capabilities, not technical

Netflix ingests 2 PB of new data weekly on S3 while keeping over 100 PB queryable and controlling storage growth. Instead of switching cloud providers, they optimized costs by identifying and patching four core efficiency leaks, stopping unnecessary processing and storage overhead.

To ingest volatile Google Sheets data into Databricks, engineers built a daily pipeline using OAuth2 and tab regex filtering. It extracts full sheets into bronze string tables and typed silver Delta Lake tables using SHA-256 row fingerprints and full overwrites to handle in-place row edits.

Databricks Serverless automates infrastructure management and scales on demand, billing per DBU-second. It suits spiky, unpredictable workloads like ad hoc SQL and incremental ETL pipelines, provisioning compute in seconds with autoscaling and Photon engine optimizations enabled by default.

AWS acquired DuckLabs to influence the engine powering modern data pipelines. Acting as a cost-efficient alternative to Spark with native Parquet support, DuckDB pairs with DuckLake and the Quack protocol to anchor an S3-based, serverless lakehouse stack that competes with Snowflake and Databricks.

Google engineered a six-layer Apache Iceberg architecture combining engines, catalogs, and an autonomous control plane. It automates file compaction below 50% of 256 MB targets and federates external catalogs like Glue and Unity to query cross-cloud tables without copying files.

To stand out, build production data projects addressing real business problems rather than toy pipelines. One strong pattern uses a Databricks Medallion Lakehouse ingesting live REST APIs, partitioning by county FIPS, and executing vectorized spatial joins to prevent memory shuffle and compute

Open Data Infrastructure decouples compute and storage using open table formats like Apache Iceberg and Delta Lake on object storage. Storing data once provides transactional lake reliability, letting teams swap engines, avoid duplicate pipelines, and support AI without vendor lock-in.

In self-managed Apache Iceberg architectures, AWS Glue optimizer handles the same table maintenance tasks as Amazon S3 Tables for identical workloads, but at a significantly higher price point. For Iceberg maintenance, Glue optimizer proved to be 15 times more expensive than S3 Tables.

The Medallion Lakehouse architecture organizes data into Bronze, Silver, and Gold tiers using open table formats like Delta Lake and Iceberg. Bronze stores raw append-only data, Silver cleans and validates entities with ACID upserts, and Gold delivers aggregated star schemas for fast BI and ML

The lakehouse architecture simplifies fragmented data stacks, unifying warehouse-grade performance, storage economics, live federation, and agent-ready context into a single foundation.

Scaling PySpark requires avoiding driver OOMs from actions like collect(), eliminating network shuffle via broadcast joins on small tables, and running compaction to solve lakehouse small-file issues. Optimize storage layouts and memory to build resilient production pipelines.

The streamhouse pattern unifies streaming and analytical data into one logical table. Real-time writes hit a low-latency Arrow tier, while a continuous tiering service compacts aging data into open Parquet lakehouse formats like Iceberg or Paimon, letting SQL engines query both tiers seamlessly.

Kafka architectures often sink real-time events to S3 for long-term retention. However, raw files fail to behave like database tables for updates and consistent reads. Open table formats like Apache Iceberg bridge this gap by adding schema, snapshot, and partition metadata over the data.

Modern AI architectures rely on lakehouses, zero-copy sharing, and metadata automation over classical ETL. Platforms must integrate tools, data products, and clear governance to capture raw schema-on-read data effectively.

Deploying a DuckLake catalog on an EC2 instance served by Quack abstracts storage and credentials from clients. Provisioned via Terraform, Quack manages S3 bindings and enables multiple concurrent clients to query and write to the data lake without handling backend secrets directly.

Production DuckLake architectures codify business logic and ingestion pipelines using Python scripts via the DuckDB API. Pipelines ingest daily Postgres snapshots, Parquet files, or streams into DuckLake tables for cleaning, modeling, and downstream analytics.

Snowflake demonstrates streaming over 1M TPS directly into Iceberg tables using the Snowpipe Streaming SDK. A server-side buffering tier handles batching and background compaction to optimize Parquet file sizes, while Iceberg v4 proposes an Adaptive Metadata Tree to reduce that write amplification.

Snowflake previewed support to access externally managed Apache Iceberg tables through Snowflake Horizon Catalog. Recent updates also added Amazon S3 Tables Iceberg REST catalog integration and CREATE OR ALTER support for dynamic Iceberg tables.

Google Cloud introduces its Lakehouse runtime catalog to help data engineers modernize Apache Hive deployments. This analytics solution enables modernization of legacy Hive architectures within cloud data lakehouse environments.

Databricks compute costs split into two bills: platform DBU charges and cloud infrastructure fees. Classic compute separates VM costs, while serverless bundles VM infrastructure directly into the DBU rate. Total spending requires tracking both DBUs and underlying cloud hardware.

Fivetran standardizes destination schemas with canonical types and metadata columns for portable SQL transformations. It provides table-level sync modes, including soft deletes and native SCD Type 2 history, while supporting open formats like Apache Iceberg directly in cloud storage.

Rednote migrated its real-time indexing pipeline from Kafka to Apache Fluss to solve read amplification from wide tables. Fluss uses columnar streaming via Apache Arrow and bypasses brokers for cold-data replays, isolating historical scans from real-time traffic to reduce bandwidth and compute

Choosing Databricks for compute differs from letting it own transformation logic. Using dbt keeps transformations in open, version-controlled SQL, ensuring portable metrics and preventing vendor lock-in across warehouses while reducing compute via state-aware runs.

Apache Hudi accelerates selective point lookups using queryable indexes stored in its metadata table. Instead of scanning entire tables, record-level and secondary indexes map keys to specific file groups during query planning, dropping a 400 GB table lookup from 977 seconds to 12 seconds.