DuckDB resources for data engineers

DSUI is an open-source, extensible data stack platform configured via YAML. It connects tools like DuckDB, Apache Airflow, dbt, ClickHouse, and MinIO via modular adapters. Teams can build custom integrations and UI workflows using TypeScript SDKs without needing an external database.

DuckDB unifies fragmented data stacks into a single lakehouse foundation, offering warehouse-grade performance, live federation, strong storage economics, and agent-ready context.

Key data engineering releases include Apache Kafka 4.4, DuckDB 2.0, Flink CDC 3.7, and Confluent's official Rust client for Kafka. Engineering highlights cover stream processing patterns like deduplication in Flink SQL, session-ordered Go pipelines, and scaling shuffle with Apache Celeborn.

DuckDB eliminates big data operational complexity by querying S3, Parquet, and Iceberg via standard SQL without JVM clusters. DuckLake optimizes catalogs by storing metadata in a SQL database rather than nested files, cutting planning latency to under a millisecond with minimal configuration.

MiniLake is a local Databricks API emulator that executes real SQL on DuckDB, writes genuine Delta files, and runs containerized spark-submit jobs. Designed for CI and local development without cloud costs, it natively supports Unity Catalog CRUD, DAG job scheduling, and official SDK clients.

This edition covers data engineering architectures like JD.com building a lakestream with Apache Fluss and Apache Hudi, Azure SQL Database Hyperscale internals, DuckDB 2.0.0 features, and how Flink's Kafka connector commits offsets to Kafka.

DuckDB can query Snowflake-managed Iceberg tables on S3 using its httpfs and iceberg extensions. Users scan the table snapshot's metadata JSON directly via iceberg_scan, then materialize data locally to cut repeated query latency from 54 seconds to 1.5 seconds.

Data teams can query Snowflake tables locally in DuckDB using Arrow Database Connectivity (ADBC) without managing Iceberg tables. By avoiding row-column serialization found in ODBC, ADBC keeps data columnar throughout transfer, speeding up data movement by up to 38x.

Snowflake Interactive Tables and Warehouses target low-latency serving layers using dedicated caching and clustering keys. However, pipelines face strict trade-offs: warehouses do not auto-suspend, tables lack UPDATE or DELETE support, and queries strictly fail after a hard 5-second timeout limit.

A new community DuckDB extension queries Snowflake directly via an ADBC driver without custom scripts. Engineers can attach Snowflake databases and enable filter pushdown to cut data transfer, joining warehouse tables directly with local Parquet files.

Greybeam transpiles Snowflake queries to DuckDB via SQLGlot, but anonymous bind variables lose their lexical order in the AST. To preserve ordering, they intercept the token stream before AST parsing, rewriting anonymous placeholders into prefixed, named bind variables for DuckDB execution.

Frequent Zapier polling keeps Snowflake warehouses running continuously, billing 60-second minimums for millisecond lookups. Offloading these queries to a lightweight engine like DuckDB cut compute costs by 93% by routing repetitive row checks away from analytical compute.

To cut embedded analytics costs, Headset used Greybeam as a wire-compatible proxy in front of Snowflake. The proxy evaluated Looker queries, routing eligible reads to managed DuckDB clusters and the rest to Snowflake, reducing overall compute spend by 92% without migrations.

Snowflake Iceberg tables store Parquet files and metadata in object storage, allowing external engines to query them without copying data. Snowflake can serve as the catalog to handle compaction and snapshots, or connect to external catalogs like AWS Glue via external volumes.

Snowflake-managed storage simplifies Iceberg tables via EXTERNALVOLUME = 'SNOWFLAKEMANAGED', removing manual cloud bucket setup. Through the Horizon catalog, external engines like DuckDB can query and write directly to the managed files using vended credentials.

MotherDuck scales single-node analytics to tens of terabytes instead of clustering smaller instances. Its Giga instance uses 192 cores and 1.5 TB of RAM, processing the 1 TB TPC-DS benchmark with a 690 ms geomean runtime. Single nodes now replace distributed clusters for large analytical jobs.

apisip is a Rust CLI that ingests OpenAPI list endpoints directly into JSONL or Parquet. It automates pagination, rate limits, and incremental loads. Instead of runtime inference, it enforces schema types from the spec, safely casting lossless values and sending unparseable data to quarantine.

Duckpad is an open-source data transformation tool and SQL workspace powered by DuckDB. Data engineers can visually compose pipelines, integrate Git versioning, and trace cell-level data lineage back to contributing source values. SQL execution runs locally on modern Linux via Landlock.

dbt v2 adds native DuckDB support, enabling serverless, offline ELT pipelines directly on local hardware without cloud warehouses. DuckDB queries CSV, Parquet, and Iceberg formats natively, running aggregations over 300M rows in seconds to replace distributed compute for typical workloads.

dbt v2 rewrites its Jinja engine in Rust, adding native SQL comprehension and local static analysis. Linting runs up to 250x faster without warehouse queries, and unit tests can run locally on DuckDB. Native checks query internal project schemas to catch pipeline errors prior to remote deployment.

DuckDB 2.0.0 adds a Python appender that buffers rows generated on the fly and flushes them in a single batch operation for faster ingestion. The release also adds secure views to prevent data leakage via query plans, asynchronous I/O, and a first-class VARIANT type for semi-structured data.

DuckDB Universe maps 244 ecosystem projects and companies, including tools like Lance, Ibis, and dbt. Snowset query data shows 96.2% of queries scan under 10 GB, enabling single-node analytics engines like DuckDB to handle typical warehouse workloads locally.

DuckDB 2.0.0 adds a Python appender that collects rows and writes them together in a single operation to optimize ingestion pipelines. Other data engineering updates include PostgreSQL-style SHOW for settings, secure views to prevent plan leakage, and setval to reset sequences after bulk loading.

duckdb-differential is a C++ extension adding versioning to DuckDB 2.0 databases via a differential filesystem. It enables snapshots, time travel, branching, and S3 replication while sharing unchanged data blocks across versions without altering native catalogs, SQL, or transactions.

Hardcharger is an open-source server for DuckDB 2.0 that replaces gRPC Flight SQL with Warp SQL, transporting SQL over standard HTTP with Arrow IPC streams. It adds native RBAC, SSO, and mTLS, running 22 TPC-H queries on 1 TB in 42.4s on Graviton5.

DuckDB and Polars outperform pandas on 1.14 billion rows. DuckDB was the only engine that finished every query, capping memory at 0.45 GB by querying Parquet directly.

DuckDB released version 2.0.0 LTS Cinnamon Teal. Key data engineering topics highlight Apache Arrow for columnar memory architectures, change-data-capture patterns via PyDebeziumAI, and handling late-arriving data in dbt snapshots alongside data governance and lineage in Snowflake.

dbt-hodor is a self-hosted alternative to dbt State that runs directly inside dbt. It reuses warehouse objects unless logic or upstream data changes. State persists in Delta tables, Postgres, or DuckDB without external services, supporting views, seeds, snapshots, and incremental tables.

GizmoData builds analytical SQL infrastructure combining DuckDB and Apache Arrow Flight SQL for remote query execution and columnar result transport. It provides execution backends in DuckDB and SQLite with support for concurrent client connections, server-side sessions, ADBC, and BI tool

DuckDB 2.0.0 LTS adds native client-server support via the Quack protocol and introduces asynchronous I/O to accelerate Parquet and CSV reads on object storage. It also integrates a first-class VARIANT type that shreds semi-structured data for columnar execution and efficient Parquet storage.