DuckDB resources for data engineers

DuckDB on an iPhone outperformed Databricks Serverless SQL on TPC-H workloads up to 100 GB. Single-node execution engines avoid the query planning and network shuffle overhead inherent to distributed clusters, making them far more cost-effective for typical real-world business datasets.

DuckDB database files can store only view definitions pointing to remote Parquet files on object storage instead of raw data. Acting as a lightweight catalog under a few hundred kilobytes, it creates a shareable semantic layer that abstracts file layouts while querying via the httpfs extension.

Production DuckLake architectures require deliberate maintenance. Because the catalog tracks data files and inlines inserts under 10 rows in a database, engineers must periodically flush inlined data to Parquet and run file compaction to prevent performance bottlenecks.

Polars 2.0 introduces out-of-core processing and makes its streaming engine default on LazyFrame.collect, spilling to disk at ~80% RAM. It adds native Arrow MapType support, join reordering, dynamic bloom filters, and first-class SQL coverage outperforming DuckDB in benchmarks.

Databricks Serverless allows running lightweight tools like DuckDB and Polars alongside Spark for medium workloads with Unity Catalog support. Benchmarks evaluate how DuckDB and Polars handle joins and window functions on Databricks Serverless compared to Spark broadcast join patterns.

To replace slow MongoDB analytics, DuckDB was deployed on the same host to handle sub-second queries. After testing JDBC batching and the community Mongo extension, custom Java table functions were chosen for ingestion to avoid C++ maintenance while streaming data without intermediate disk files.

Hands-on workshops detail how to wire up data pipelines and run agents in production using DuckDB and MotherDuck. Engineers build conversational architectures to profile raw files, generate schemas, execute transformations, run data quality checks, and host MCP servers for automated workloads.

AWS acquired DuckDB Labs instead of Polars, a move that stirred concerns over open source independence. The author argues that developer goodwill heavily influenced the cloud giant's choice to bring the analytical engine in-house over alternative tools.

Engineers still seek deep technical mastery despite AI automating high-level code. Demand remains strong for foundational topics like DuckDB, DuckLake, and Apache Arrow. Success now requires understanding core internals, memory architecture, and low-level mechanics to effectively direct AI tools.

To connect AI agents to QuickBooks, split reads from writes. Bruin and ingestr load QuickBooks data into BigQuery or DuckDB to model metrics via SQL, while Intuit's MCP server handles writes. This architecture avoids API rate limits and ensures agents query reliable, pre-reconciled data.

Speed up string aggregations by moving repeated strings into dimension tables with narrow integer keys. Hash aggregation on fixed-width integers reduces CPU overhead and memory footprint compared to variable-length text. The strings are joined back only after grouping.

Data engineering updates highlight DuckDB 1.5.6, Parquet X-ray, and billion-row record linkage with Splink 5. Key analyses cover streaming pipeline cost efficiency, PostgreSQL row-level security performance, handling hot shards, and building analytical MySQL replicas with Parquet and DuckDB.

Aurora PostgreSQL now embeds DuckDB to directly query Apache Iceberg and Parquet in S3 without ETL pipelines. Using the aurora_analytics extension, it supports foreign tables with predicate pushdown and column pruning to join operational and lake data.

The edition highlights Kafka and stream processing developments, focusing on broker architectures and Flink SQL patterns. Notable releases include Apache Fluss v1.0 and Khaos for failure testing, while new alternatives like blob-stream target cost over latency.

TypeSafe AI released Jev, enabling DuckDB extensions to filter, classify, and score Parquet and SQL tables via natural language. Extensions process data in 2,048-row chunks and push down standard SQL predicates first, preventing unnecessary third-party API calls on rejected rows.

Splink 5.0 improves record linkage at scale with chunked execution in predict(), enabling multi-machine DuckDB processing and progress updates. Dependencies are stripped down to core sqlglot, duckdb, and pyarrow, while incremental matching adds predictwithin() and predictbetween() APIs.

🎙️ Altertable — DuckDB serves as a critical component in agentic data architectures, enabling AI agents to access, prepare, and analyze data efficiently. In modern analytics stacks, it functions as an embedded engine for reliable agentic processing and context engineering across analytical workloads.

🎙️ Altertable — Taktile integrates DuckDB with Apache Iceberg to execute fast analytical workloads over an open lakehouse format. The architecture leverages DuckDB as an embedded OLAP query engine directly atop Iceberg tables, providing high-performance query execution and scalable data engineering patterns.

DuckDB 1.5.6 delivers critical bugfixes and performance updates. For data engineers, it fixes swapped min/max stats on 128-bit decimals, prevents MultiFileReader NULL MAP keys, and enables ALP compression on smaller block sizes. It also adds ADBC Statistics API and duckdb:// URI support.

Vector Gateway Interface applies HTTP freshness rules to DuckDB SQL functions, letting API-backed workers cache results across queries. It caches table and scalar outputs based on worker-defined TTLs, cutting external API calls when repetitive lookups outweigh cache lookup overhead.

🎙️ André Noujaïm — Combining DuckDB with dbt enables a lightweight approach to analytics engineering that reduces the need for complex, traditional data pipelines. By moving compute closer to storage and leveraging live federation, teams can simplify transformations within an AI-native lakehouse architecture.

DuckDB queries Hugging Face datasets directly via the hf:// protocol over HTTP without downloading files locally. It performs remote columnar projections on Parquet files, supports glob patterns across partitions, and can target the auto-generated ~parquet branch for fast scans.

Recent data engineering updates highlight DuckDB shipping inside dbt v2 and the release of Apache Fluss 1.0 for real-time AI architectures. Modern pipeline tools also focus on Polars query profiling, Cassandra upgrades, and OpenTelemetry adaptive tail sampling for high-throughput data systems.

The DuckDB ADBC extension enables DuckDB to query external systems directly using ADBC drivers, including PostgreSQL, MySQL, Snowflake, Trino, and Arrow Flight SQL. It registers internally as adbc_scanner and exposes an ATTACH storage type for federated data access.

🎙️ The Joe Reis Show — DuckDB co-creator Hannes Mühleisen discusses the analytical engine's evolution, architecture, and DuckLabs' acquisition by AWS. The episode covers in-process OLAP data engineering workflows and the database's future roadmap.

Querying billions of Common Crawl records does not require petabyte scans. By targeting Hive-partitioned Parquet files on S3, DuckDB uses row-group statistics to prune data. It filters by sort key directly over S3, returning targeted results across years of crawl data in minutes.

🎙️ The Joe Reis Show — DuckDB co-creator Hannes Mühleisen discusses the analytical database's evolution and future. The conversation covers the open-source engine's trajectory alongside DuckLabs following its acquisition by AWS, offering key insights for data engineers on embedded analytics architectures.

dbt v2 ships with a built-in DuckDB adapter running on its Rust engine via ADBC drivers. The release adds DuckLake and Iceberg REST catalog support, plus an Information Schema feature that outputs project metadata as Parquet files, enabling fast SQL queries on manifests directly with DuckDB.

While open table formats like Iceberg decouple compute and storage, connecting client apps to diverse engines remains fractured. Apache Arrow and ADBC solve this by providing a standardized columnar API, eliminating costly row-to-column conversions and yielding up to 38x transfer speedups over ODBC.

Orchestra announced real-time anomaly detection for dbt operations to alert when models run long. Platform updates include Google Kubernetes integration, task pausing, and pipeline environment controls, while the broader data ecosystem saw major funding for Temporal and Streamhouse adoption.