MotherDuck resources for data engineers

Data roles are shifting from core reporting to operational automation and data ingestion. Engineers must close data action loops using domain data, virtual knowledge graphs, and LLM-driven agents. Understanding pipeline trade-offs remains essential as automation absorbs routine tasks.

MotherDuck introduced Dive Apps, allowing visualizations to trigger Flights, a serverless Python runtime. This architecture enables two-way data pipelines that execute write-backs into source systems directly from warehouse-backed interfaces.

MotherDuck scales single-node analytics to tens of terabytes instead of clustering smaller instances. Its Giga instance uses 192 cores and 1.5 TB of RAM, processing the 1 TB TPC-DS benchmark with a 690 ms geomean runtime. Single nodes now replace distributed clusters for large analytical jobs.

DuckDB 2.0 introduces a PEG parser to eliminate shift-reduce conflicts and allow dynamic grammar extensions. For lakehouse pipelines, DuckLake now supports file cleanup routines like purging orphaned Parquet files from S3 and flushing inlined catalog writes directly to storage.

Jevaro 0.5.0 adds bidirectional Arrow streaming to TypeSafe's Jev API using multipart requests. By ingesting raw binary data via Arrow IPC streams, it bypasses Base64 overhead and processes up to 2,000 states per second, letting DuckDB and Polars tables seamlessly pipe data in and out.

MotherDuck added image support to promptjev(), querying BLOB columns or Parquet directly in SQL. It returns typed outputs and confidence scores that can be joined and grouped with tabular data. Load images via readblob or S3, then process visual pipelines alongside standard data models.

Self-serve analytics powered by AI agents and MCP servers lets users query warehouses without SQL, but it risks metric fragmentation. Schema introspection only solves baseline modeling. Data teams must establish context layers and human-governed canonical definitions to enforce consensus.

MotherDuck is expanding its DuckDB serverless data warehouse to Azure, letting teams run isolated compute instances next to Azure Blob Storage to cut cross-cloud egress. It integrates natively with Power BI and Entra ID while supporting an MCP server for AI agents.

MotherDuck integrates Jev via prompt_jev() to execute fast decision-model evaluations directly in SQL queries on every row. It classifies text into closed sets like scores and labels alongside telemetry spans, separating plain SQL counts from subjective judgment calls.

Self-serve analytics and AI tools often cause metric drift, yielding conflicting results like different calculations for net revenue retention across teams. The solution requires establishing standardized metric definitions and feeding accurate business context into AI systems to ensure consistency.

Hands-on workshops detail how to wire up data pipelines and run agents in production using DuckDB and MotherDuck. Engineers build conversational architectures to profile raw files, generate schemas, execute transformations, run data quality checks, and host MCP servers for automated workloads.

MotherDuck introduces prompt_jev() to query unstructured text columns like agent traces, comments, and job ads directly in SQL. Running roughly 20 times faster than standard LLM calls at lower cost, it lets data engineers extract insights and group business metrics within a single SQL pipeline.

Jev acts as a fast SQL decision function to classify unstructured text into typed choices without token-by-token LLM generation. MotherDuck's prompt_jev() processed 100,000 text rows in 82.2 seconds for $0.50, outputting a struct with the chosen label and confidence scores directly in the query.

MotherDuck introduced prompt_jev(), a SQL-native classification function returning labels and confidence scores to filter, join, and aggregate directly in queries.

Querying billions of Common Crawl records does not require petabyte scans. By targeting Hive-partitioned Parquet files on S3, DuckDB uses row-group statistics to prune data. It filters by sort key directly over S3, returning targeted results across years of crawl data in minutes.

MotherDuck now lets you publish declarative dbt Charts as interactive MotherDuck Dives. Via an on-run-end hook, a serverless Python Flight compiles chart YAML into Vega Lite and publishes live-updating dashboards alongside data pipelines.

MotherDuck integrated Jev, enabling in-database text classification via the prompt_jev() SQL scalar function. It matches frontier LLM accuracy while running 50x faster at 1% of the cost, classifying 100,000 rows in 40 seconds to allow direct filtering and joins.

Effective data modeling outperforms raw LLM upgrades for text-to-SQL accuracy. dbt Labs found schema modeling alone raised accuracy from 64.5% to 90.0% without a semantic layer, preventing subtle join grain errors where LLMs aggregate non-additive metrics.

DuckDB v2.0-alpha introduces client/server mode via Quack, async S3 I/O, and an efficient VARIANT type. Additionally, the MSSQL extension v0.2.3 optimizes bulk loading via parallel writers and schema tuning, cutting load times for 38M rows from 933s to 96s.

Choosing the right database depends on access patterns and scale. While Postgres handles typical OLTP workloads, OLAP engines like DuckDB use columnar disk layouts to cut latency on heavy aggregations across millions of rows. SQLite fits single-file apps, but lacks concurrent write scaling.

Postgres suits OLTP workloads, but analytical queries demand OLAP architectures. DuckDB optimizes aggregations through columnar storage, vectorized execution, and predicate pushdown, reading only needed columns to cut latency and disk I/O when processing millions of rows.

MotherDuck now offers commercial support for open-source DuckDB in production. Data engineering teams get SLA-backed response times, architectural reviews, query tuning, and upgrade validation to scale embedded analytics and analytical workloads reliably.

DuckDB 2.0 boosts S3 queries via async I/O: a thread pool prefetches Parquet row groups so workers decode continuously without idle CPU time. It also overhauls recursive CTEs, indexing deep parent-child hierarchies once instead of re-reading tables each round.

DuckDB can query petabyte-scale Common Crawl archives directly on S3 using Hive-partitioned Parquet files and URL indexes. By evaluating Parquet row-group statistics via DuckDB, queries skip unneeded files to inspect millions of hostnames in minutes without spinning up a distributed cluster.

AWS acquired DuckLabs to influence the engine powering modern data pipelines. Acting as a cost-efficient alternative to Spark with native Parquet support, DuckDB pairs with DuckLake and the Quack protocol to anchor an S3-based, serverless lakehouse stack that competes with Snowflake and Databricks.

Context layers improve autonomous agent workflows and SQL accuracy by capturing unstructured business documentation alongside metadata. They automate context construction from query history and dbt models, leaving humans to curate context and handle data pipeline verification.

AWS agreed to acquire DuckLabs, keeping DuckDB open-source under the independent DuckDB Foundation and MIT license. The engine offers fast in-memory SQL on Parquet and Iceberg, giving AWS strategic leverage and potential hosting paths across EC2, ECS, and Lambda.

AI agents now drive data platform workloads, running 29 times more queries than humans. Their query patterns are bursty and experimental, with a median delay of four seconds and smaller scans, requiring platforms optimized for high concurrency, fast autoscaling, and efficient local caching.

AWS acquired DuckLabs, the team behind DuckDB, while Motherduck bought tower.dev for a python runtime for ingest. NVIDIA agreed to buy Hugging Face for $13bn. AWS Glue jobs can now fetch logs, and Power BI supports partition-specific dataset refreshes.

Run agentic SQL locally by pairing a quantized Qwen 3.8 27B model with DuckDB on a laptop with 16GB RAM. This architecture enables zero-token-cost data analysis, query optimization, and background automation tasks while scaling to MotherDuck via simple configuration.