MotherDuck resources for data engineers

MotherDuck integrates Jev via prompt_jev() to execute fast decision-model evaluations directly in SQL queries on every row. It classifies text into closed sets like scores and labels alongside telemetry spans, separating plain SQL counts from subjective judgment calls.

Self-serve analytics and AI tools often cause metric drift, yielding conflicting results like different calculations for net revenue retention across teams. The solution requires establishing standardized metric definitions and feeding accurate business context into AI systems to ensure consistency.

Hands-on workshops detail how to wire up data pipelines and run agents in production using DuckDB and MotherDuck. Engineers build conversational architectures to profile raw files, generate schemas, execute transformations, run data quality checks, and host MCP servers for automated workloads.

MotherDuck introduces prompt_jev() to query unstructured text columns like agent traces, comments, and job ads directly in SQL. Running roughly 20 times faster than standard LLM calls at lower cost, it lets data engineers extract insights and group business metrics within a single SQL pipeline.

Data Outpost examines modern data architectures adapting to autonomous agents that read and write data. Sessions focus on building scalable insight platforms, managing automated semantic layers, integrating operational schemas, and handling the oversight, latency, and cost constraints of production

Jev acts as a fast SQL decision function to classify unstructured text into typed choices without token-by-token LLM generation. MotherDuck's prompt_jev() processed 100,000 text rows in 82.2 seconds for $0.50, outputting a struct with the chosen label and confidence scores directly in the query.

MotherDuck introduced prompt_jev(), a SQL-native classification function returning labels and confidence scores to filter, join, and aggregate directly in queries.

Querying billions of Common Crawl records does not require petabyte scans. By targeting Hive-partitioned Parquet files on S3, DuckDB uses row-group statistics to prune data. It filters by sort key directly over S3, returning targeted results across years of crawl data in minutes.

MotherDuck now lets you publish declarative dbt Charts as interactive MotherDuck Dives. Via an on-run-end hook, a serverless Python Flight compiles chart YAML into Vega Lite and publishes live-updating dashboards alongside data pipelines.

MotherDuck integrated Jev, enabling in-database text classification via the prompt_jev() SQL scalar function. It matches frontier LLM accuracy while running 50x faster at 1% of the cost, classifying 100,000 rows in 40 seconds to allow direct filtering and joins.

Learn how to build data architectures for autonomous agents. Sessions focus on constructing a context layer to cut token costs and reduce hallucinations when querying data.

Big Data London hosts an evening event featuring MotherDuck, bringing together the data community to unwind and connect after conference sessions. Attendees can network with peers in analytical engineering and serverless data warehousing in an informal setting.

Connect with fellow data engineers and analytics professionals at the Seattle Data Happy Hour. Before focusing on optimizing data pipelines for the season, gather for informal networking and discussions on data tooling, architecture, and analytics engineering without formal presentations.

Effective data modeling outperforms raw LLM upgrades for text-to-SQL accuracy. dbt Labs found schema modeling alone raised accuracy from 64.5% to 90.0% without a semantic layer, preventing subtle join grain errors where LLMs aggregate non-additive metrics.

DuckDB v2.0-alpha introduces client/server mode via Quack, async S3 I/O, and an efficient VARIANT type. Additionally, the MSSQL extension v0.2.3 optimizes bulk loading via parallel writers and schema tuning, cutting load times for 38M rows from 933s to 96s.

Choosing the right database depends on access patterns and scale. While Postgres handles typical OLTP workloads, OLAP engines like DuckDB use columnar disk layouts to cut latency on heavy aggregations across millions of rows. SQLite fits single-file apps, but lacks concurrent write scaling.

Postgres suits OLTP workloads, but analytical queries demand OLAP architectures. DuckDB optimizes aggregations through columnar storage, vectorized execution, and predicate pushdown, reading only needed columns to cut latency and disk I/O when processing millions of rows.

MotherDuck now offers commercial support for open-source DuckDB in production. Data engineering teams get SLA-backed response times, architectural reviews, query tuning, and upgrade validation to scale embedded analytics and analytical workloads reliably.

DuckDB 2.0 boosts S3 queries via async I/O: a thread pool prefetches Parquet row groups so workers decode continuously without idle CPU time. It also overhauls recursive CTEs, indexing deep parent-child hierarchies once instead of re-reading tables each round.

DuckDB can query petabyte-scale Common Crawl archives directly on S3 using Hive-partitioned Parquet files and URL indexes. By evaluating Parquet row-group statistics via DuckDB, queries skip unneeded files to inspect millions of hostnames in minutes without spinning up a distributed cluster.

AWS acquired DuckLabs to influence the engine powering modern data pipelines. Acting as a cost-efficient alternative to Spark with native Parquet support, DuckDB pairs with DuckLake and the Quack protocol to anchor an S3-based, serverless lakehouse stack that competes with Snowflake and Databricks.

Context layers improve autonomous agent workflows and SQL accuracy by capturing unstructured business documentation alongside metadata. They automate context construction from query history and dbt models, leaving humans to curate context and handle data pipeline verification.

AWS agreed to acquire DuckLabs, keeping DuckDB open-source under the independent DuckDB Foundation and MIT license. The engine offers fast in-memory SQL on Parquet and Iceberg, giving AWS strategic leverage and potential hosting paths across EC2, ECS, and Lambda.

AI agents now drive data platform workloads, running 29 times more queries than humans. Their query patterns are bursty and experimental, with a median delay of four seconds and smaller scans, requiring platforms optimized for high concurrency, fast autoscaling, and efficient local caching.

AWS acquired DuckLabs, the team behind DuckDB, while Motherduck bought tower.dev for a python runtime for ingest. NVIDIA agreed to buy Hugging Face for $13bn. AWS Glue jobs can now fetch logs, and Power BI supports partition-specific dataset refreshes.

Run agentic SQL locally by pairing a quantized Qwen 3.8 27B model with DuckDB on a laptop with 16GB RAM. This architecture enables zero-token-cost data analysis, query optimization, and background automation tasks while scaling to MotherDuck via simple configuration.

MotherDuck launched a CLI with embedded DuckDB to run queries, manage Python data pipelines, and deploy dashboards from the terminal. Designed for CI and AI agents, it enables local data processing, piping, and automated workflows without burning model tokens on intermediate query results.

Production DuckLake architectures codify business logic and ingestion pipelines using Python scripts via the DuckDB API. Pipelines ingest daily Postgres snapshots, Parquet files, or streams into DuckLake tables for cleaning, modeling, and downstream analytics.

Amazon acquired Duck Labs, the team behind DuckDB, to expand its database ecosystem. DuckDB will keep its open-source license under the DuckDB Foundation. Proving a single-node engine can outpace distributed systems, MotherDuck will continue running serverless DuckDB and offering support.

🎙️ It's About Data — Amazon has signed an agreement to acquire DuckLabs, the developer of the DuckDB analytics database engine. This move brings DuckDB's in-process analytical query processing capabilities directly under the AWS ecosystem.