MotherDuck resources for data engineers

Choosing the right database depends on access patterns and scale. While Postgres handles typical OLTP workloads, OLAP engines like DuckDB use columnar disk layouts to cut latency on heavy aggregations across millions of rows. SQLite fits single-file apps, but lacks concurrent write scaling.

DuckDB 2.0 boosts S3 queries via async I/O: a thread pool prefetches Parquet row groups so workers decode continuously without idle CPU time. It also overhauls recursive CTEs, indexing deep parent-child hierarchies once instead of re-reading tables each round.

AWS acquired DuckLabs to influence the engine powering modern data pipelines. Acting as a cost-efficient alternative to Spark with native Parquet support, DuckDB pairs with DuckLake and the Quack protocol to anchor an S3-based, serverless lakehouse stack that competes with Snowflake and Databricks.

Context layers improve autonomous agent workflows and SQL accuracy by capturing unstructured business documentation alongside metadata. They automate context construction from query history and dbt models, leaving humans to curate context and handle data pipeline verification.

AWS agreed to acquire DuckLabs, keeping DuckDB open-source under the independent DuckDB Foundation and MIT license. The engine offers fast in-memory SQL on Parquet and Iceberg, giving AWS strategic leverage and potential hosting paths across EC2, ECS, and Lambda.

AI agents now drive data platform workloads, running 29 times more queries than humans. Their query patterns are bursty and experimental, with a median delay of four seconds and smaller scans, requiring platforms optimized for high concurrency, fast autoscaling, and efficient local caching.

AWS acquired DuckLabs, the team behind DuckDB, while Motherduck bought tower.dev for a python runtime for ingest. NVIDIA agreed to buy Hugging Face for $13bn. AWS Glue jobs can now fetch logs, and Power BI supports partition-specific dataset refreshes.

Run agentic SQL locally by pairing a quantized Qwen 3.8 27B model with DuckDB on a laptop with 16GB RAM. This architecture enables zero-token-cost data analysis, query optimization, and background automation tasks while scaling to MotherDuck via simple configuration.

MotherDuck launched a CLI with embedded DuckDB to run queries, manage Python data pipelines, and deploy dashboards from the terminal. Designed for CI and AI agents, it enables local data processing, piping, and automated workflows without burning model tokens on intermediate query results.

Production DuckLake architectures codify business logic and ingestion pipelines using Python scripts via the DuckDB API. Pipelines ingest daily Postgres snapshots, Parquet files, or streams into DuckLake tables for cleaning, modeling, and downstream analytics.

Amazon acquired Duck Labs, the team behind DuckDB, to expand its database ecosystem. DuckDB will keep its open-source license under the DuckDB Foundation. Proving a single-node engine can outpace distributed systems, MotherDuck will continue running serverless DuckDB and offering support.

🎙️ It's About Data — Amazon has signed an agreement to acquire DuckLabs, the developer of the DuckDB analytics database engine. This move brings DuckDB's in-process analytical query processing capabilities directly under the AWS ecosystem.

MotherDuck acquired Tower, a managed runtime for Python pipelines, to bring its execution layer in-house. Tower underpins MotherDuck Flights, enabling AI agents to package, sandbox, schedule, and run data pipelines directly against DuckDB.

MotherDuck acquired Tower to provide runtime infrastructure for LLM-built data pipelines. Tower adds sandboxing, scheduling, and observability to MotherDuck Flights, letting engineers orchestrate AI data agents and turn prompt-generated pipelines into callable Data APIs.

AI agents can build semantic layers using Malloy, but using Markdown and SQL proved faster and more token-efficient. When implementations are easily regenerated, durable eval sets and data tests matter more than the semantic layer itself.

Conductor remains the top tool for orchestrating multiple coding agents across repositories. It provides essential visual overviews, fresh agent contexts, and file tracking for high-volume PR workflows, outperforming alternatives like Solo, Xirp, and Kiro Crew despite high CPU usage.

MotherDuck integrates with Replit to quickly build scalable data apps analyzing tens of millions of rows. Leveraging its Postgres-compatible API and serverless Python Flights, MotherDuck handles bulk historical ingestion and analytical queries while scaling compute down to zero when idle.

Migrating HubSpot pipelines from Fivetran to dlt requires reverse-engineering vendor schemas and mapping APIs. Declarative REST sources handle pagination, while custom @dlt.resource functions manage batch associations and fan-outs, with SQL transformations reproducing downstream mart models.

DuckDB ecosystem updates highlight the new ADBC extension for querying over 30 external databases using Apache Arrow for fast columnar transfers. Upcoming v2.0 introduces asynchronous I/O thread pools, accelerating remote Parquet queries by 3x and CSV queries by nearly 20x.

Vercel v0 and MotherDuck bridge the gap between rapid prototyping and production data apps. Users can build full-stack analytical dashboards using natural language prompts to ingest large datasets into MotherDuck and deploy to Vercel. A data app is one that needs to analyze a large amount of data.

Open table formats like Apache Iceberg, Delta Lake, and Hudi bring database-like features to cheap object storage. DuckDB serves as a lightweight query engine to directly read these formats. "Open Table Format bundles distributed files into manageable tables with database-like features."

DuckDB serves as a fast log parser using SQL to query unstructured text, JSON, and pipeline logs. These logs are the lifeblood of our data platforms, but parsing and analyzing them efficiently remains a persistent challenge. Engineers can scale processing beyond local resources using MotherDuck.

DuckDB handles analytics through embedded query capabilities and virtualized data access. Teams use it for local-first pipelines, processing trillions of records, and replacing heavy Spark clusters. FinQore reduced pipeline times from 8hr to 8min while Okta queried massive datasets efficiently.

DuckDB simplifies geospatial engineering by replacing complex GIS stacks with spatial SQL capabilities. It handles spatial joins, vector data, and geometric types efficiently using optimized spatial indexing.

Bluesky runs on the open AT Protocol, where each user data repo is stored in an SQLite database. Data engineers can query posts and handles directly via open APIs using DuckDB or stream Jetstreams using Python. Each user's data is implemented and stored with a single SQLite database.

A live demonstration shows building an end-to-end data pipeline using a single unified schema. An AI agent uses dltHub to generate and deploy ingestion pipelines from a REST API into MotherDuck serverless warehouse.

MotherDuck released a signup API allowing AI agents to directly provision cloud data warehouses. Agents issue an unauthenticated POST request to acquire tokens, run DuckDB analytics, and deploy hosted Python pipelines.

AI agents can now provision data infrastructure autonomously via no-auth endpoints like MotherDuck and Neon. Agents can now provision real infrastructure by themselves, and that quietly breaks something.

OpenAI slashed GPT 5.6 Luna costs, enabling agentic analytics for under half a penny per answer. On an agentic SQL benchmark, Luna achieved 99.8% accuracy. Small models paired with fast analytical database engines eliminate LLM bottlenecks, making transactional databases the new UX bottleneck.

The Post-Dashboard Index evaluates team readiness as analytics consumption shifts from humans to AI agents. It benchmarks data teams against peers, noting your next analytics consumer is an agent while platforms like Together AI execute agent queries on MotherDuck.