dbt resources for data engineers

This repository compiles case studies on analytics agents, focusing on context preparation, retrieval, permissions, and evaluations. Teams integrate dbt models, SQL context tables, and domain documentation to improve data discovery while controlling cost and latency across production pipelines.

Orchestra launched AI Incident Management to triage data issues and open PRs. New orchestration features include Google Kubernetes integrations, pausing pipeline tasks, environment branch restrictions, and automated alerts for slow dbt models.

This dbt package analyzes compute, storage, and query patterns across Snowflake, Databricks, BigQuery, and Redshift to generate prioritized cost-saving recommendations. It offers run-operation macros and builds gold-layer marts when enabled via environment variables.

Fivetran merged with dbt Labs to unite automated data movement and transformation. The platform integrates over 750 managed connectors with dbt modeling, semantic metrics, and open formats like Apache Iceberg, streamlining ELT pipelines and providing tested data models for analytics and AI.

🎙️ The Joe Reis Show — Analytics engineering is shifting under agentic AI. Teams must move past dashboard maintenance and SQL ticket queues toward context engineering, using tools like dbt for agentic abstractions rather than acting as pure pipeline mechanics. Traditional BI faces disruption as architectures adapt.

OpenRouter uses ClickHouse Cloud to ingest high-volume telemetry via asynchronous inserts and ClickPipes. Materialized views aggregate raw logs for near real-time routing and public rankings, while a dbt medallion warehouse on ClickHouse powers internal analytics and automated agents.

Apache Ossie provides an open YAML spec for semantic models to stop metric drift across tools. A new converter translates Tableau workbook XML into Ossie format, generating Snowflake semantic views, dbt MetricFlow metrics, and Power BI models while preserving formulas and relationships.

Frontier AI models no longer require external dbt skills for standard analytics engineering pipelines. In an evaluation on Snowflake, modern agents cleared data checks natively; failures stemmed from project processes rather than missing dbt syntax.

dbt Charts introduces an open-source, dashboards-as-code spec designed for AI agents to write and engineers to audit. Aiming to bypass complex BI layers, it lets teams rely more on relational data architectures instead of pre-joining models solely to satisfy traditional BI tools.

🎙️ The Analytics Engineering Podcast (dbt) — dbt Charts introduces a human-readable spec for dashboards that AI can generate and data teams can audit. Dave Fowler calls it "the Markdown of dashboards" to tackle BI bottlenecks and unreviewable generated code, while exploring how AI may eventually reduce reliance on dimensional modeling.

Decision models like Jev prioritize fast answers paired with calibrated probabilities rather than generating text. Data engineers can integrate these models into data analytics pipelines using the dbt-decisions package, linking AI decision outputs directly with a semantic layer inside dbt workflows.

BigQuery lets data teams call LLMs in SQL queries via AI functions instead of Python APIs. Integrating calls like AI.GENERATE into dbt incremental models prevents expensive full-refresh recomputation. Output schemas force typed JSON, enabling validation with standard dbt tests.

Sigma manages its semantic layer as versioned dbt code orchestrated by Dagster across 2,883 models. Custom packages compile definitions into Sigma data models, Snowflake semantic views, and Cortex agents. Dagster automates state-aware dbt builds and syncs them through Sigma's API.

DoorDash consolidated its analytics architecture by decoupling the semantic layer from consumption. Data stored in Apache Iceberg and Snowflake is modeled via dbt metrics, serving ThoughtSpot and AI agents centrally. DoorDash treats semantic definitions as version-controlled code.

AWS architects proposed an architecture pairing the dbt Semantic Layer with the Amazon Redshift MCP Server to stop AI agents from writing raw SQL. MetricFlow compiles certified metrics while Redshift enforces read-only execution, preventing hallucinations and silent join errors.

Twilio replaced Kafka, Spark Streaming, and Elasticsearch with a consolidated ClickHouse pipeline managed by dbt. Using the dbt-clickhouse adapter, materialized views handle streaming transformations, while GitOps and Argo Workflows enable automated testing and multi-region deployment.

KOHO built KoDa, an LLM self-serve tool connecting Claude to Redshift via MCP. It queries only governed dbt marts without PII or raw data. The team maintains an eval council of 375 real analyst questions to benchmark quality, proving foundational data modeling and evals matter more than the LLM.

Ecolab used Datafold's migration agent to replace fragile Snowflake stored procedures and tasks with over 40 modular dbt Mesh projects. Automated data diffing verified parity against production data, cutting code 6x and data connections 4x across the pipeline.

LangChain built a layered context architecture for AI analytics agents. They sync detailed dbt docs into BigQuery, expose DAG SQL dependencies, and use a semantic layer for metrics. Context guides are versioned in Git and imported via CI/CD, boosting self-serve questions by 40x.

Sweetwater scaled its governed dbt platform by integrating Datadog for proactive pipeline observability. The data team monitors Kafka inserts, warehouse compute, and table freshness to catch pipeline issues like mis-scheduled legacy scripts before reporting breaks.

Checkr solved LLM metric drift across 6,000 dbt models by pairing Snowflake with a semantic layer and structured context. Using eval loops, they feed synonyms and business rules into Omni to prevent incorrect joins. Context should be added surgically to fix specific query failures.

Guitar Center rebuilt its data stack using Snowflake and managed dbt Core, replacing legacy stored procedures with CI/CD and automated tests. Scoped builds and smaller warehouses cut Snowflake compute costs by 40% and Airflow usage by 50%, speeding up pipeline delivery from months to days.

dbt Labs launched dbt Charts, an open-source charting language configured via YAML that lives directly in repositories. Designed around an underlying grammar, it automatically adjusts axis label density and flags charts that are too dense to render legibly before they reach production.

dbt Labs launched dbt Charts, an open-source YAML language for defining dashboards alongside dbt models. Stored in a charts directory, configurations run through standard git workflows and tests. It operates via the dct CLI, integrating analytics engineering and BI into unified deployments.

Keep built a real-time data platform using Artie for CDC from Postgres and MongoDB into Snowflake. Snowflake dynamic tables, streams, and tasks flatten JSON and trigger processing, while dbt governs models and deduplication, cutting credit underwriting latency to under 60 seconds.

AI agents require shared context across structured and unstructured data to act reliably. Fivetran Context Layer addresses this by indexing docs and data sources into warehouse tables called Agents Schema, letting agents query unified ontologies directly via MCP instead of proprietary APIs.

dbt v2 adds native DuckDB support, enabling serverless, offline ELT pipelines directly on local hardware without cloud warehouses. DuckDB queries CSV, Parquet, and Iceberg formats natively, running aggregations over 300M rows in seconds to replace distributed compute for typical workloads.

Grounding AI analysts with a semantic layer achieved 96-100% accuracy, versus 80-91% with raw SQL against database schemas. Reliable text-to-SQL systems require data modeling, tests, and metadata context over raw schema access.

Lake Compute lets data teams run low-complexity dbt models on cheaper lake engines instead of expensive data warehouses. Built on Apache Iceberg and open REST catalogs, this architecture cuts compute costs model by model without disrupting downstream DAGs or BI tools.

AI agents are reviving the dbt Semantic Layer by requiring governed data paths. To prevent cascading pipeline errors in multi-step LLM chains, dbt is developing "semantic SQL" to expose metrics as a standard database interface while pushing pre-aggregations upstream to reduce warehouse spend.