Modeling resources for data engineers

🎙️ Practical Data Modeling — Data modeling requires controlling dataset grain when aggregating relationships. Joining multi-product tickets or message transcripts inflates row counts, meaning grouped totals will not sum to distinct totals. Engineers must preserve underlying counts rather than averaging pre-aggregated metrics.

Google Cloud is expanding Knowledge Catalog to ground data agents in a unified context graph. The architecture connects trusted semantic models across analytics pipelines, enabling engineers to link disparate metadata frameworks and deliver consistent context to automated systems.

Data engineers are shifting from manual modeling to building automated systems that continuously maintain semantic layers. Using production query feedback and Semantic SQL, these architectures let consumers combine governed measures while agents propose model updates and pre-aggregations.

🎙️ The Joe Reis Show — AI cannot solve poor data modeling. Feeding an entire enterprise schema into an LLM produces data slop. Instead, agents are shifting pipelines from bottom-up ETL to deterministic top-down domain modeling, highlighting semantic layers and Malloy to separate modeling logic from queries.

🎙️ It's About Data® — Jessica Talisman explains how organizations compromise data modeling and ontology architectures in their rush to deploy AI systems. Rushing AI initiatives without solid foundational engineering leads teams to build flawed, low-quality ontologies that degrade system performance.

SignalWeave normalizes heterogeneous AI telemetry from tools like LangChain and OpenTelemetry into canonical signals. Using standard Python, it processes JSON/JSONL via CLI pipes to aggregate reliability, quality, and cost metrics by run ID.

BigQuery measures has reached general availability to unify business intelligence metrics and knowledge graphs. This capability enhances data analytics architectures by standardizing metric definitions and integrating graph structures to boost accuracy for AI workloads.

Survey data shows 59% of teams use AI analytics, with 50% wanting AI agents querying data via MCP. Infrastructure control remains critical, as 44% require running on their own servers. Yet 31% do not know what a semantic layer is, despite it being essential for trustworthy AI.

Sigma manages its semantic layer as versioned dbt code orchestrated by Dagster across 2,883 models. Custom packages compile definitions into Sigma data models, Snowflake semantic views, and Cortex agents. Dagster automates state-aware dbt builds and syncs them through Sigma's API.

DoorDash consolidated its analytics architecture by decoupling the semantic layer from consumption. Data stored in Apache Iceberg and Snowflake is modeled via dbt metrics, serving ThoughtSpot and AI agents centrally. DoorDash treats semantic definitions as version-controlled code.

AWS architects proposed an architecture pairing the dbt Semantic Layer with the Amazon Redshift MCP Server to stop AI agents from writing raw SQL. MetricFlow compiles certified metrics while Redshift enforces read-only execution, preventing hallucinations and silent join errors.

Checkr solved LLM metric drift across 6,000 dbt models by pairing Snowflake with a semantic layer and structured context. Using eval loops, they feed synonyms and business rules into Omni to prevent incorrect joins. Context should be added surgically to fix specific query failures.

Grounding AI analysts with a semantic layer achieved 96-100% accuracy, versus 80-91% with raw SQL against database schemas. Reliable text-to-SQL systems require data modeling, tests, and metadata context over raw schema access.

AI agents are reviving the dbt Semantic Layer by requiring governed data paths. To prevent cascading pipeline errors in multi-step LLM chains, dbt is developing "semantic SQL" to expose metrics as a standard database interface while pushing pre-aggregations upstream to reduce warehouse spend.

To stop LLMs from hallucinating metrics, agents use a compiled semantic bundle rather than direct SQL. The pipeline validates terms, fetches certified logic, and queries warehouse tables with forced filters via an immutable in-memory bundle.

Verily uses dbt and a medallion architecture to process nested FHIR healthcare data for wastewater pathogen monitoring. The team created reusable dbt macros to standardize FHIR resources across datasets, capping composite keys at three columns to optimize data modeling and downstream analytics.

Bolt scaled its analytics engineering by standardizing dbt architectures into strict intermediate shapes (spine, feature, metrics) and exposed marts. Project dependencies flow strictly one-way from horizontal to vertical domains, and models enforce single ownership with no shared logic.

Semantic layers must provide a grammar instead of predefined metrics for AI agents. Rather than hardcoding every query, define base measures, context, and composable rules like time shifts or aggregations. Tools like SLayer compile these into correct, deterministic SQL.

🎙️ It's About Data® — Jessica Talisman discusses her book, "Ontology Pipeline: A Framework for Knowledge Engineering." The work outlines data architecture concepts and systematic patterns for knowledge engineering, helping data engineers structure knowledge graphs and formal semantic frameworks effectively.

LookML metadata indexing in Google Cloud Knowledge Catalog is now GA. This integration unifies enterprise context and semantics across data architectures, enabling automated cataloging and governance of business intelligence assets directly within the metadata platform.

WMCHealth designed a dbt pattern using an environment variable to toggle models between two execution modes from one codebase. Setting the flag filters to active records every five minutes, cutting runtime 93.6% over full historical incremental runs. Downstream models inherit scope via inner joins.

Green Dot created Meta:Context, embedding a five-layer schema in dbt semantic layer YAML meta blocks. This delivers typed business context alongside query metrics via API calls, governed by CI and pull requests to guide AI agents.

Greenpeace used a 3-layer dbt package architecture to unify transformations across 25 autonomous entities on BigQuery and Fivetran. Shared dbt packages hold 75% of the logic, standardizing metrics and semantic definitions while allowing automated CI rollouts to 170 projects.

AI data agents struggle with complex analytics across disparate databases and unstructured text. In the Data Agent Bench, top agents achieved only a 61% pass@1 rate. Adding a semantic layer significantly improves query accuracy, reinforcing the need for curated data modeling over raw text-to-SQL.

🎙️ The Analytics Engineering Podcast (dbt) — AI data agents struggle with raw data, often pulling everything into pandas instead of pushing down compute. Tested on messy benchmarks, top agents solve only 61% of queries on the first try. Data engineers must provide gold-layer tables and robust semantic layers to enable reliable queries.

Database time travel only captures system ingestion time, failing to accurately model late-arriving data. Bitemporal modeling resolves this by tracking both business effective time and system belief time. This pattern enables reproducible point-in-time replays alongside corrected historical views.

🎙️ Practical Data Modeling — Data modeling requires separating data forms from physical storage formats. Unifying concepts like support tickets across CRM records, JSON chats, and transcripts demands explicit identifiers and attributes, ensuring downstream pipelines distinguish derived data from raw facts.

To build a dynamic SaaS demo pipeline, Cursor was used to generate synthetic data with Python, build staging and mart models via dbt to keep historical tables fresh, and configure data modeling in Omni. Cursor streamlined context switching across data generation, dbt transformations, and UI code.

Data platforms evolved across warehouses, lakes, and lakehouses to solve storage and compute, but omitted semantic context. AI agents lack human intuition to interpret fragmented schemas, turning unmodeled business rules into hallucinations.

Cheap LLMs fail on analytics queries when business definitions are ambiguous, hallucinating assumptions as facts. Feeding a semantic model with curated metrics and datasets solves this: cheaper models with defined warehouse schemas outperformed flagship models lacking that harness context.