Transformation resources for data engineers

Frontier AI models no longer require external dbt skills for standard analytics engineering pipelines. In an evaluation on Snowflake, modern agents cleared data checks natively; failures stemmed from project processes rather than missing dbt syntax.

Skipping data engineering for AI is a disaster. Pointing LLMs at messy source tables fails; solid modeling remains essential. Meanwhile, AI tools like Cortex Code accelerate SQL Server to Snowflake migrations, converting code to dbt projects or stored procedures in hours rather than weeks.

An AI harness governs how models generate data engineering pipelines by enforcing contracts, metadata, and operational controls. Rather than letting AI guess logic, the harness defines the output grain, metric rules, and validation checks before generating transformations.

🎙️ Practical Data Modeling — Data modeling requires controlling dataset grain when aggregating relationships. Joining multi-product tickets or message transcripts inflates row counts, meaning grouped totals will not sum to distinct totals. Engineers must preserve underlying counts rather than averaging pre-aggregated metrics.

🎙️ The Joe Reis Show — AI cannot solve poor data modeling. Feeding an entire enterprise schema into an LLM produces data slop. Instead, agents are shifting pipelines from bottom-up ETL to deterministic top-down domain modeling, highlighting semantic layers and Malloy to separate modeling logic from queries.

Silent pipeline bugs often stem from boundary mismatches and unwritten definitions rather than crashes. A healthy job run does not guarantee correctness. Robust platforms centralize business rules across modular models and use external reconciliation checks to catch silent join drops.

Apply Little's Law in SQL to detect pipeline data bloat. By multiplying weekly deal arrivals by average cycle time, you calculate expected open deals versus actual records. A second query calculates the p90 cycle duration to segment and isolate stale, dead inventory skewing pipeline metrics.

Snowflake introduced QUERYATTRIBUTIONHISTORY to track compute costs per query, but testing reveals inflated numbers. Idle time is excluded, and warehouse IDs can mismatch. For precise attribution, engineers join QUERYHISTORY with WAREHOUSEMETERING_HISTORY and warehouse suspend events.

Snowflake ASOF JOINs optimize time-relative joins by replacing slow Cartesian-style range joins with partition-aware sorting and proximity searches. Using MATCH_CONDITION, it pairs records with the nearest timestamp, dropping query runtime from over 25 minutes down to 14 seconds.

Snowflake's MAXBY and MINBY aggregate functions simplify finding values associated with another column's extrema, replacing complex self-joins or window functions. Passing an optional third integer returns an array representing the top N values per group.

AI agents querying revenue data often fail on fan-out joins, time zones, and test accounts despite valid SQL. Test them against frozen golden datasets, comparing output values within tolerances rather than syntax. Build an exam on reconciled numbers to catch semantic modeling errors early.

SignalWeave normalizes heterogeneous AI telemetry from tools like LangChain and OpenTelemetry into canonical signals. Using standard Python, it processes JSON/JSONL via CLI pipes to aggregate reliability, quality, and cost metrics by run ID.

Duckpad is an open-source data transformation tool and SQL workspace powered by DuckDB. Data engineers can visually compose pipelines, integrate Git versioning, and trace cell-level data lineage back to contributing source values. SQL execution runs locally on modern Linux via Landlock.

BigQuery lets data teams call LLMs in SQL queries via AI functions instead of Python APIs. Integrating calls like AI.GENERATE into dbt incremental models prevents expensive full-refresh recomputation. Output schemas force typed JSON, enabling validation with standard dbt tests.

DoorDash consolidated its analytics architecture by decoupling the semantic layer from consumption. Data stored in Apache Iceberg and Snowflake is modeled via dbt metrics, serving ThoughtSpot and AI agents centrally. DoorDash treats semantic definitions as version-controlled code.

Ecolab used Datafold's migration agent to replace fragile Snowflake stored procedures and tasks with over 40 modular dbt Mesh projects. Automated data diffing verified parity against production data, cutting code 6x and data connections 4x across the pipeline.

Guitar Center rebuilt its data stack using Snowflake and managed dbt Core, replacing legacy stored procedures with CI/CD and automated tests. Scoped builds and smaller warehouses cut Snowflake compute costs by 40% and Airflow usage by 50%, speeding up pipeline delivery from months to days.

Keep built a real-time data platform using Artie for CDC from Postgres and MongoDB into Snowflake. Snowflake dynamic tables, streams, and tasks flatten JSON and trigger processing, while dbt governs models and deduplication, cutting credit underwriting latency to under 60 seconds.

dbt v2 adds native DuckDB support, enabling serverless, offline ELT pipelines directly on local hardware without cloud warehouses. DuckDB queries CSV, Parquet, and Iceberg formats natively, running aggregations over 300M rows in seconds to replace distributed compute for typical workloads.

Integrating NetSuite with Snowflake decouples operational processing from analytical reporting. Ingestion pipelines replicate ERP records into Snowflake for cross-domain modeling, while reverse ETL writes approved values back into NetSuite via web services using explicit object permissions.

Verily uses dbt and a medallion architecture to process nested FHIR healthcare data for wastewater pathogen monitoring. The team created reusable dbt macros to standardize FHIR resources across datasets, capping composite keys at three columns to optimize data modeling and downstream analytics.

Bolt scaled its analytics engineering by standardizing dbt architectures into strict intermediate shapes (spine, feature, metrics) and exposed marts. Project dependencies flow strictly one-way from horizontal to vertical domains, and models enforce single ownership with no shared logic.

dbt demonstrated that pre-compressing Gong transcripts with warehouse-native AI functions and serving them via dbt MCP cut cost 48.5% and run time 43.2%. Moving context engineering upstream into SQL pipelines avoids vendor API limits and eliminates duplicate LLM token consumption.

Riot Games used an agent assembly line to transform 18TB of raw gaming data into dbt silver and gold models. The system automates column profiling, SQL authoring, and multi-level validation against catalogs, drastically lowering the marginal engineering cost of building new data models.

PetScreening connected an in-house pet tag printer as a dbt exposure using a medallion architecture. Because bad joins physically waste materials, the team uses strict dbt tests for validation. The same gold-layer model powers BI dashboards and prints over 1,000 tags daily across two machines.

WebstaurantStore automated a bronze staging layer for over 700 sources using Snowflake Cortex and dbt. Cortex LLMs propose normalized column names from raw OLTP metadata into a review table. Once audited, dbt Jinja macros generate production SQL and YAML models.

Compare the Market shifted from a 4,000-model dbt monorepo to a multi-repo dbt Mesh pattern. Transitioning to domain ownership and tested data models cut report delivery from six weeks to under one week, raised test coverage past 75%, and increased model reuse to 3.0x.

Obie Insurance rebuilt its data stack into a monorepo dbt project using Apache Iceberg lakehouse architecture and Snowflake catalog integration. Utilizing dbt State for state-aware orchestration and model reuse drove 40% cost savings and reduced month-end close from 25 hours to 2 hours.

Greenpeace used a 3-layer dbt package architecture to unify transformations across 25 autonomous entities on BigQuery and Fivetran. Shared dbt packages hold 75% of the logic, standardizing metrics and semantic definitions while allowing automated CI rollouts to 170 projects.

Verisk migrated billions of rows of legacy mainframe data to Snowflake and dbt, replacing brittle notebooks and uncoordinated tasks with modular staging, intermediate, and mart layers. Automated tests, macros, and versioned lineage now ensure pipeline integrity and support AI workflows.