LLM resources for data engineers

Snowflake release updates highlight web research in Snowflake CoWork alongside storage and table enhancements. Data engineers gain General Availability for Tri-Secret Secure on Apache Iceberg tables and UUID data types in hybrid tables, plus GCP support for hybrid tables in Preview.

Frontier AI models no longer require external dbt skills for standard analytics engineering pipelines. In an evaluation on Snowflake, modern agents cleared data checks natively; failures stemmed from project processes rather than missing dbt syntax.

Snowflake announced Cortex Training in preview, alongside hybrid table support for Google Cloud and External OAuth for Snowflake Postgres. Recent updates also expand data engineering tooling, including DCM project task graphs, Time Travel for Apache Iceberg tables, and secondary indexes.

Skipping data engineering for AI is a disaster. Pointing LLMs at messy source tables fails; solid modeling remains essential. Meanwhile, AI tools like Cortex Code accelerate SQL Server to Snowflake migrations, converting code to dbt projects or stored procedures in hours rather than weeks.

dbt Charts introduces an open-source, dashboards-as-code spec designed for AI agents to write and engineers to audit. Aiming to bypass complex BI layers, it lets teams rely more on relational data architectures instead of pre-joining models solely to satisfy traditional BI tools.

An AI harness governs how models generate data engineering pipelines by enforcing contracts, metadata, and operational controls. Rather than letting AI guess logic, the harness defines the output grain, metric rules, and validation checks before generating transformations.

Snowflake rolled out CoCo Desktop v1.23.1 alongside continuous and bulk file loading capabilities. Engineering updates also highlight Apache Iceberg table time travel and task graph enhancements.

🎙️ The Analytics Engineering Podcast (dbt) — dbt Charts introduces a human-readable spec for dashboards that AI can generate and data teams can audit. Dave Fowler calls it "the Markdown of dashboards" to tackle BI bottlenecks and unreviewable generated code, while exploring how AI may eventually reduce reliance on dimensional modeling.

Split data reconciliation into two steps. First, use scheduled deterministic checks like row counts, measure sums, and join-key coverage to detect raw gaps. Then, apply an AI layer over historical resolution logs to triage expected noise and explain cross-platform pipeline discrepancies.

Data engineering hiring is recovering, but employers now expect more than traditional SQL, Python, and Airflow pipelines. Engineers must build AI-enabled systems and master tools like Databricks, dbt, and AWS, even as junior opportunities remain limited to just 5.5% of openings.

Data roles are shifting from core reporting to operational automation and data ingestion. Engineers must close data action loops using domain data, virtual knowledge graphs, and LLM-driven agents. Understanding pipeline trade-offs remains essential as automation absorbs routine tasks.

Nao launched a cloud UI for its open-source agentic BI builder. It syncs warehouse metadata across systems like Snowflake and ClickHouse to build a governed context layer of schemas, metrics, and rules. It features offline testing and MCP integration to serve governed data directly to AI agents.

Data engineers are shifting from manual modeling to building automated systems that continuously maintain semantic layers. Using production query feedback and Semantic SQL, these architectures let consumers combine governed measures while agents propose model updates and pre-aggregations.

🎙️ The Joe Reis Show — AI cannot solve poor data modeling. Feeding an entire enterprise schema into an LLM produces data slop. Instead, agents are shifting pipelines from bottom-up ETL to deterministic top-down domain modeling, highlighting semantic layers and Malloy to separate modeling logic from queries.

MotherDuck introduced Dive Apps, allowing visualizations to trigger Flights, a serverless Python runtime. This architecture enables two-way data pipelines that execute write-backs into source systems directly from warehouse-backed interfaces.

Snowflake released AIPARSEDOCUMENT in AGENTIC mode for intelligent document processing in public preview. Concurrently, data engineers get preview access to Time Travel for Apache Iceberg tables and DEFINE TASK GRAPH in DCM Projects for orchestrating complex pipelines.

This edition covers data engineering architectures like JD.com building a lakestream with Apache Fluss and Apache Hudi, Azure SQL Database Hyperscale internals, DuckDB 2.0.0 features, and how Flink's Kafka connector commits offsets to Kafka.

Decision models like Jev prioritize fast answers paired with calibrated probabilities rather than generating text. Data engineers can integrate these models into data analytics pipelines using the dbt-decisions package, linking AI decision outputs directly with a semantic layer inside dbt workflows.

AI agents querying revenue data often fail on fan-out joins, time zones, and test accounts despite valid SQL. Test them against frozen golden datasets, comparing output values within tolerances rather than syntax. Build an exam on reconciled numbers to catch semantic modeling errors early.

A/B testing GLM-5.2 against Claude Opus revealed that agent harnesses matter as much as model intelligence. GLM ran faster per call but took 14% longer overall, once executing the same SQL query 254 times due to ambiguous tool contracts and missing limits. Engineers must bound repetitive calls.

Snowflake released CoCo Desktop v1.23.0 alongside continuous and bulk file loading updates. Recent data engineering enhancements include General Availability for secondary indexes and preview support for DCM Projects task graphs and Time Travel on Apache Iceberg tables.

Snowflake added Cortex AI usage tracking to Native App usage history. The release updates telemetry and governance for data engineers running AI workloads in apps, alongside previews for Apache Iceberg Time Travel and DEFINE TASK GRAPH in DCM Projects to manage pipelines.

Enterprise AI bottlenecks have shifted from model reasoning to reliable data architectures. Instead of siloed vector databases, teams need unified platforms pairing SQL with native vector search and open formats like Apache Iceberg to deliver minimal, high-quality context directly to agents.

Omni introduced viz layers for X/Y charts, letting engineers configure independent mark types, axes, and aggregations per measure. Smart aggregation automatically enables required query totals to handle varying levels of detail without custom code.

Decision models reshape ETL by offering fast, typed transformations instead of text generation. OpenAI's Decisions API evaluates multimodal inputs like images via predicates, choices, and scores with zero output tokens, enabling flexible, low-latency automated pipeline actions.

Survey data shows 59% of teams use AI analytics, with 50% wanting AI agents querying data via MCP. Infrastructure control remains critical, as 44% require running on their own servers. Yet 31% do not know what a semantic layer is, despite it being essential for trustworthy AI.

Optimizing an analytics agent across 59 data warehouse marts improved accuracy from 49.6% to 96.5%. While basic schema profiling lowered query costs, reaching top performance required strict schema documentation, table grain definitions, join rules, and constrained semantic context.

Snowflake updated Cortex Agents Coding Agent with thinking effort controls, system instructions, and stage mounts. Recent releases also added Time Travel for Apache Iceberg tables and bulk loading with CoCo.

Snowflake added Cortex Code skills for interactive analytics alongside continuous and bulk file loading with CoCo. Data platform capabilities expand with preview support for Time Travel on Apache Iceberg tables and atomic deployments for task graphs.

Poorly written SQL queries drive up BigQuery costs. The open-source BigQuery FinOps Agent uses a multi-agent workflow to collect table metadata, refactor code, and validate output schemas. Using dry-runs, it cuts processed bytes while ensuring the behavior remains identical.