Agentic resources for data engineers

Frontier AI models no longer require external dbt skills for standard analytics engineering pipelines. In an evaluation on Snowflake, modern agents cleared data checks natively; failures stemmed from project processes rather than missing dbt syntax.

Skipping data engineering for AI is a disaster. Pointing LLMs at messy source tables fails; solid modeling remains essential. Meanwhile, AI tools like Cortex Code accelerate SQL Server to Snowflake migrations, converting code to dbt projects or stored procedures in hours rather than weeks.

dbt Charts introduces an open-source, dashboards-as-code spec designed for AI agents to write and engineers to audit. Aiming to bypass complex BI layers, it lets teams rely more on relational data architectures instead of pre-joining models solely to satisfy traditional BI tools.

An AI harness governs how models generate data engineering pipelines by enforcing contracts, metadata, and operational controls. Rather than letting AI guess logic, the harness defines the output grain, metric rules, and validation checks before generating transformations.

Snowflake rolled out CoCo Desktop v1.23.1 alongside continuous and bulk file loading capabilities. Engineering updates also highlight Apache Iceberg table time travel and task graph enhancements.

Apache Flink Agents 0.4.0 brings AI agents into streaming pipelines, providing distributed execution and fault tolerance. The update adds sub-agents with durable execution, dynamic model routing, and the unified Decision Model API for scoring against business logic.

🎙️ The Analytics Engineering Podcast (dbt) — dbt Charts introduces a human-readable spec for dashboards that AI can generate and data teams can audit. Dave Fowler calls it "the Markdown of dashboards" to tackle BI bottlenecks and unreviewable generated code, while exploring how AI may eventually reduce reliance on dimensional modeling.

Data engineering hiring is recovering, but employers now expect more than traditional SQL, Python, and Airflow pipelines. Engineers must build AI-enabled systems and master tools like Databricks, dbt, and AWS, even as junior opportunities remain limited to just 5.5% of openings.

Tares ingests distributed events like logs, metrics, deploys, and database rows into unified, time-ordered entity timelines. Event-driven triggers wake agents via MCP or webhooks with pre-correlated context, persisting state and decisions across pipeline runs.

Data roles are shifting from core reporting to operational automation and data ingestion. Engineers must close data action loops using domain data, virtual knowledge graphs, and LLM-driven agents. Understanding pipeline trade-offs remains essential as automation absorbs routine tasks.

Google Cloud is expanding Knowledge Catalog to ground data agents in a unified context graph. The architecture connects trusted semantic models across analytics pipelines, enabling engineers to link disparate metadata frameworks and deliver consistent context to automated systems.

AI integration requires strong data engineering foundations. Fivetran advocates for open formats, portable storage, and reliable data pipelines to power AI agents securely, noting that data should not be trapped in closed systems. Reliable, well-governed data pipelines make AI accountable.

Nao launched a cloud UI for its open-source agentic BI builder. It syncs warehouse metadata across systems like Snowflake and ClickHouse to build a governed context layer of schemas, metrics, and rules. It features offline testing and MCP integration to serve governed data directly to AI agents.

Data engineers are shifting from manual modeling to building automated systems that continuously maintain semantic layers. Using production query feedback and Semantic SQL, these architectures let consumers combine governed measures while agents propose model updates and pre-aggregations.

🎙️ The Joe Reis Show — AI cannot solve poor data modeling. Feeding an entire enterprise schema into an LLM produces data slop. Instead, agents are shifting pipelines from bottom-up ETL to deterministic top-down domain modeling, highlighting semantic layers and Malloy to separate modeling logic from queries.

MotherDuck introduced Dive Apps, allowing visualizations to trigger Flights, a serverless Python runtime. This architecture enables two-way data pipelines that execute write-backs into source systems directly from warehouse-backed interfaces.

Snowflake launched SnowPR to benchmark AI coding agents on its 43-million-line Bazel monorepo. Across 65 real tasks spanning Java control plane and C++ execution engine services, Cortex Code in REM mode solved 62 tasks at low cost, handling complex setups with multi-target builds and FoundationDB.

ClickHouse introduced GroeneAI, an autonomous agent resolving CI test failures across the codebase. It reproduces bugs, builds ClickHouse, runs tests, and opens pull requests without humans in the loop, targeting database stability and continuous integration hygiene. Maintainers review each fix.

MiniLake is a local Databricks API emulator that executes real SQL on DuckDB, writes genuine Delta files, and runs containerized spark-submit jobs. Designed for CI and local development without cloud costs, it natively supports Unity Catalog CRUD, DAG job scheduling, and official SDK clients.

Snowflake released AIPARSEDOCUMENT in AGENTIC mode for intelligent document processing in public preview. Concurrently, data engineers get preview access to Time Travel for Apache Iceberg tables and DEFINE TASK GRAPH in DCM Projects for orchestrating complex pipelines.

AI agents querying revenue data often fail on fan-out joins, time zones, and test accounts despite valid SQL. Test them against frozen golden datasets, comparing output values within tolerances rather than syntax. Build an exam on reconciled numbers to catch semantic modeling errors early.

A/B testing GLM-5.2 against Claude Opus revealed that agent harnesses matter as much as model intelligence. GLM ran faster per call but took 14% longer overall, once executing the same SQL query 254 times due to ambiguous tool contracts and missing limits. Engineers must bound repetitive calls.

Snowflake released CoCo Desktop v1.23.0 alongside continuous and bulk file loading updates. Recent data engineering enhancements include General Availability for secondary indexes and preview support for DCM Projects task graphs and Time Travel on Apache Iceberg tables.

Enterprise AI bottlenecks have shifted from model reasoning to reliable data architectures. Instead of siloed vector databases, teams need unified platforms pairing SQL with native vector search and open formats like Apache Iceberg to deliver minimal, high-quality context directly to agents.

Survey data shows 59% of teams use AI analytics, with 50% wanting AI agents querying data via MCP. Infrastructure control remains critical, as 44% require running on their own servers. Yet 31% do not know what a semantic layer is, despite it being essential for trustworthy AI.

Optimizing an analytics agent across 59 data warehouse marts improved accuracy from 49.6% to 96.5%. While basic schema profiling lowered query costs, reaching top performance required strict schema documentation, table grain definitions, join rules, and constrained semantic context.

Snowflake updated Cortex Agents Coding Agent with thinking effort controls, system instructions, and stage mounts. Recent releases also added Time Travel for Apache Iceberg tables and bulk loading with CoCo.

Amazon OpenSearch Service's Search Relevance Agent automates query tuning pipelines. It analyzes behavioral UBI indexes, adjusts DSL field weights and scoring, and validates improvements via offline SRW experiments measuring NDCG, MAP, and Precision through a secure MCP server interface.

Hex raised $100M alongside releasing Hex 4.0, shifting its analytics architecture toward an agent-native data platform.

Poorly written SQL queries drive up BigQuery costs. The open-source BigQuery FinOps Agent uses a multi-agent workflow to collect table metadata, refactor code, and validate output schemas. Using dry-runs, it cuts processed bytes while ensuring the behavior remains identical.