Agentic resources for data engineers

🎙️ DataTalksClub ⬛ — Building production AI platforms requires managing evaluation datasets, LLM memory, and tool orchestration. Key technical patterns include mitigating context window limits, optimizing tool calling, and overcoming retrieval degradation like the lost in the middle problem in RAG pipelines.

AI systems require computer vision as core data infrastructure to process massive uncurated visual streams. Edge architectures must run low-latency vision-language-action pipelines locally because shipping high-throughput sensor frames to cloud datacenters introduces critical latency.

AI agents require data platforms to evolve beyond static, clean datasets toward fresh, reliable, contextual, and observable data. Engineering modern architectures for agentic workflows demands real-time data pipelines, robust governance, automated data quality, and agent-ready discovery mechanisms.

dltHub launched job-inspector in public preview to diagnose failed pipeline jobs. Running with read-only telemetry access, the background agent analyzes logs and definitions to post the root cause and evidence to Slack. It logs findings to a Context Graph to improve future failure analysis.

Snowflake previewed Snowflake Decision with AI_COMPLETE for automated decisions. Supporting updates for data pipelines include dynamic table incremental refreshes after failover and Snowpipe Streaming support for partitioned Apache Iceberg tables.

Wire 4.x introduces a multi-agent Release Director Model for Claude Code and Gemini CLI, orchestrating warehouse migrations and analytics delivery. It adds dbt Charts, Looker-to-Omni migrations, and support for dbt Labs' Agents Schema to house agent context directly inside data warehouses.

AI agents still require Kubernetes to run, scale, and recover reliably in production. However, because agents spend up to 90% of their time idle, assigning dedicated pods wastes resources; new patterns like Agent Substrate decouple agent sessions from pods to efficiently pack workloads.

Infino launched a unified agent retrieval layer that queries Parquet in object storage directly via SQL. By combining vector search, keyword search, joins, and aggregates into one query over a single copy of data, it eliminates fragmented ETL pipelines, sync jobs, and multi-database sprawl.

Data architectures often confuse system permissions with authority. Automated agents defer strictly to documented role schemas, missing informal engineering influence. This gap drives technical debt and rework, proving that ontologies must resolve real decision authority across operational domains.

🎙️ DataFramed — RADAR 11x explores the evolving engineering landscape, focusing on AI-driven development life cycles, agentic workflows, and production hurdles. Engineering leaders address critical operational patterns like cutting token costs and navigating why AI pilots stall before reaching production.

Modern semantic layers often require excessive glue code for data pipelines and agents. SLayer offers an open-source alternative under the MIT license, combining context tooling with an integrated query engine capable of handling complex multi-stage analytical queries out of the box.

SAP is acquiring TechWolf to feed its three-layer context graph for skills and work into HR AI agents. The platform ingests business application data, infers skills, and writes them back to SAP SuccessFactors, optimizing LLM token efficiency and agent query grounding.

MotherDuck integrates Jev via prompt_jev() to execute fast decision-model evaluations directly in SQL queries on every row. It classifies text into closed sets like scores and labels alongside telemetry spans, separating plain SQL counts from subjective judgment calls.

SAP is embedding autonomous Joule AI agents across its Cloud ERP backend, automating data pipelines for billing reconciliation, statutory reporting, and transactions. Agents inherit built-in database governance, access controls, and compliance rules directly from core ERP systems.

AI agents break data architecture assumptions by automating rapid system exploitation. Teams must secure execution harnesses, separate read and write permissions, isolate message queues and shared databases, and stop using permanent credentials to limit blast radius across environments.

Databricks and Snowflake drive growth by tying compute engines directly to data state, warehousing, and AI tools. Databricks expands beyond core lakehouse analytics into serverless operational Postgres with Lakebase while scaling AI gateways to sustain massive revenue and enterprise valuations.

Orchestra is developing an agentic control plane for data and AI, providing a centralized architecture to build, run, monitor, and govern data pipelines alongside AI agents.

Snowflake announced the general availability of the Cortex Agent code execution tool. This feature enhances agentic workflows by enabling automated code execution directly within the Snowflake data platform.

Snowflake announced the public preview of Agentic Marketplace Discovery in its feature updates. Recent data engineering updates also include Iceberg partition evolution, Elastic Channels for Snowpipe Streaming, and dynamic table incremental refresh support for non-deterministic functions.

AI context layers degrade as schemas and metrics drift. Prevent stale metadata using four loops: local task fixes, CI pipeline validation, scheduled audits, and channel feedback. Every loop requires a tracked pull request and human review. Agents only propose changes, never silently editing context.

Snowflake introduced AI Gateway for Advertising, connecting Cortex Agents to Meta and TikTok via Model Context Protocol. It bridges warehouse data like transactions and KPIs with live platform signals to automate cross-channel campaign analysis and optimization.

Amazon Aurora PostgreSQL now supports direct querying of Apache Iceberg and Parquet data lake files. Other data engineering highlights include extending Apache DataFusion across machines, using ClickHouse for metric discovery at LinkedIn, and DuckDB 2.0 adding dynamic ATTACH statements.

Orchestra introduced .oml and rolled out anomaly detection for long-running dbt models alongside an enterprise audit trail. The platform added state machine task pausing, environment branch controls, augmented managed agents, and Google Kubernetes integration.

Hands-on workshops detail how to wire up data pipelines and run agents in production using DuckDB and MotherDuck. Engineers build conversational architectures to profile raw files, generate schemas, execute transformations, run data quality checks, and host MCP servers for automated workloads.

Data engineering oscillates through recurring trade-offs like centralization versus decentralization and batch versus streaming. As bottlenecks emerge, teams decentralize pipelines and compute, only to later recentralize metadata, governance, and semantic layers when coordination breaks down.

Marimo Studio transforms reactive Python notebooks into interactive apps, dashboards, and static reports. It keeps computation, data, and controls centralized in the notebook while serving distinct frontend views. Notebook cells update dependently as input controls change.

Big Data LDN highlighted that AI agents require strong data governance, context, and semantics to scale into production. Operationalizing agentic systems demands investments in semantic layers, ontologies, and trusted metadata architectures to close critical foundation gaps.

Spec-driven data engineering replaces ad-hoc AI prompts with formal specifications, business concepts, and engineering standards. Using agentic AI in Snowflake, engineers transition from subjective trial and error to repeatable pipelines, transforming one-off assistance into measurable

The Rust SDK for BigQuery is now generally available. Additionally, the Data Engineering Agent supports the gemini-3.7-flash model across us, eu, and global multi-regional endpoints.

Snowflake has released observability support for Cortex Agents within Snowflake Native Apps. This update expands telemetry and tracking for agentic workflows deployed natively, complementing data pipeline lineage tracking.