LLM resources for data engineers

Denver data leaders are gathering to discuss scaling data-driven organizations through robust infrastructure and team incentives. The session explores how AI is actively reshaping workflows, evolving data team roles, and transitioning analytical pipelines into collaborative multiplayer analytics.

dltHub launched job-inspector in public preview to diagnose failed pipeline jobs. Running with read-only telemetry access, the background agent analyzes logs and definitions to post the root cause and evidence to Slack. It logs findings to a Context Graph to improve future failure analysis.

Snowflake previewed Snowflake Decision with AI_COMPLETE for automated decisions. Supporting updates for data pipelines include dynamic table incremental refreshes after failover and Snowpipe Streaming support for partitioned Apache Iceberg tables.

Wire 4.x introduces a multi-agent Release Director Model for Claude Code and Gemini CLI, orchestrating warehouse migrations and analytics delivery. It adds dbt Charts, Looker-to-Omni migrations, and support for dbt Labs' Agents Schema to house agent context directly inside data warehouses.

Snowflake cost optimization requires granular tracking across compute pools and Cortex AI functions. Simple fixes like right-sizing warehouses and reducing job frequencies yield the highest ROI, while observability views help detect token spikes before limits are breached.

San Diego Data Foundations focuses on engineering data platforms ready for AI. Engineers will explore managing production Cassandra clusters with AI-driven operations for faster incident triage.

MotherDuck integrates Jev via prompt_jev() to execute fast decision-model evaluations directly in SQL queries on every row. It classifies text into closed sets like scores and labels alongside telemetry spans, separating plain SQL counts from subjective judgment calls.

Using Claude with Model Context Protocol (MCP) replaces BI interfaces by querying warehouses directly, but shifts pipeline maintenance to you. Without a semantic layer, ad-hoc queries yield inconsistent metrics, and schema changes silently break data.

The nao community session highlights custom stories for building vibe-coded applications aligned with user design systems. Data engineers can leverage new migration tools to transfer Metabase and Tableau dashboards into nao, focusing on personal BI, agentic analytics, and context engineering

Snowflake announced the general availability of the Cortex Agent code execution tool. This feature enhances agentic workflows by enabling automated code execution directly within the Snowflake data platform.

Snowflake announced the public preview of Agentic Marketplace Discovery in its feature updates. Recent data engineering updates also include Iceberg partition evolution, Elastic Channels for Snowpipe Streaming, and dynamic table incremental refresh support for non-deterministic functions.

AI context layers degrade as schemas and metrics drift. Prevent stale metadata using four loops: local task fixes, CI pipeline validation, scheduled audits, and channel feedback. Every loop requires a tracked pull request and human review. Agents only propose changes, never silently editing context.

Executable UDFs are now generally available on ClickHouse Cloud across AWS, GCP, and Azure. Data engineers can deploy precompiled binaries in Rust, Go, C++, or JavaScript alongside Python, manage resources via Terraform, and leverage outbound network access and per-query execution metrics.

jev4pg adds natural language to SQL and semantic operators to PostgreSQL. It features 41 JEV operators for filtering, extraction, and verification. PostgreSQL manages joins and windows while JEV uses parallel execution, input deduplication, and evidence caching for efficient query planning.

Hands-on workshops detail how to wire up data pipelines and run agents in production using DuckDB and MotherDuck. Engineers build conversational architectures to profile raw files, generate schemas, execute transformations, run data quality checks, and host MCP servers for automated workloads.

MotherDuck introduces prompt_jev() to query unstructured text columns like agent traces, comments, and job ads directly in SQL. Running roughly 20 times faster than standard LLM calls at lower cost, it lets data engineers extract insights and group business metrics within a single SQL pipeline.

Spec-driven data engineering replaces ad-hoc AI prompts with formal specifications, business concepts, and engineering standards. Using agentic AI in Snowflake, engineers transition from subjective trial and error to repeatable pipelines, transforming one-off assistance into measurable

The Rust SDK for BigQuery is now generally available. Additionally, the Data Engineering Agent supports the gemini-3.7-flash model across us, eu, and global multi-regional endpoints.

Snowflake has released observability support for Cortex Agents within Snowflake Native Apps. This update expands telemetry and tracking for agentic workflows deployed natively, complementing data pipeline lineage tracking.

To connect AI agents to QuickBooks, split reads from writes. Bruin and ingestr load QuickBooks data into BigQuery or DuckDB to model metrics via SQL, while Intuit's MCP server handles writes. This architecture avoids API rate limits and ensures agents query reliable, pre-reconciled data.

Astronomer launched the astronomer-data plugin for ChatGPT, connecting to managed or self-hosted Apache Airflow. Data engineers can debug DAG failures from task logs, scaffold pipelines, query data warehouses, and trace lineage using natural language.

A semantic layer translates raw tables into standardized metrics, dimensions, and business logic. It provides machine-readable context for AI models, reducing hallucinations and avoiding duplicate SQL definitions across dashboards. Modern workflows now infer definitions directly from query history.

Orchestra Labs introduced Orchestra Markup Language (.oml) for pipeline orchestration. Unlike plain YAML, it enforces schema validation and real-time typing at write-time, eliminating agent errors and cutting config generation time by over 90%.

AI agents require explicit metadata because, unlike humans, they lack ingrained business knowledge. Using raw schemas yields shallow descriptions, so engineering context pipelines requires feeding lineage, upstream queries, and glossary terms left-to-right across transformations.

🎙️ The Data Flowcast by Astronomer — Airflow orchestrates local LLM pipelines for sensitive data, running GPU inference via KubernetesPodOperator into feature stores. DAGs manage ingest, Pydantic validation to stop silent failures, and PEFT fine-tuning. Production requires strict pinning and tracing adapter provenance per row.

Data engineering updates highlight DuckDB 1.5.6, Parquet X-ray, and billion-row record linkage with Splink 5. Key analyses cover streaming pipeline cost efficiency, PostgreSQL row-level security performance, handling hot shards, and building analytical MySQL replicas with Parquet and DuckDB.

Custom data apps replace static dashboards by pairing tailored views with row-level permissions and a governed metric layer. Unlike internal workflow tools, they focus on operational decisions.

Snowflake released CoCo Desktop v1.21.6 alongside major data engineering updates. Key additions include GA for inline stored procedures and check constraints on hybrid tables, plus Snowpipe Streaming support for partitioned Apache Iceberg tables and incremental refreshes on dynamic tables.

Snowflake announced general availability for Semantic Studio alongside DCM Projects DEFINE SEMANTIC VIEW. Data engineers also get general availability for Optimized Refresh and RPO Assurance on failover groups, plus previews for DCM masking policy attachments and a Spark Monitoring UI.

Data engineering teams are updating workflows and architectures to adapt to AI. Tools like MotherDuck and dltHub are evolving to meet these demands, shifting requirements for engineers.