Governance resources for data engineers

AI agents require data platforms to evolve beyond static, clean datasets toward fresh, reliable, contextual, and observable data. Engineering modern architectures for agentic workflows demands real-time data pipelines, robust governance, automated data quality, and agent-ready discovery mechanisms.

To build reliable healthcare AI, data engineers pair automated ingestion via Fivetran with governed transformations and lineage using dbt Labs. Integrated with Snowflake, Tuva Health, and Omni, this architecture establishes shared data definitions and strict governance for conversational analytics.

Snowflake completed an independent NZISM Restricted assessment on AWS in Auckland, enabling New Zealand public sector data pipelines to meet local security compliance while centralizing workloads and deploying AI models securely.

Data architectures often confuse system permissions with authority. Automated agents defer strictly to documented role schemas, missing informal engineering influence. This gap drives technical debt and rework, proving that ontologies must resolve real decision authority across operational domains.

Snowflake previewed the ability to enforce CoCo policy with MDM. Related updates include incremental refresh support for dynamic tables, partition evolution for Apache Iceberg tables, and Snowpipe Streaming monitoring via event tables.

SAP is embedding autonomous Joule AI agents across its Cloud ERP backend, automating data pipelines for billing reconciliation, statutory reporting, and transactions. Agents inherit built-in database governance, access controls, and compliance rules directly from core ERP systems.

Omni built its semantic layer as an in-memory data structure rather than raw text. Using Apache Calcite, it translates structured SQL expressions into relational algebra to generate performant queries, relying on a core extendFrom architecture to balance governance with user-defined extensions.

Using Claude with Model Context Protocol (MCP) replaces BI interfaces by querying warehouses directly, but shifts pipeline maintenance to you. Without a semantic layer, ad-hoc queries yield inconsistent metrics, and schema changes silently break data.

Self-serve analytics and AI tools often cause metric drift, yielding conflicting results like different calculations for net revenue retention across teams. The solution requires establishing standardized metric definitions and feeding accurate business context into AI systems to ensure consistency.

El evento aborda la arquitectura y gobierno de datos, calidad, linaje, catálogos de metadatos, seguridad e integración analítica. Presenta casos reales sobre la gestión de datos orientada a inteligencia artificial, destacando la trazabilidad técnica y la madurez en plataformas de datos.

AI context layers degrade as schemas and metrics drift. Prevent stale metadata using four loops: local task fixes, CI pipeline validation, scheduled audits, and channel feedback. Every loop requires a tracked pull request and human review. Agents only propose changes, never silently editing context.

Snowflake introduced AI Gateway for Advertising, connecting Cortex Agents to Meta and TikTok via Model Context Protocol. It bridges warehouse data like transactions and KPIs with live platform signals to automate cross-channel campaign analysis and optimization.

Data engineering oscillates through recurring trade-offs like centralization versus decentralization and batch versus streaming. As bottlenecks emerge, teams decentralize pipelines and compute, only to later recentralize metadata, governance, and semantic layers when coordination breaks down.

Platform engineering perfected infrastructure like Kafka and Snowflake, but data payloads still break downstream. Supercargo enforces shift-left data contracts and CI/CD gates to inspect source data models before pull requests merge, protecting pipelines from schema drift and unencrypted PII.

Big Data LDN highlighted that AI agents require strong data governance, context, and semantics to scale into production. Operationalizing agentic systems demands investments in semantic layers, ontologies, and trusted metadata architectures to close critical foundation gaps.

Tabby is hosting a technical meetup covering pipelines, warehousing, feature stores, and operating multi-region infrastructure under payment load. Sessions focus on aligning AI strategy with data platforms, embedding governance directly into pipelines via metadata and lineage, and the Tabby DWH

Snowflake introduced User-managed Restricted Session Scope in CoCo as generally available. Alongside updates like zero-copy support for all table formats and write support for nested namespaces, this feature enhances session security and access control across data engineering workflows.

A semantic layer translates raw tables into standardized metrics, dimensions, and business logic. It provides machine-readable context for AI models, reducing hallucinations and avoiding duplicate SQL definitions across dashboards. Modern workflows now infer definitions directly from query history.

AI agents require explicit metadata because, unlike humans, they lack ingrained business knowledge. Using raw schemas yields shallow descriptions, so engineering context pipelines requires feeding lineage, upstream queries, and glossary terms left-to-right across transformations.

BigQuery pipelines now support automated metadata enrichment and Knowledge Catalog data quality scorecard integration. Additionally, the Data Engineering Agent can proactively generate semantic metadata for pipeline assets. These features are generally available.

Public sector operations rely on underlying data infrastructure to collect, validate, transform, and exchange records across legacy ERP systems, GIS, and databases. Reliable government services require robust pipelines that maintain data quality across integrated administrative workflows.

Freelancers suit scoped data jobs connecting simple sources, but lack continuity. Managed services handle ongoing ETL pipeline breaks, semantic governance, source schema changes, and metric upkeep when teams lack internal data engineers to review SQL or maintain dashboards long-term.

Custom data apps replace static dashboards by pairing tailored views with row-level permissions and a governed metric layer. Unlike internal workflow tools, they focus on operational decisions.

Managed analytics services handle dashboard maintenance, pipeline upkeep, and source schema changes. A reliable service monitors daily data refreshes, prevents schema drift when source APIs change, and fixes broken pipelines so teams avoid returning to spreadsheet exports.

Avoid duplicating system facts in docs for AI agents to prevent drift. Point agents to inherent metadata like code, configs, and logs. Add explicit context only for missing business definitions, and establish a single source of truth since conflicts produce confident errors.

Data Outpost examines modern data architectures adapting to autonomous agents that read and write data. Sessions focus on building scalable insight platforms, managing automated semantic layers, integrating operational schemas, and handling the oversight, latency, and cost constraints of production

BigQuery launched the Security center in GA to manage row- and column-level security policies and governance tags. Additionally, query execution plans now display steps for UPDATE, DELETE, MERGE, and EXPORT operations, improving performance optimization for DML and data export pipelines.

🎙️ It's About Data® — Tim Gasper examines the core drivers of data governance across structured and unstructured data architectures, evaluating both incentives and enforcement strategies. The discussion also addresses whether the industry is facing a genuine SaaSpocalypse.

Snowflake released a preview of ATTACH MASKING POLICY for DCM Projects. The update expands data governance and column-level security within declarative project workflows, alongside general availability for DEFINE SEMANTIC VIEW.

Treating OpenTelemetry as enterprise data shifts observability from incident triage to business analytics. Combining telemetry with open formats like Apache Iceberg and Snowflake allows data engineers to govern logs, metrics, and traces alongside customer tables for AI and impact modeling.