Governance resources for data engineers

Amazon SageMaker Unified Studio introduces designations, named permission sets backed by the Cedar policy language to enforce least privilege within shared projects. Administrators can isolate scheduled ETL pipelines, notebook compute, and catalog stewardship across personas in a single project.

Salesforce cannot ingest granular event telemetry at scale or model complex predictive insights due to strict API limits and compute constraints. Reverse ETL pipelines bridge this by syncing warehouse-modeled data and external IDs back to CRMs through governed, automated workflows.

Data engineering teams must manage semantics, not just storage. Modern AI requires treating business policies and context as core data assets to prevent hallucinations caused by outdated pipelines. As leaders note, if you have chaos, AI is just going to amplify that chaos. Keep humans over the loop.

🎙️ The Analytics Engineering Podcast (dbt) — dbt Charts introduces a human-readable spec for dashboards that AI can generate and data teams can audit. Dave Fowler calls it "the Markdown of dashboards" to tackle BI bottlenecks and unreviewable generated code, while exploring how AI may eventually reduce reliance on dimensional modeling.

Data roles are shifting from core reporting to operational automation and data ingestion. Engineers must close data action loops using domain data, virtual knowledge graphs, and LLM-driven agents. Understanding pipeline trade-offs remains essential as automation absorbs routine tasks.

AI integration requires strong data engineering foundations. Fivetran advocates for open formats, portable storage, and reliable data pipelines to power AI agents securely, noting that data should not be trapped in closed systems. Reliable, well-governed data pipelines make AI accountable.

Silent pipeline bugs often stem from boundary mismatches and unwritten definitions rather than crashes. A healthy job run does not guarantee correctness. Robust platforms centralize business rules across modular models and use external reconciliation checks to catch silent join drops.

Waiting for a fully modernized data platform delays AI value. Teams should run AI delivery and infrastructure modernization in parallel, using a unified policy layer to govern data across legacy systems in place while migration pipelines continue independently.

Snowflake release notes detail platform updates, including Apache Iceberg table enhancements like Time Travel preview, secondary indexes, and DCM DEFINE TASK GRAPH for workflow orchestration. Other features include declarative feature stores and bulk file loading to optimize data pipelines.

AI agents querying revenue data often fail on fan-out joins, time zones, and test accounts despite valid SQL. Test them against frozen golden datasets, comparing output values within tolerances rather than syntax. Build an exam on reconciled numbers to catch semantic modeling errors early.

Choose SAP OData patterns by workload: use CDS views for read-heavy analytics, and custom RAP services for complex transactional logic. Crossing hybrid boundaries via Cloud Connector adds latency, making metadata caching and tuning expand payloads critical for remote BTP performance.

Enterprise AI bottlenecks have shifted from model reasoning to reliable data architectures. Instead of siloed vector databases, teams need unified platforms pairing SQL with native vector search and open formats like Apache Iceberg to deliver minimal, high-quality context directly to agents.

Survey data shows 59% of teams use AI analytics, with 50% wanting AI agents querying data via MCP. Infrastructure control remains critical, as 44% require running on their own servers. Yet 31% do not know what a semantic layer is, despite it being essential for trustworthy AI.

Amazon OpenSearch Serverless updated its time-series architecture, separating stateless compute units from S3-backed storage. It autoscales compute within seconds and scales to zero when idle, using automated index tiering to move aging data from fast storage to low-cost S3.

Data catalogs organize structural metadata and schemas across warehouses and lakes. Technical catalogs track schemas, access policies, and column-level lineage, enabling tools like Iceberg and Glue to manage state and evaluate the impact of schema changes across pipelines.

DoorDash consolidated its analytics architecture by decoupling the semantic layer from consumption. Data stored in Apache Iceberg and Snowflake is modeled via dbt metrics, serving ThoughtSpot and AI agents centrally. DoorDash treats semantic definitions as version-controlled code.

AWS architects proposed an architecture pairing the dbt Semantic Layer with the Amazon Redshift MCP Server to stop AI agents from writing raw SQL. MetricFlow compiles certified metrics while Redshift enforces read-only execution, preventing hallucinations and silent join errors.

KOHO built KoDa, an LLM self-serve tool connecting Claude to Redshift via MCP. It queries only governed dbt marts without PII or raw data. The team maintains an eval council of 375 real analyst questions to benchmark quality, proving foundational data modeling and evals matter more than the LLM.

Guitar Center rebuilt its data stack using Snowflake and managed dbt Core, replacing legacy stored procedures with CI/CD and automated tests. Scoped builds and smaller warehouses cut Snowflake compute costs by 40% and Airflow usage by 50%, speeding up pipeline delivery from months to days.

AI agents require shared context across structured and unstructured data to act reliably. Fivetran Context Layer addresses this by indexing docs and data sources into warehouse tables called Agents Schema, letting agents query unified ontologies directly via MCP instead of proprietary APIs.

Grounding AI analysts with a semantic layer achieved 96-100% accuracy, versus 80-91% with raw SQL against database schemas. Reliable text-to-SQL systems require data modeling, tests, and metadata context over raw schema access.

KOHO built KoDa, an internal tool pairing Claude and Redshift MCP with 1,400 governed dbt models. Restricting access to certified marts prevents PII exposure while an evaluation council logs and audits query accuracy. Reliable self-serve AI relies on robust semantic foundations over LLMs.

EF Education First tackled Snowflake costs by building a dbt scoring system tracking model runtime, credits, and throughput. They paired account usage data with BI logs into a semantic layer queried via Snowflake Cortex Agent, driving a 40 percent efficiency gain on heavy pipelines.

dbt Charts Cloud supports public dashboards defined in versioned YAML, modeled via dbt. Access requires dual-level opt-in controls across organization and project settings, letting external users view curated analytics without logins while admins retain kill-switch controls.

Kramp scaled dbt Mesh on Google BigQuery using Data Project Factory, a Terraform module that provisions GCP projects, dbt repositories, and access controls for domain teams. This self-serve architecture cut onboarding from days to minutes across 25 projects while enforcing CI governance.

Bolt scaled its analytics engineering by standardizing dbt architectures into strict intermediate shapes (spine, feature, metrics) and exposed marts. Project dependencies flow strictly one-way from horizontal to vertical domains, and models enforce single ownership with no shared logic.

Okta solved AI context fragmentation by moving agent governance into dbt. Analytics engineers manage metadata, markdown context, and semantic views via pull requests. Context syncs to Snowflake Agents Schema, serving Cortex and Claude from one unified source of truth.

Delivery Hero scaled dbt across 45 domain teams processing 60 petabytes. Using Astronomer Cosmos and custom CI scripts, the platform converts dbt manifests into Airflow DAGs with priority weights, enforcing model ownership tags and central policy macros directly in CI pipelines.

Snowflake introduced ingress policies in preview alongside server release 10.38. The security feature provides data engineers and administrators with inbound network traffic governance to restrict and control incoming connections to data platform architectures.

Snowflake announced the general availability of the Trust Center Data Security scanner package. Recent platform updates also include Snowpipe Streaming SDK v1.9.1, Apache Iceberg v3 table upgrades, Parquet Direct, and dynamic tables continuing incremental refreshes after failovers.