dataaaaa!

Snowflake DCM Projects provide native declarative infrastructure-as-code using SQL syntax. Users define objects using DEFINE instead of CREATE, following a plan-and-deploy loop where state is tracked in Snowflake. It pairs with inherited grants to replace complex future grant configurations.

Scaling personal knowledge graphs to the enterprise requires typed graphs, data pipelines, and object storage. Unstructured Markdown lacks schema guarantees, multi-user writes, and versioning.

dbt Charts open-sources a declarative YAML framework for dashboards, letting AI agents build governed BI via code. It stores dashboard definitions alongside dbt models in Git, enabling CI validation and rendering to static assets like SVG or HTML via a dedicated CLI tool.

Snowflake announced general availability for inherited grants and container-level MANAGE GRANTS privileges. This update simplifies access control architectures by streamlining permission inheritance across data containers. Other features include dbt project Slim CI and Iceberg Scan Plan API.

Kestra 2.0 expands plugins beyond integrations to platform internals, adding custom policy rules, file renderers, and log data stores. Notably, flow-level pluginDefaults is removed, and open source now supports external log data stores to streamline migrations.

Databricks Agent Bricks manages production AI agents directly on Unity Catalog. It embeds model access, context, and governance into existing data infrastructure, solving security and deployment debt by enforcing data access via the user permissions and lineage already governing production tables.

Quickly built internal tools carry hidden long-term technical debt. Upkeep and maintenance consume 60% to 80% of total lifetime costs. Engineers provide irreplaceable value by evaluating architecture fit and life-cycle costs rather than just immediate delivery.

A context layer feeds AI agents unstructured business documentation alongside structured data. Unlike a semantic layer defined in dbt or LookML, it captures query history, joins, and internal notes from Slack or Notion to refine SQL accuracy and verify automated data transformations.

Snowflake introduced data lineage for streams, enhancing visibility into CDC pipelines. This update helps data engineers track pipeline dependencies and schema transformations across change tracking objects directly in the data platform.

Snowflake introduced external secret providers in preview for managing credentials. Additionally, dbt Projects on Snowflake now supports Slim CI and defer to production using dbt artifacts. Data engineers can also manage hybrid table constraints online and leverage external lineage features.

🎙️ It's About Data — Justin Borgman argues that building data products requires context engineering, emphasizing that context engineering is a human in the loop process. Data engineers must integrate domain knowledge directly into data architectures.

Focus on data distribution over storage or collection. Effective data architecture builds around primitives that preserve optionality rather than fixed answers, ensuring systems adapt as business needs shift. Treat data strategy as continuous work rather than a static roadmap.

AI generates rapid data queries, creating ungoverned code sprawl across pipelines. Data engineers face untraceable context and dropped table filters, breaking observability and metric definitions. Traditional BI cannot solve this governance breakdown as non-technical users bypass rigid platforms.

ChatGPT generates Python scripts via Code Interpreter, while Claude renders interactive artifacts via MCP connectors. Both suit prototyping, but data engineers cannot use them as production dashboards due to missing semantic layers, absent governance, lack of SLAs, and manual schema upkeep.

Embedded analytics adoption fails when users do not trust data or reach insights fast. Data engineers should implement a governed semantic layer and tenant-safe row-level security to prevent metric drift and cache issues. Track behavioral events rather than seat counts to measure real usage.

Claude builds interactive dashboards via MCP connectors to query live SQL databases and warehouses. While effective for prototyping, it lacks production data engineering requirements: there is no shared semantic layer, unified access control, or automated pipeline maintenance when schemas change.

Snowflake published updates for Snowflake Data Clean Rooms alongside several data engineering features. Key platform enhancements include general availability for online modification of hybrid table constraints, external lineage tracking, and second generation Openflow runtimes and deployments.

Salesforce launched Enterprise AI Harness, integrating Data 360, Informatica, and MuleSoft into a unified control plane to halt siloed data. It unifies context, API orchestration, and governance across batch and real-time pipelines, letting teams deploy only the integrations they need.

Snowflake introduced a secure by default approach for scanner package enablement in Trust Center. This release enhances security automation for data engineers and platform administrators managing Snowflake accounts. Related updates include online constraint modifications for hybrid tables.

Snowflake launched Intent-Driven Governance, a spec-driven workflow in Snowflake CoCo. It converts plain-English intent into tags, classification profiles, and masking DDL through stage-gated approvals. It features version-controlled artifacts, non-mutating prechecks, and scheduled drift detection.

Lift-and-shift rewrites to Databricks spike DBU costs by turning unoptimized queries into metered waste. Avoid line-by-line translation. Instead, audit portfolios, kill zombie jobs, and consolidate redundant filter variants into unified Gold tables to migrate business capabilities, not technical

Pointing AI agents at raw warehouse tables causes inconsistent metrics. To ensure reliability, teams must implement a semantic layer and tested data models so agents query validated business definitions instead of guessing schema nuances. Exploration can be probabilistic, but published BI cannot.

me&u migrated four BI tools to Omni to unify their regionalized Snowflake data stack. By implementing YAML-based semantic modeling in Git alongside dbt, the team established governed joins and definitions that support direct warehouse querying, self-service dashboards, and Claude via MCP.

Centralizing data is not enough; retail and CPG pipelines require dbt to validate data integrity. Automated dbt tests catch broken feeds, while the dbt Semantic Layer defines shared metrics. Lineage graphs via dbt Explorer trace downstream numbers back to source systems for reliable automation.

Data catalogs are shifting from human-facing documentation to agentic infrastructure. To supply the necessary context, metadata lakehouses store query history, lineage, and usage patterns as open Iceberg tables queryable via SQL, allowing agents to automate documentation and metadata generation.