Catalogs resources for data engineers

Snowflake announced the general availability of Horizon Iceberg REST Catalog telemetry, expanding monitoring for Apache Iceberg metadata operations. Related updates include general availability for Apache Iceberg v3 table upgrades and Tri-Secret Secure storage support.

Google Cloud is expanding Knowledge Catalog to ground data agents in a unified context graph. The architecture connects trusted semantic models across analytics pipelines, enabling engineers to link disparate metadata frameworks and deliver consistent context to automated systems.

MiniLake is a local Databricks API emulator that executes real SQL on DuckDB, writes genuine Delta files, and runs containerized spark-submit jobs. Designed for CI and local development without cloud costs, it natively supports Unity Catalog CRUD, DAG job scheduling, and official SDK clients.

Using pyIceberg with AWS Glue streamlines data lake management on S3. Configuring client properties allows shared credentials across Glue and S3 to define schemas, create tables, and query Iceberg metadata directly using DuckDB.

The platform unifies fragmented data stacks into a single lakehouse architecture. It delivers warehouse-grade performance, storage economics, live data federation, and agent-ready context.

Optimizing an analytics agent across 59 data warehouse marts improved accuracy from 49.6% to 96.5%. While basic schema profiling lowered query costs, reaching top performance required strict schema documentation, table grain definitions, join rules, and constrained semantic context.

Data catalogs organize structural metadata and schemas across warehouses and lakes. Technical catalogs track schemas, access policies, and column-level lineage, enabling tools like Iceberg and Glue to manage state and evaluate the impact of schema changes across pipelines.

LangChain built a layered context architecture for AI analytics agents. They sync detailed dbt docs into BigQuery, expose DAG SQL dependencies, and use a semantic layer for metrics. Context guides are versioned in Git and imported via CI/CD, boosting self-serve questions by 40x.

AI agents require shared context across structured and unstructured data to act reliably. Fivetran Context Layer addresses this by indexing docs and data sources into warehouse tables called Agents Schema, letting agents query unified ontologies directly via MCP instead of proprietary APIs.

Snowflake Horizon now supports external Apache Iceberg writes natively. Coupled with dbt catalog configs and DuckDB upstream fixes, teams can build Iceberg tables in object storage and query governed data locally in DuckDB without touching Snowflake compute warehouses.

LookML metadata indexing in Google Cloud Knowledge Catalog is now GA. This integration unifies enterprise context and semantics across data architectures, enabling automated cataloging and governance of business intelligence assets directly within the metadata platform.

AlphaSense paired Euno context graphs with dbt to rebuild legacy data models into a governed star schema. Using Cursor, they refactored a 287-field table into 13 tested dbt models in hours instead of weeks, shifting business logic from BI back into the data layer.

Recruit Holdings improved its text-to-SQL agent by encoding UI actions and event logs into a YAML domain context schema. Front-end logging code and QA tests are generated directly from this file to prevent schema drift, enabling accurate mapping between UI elements and analytics event IDs.

AWS enables cross-Region data access using Lake Formation and IAM Identity Center Trusted Identity Propagation. Queries across S3 and Redshift are authorized per individual user instead of shared IAM roles, letting engines like Athena trace every cross-Region request directly in AWS CloudTrail.

Google Cloud Knowledge Catalog introduces self-learning context enrichment for AI agents in data analytics architectures. The capability automates metadata enrichment to improve contextual data discovery and retrieval pipelines.

DuckLake 1.1 adds nested schemas, Iceberg-aligned epoch partition transforms, and exact string statistics to avoid full scans. Catalogs now store Parquet row group counts directly, letting engines plan distributed scans without fetching file footers from object storage.

Snowflake previewed Horizon Context, adding metadata connectors across 18 sources to track lineage and reuse business definitions for AI. Column description propagation carries documentation downstream across views when values pass through unchanged, reducing pipeline maintenance.

🎙️ Data Engineering Podcast — Cross-platform metadata architectures are critical to reconcile business logic across databases, pipelines, and BI tools. MetaKarta provides a Semantic Hub offering deep lineage and configuration management across systems to prevent semantic drift in downstream analytics and agentic AI.

AI shifted the analytics bottleneck from SQL writing to validation. To ensure accurate AI queries, data engineers must build a semantic layer using transforms, metadata, and standardized metrics. Reliable AI output relies on curated data schemas and end-to-end lineage rather than model logic.

OLCF is an open-source lakehouse platform creating an Apache Spark Standalone cluster across custom machines, cloud VMs, or Kubernetes. It extends Microsoft Fabric by using olcf.run_remote() to offload notebook compute to owned Spark clusters while directly reading and writing OneLake.

🎙️ DataGen — Modern data management architectures leverage Federated Knowledge Graph technology to optimize data catalog performance. Integrated platforms apply AI directly to governance workflows, automating descriptions and tagging to shift search paradigms toward conversational interfaces.

Snowflake announced Horizon Context in preview along with named workload identity management. Recent data engineering updates also include Snowpipe Streaming SDK v1.9.0 and data lineage support for storage integrations, external tables, and Cortex Search services.

Databricks enables building a document knowledge base for agentic RAG using vector search. Unity Catalog governs the pipeline, where "Unity Catalog grants the index, and the row filter stops at the table," securing vector search indexes alongside underlying table data.

DuckDB database files can store only view definitions pointing to remote Parquet files on object storage instead of raw data. Acting as a lightweight catalog under a few hundred kilobytes, it creates a shareable semantic layer that abstracts file layouts while querying via the httpfs extension.

Google Cloud details how Cloud Spanner powers its managed Apache Iceberg Lakehouse runtime catalog at scale, providing a robust architecture for high-concurrency metadata operations and distributed data lake analytics.

Snowflake announced the public preview of Agentic Marketplace Discovery in its feature updates. Recent data engineering updates also include Iceberg partition evolution, Elastic Channels for Snowpipe Streaming, and dynamic table incremental refresh support for non-deterministic functions.

Amazon EMR 8.1.0 introduces RedirectingSessionCatalog to solve multi-format and cross-account Spark queries. It automatically detects table formats from AWS Glue metadata to route Iceberg, Delta Lake, Hudi, and Hive tables dynamically, enabling cross-account joins without copying data.

AI agents require explicit metadata because, unlike humans, they lack ingrained business knowledge. Using raw schemas yields shallow descriptions, so engineering context pipelines requires feeding lineage, upstream queries, and glossary terms left-to-right across transformations.

BigQuery pipelines now support automated metadata enrichment and Knowledge Catalog data quality scorecard integration. Additionally, the Data Engineering Agent can proactively generate semantic metadata for pipeline assets. These features are generally available.

ClickHouse now writes Apache Iceberg tables directly to Microsoft OneLake via Iceberg REST-based Table APIs. This integration automatically registers tables in the OneLake catalog under native access controls, allowing downstream engines like Fabric Spark and Power BI to query ClickHouse output.