Catalogs resources for data engineers

DuckDB database files can store only view definitions pointing to remote Parquet files on object storage instead of raw data. Acting as a lightweight catalog under a few hundred kilobytes, it creates a shareable semantic layer that abstracts file layouts while querying via the httpfs extension.

Google Cloud details how Cloud Spanner powers its managed Apache Iceberg Lakehouse runtime catalog at scale, providing a robust architecture for high-concurrency metadata operations and distributed data lake analytics.

Snowflake announced the public preview of Agentic Marketplace Discovery in its feature updates. Recent data engineering updates also include Iceberg partition evolution, Elastic Channels for Snowpipe Streaming, and dynamic table incremental refresh support for non-deterministic functions.

AI agents require explicit metadata because, unlike humans, they lack ingrained business knowledge. Using raw schemas yields shallow descriptions, so engineering context pipelines requires feeding lineage, upstream queries, and glossary terms left-to-right across transformations.

BigQuery pipelines now support automated metadata enrichment and Knowledge Catalog data quality scorecard integration. Additionally, the Data Engineering Agent can proactively generate semantic metadata for pipeline assets. These features are generally available.

ClickHouse now writes Apache Iceberg tables directly to Microsoft OneLake via Iceberg REST-based Table APIs. This integration automatically registers tables in the OneLake catalog under native access controls, allowing downstream engines like Fabric Spark and Power BI to query ClickHouse output.

Onboarding an AI data agent requires a four-phase hiring model: orientation, onboard, internship, and hire. To prevent production failures, teams scaffold context from warehouse schemas, define repo-level AGENTS.md boundaries, restrict tool permissions, and require test gates before granting write

Apache Iceberg C++ 0.4.0 adds Iceberg v3 support, including row lineage and deletion vectors. It exposes new table update APIs like RowDelta, adds parallel manifest operations and lazy scan planning, and introduces SigV4 and ResolvingFileIO to the REST catalog while standardizing on CMake builds.

dbt v2 ships with a built-in DuckDB adapter running on its Rust engine via ADBC drivers. The release adds DuckLake and Iceberg REST catalog support, plus an Information Schema feature that outputs project metadata as Parquet files, enabling fast SQL queries on manifests directly with DuckDB.

Data platforms split into physical infrastructure and derived assets like metrics, features, and AI models. Disconnected tools break end-to-end lineage across catalogs, semantic layers, and models.

The City of Toronto Open Data team is collaborating with Toronto Police Services to showcase public data operations. The session covers data pipeline management and catalog architecture across more than 45 public datasets hosted on the police data portal, emphasizing data accessibility and civic

The Apache Iceberg meetup explores core lakehouse data engineering patterns, focusing on Write-Audit-Publish workflows via branching and tagging. Sessions address migrating from Hive Metastore to Lakekeeper REST catalogs for multi-tenant isolation, plus deploying Iceberg engines on Kubernetes.

Databricks and Corteva are hosting a technical session and hackathon focused on core data engineering tools. Participants will explore the Lakehouse architecture, Unity Catalog for governance, Genie, and AI/BI Dashboards through hands-on exercises and real-world engineering challenges.

Apache Polaris advances open lakehouse architectures through integrations like Apache Ossie, which manages shared semantic models alongside physical tables and views. Additionally, a leaf-wins hierarchical resolution model enables dynamic, path-level storage overrides at namespace and table levels.

This meetup focuses on open lakehouse and AI architectures, exploring composable lakehouse building blocks with Unity Catalog for hybrid and on-premises deployments.

Data leaders are gathering alongside the Snowflake World Tour NYC for an industry dinner hosted by Matia, a unified DataOps platform covering ETL, reverse ETL, data cataloging, and observability.

Delta Lake UniForm writes Iceberg metadata alongside Delta logs over identical Parquet files, letting Snowflake read Databricks tables directly. Linking Snowflake to Unity Catalog endpoints eliminates duplicate storage pipelines, though cross-platform reads depend on catalog metadata refresh cycles.

Having a graph model does not require running a graph database. Known relational queries with few joins fit existing warehouses. Graph databases become essential for open-ended, multi-hop traversals with unpredictable paths where relational joins fail to scale.

To enable reliable AI agents, Webflow rebuilt its enterprise data warehouse using Kimball dimensional modeling. Standardized modular star schemas unify core entities, dimensions, and facts into governed data marts, eliminating tribal knowledge and conflicting metric definitions across models.

Databricks Genie combines Genie One, Genie Agents, and Genie Code over Genie Ontology, an intelligence layer on Unity Catalog that maps table relationships and metadata. Genie Code builds pipelines and agents, while dedicated SQL warehouses are required to manage underlying compute costs.

Snowflake updated catalog-linked databases with write support for nested namespaces, expanding data lakehouse integration. Concurrent releases also introduced zero-copy support for all table formats in interactive warehouses and an Iceberg Scan Plan API in Horizon Catalog.

Snowflake introduced the preview of Horizon Catalog Explorer UI alongside key data engineering features, including general availability of EXPLAIN CHANGES to predict dynamic table refreshes and zero-copy support for all table formats in interactive warehouses.

🎙️ Data Engineering Podcast — Data engineering is shifting from serving human analysts to feeding LLMs and agents. Because AI systems cannot reliably fill in missing information, data teams must rigorously model, validate, and benchmark context across semantic layers, catalogs, and logs to ensure dependable outcomes.

RustFS launched open-source S3 Tables, embedding an Apache Iceberg REST Catalog into its storage kernel. One process on port 9000 serves S3 and Iceberg REST protocols.

Snowflake released the Iceberg Scan Plan API in Horizon Catalog in public preview. This enhances data lakehouse integration by exposing Iceberg table scan planning directly through the catalog.

Data catalogs are shifting from human-facing documentation to agentic infrastructure. To supply the necessary context, metadata lakehouses store query history, lineage, and usage patterns as open Iceberg tables queryable via SQL, allowing agents to automate documentation and metadata generation.

Apache Iceberg Python 0.12.0 adds REST catalog view support and conflict-validated commit retries on concurrent writes. Data engineers can now use incremental append scans to read snapshot changes directly into Arrow, alongside native support for v3 geometry and geography types.

Google engineered a six-layer Apache Iceberg architecture combining engines, catalogs, and an autonomous control plane. It automates file compaction below 50% of 256 MB targets and federates external catalogs like Glue and Unity to query cross-cloud tables without copying files.

Context layers improve autonomous agent workflows and SQL accuracy by capturing unstructured business documentation alongside metadata. They automate context construction from query history and dbt models, leaving humans to curate context and handle data pipeline verification.

Column-level lineage maps each output column to its source columns and transforms. Frameworks like Bruin parse SQL code to enable pre-deployment impact analysis, while catalogs reconstruct graphs from query logs for root-cause debugging.