Catalogs resources for data engineers

RustFS launched open-source S3 Tables, embedding an Apache Iceberg REST Catalog into its storage kernel. One process on port 9000 serves S3 and Iceberg REST protocols.

Snowflake released the Iceberg Scan Plan API in Horizon Catalog in public preview. This enhances data lakehouse integration by exposing Iceberg table scan planning directly through the catalog.

Data catalogs are shifting from human-facing documentation to agentic infrastructure. To supply the necessary context, metadata lakehouses store query history, lineage, and usage patterns as open Iceberg tables queryable via SQL, allowing agents to automate documentation and metadata generation.

Apache Iceberg Python 0.12.0 adds REST catalog view support and conflict-validated commit retries on concurrent writes. Data engineers can now use incremental append scans to read snapshot changes directly into Arrow, alongside native support for v3 geometry and geography types.

Google engineered a six-layer Apache Iceberg architecture combining engines, catalogs, and an autonomous control plane. It automates file compaction below 50% of 256 MB targets and federates external catalogs like Glue and Unity to query cross-cloud tables without copying files.

Context layers improve autonomous agent workflows and SQL accuracy by capturing unstructured business documentation alongside metadata. They automate context construction from query history and dbt models, leaving humans to curate context and handle data pipeline verification.

Snowflake announced general availability for External lineage alongside data lineage for Cortex Agents. Data engineers also gain GA features including DCM Projects, multi-value tags, and an Openflow Connector for SQL Server CDC, plus REST catalog integration for Amazon S3 Tables Iceberg support.

Snowflake released Snowflake-provided tags in public preview, expanding built-in metadata management. Recent platform updates also introduced multi-value tags and Apache Iceberg REST catalog integrations, streamlining data governance and cataloging across data engineering pipelines.

Modern AI architectures rely on lakehouses, zero-copy sharing, and metadata automation over classical ETL. Platforms must integrate tools, data products, and clear governance to capture raw schema-on-read data effectively.

Deploying a DuckLake catalog on an EC2 instance served by Quack abstracts storage and credentials from clients. Provisioned via Terraform, Quack manages S3 bindings and enables multiple concurrent clients to query and write to the data lake without handling backend secrets directly.

To document raw tables across 70 dlt pipelines in Snowflake, an agent inspects pipeline sources and live INFORMATION_SCHEMA schemas to generate metadata JSON files. A deterministic runtime function reads the JSON to execute COMMENT ON TABLE and COMMENT ON COLUMN statements after each load.

Google Cloud explores using OKF bundles with Knowledge Catalog to serve context for data analytics agents across an organization. This architecture scales organizational context and metadata discovery to support automated data workflows.

Snowflake announced the general availability of multi-value tags for enhanced metadata management and governance. Recent updates also introduced custom incremental dynamic tables and Apache Iceberg REST catalog integration with Amazon S3 Tables.

Lineage tools depend on your stack. Parsing pipeline SQL allows column-level impact analysis in CI before code merges, whereas catalogs reconstruct lineage retrospectively from warehouse query logs. Tools like OpenMetadata handle heterogeneous stacks, while dbt offers model-level DAGs.

Bruin CLI generates a version-controlled AI context layer for data warehouses. Using bruin import database and bruin ai enhance, it maps schemas into local YAML files containing descriptions, tags, and executable data quality checks like unique and not_null.

A survey of 111 data practitioners shows that tooling alone fails to solve data quality or architecture challenges, as 42% still cite quality as a primary issue. Most teams run governance under IT, where unclear ownership and lack of adoption hinder pipeline reliability and process accountability.

Snowflake previewed support to access externally managed Apache Iceberg tables through Snowflake Horizon Catalog. Recent updates also added Amazon S3 Tables Iceberg REST catalog integration and CREATE OR ALTER support for dynamic Iceberg tables.

Google Cloud introduces its Lakehouse runtime catalog to help data engineers modernize Apache Hive deployments. This analytics solution enables modernization of legacy Hive architectures within cloud data lakehouse environments.

AI systems require dependable, well-governed data pipelines to prevent scaling errors. Quality is an ongoing discipline supported by validation rules and continuous monitoring.

Enterprise AI systems increasingly fail due to fragmented context across pipelines and agents. While data contracts and context graphs emerge to link semantics and execution lineage, operational barriers rise 14% post-peak. Engineers lack deep metadata to reconstruct agent decisions end-to-end.

Open table formats like Apache Iceberg, Delta Lake, and Hudi bring database-like features to cheap object storage. DuckDB serves as a lightweight query engine to directly read these formats. "Open Table Format bundles distributed files into manageable tables with database-like features."

AWS S3 Tables and Cloudflare R2 reshape Lakehouse architectures via the ICE stack, decoupling compute and storage. Open table formats like Apache Iceberg enable database features over distributed files.

Modern BI uses data engineering techniques like ELT and Python to simplify cloud architectures. ELT gives you a significant advantage in speed. Load incrementally with business keys, centralize metadata in data catalogs, and replace imperative pipelines with declarative container orchestration.

Snowflake integration for Amazon S3 Tables via the Iceberg REST catalog is generally available. This update expands Snowflake architectural capabilities for managing external Apache Iceberg open table formats directly alongside S3 object storage infrastructure.

Starting September 30, 2026, all new Databricks workspaces will be provisioned as Unity Catalog-only without access to the legacy Hive metastore. Even on Unity Catalog, clusters connect to Hive metastore by default. To secure data, run spark.databricks.unityCatalogOnlyMode True on clusters.

The Paris Open Lakehouse and AI Meetup focuses on open source data engineering and lakehouse architectures. Technical sessions cover Unity Catalog, composable lakehouse building blocks for hybrid deployments, and Omnigent, a meta-harness for AI agents that integrates MLflow and OpenTelemetry traces.

Data architecture suffers from adding redundant layers. A true layer earns its place by changing data state, divided into build, consume, or operate functions. "A pipeline without a semantic model isn't lacking a nice-to-have; it's lacking a shared definition of truth." Focus on purpose-based zones.

Snowflake has officially launched General Availability for consuming Delta Shares within the Horizon Catalog. Data engineers can now directly access and integrate external open-format Delta Lake data provider shares alongside native governance features without copying or moving raw data assets

Zero-copy architectures hide subtle network and compute tradeoffs. Catalog federation and sharing protocols move metadata or signed references, while federated queries pull remote rows per execution. The same work repeats on every execution, because nothing persists.

A production-grade DuckLake architecture uses DuckDB for compute, S3 for Parquet file storage, and Postgres for the catalog database. DuckLake treats metadata as an ACID database problem rather than reading scattered JSON files. DuckLake keeps catalog and metadata in an ACID SQL database.