Open infrastructure resources for data engineers

Snowflake Horizon Catalog has reached general availability for accessing externally managed Apache Iceberg tables. The release also adds general availability for Horizon Iceberg REST Catalog telemetry, improving external catalog integrations and data lakehouse architectures.

dbt Charts introduces an open-source, dashboards-as-code spec designed for AI agents to write and engineers to audit. Aiming to bypass complex BI layers, it lets teams rely more on relational data architectures instead of pre-joining models solely to satisfy traditional BI tools.

Tares ingests distributed events like logs, metrics, deploys, and database rows into unified, time-ordered entity timelines. Event-driven triggers wake agents via MCP or webhooks with pre-correlated context, persisting state and decisions across pipeline runs.

Key data engineering releases include Apache Kafka 4.4, DuckDB 2.0, Flink CDC 3.7, and Confluent's official Rust client for Kafka. Engineering highlights cover stream processing patterns like deduplication in Flink SQL, session-ordered Go pipelines, and scaling shuffle with Apache Celeborn.

Google Cloud details lakehouse interoperability for querying Databricks Apache Iceberg tables across environments. The architecture enables data engineers to run cross-platform analytics directly on open table formats without data movement.

AI integration requires strong data engineering foundations. Fivetran advocates for open formats, portable storage, and reliable data pipelines to power AI agents securely, noting that data should not be trapped in closed systems. Reliable, well-governed data pipelines make AI accountable.

🎙️ The Data Flowcast by Astronomer — HCL Tech relies on Apache Airflow for cloud data migrations, managing on-prem extraction, watermarking, and loading into Snowflake and Databricks. The team generates dynamic DAGs from JSON configs for micro-batching and uses blue-green deployment patterns across environments.

MiniLake is a local Databricks API emulator that executes real SQL on DuckDB, writes genuine Delta files, and runs containerized spark-submit jobs. Designed for CI and local development without cloud costs, it natively supports Unity Catalog CRUD, DAG job scheduling, and official SDK clients.

Snowflake Iceberg tables decouple compute from storage by using external S3 volumes. This setup uses AWS IAM roles, STS, and external IDs to securely access parquet data and metadata, enabling cross-engine queries in DuckDB or Spark without proprietary storage lock-in.

DuckDB can query Snowflake-managed Iceberg tables on S3 using its httpfs and iceberg extensions. Users scan the table snapshot's metadata JSON directly via iceberg_scan, then materialize data locally to cut repeated query latency from 54 seconds to 1.5 seconds.

Using pyIceberg with AWS Glue streamlines data lake management on S3. Configuring client properties allows shared credentials across Glue and S3 to define schemas, create tables, and query Iceberg metadata directly using DuckDB.

Data teams can query Snowflake tables locally in DuckDB using Arrow Database Connectivity (ADBC) without managing Iceberg tables. By avoiding row-column serialization found in ODBC, ADBC keeps data columnar throughout transfer, speeding up data movement by up to 38x.

A new community DuckDB extension queries Snowflake directly via an ADBC driver without custom scripts. Engineers can attach Snowflake databases and enable filter pushdown to cut data transfer, joining warehouse tables directly with local Parquet files.

Greybeam transpiles Snowflake queries to DuckDB via SQLGlot, but anonymous bind variables lose their lexical order in the AST. To preserve ordering, they intercept the token stream before AST parsing, rewriting anonymous placeholders into prefixed, named bind variables for DuckDB execution.

Modern data orchestration demands cross-engine interoperability and warehouse cost visibility. Declarative tools like Orchestra speed up delivery via YAML or GUI interfaces and serverless task execution, while Apache Airflow remains the standard for complex, Python-defined DAG dependencies.

Snowflake Iceberg tables store Parquet files and metadata in object storage, allowing external engines to query them without copying data. Snowflake can serve as the catalog to handle compaction and snapshots, or connect to external catalogs like AWS Glue via external volumes.

Snowflake-managed storage simplifies Iceberg tables via EXTERNALVOLUME = 'SNOWFLAKEMANAGED', removing manual cloud bucket setup. Through the Horizon catalog, external engines like DuckDB can query and write directly to the managed files using vended credentials.

SignalWeave normalizes heterogeneous AI telemetry from tools like LangChain and OpenTelemetry into canonical signals. Using standard Python, it processes JSON/JSONL via CLI pipes to aggregate reliability, quality, and cost metrics by run ID.

Enterprise AI bottlenecks have shifted from model reasoning to reliable data architectures. Instead of siloed vector databases, teams need unified platforms pairing SQL with native vector search and open formats like Apache Iceberg to deliver minimal, high-quality context directly to agents.

apisip is a Rust CLI that ingests OpenAPI list endpoints directly into JSONL or Parquet. It automates pagination, rate limits, and incremental loads. Instead of runtime inference, it enforces schema types from the spec, safely casting lossless values and sending unparseable data to quarantine.

Duckpad is an open-source data transformation tool and SQL workspace powered by DuckDB. Data engineers can visually compose pipelines, integrate Git versioning, and trace cell-level data lineage back to contributing source values. SQL execution runs locally on modern Linux via Landlock.

Run open-weight LLMs locally to cut token costs and protect data privacy. Use quantization like 4-bit to shrink tensor memory footprints for consumer GPUs, and orchestrate automated pipelines using tools like Kestra for scheduled batch inference and continuous tasks.

Data catalogs organize structural metadata and schemas across warehouses and lakes. Technical catalogs track schemas, access policies, and column-level lineage, enabling tools like Iceberg and Glue to manage state and evaluate the impact of schema changes across pipelines.

dbt Labs launched dbt Charts, an open-source YAML language for defining dashboards alongside dbt models. Stored in a charts directory, configurations run through standard git workflows and tests. It operates via the dct CLI, integrating analytics engineering and BI into unified deployments.

ADBC standardizes database access by returning Apache Arrow columnar data to bypass row-oriented protocol bottlenecks. While native engines like DuckDB enable zero-copy transfers, drivers face severe type-mapping mismatches across engines like Postgres, BigQuery, and Spark. Test schemas early.

Lake Compute lets data teams run low-complexity dbt models on cheaper lake engines instead of expensive data warehouses. Built on Apache Iceberg and open REST catalogs, this architecture cuts compute costs model by model without disrupting downstream DAGs or BI tools.

Snowflake Horizon now supports external Apache Iceberg writes natively. Coupled with dbt catalog configs and DuckDB upstream fixes, teams can build Iceberg tables in object storage and query governed data locally in DuckDB without touching Snowflake compute warehouses.

A catalogs.yml v2 spec decouples dbt Iceberg configurations from engine-specific storage, enabling cross-platform dbt Mesh across Snowflake and Databricks. Catalog federation lets engines query shared Iceberg tables without data duplication, while DuckDB routes specific models.

Verily uses dbt and a medallion architecture to process nested FHIR healthcare data for wastewater pathogen monitoring. The team created reusable dbt macros to standardize FHIR resources across datasets, capping composite keys at three columns to optimize data modeling and downstream analytics.

dbt v2 converges the ecosystem onto one engine with faster parsing and unified binaries. Built on the ADBC standard, shared adapters eliminate dependency conflicts across warehouses like Snowflake and Databricks. Docs v2 also replaces manifest.json with queryable Parquet files for lineage analysis.