Open infrastructure resources for data engineers

ClickHouse integrates with Postgres to unite OLTP and OLAP workloads. Tools include pgclickhouse to query ClickHouse via native Postgres SQL with query pushdowns, and chdbhook to import and export data lake files through Postgres COPY commands across cloud object storage.

Infino launched a unified agent retrieval layer that queries Parquet in object storage directly via SQL. By combining vector search, keyword search, joins, and aggregates into one query over a single copy of data, it eliminates fragmented ETL pipelines, sync jobs, and multi-database sprawl.

Modern semantic layers often require excessive glue code for data pipelines and agents. SLayer offers an open-source alternative under the MIT license, combining context tooling with an integrated query engine capable of handling complex multi-stage analytical queries out of the box.

Right-sizing analytics pipelines prevents early technical debt while avoiding premature distributed complexity. Starting with single-node tools like Polars, Postgres, and DuckDB enables high performance before migrating to Databricks or Snowflake.

Parseable is a Rust-based, stateless columnar data lake for telemetry data. Built on open Parquet and object storage native architecture, it unifies logs, metrics, and traces while managing high-cardinality workloads at scale with SQL querying.

Amazon Aurora PostgreSQL now supports direct querying of Apache Iceberg and Parquet data lake files. Other data engineering highlights include extending Apache DataFusion across machines, using ClickHouse for metric discovery at LinkedIn, and DuckDB 2.0 adding dynamic ATTACH statements.

Streamhouse shifts data architectures from topic-centric pipelines to shared, table-centric storage. Powered by Lakestream, it unifies real-time streaming and historical lakehouse data into single logical tables, preventing redundant state maintenance across operational and analytical workloads.

Apache Iceberg 1.12.0 expands semi-structured and geospatial data capabilities. Flink and Kafka Connect can now write shredded Variant columns alongside Spark. The release also adds a Flink maintenance task to convert equality deletes into positional deletion vectors for faster streaming reads.

AWS acquired DuckDB Labs instead of Polars, a move that stirred concerns over open source independence. The author argues that developer goodwill heavily influenced the cloud giant's choice to bring the analytical engine in-house over alternative tools.

AI labs are slowing down not from training data exhaustion, but to hinder open-source alternatives. Using LoRA and quantization, developers match monolithic APIs at lower infrastructure costs, prompting closed providers to lobby for strict compute limits to protect their cloud margins.

Cloudflare Basin is a serverless data platform built on Apache Iceberg and R2. It pairs managed pipelines and metadata catalogs with a distributed SQL engine to ingest and query analytics data, letting engineers process open Iceberg tables with zero egress fees across external engines.

redb.Route is a .NET integration framework acting like Apache Camel. It uses a C# DSL to build From-Process-To pipelines connecting Kafka, RabbitMQ, Redis, SQL, and S3 without message-bus lock-in. It supports 29 external transports, hot-reloading, and token streaming.

Embrasure Flow is a Rust service that streams PostgreSQL changes into Apache Iceberg. It captures initial COPY and logical replication, writing Parquet data with v2 position deletes or v3 deletion vectors to S3 storage via an Iceberg REST catalog, while handling local compaction.

A benchmark evaluates Cube Core, MetricFlow, Malloy, and SLayer across 54 features. SLayer and Malloy lead in dynamic query-time expressiveness, while Cube and MetricFlow rely on configuration-time modeling.

Dbc 0.3.1 updates the CLI tool for managing ADBC drivers, fixing Windows output and unsupported lockfiles. Recent driver additions include Apache Spark, DataFusion, and Presto, streamlining Arrow-native database connectivity for data engineers. The tool is also now available via npm.

Astronomer launched the astronomer-data plugin for ChatGPT, connecting to managed or self-hosted Apache Airflow. Data engineers can debug DAG failures from task logs, scaffold pipelines, query data warehouses, and trace lineage using natural language.

Apache Iceberg Go 0.7.0 adds Iceberg V3 support, introducing deletion vectors, variant shredding, and geometry types. It also implements REST server-side scan planning delegation, incremental changelog scans, and full metadata tables to optimize distributed queries and table maintenance.

Aurora PostgreSQL now embeds DuckDB to directly query Apache Iceberg and Parquet in S3 without ETL pipelines. Using the aurora_analytics extension, it supports foreign tables with predicate pushdown and column pruning to join operational and lake data.

Updates to pgclickhouse and chdb improve Postgres-ClickHouse integration with a new checkencoding option. It handles invalid byte sequences across foreign tables and chdb queries using fail, remove, replace, or truncate handlers.

Apache Ossie now supports Microsoft ecosystem integration via a Power BI converter translating to TMDL. This enables bidirectional conversions with Snowflake Semantic Views while adding DAX query support, preserving metric logic without duplicating underlying data across platforms.

OSA Con features technical sessions on open-source data infrastructure and AI workloads. Data engineers explore AI agents generating Apache Spark pipelines, serverless eventing for data flows, and the Presto C++ engine, alongside architectures like shared data lakes and production ClickHouse on

Treating OpenTelemetry as enterprise data shifts observability from incident triage to business analytics. Combining telemetry with open formats like Apache Iceberg and Snowflake allows data engineers to govern logs, metrics, and traces alongside customer tables for AI and impact modeling.

The edition highlights Kafka and stream processing developments, focusing on broker architectures and Flink SQL patterns. Notable releases include Apache Fluss v1.0 and Khaos for failure testing, while new alternatives like blob-stream target cost over latency.

TypeSafe AI's Jev outputs typed decisions as JSON. Representing these outputs via Apache Arrow schemas enables zero-copy ingestion into engines like DuckDB and Polars, avoiding serialization overhead. However, analytical batch evaluation remains bottlenecked by the API's single-state request design.

Fivetran leads Snowflake ingestion tools at 12.7% of surveyed companies, followed by Azure Data Factory at 7.9% and Informatica at 6.5%. Most teams bypass standard ETL vendors, relying instead on native ingestion like COPY INTO, Snowpipe, or custom code.

Apache Arrow Rust 60.0.0 adds Parquet ALP float encoding and external page index caching to cut scan I/O. Writers now record NDV statistics for smarter query planning, alongside 8x faster decimal string casting.

🎙️ Altertable — Taktile integrates DuckDB with Apache Iceberg to execute fast analytical workloads over an open lakehouse format. The architecture leverages DuckDB as an embedded OLAP query engine directly atop Iceberg tables, providing high-performance query execution and scalable data engineering patterns.

ClickHouse expanded its Microsoft collaboration with a native Fabric workload for sub-second queries directly on OneLake data. It also launched GA for OneLake read, public preview for OneLake write, and Bring Your Own Cloud availability in the Azure Marketplace.

Paradime State adds state-aware orchestration to Bolt via dbt Core v2.0. Its decision engine evaluates upstream freshness and SQL logic fingerprints rather than raw text to skip, clone, or build models, avoiding full pipeline rebuilds without per-skip fees.