Real-time resources for data engineers

AI systems require computer vision as core data infrastructure to process massive uncurated visual streams. Edge architectures must run low-latency vision-language-action pipelines locally because shipping high-throughput sensor frames to cloud datacenters introduces critical latency.

ClickHouse integrates with Postgres to unite OLTP and OLAP workloads. Tools include pgclickhouse to query ClickHouse via native Postgres SQL with query pushdowns, and chdbhook to import and export data lake files through Postgres COPY commands across cloud object storage.

Implement the streamhouse pattern locally using Apache Fluss, Apache Flink, and Apache Iceberg to enable direct querying of streaming data. This architecture integrates Fluss for real-time storage with Flink processing and Iceberg lakehouse tables.

Readyset compiles SQL into an incrementally updated streaming dataflow graph. Because the engine requires binary equality joins and flat structures, a multi-pass rewrite pipeline normalizes queries, inlines derived tables, and eliminates correlated subqueries to avoid intermediate materialization.

Databricks Lakeflow complements Fivetran and dbt within a unified architecture. Fivetran automates SaaS and database ingestion into open formats like Delta Lake, Lakeflow processes Kafka event streams with Spark Declarative Pipelines, and dbt models SQL business logic with built-in testing.

ClickHouse Cloud outperformed Databricks Serverless SQL on continuous ingestion benchmarks, achieving 752x higher performance per dollar while streaming 113.2 billion quotes. Its efficient data preparation and real-time pre-aggregations reduce query cost and eliminate pipeline lag.

Streamhouse shifts data architectures from topic-centric pipelines to shared, table-centric storage. Powered by Lakestream, it unifies real-time streaming and historical lakehouse data into single logical tables, preventing redundant state maintenance across operational and analytical workloads.

Data engineering oscillates through recurring trade-offs like centralization versus decentralization and batch versus streaming. As bottlenecks emerge, teams decentralize pipelines and compute, only to later recentralize metadata, governance, and semantic layers when coordination breaks down.

Snowflake announced improved data freshness for selected ORGANIZATION_USAGE history views. Recent updates also expand data pipeline capabilities, adding support for the VALUES clause and non-deterministic aggregate functions during dynamic table incremental refresh.

LinkedIn expanded ClickHouse from distributed tracing to metric discovery and analytics. The consolidated metadata index handles 150k+ queries per minute across 13B+ metrics with 68ms average latency, slashing memory use to one-fifth and compute to two-thirds.

🎙️ DataGen — ClickHouse evolves into a broader data and AI platform, acquiring Langfuse to build an Agentic Data Stack for AI observability. The real-time database expands its core data engineering capabilities by integrating PostgreSQL support, Change Data Capture (CDC), and native streaming queries.

Building a Postgres CDC connector relies on reading the Write-Ahead Log via logical decoding and replication slots. If a consumer halts and fails to acknowledge its slot, Postgres retains unbounded WAL files, filling disk space and halting writes across the production database.

The edition highlights Kafka and stream processing developments, focusing on broker architectures and Flink SQL patterns. Notable releases include Apache Fluss v1.0 and Khaos for failure testing, while new alternatives like blob-stream target cost over latency.

BigQuery released in Preview the INFORMATIONSCHEMA.FAILOVERHISTORY view. Data engineers can query it to retrieve a near real-time list of failover events for reservations within an administration project that use managed disaster recovery.

Jevstiller acts as a drop-in proxy, routing inference to a 15 ms local CPU model instead of a 300 ms vendor API call. Using Clopper-Pearson bounds and fixed-sequence testing on calibration data, it guarantees a strict target agreement rate with the vendor model across requests.

ClickHouse expanded its Microsoft collaboration with a native Fabric workload for sub-second queries directly on OneLake data. It also launched GA for OneLake read, public preview for OneLake write, and Bring Your Own Cloud availability in the Azure Marketplace.

ClickHouse is now a native workload in Microsoft Fabric, acting as an acceleration layer for OneLake. It syncs OneLake tables into ClickHouse Cloud without custom ETL pipelines, enabling sub-second analytical queries on billions of rows at high concurrency without consuming Fabric capacity.

BigQuery continuous queries now support writing output directly into Apache Iceberg managed tables using INSERT DML statements, enabling real-time streaming into open-format lakehouses. Additionally, the BigQuery Data Transfer Service MCP server is generally available.

Data engineers must clarify real-time requests before building complex streaming pipelines. Stakeholders often only need hourly or daily batch updates. True sub-second latency adds severe compute costs and complexity, typically serving automated systems like fraud detection rather than humans.

The technical sessions focus on building production data infrastructure for human and AI-driven queries. Topics include real-time analytics architectures, open lakehouse designs, serving layers, and workload isolation, addressing query performance, concurrency, and scale for agentic workflows.

Snowflake announced event table monitoring for Snowpipe Streaming, enhancing ingestion pipeline observability. Recent ingestion updates also include general availability for partitioned Apache Iceberg tables, Elastic Channels, and dynamic table incremental refresh support for VALUES clauses.

Snowflake announced the general availability of Snowpipe Streaming support for partitioned Apache Iceberg tables. Data engineers can now stream low-latency data directly into partitioned open table formats, aligning with recent updates like partition evolution and Elastic Channels.

Recent data engineering updates highlight DuckDB shipping inside dbt v2 and the release of Apache Fluss 1.0 for real-time AI architectures. Modern pipeline tools also focus on Polars query profiling, Cassandra upgrades, and OpenTelemetry adaptive tail sampling for high-throughput data systems.

Real-time ETL spans batch, micro-batch, log-based CDC, and event streaming via tools like Kafka. Selection hinges on required latency rather than raw speed: log-based CDC efficiently reads database transaction logs, while true streaming continuously processes events for sub-second workloads.

Databricks launched Lakebase, combining serverless Postgres with open lakehouse storage to blur OLTP and OLAP. By enabling instant branching, scale-to-zero compute, and native zero-ETL access for Spark and Delta Lake, it removes brittle CDC pipelines and offers unified governance via Unity Catalog.

Fountain rebuilt its data plane on ClickHouse Cloud, replacing a BigQuery and S3/Iceberg batch stack. Using ClickPipes CDC to stream Postgres and MongoDB data into 2,000+ materialized views, it cut latency from three hours to under two minutes while reducing costs by 66%.

AI agents turn analytical queries into production dependencies where stale data causes operational failures. System requirements now mirror application infrastructure: low latency, high concurrency, and strict consistency. Better data modeling and unified storage layers are vital for reliability.

Change Data Capture extends beyond updating transactional databases to encompass streaming data, API calls, and write-ahead logs. The discussion explores CDC architecture, use cases, and tools like Estuary built for streaming, updating, and materializing pipeline data efficiently.

Build a local real-time pipeline by publishing raw events to Kafka, deduplicating them with Flink SQL, and streaming the results into DuckLake. A consumer writes batches every 50 records or 5 seconds, using DuckLake row inlining to avoid small Parquet files.

Apache Fluss 1.0 delivers real-time storage for streaming and AI workloads. It adds a Rust-based HTTP Gateway for REST operations and lakehouse ingestion, native Python clients for feature lookups, and support for Flink, Spark, Paimon, and Apache Hudi.