Event streaming resources for data engineers

Implement the streamhouse pattern locally using Apache Fluss, Apache Flink, and Apache Iceberg to enable direct querying of streaming data. This architecture integrates Fluss for real-time storage with Flink processing and Iceberg lakehouse tables.

Databricks Lakeflow complements Fivetran and dbt within a unified architecture. Fivetran automates SaaS and database ingestion into open formats like Delta Lake, Lakeflow processes Kafka event streams with Spark Declarative Pipelines, and dbt models SQL business logic with built-in testing.

Streamhouse shifts data architectures from topic-centric pipelines to shared, table-centric storage. Powered by Lakestream, it unifies real-time streaming and historical lakehouse data into single logical tables, preventing redundant state maintenance across operational and analytical workloads.

redb.Route is a .NET integration framework acting like Apache Camel. It uses a C# DSL to build From-Process-To pipelines connecting Kafka, RabbitMQ, Redis, SQL, and S3 without message-bus lock-in. It supports 29 external transports, hot-reloading, and token streaming.

The edition highlights Kafka and stream processing developments, focusing on broker architectures and Flink SQL patterns. Notable releases include Apache Fluss v1.0 and Khaos for failure testing, while new alternatives like blob-stream target cost over latency.

Real-time ETL spans batch, micro-batch, log-based CDC, and event streaming via tools like Kafka. Selection hinges on required latency rather than raw speed: log-based CDC efficiently reads database transaction logs, while true streaming continuously processes events for sub-second workloads.

Build a local real-time pipeline by publishing raw events to Kafka, deduplicating them with Flink SQL, and streaming the results into DuckLake. A consumer writes batches every 50 records or 5 seconds, using DuckLake row inlining to avoid small Parquet files.

Apache Fluss 1.0 delivers real-time storage for streaming and AI workloads. It adds a Rust-based HTTP Gateway for REST operations and lakehouse ingestion, native Python clients for feature lookups, and support for Flink, Spark, Paimon, and Apache Hudi.

Engineers explore scalable architectures, bridging online streaming logic with offline replay across millions of events. Discussions focus on Spark 4.0 features like Real-Time Mode and transformWithState versus Apache Flink, alongside managing high-volume AI agent workloads on modern data

The Kafka User Group session covers event-driven architectures with Apache Kafka, focusing on publishing database changes to downstream services. It also explores using Kafka as the core messaging backbone for multi-service superapps, alongside Atlas Search indexes for handling near-match queries.

Real-time data engineering patterns center on streaming architectures and change data capture. Delivery Hero routes Postgres CDC events into Kafka to feed its live ML feature store, while Zalando abstracts Kafka via Nakadi to swap topics seamlessly.

Stream Forward covers production Apache Flink engineering, addressing state management, checkpoint tuning, backpressure debugging, and job graph optimization. Sessions explore streamhouse patterns pairing Flink compute with Fluss real-time storage and Paimon, alongside real-time feature pipelines.

Stream Forward covers production Apache Flink engineering, addressing state management, checkpoint tuning, backpressure debugging, and job graph optimization. It details modern patterns like streamhouse architectures using Flink, Fluss, and Paimon, alongside real-time AI and stream ETL migrations.

Engineers explore Apache Flink in production for real-time streaming architectures. Adyen shares how Flink powers real-time feature computation for machine learning in payment processing.

Build a custom robotics data stack using Foxglove's API to collect, debug, curate, and search telemetry. Develop extensions for custom message types, visualizations, and transformations. Leverage webhooks to construct event-driven workflows and embed the platform directly into custom applications.

This session covers modern data engineering architectures on Google Cloud Platform, focusing on scalable analytics, data lake design, and real-time data streaming.

The Streamhouse Working Group introduced an open architecture pattern designed to serve continuously updated enterprise state to production apps and AI agents. It relies on CDC, event streams, stream processing, and open table formats rather than periodic batching.

A technical meetup focused on real-time stream processing covers Apache Flink architectures and the streaming ecosystem. Sessions explore managing Flink state schema evolutions for bot detection, running real-time ad pacing workloads, and using Apache Fluss as a modern streaming analytics layer.

Diskless Kafka decouples compute from storage by shifting durable logs to cloud object stores via KIP-1150. While retaining Kafka APIs and offset ordering, it allows brokers to scale independently without managing local disk partitions or moving datasets.

Data engineering developments highlight Diskless Kafka architectures that separate storage from brokers. Other focal points include DuckDB 2.0 performance, executing multi-day PostgreSQL CDC backfills during writes, and optimizing Spark resource waste across the stack.

Apache Iggy and Kafka both use partitioned, append-only logs with offsets and consumer groups. However, while Kafka runs on the JVM, Iggy uses Rust with a thread-per-core architecture on io_uring. The key architectural divergence lies in how each engine defines and guarantees successful writes.

Lyft migrated its streaming fleet from an in-house Kubernetes operator to the open-source Apache Flink Kubernetes Operator to eliminate custom code debt. The shift unlocked native autoscaling and robust reconciliation loops without disrupting existing stateful job workflows.

The streamhouse pattern unifies streaming and analytical data into one logical table. Real-time writes hit a low-latency Arrow tier, while a continuous tiering service compacts aging data into open Parquet lakehouse formats like Iceberg or Paimon, letting SQL engines query both tiers seamlessly.

Apache Spark 4.1.0 introduces Spark Declarative Pipelines to manage execution graphs, dependencies, and retries. Structured Streaming adds Real-Time Mode for sub-second latency. PySpark adds Arrow-native UDFs, SQL Scripting is GA by default, and VARIANT type GA brings Parquet shredding support.

Kafka architectures often sink real-time events to S3 for long-term retention. However, raw files fail to behave like database tables for updates and consistent reads. Open table formats like Apache Iceberg bridge this gap by adding schema, snapshot, and partition metadata over the data.

Recent data engineering updates highlight the release of Spark 4.1.0 and Apache DataFusion 55.0.0. Technical deep dives focus on computing time in queue for Apache Hudi pipelines at petabyte scale, streaming table patterns with Kafka and Fluss, and replacing Fivetran for Snowflake ingestion.

GoalFlow treats non-linear funnels as graphs defined in YAML. Using NetworkX, it validates dependencies and generates pipeline jobs, typed schemas, and backfill scripts directly from the specification.

Key updates cover stream architecture and storage engineering: Khaos provides tools for Kafka load testing and chaos engineering, while DuckLabs joins AWS with its open source projects intact. Other deep dives evaluate RocksDB amplification factors and shared log composition via LogDrive.

Uber’s payments platform uses immutable money orders and double-entry bookkeeping to process transactions. It relies on Amazon DynamoDB for strongly consistent ledger balances across 1.2B+ entities and Apache Kafka as an async messaging bus between stateless microservices.

August data engineering roundups highlight streaming, query engines, and platform patterns. Apache Fluss became a top-level project, DuckDB 2.0 was previewed, and Netflix detailed streaming joins in Flink alongside its real-time distributed graph architecture.