Event streaming resources for data engineers

A technical meetup focused on real-time stream processing covers Apache Flink architectures and the streaming ecosystem. Sessions explore managing Flink state schema evolutions for bot detection, running real-time ad pacing workloads, and using Apache Fluss as a modern streaming analytics layer.

Diskless Kafka decouples compute from storage by shifting durable logs to cloud object stores via KIP-1150. While retaining Kafka APIs and offset ordering, it allows brokers to scale independently without managing local disk partitions or moving datasets.

Apache Iggy and Kafka both use partitioned, append-only logs with offsets and consumer groups. However, while Kafka runs on the JVM, Iggy uses Rust with a thread-per-core architecture on io_uring. The key architectural divergence lies in how each engine defines and guarantees successful writes.

Lyft migrated its streaming fleet from an in-house Kubernetes operator to the open-source Apache Flink Kubernetes Operator to eliminate custom code debt. The shift unlocked native autoscaling and robust reconciliation loops without disrupting existing stateful job workflows.

The streamhouse pattern unifies streaming and analytical data into one logical table. Real-time writes hit a low-latency Arrow tier, while a continuous tiering service compacts aging data into open Parquet lakehouse formats like Iceberg or Paimon, letting SQL engines query both tiers seamlessly.

Apache Spark 4.1.0 introduces Spark Declarative Pipelines to manage execution graphs, dependencies, and retries. Structured Streaming adds Real-Time Mode for sub-second latency. PySpark adds Arrow-native UDFs, SQL Scripting is GA by default, and VARIANT type GA brings Parquet shredding support.

Kafka architectures often sink real-time events to S3 for long-term retention. However, raw files fail to behave like database tables for updates and consistent reads. Open table formats like Apache Iceberg bridge this gap by adding schema, snapshot, and partition metadata over the data.

GoalFlow treats non-linear funnels as graphs defined in YAML. Using NetworkX, it validates dependencies and generates pipeline jobs, typed schemas, and backfill scripts directly from the specification.

Uber’s payments platform uses immutable money orders and double-entry bookkeeping to process transactions. It relies on Amazon DynamoDB for strongly consistent ledger balances across 1.2B+ entities and Apache Kafka as an async messaging bus between stateless microservices.

August data engineering roundups highlight streaming, query engines, and platform patterns. Apache Fluss became a top-level project, DuckDB 2.0 was previewed, and Netflix detailed streaming joins in Flink alongside its real-time distributed graph architecture.

Rednote migrated its real-time indexing pipeline from Kafka to Apache Fluss to solve read amplification from wide tables. Fluss uses columnar streaming via Apache Arrow and bypasses brokers for cold-data replays, isolating historical scans from real-time traffic to reduce bandwidth and compute

Netflix evolved its ads event architecture for live streaming traffic surges by replacing a synchronous database write on the ad-serving critical path with stateful stream processing.

CDC does not eliminate dual-writes; it merely shifts them downstream. Updating targets and committing pipeline offsets are independent, non-transactional acts. Without idempotent receivers, dual-write failure modes persist.

Apache Fluss provides lakehouse-native streaming storage, uniting message queues, key-value stores, and lakehouses into one substrate. It externalizes stream state and offers Arrow-based streaming with Iceberg, Paimon, and Lance integrations.

NATS 2.12 introduces atomic batch publishing in JetStream, giving all-or-nothing consistency guarantees for multi-message writes. Staged on the server until a commit header is sent, batches prevent partial writes. "Either every message in your batch commits together, or none of them do."

Bluesky runs on the open AT Protocol, where each user data repo is stored in an SQLite database. Data engineers can query posts and handles directly via open APIs using DuckDB or stream Jetstreams using Python. Each user's data is implemented and stored with a single SQLite database.

This architecture detects streaming price anomalies using Kafka, Flink, and Spark. Flink processes Kafka event streams, calculates rolling Z-scores using historical baselines built by Spark, and routes anomalies to Claude to generate human-readable explanations displayed in Streamlit.

Apache Fluss graduated to an ASF Top Level Project, providing unified streaming storage for real-time lakehouses. Built on Apache Arrow, it enables server-side column and partition pruning to cut data transfer costs and bridge real-time streams with historical tables.

The streaming cold tier processes up to 1,250 reads per second globally, delivering a p50 latency of ~150 ms and a p99 of ~500 ms. Once a time slice crosses the user-configured cold-tier boundary, TimeSeries stops accepting writes for that time slice making it immutable.

OpenData Log is an open-source, object-native log built in Rust for routing rather than funneling data. Unlike Kafka, it replaces topic-partitions with key-oriented indexing via an LSM tree to manage millions of individual logs with low read amplification.

Streaming log tech enters a third wave by abandoning Kafka protocol compatibility. Apache Iggy uses Rust and thread-per-core io_uring architecture. S2 provides a low-level HTTP API without partitions. OpenData Log optimizes object-native storage for funnelling and routing workloads.

To handle 145 billion daily events, Atlassian migrated StreamHub from Amazon Kinesis to AWS Managed Streaming for Apache Kafka. Using Kafka Tiered Storage offloaded historical data to S3, reducing EBS costs while supporting multi-day retention.

The July 2026 data ecosystem round-up highlights key event streaming developments. Atlassian detailed transitioning from Kinesis to Kafka for 145 billion daily events, while Confluent covered cutting Flink OOMKills by 91.2%. Other topics include Apache Fluss graduating to an ASF top-level project.

To build a real-time crypto pipeline using Redpanda and ClickHouse, the architecture decouples WebSocket ingestion from downstream consumers to prevent data loss during gap recovery. The consumer detects ID gaps per symbol and fills bounded missing ranges from Binance REST.

latlng is an open-source spatial database written in Rust for indexing moving objects and geofencing. It offers R-tree spatial indexing and streams events via WebSockets or Webhooks. latlng compiles a single Rust core into the form that fits where you run it—natively or via WebAssembly.

Cloudflare has acquired Arroyo to bring serverless SQL stream processing to its developer platform. Built in Rust, Arroyo provides stateful aggregations and joins. The engine will remain open-source and self-hostable.

Netflix built a streaming-first service topology system that processes millions of flow records per second using reactive streams with backpressure to prevent crashes under load. The architecture physically isolates network, IPC, and tracing layers to enable independent optimization.

CDC streaming reads database logs to emit inserts, updates, and deletes as they happen. It avoids the limits of batch polling, where hard deletes are invisible and timestamp bugs create missing rows. By using the database log as the source of truth, it ensures low-latency data delivery.

StreamFusion accelerates Apache Flink SQL by executing operators natively via Rust and DataFusion over JNI. It targets stateful streaming tasks like windowed joins and changelog processing.

Debezium is an open-source platform for log-based change data capture. It monitors database transaction logs to stream row-level changes into Kafka without adding query load. Debezium is the open-source standard for turning database transactions into event streams.