Kafka resources for data engineers

Key data engineering releases include Apache Kafka 4.4, DuckDB 2.0, Flink CDC 3.7, and Confluent's official Rust client for Kafka. Engineering highlights cover stream processing patterns like deduplication in Flink SQL, session-ordered Go pipelines, and scaling shuffle with Apache Celeborn.

JD.com built a Lakestream using Apache Fluss and Apache Hudi to unify streaming and lakehouse data. Fresh data in Fluss tables tiers directly into Hudi, cutting batch compute costs over 80% and storage by 50% while ending duplicate pipelines and reducing wasted network bandwidth from row-based

Netflix rebuilt its Ads Event Publisher around stateful stream processing with Apache Flink. To handle massive live traffic spikes, it replaced synchronous key-value store writes with a real-time streaming join of Kafka streams using a custom KeyedCoProcessFunction over a 60-minute window.

ClickHouse Cloud outperformed Redshift Serverless with 1,996x better performance per dollar during continuous ingestion of 113.2 billion rows. It leverages asynchronous inserts and pre-aggregations to optimize data preparation and minimize scanning costs for real-time streaming queries.

Amazon Aurora PostgreSQL now supports direct querying of Apache Iceberg and Parquet data lake files. Other data engineering highlights include extending Apache DataFusion across machines, using ClickHouse for metric discovery at LinkedIn, and DuckDB 2.0 adding dynamic ATTACH statements.

redb.Route is a .NET integration framework acting like Apache Camel. It uses a C# DSL to build From-Process-To pipelines connecting Kafka, RabbitMQ, Redis, SQL, and S3 without message-bus lock-in. It supports 29 external transports, hot-reloading, and token streaming.

Apache Kafka 4.0 on Amazon MSK introduces the KIP-848 consumer rebalance protocol, shifting coordination logic from clients to brokers. This incremental, server-driven rebalancing removes global synchronization barriers, letting unaffected consumers process data continuously during group changes.

The edition highlights Kafka and stream processing developments, focusing on broker architectures and Flink SQL patterns. Notable releases include Apache Fluss v1.0 and Khaos for failure testing, while new alternatives like blob-stream target cost over latency.

Real-time ETL spans batch, micro-batch, log-based CDC, and event streaming via tools like Kafka. Selection hinges on required latency rather than raw speed: log-based CDC efficiently reads database transaction logs, while true streaming continuously processes events for sub-second workloads.

Data engineering is shifting from rigid ETL toward autonomous pipelines and open lakehouses. Teams are adopting formats like Apache Iceberg and Delta Lake alongside Kafka and Flink for unified streaming, reducing manual SQL scripting in favor of data reliability engineering and FinOps.

Scaling Kafka consumer groups on MSK can exceed the internal 1 MB metadata record limit, causing a RecordTooLargeException during rebalances. Prevent rebalance retry loops by raising max.message.bytes on consumeroffsets and keeping replica.fetch.max.bytes matched.

Build a local real-time pipeline by publishing raw events to Kafka, deduplicating them with Flink SQL, and streaming the results into DuckLake. A consumer writes batches every 50 records or 5 seconds, using DuckLake row inlining to avoid small Parquet files.

Snowflake launched Data Connectivity Proxy (DCP) for Openflow, allowing secure ingestion from private databases and Kafka brokers. Delivered as a Docker agent, it uses outbound TLS on port 443 with mTLS tunnels to bypass inbound firewall rules without inspecting data or storing source credentials.

Diskless Kafka decouples compute from storage by shifting durable logs to cloud object stores via KIP-1150. While retaining Kafka APIs and offset ordering, it allows brokers to scale independently without managing local disk partitions or moving datasets.

Data engineering developments highlight Diskless Kafka architectures that separate storage from brokers. Other focal points include DuckDB 2.0 performance, executing multi-day PostgreSQL CDC backfills during writes, and optimizing Spark resource waste across the stack.

Kestra 2.0 decouples its control and data planes, allowing workers to run anywhere via outbound-only gRPC streams without direct database access. The engine also decouples the queue from the repository, unifying the architecture while supporting backends like Postgres, Kafka, Redis, or AMQP.

Kestra 2.0 introduces a unified orchestration engine with decoupled queues and repositories. Workers now run anywhere using outbound gRPC with no database access. Data engineers can keep Postgres and switch queues to Kafka or Redis without touching flows, keeping execution next to the data.

Apache Iggy and Kafka both use partitioned, append-only logs with offsets and consumer groups. However, while Kafka runs on the JVM, Iggy uses Rust with a thread-per-core architecture on io_uring. The key architectural divergence lies in how each engine defines and guarantees successful writes.

Key data engineering topics highlight optimized data pipelines and storage architectures, including direct S3-to-GPU data loading for machine learning, database proxy patterns via ZGateway for ZippyDB, and distributed processing updates from Apache Kafka versus Iggy and Polars 2.0.

Kestra 2.0 decouples queue and repository backends under a unified engine. Data engineers can run Postgres for both, add AMQP/Redis for lower latency, or use Kafka for horizontal scale paired with Postgres or Elasticsearch, eliminating the operational complexity of Kafka Streams.

Kestra 2.0 rewrote its orchestration engine to decouple workers from the database via a new Controller service over gRPC. It unbundles the queue and repository, letting engineers choose backends like Kafka, Redis, or JDBC independently to eliminate database bottlenecks.

Kafka architectures often sink real-time events to S3 for long-term retention. However, raw files fail to behave like database tables for updates and consistent reads. Open table formats like Apache Iceberg bridge this gap by adding schema, snapshot, and partition metadata over the data.

Recent data engineering updates highlight the release of Spark 4.1.0 and Apache DataFusion 55.0.0. Technical deep dives focus on computing time in queue for Apache Hudi pipelines at petabyte scale, streaming table patterns with Kafka and Fluss, and replacing Fivetran for Snowflake ingestion.

Key updates cover stream architecture and storage engineering: Khaos provides tools for Kafka load testing and chaos engineering, while DuckLabs joins AWS with its open source projects intact. Other deep dives evaluate RocksDB amplification factors and shared log composition via LogDrive.

Uber’s payments platform uses immutable money orders and double-entry bookkeeping to process transactions. It relies on Amazon DynamoDB for strongly consistent ledger balances across 1.2B+ entities and Apache Kafka as an async messaging bus between stateless microservices.

Kestra released a DigitalOcean plugin enabling declarative orchestration for Droplets, DOKS, and managed databases like Kafka and Postgres. It automates ephemeral compute pipelines and teardowns, letting engineers orchestrate batch infrastructure with secure secret handling and state polling.

Rednote migrated its real-time indexing pipeline from Kafka to Apache Fluss to solve read amplification from wide tables. Fluss uses columnar streaming via Apache Arrow and bypasses brokers for cold-data replays, isolating historical scans from real-time traffic to reduce bandwidth and compute

This architecture detects streaming price anomalies using Kafka, Flink, and Spark. Flink processes Kafka event streams, calculates rolling Z-scores using historical baselines built by Spark, and routes anomalies to Claude to generate human-readable explanations displayed in Streamlit.

This end-to-end recommendation platform processes e-commerce events via PostgreSQL, MinIO, Debezium, and Kafka. Spark builds Iceberg tables while Flink processes real-time streams. Airflow orchestrates pipelines, feeding Feast with PostgreSQL offline and Redis online feature stores.

Recent engineering highlights focus on scaling stream architectures from Kafka and Flink into Cassandra cold storage, scaling Polars queries to 16 billion rows, and integrating DuckLake with DataFusion. Tooling updates also spotlight pyhctsa for time-series analysis and graph-based data engines.