Kafka resources for data engineers

Amazon Aurora PostgreSQL now supports direct querying of Apache Iceberg and Parquet data lake files. Other data engineering highlights include extending Apache DataFusion across machines, using ClickHouse for metric discovery at LinkedIn, and DuckDB 2.0 adding dynamic ATTACH statements.

redb.Route is a .NET integration framework acting like Apache Camel. It uses a C# DSL to build From-Process-To pipelines connecting Kafka, RabbitMQ, Redis, SQL, and S3 without message-bus lock-in. It supports 29 external transports, hot-reloading, and token streaming.

The edition highlights Kafka and stream processing developments, focusing on broker architectures and Flink SQL patterns. Notable releases include Apache Fluss v1.0 and Khaos for failure testing, while new alternatives like blob-stream target cost over latency.

Real-time ETL spans batch, micro-batch, log-based CDC, and event streaming via tools like Kafka. Selection hinges on required latency rather than raw speed: log-based CDC efficiently reads database transaction logs, while true streaming continuously processes events for sub-second workloads.

Data engineering is shifting from rigid ETL toward autonomous pipelines and open lakehouses. Teams are adopting formats like Apache Iceberg and Delta Lake alongside Kafka and Flink for unified streaming, reducing manual SQL scripting in favor of data reliability engineering and FinOps.

Build a local real-time pipeline by publishing raw events to Kafka, deduplicating them with Flink SQL, and streaming the results into DuckLake. A consumer writes batches every 50 records or 5 seconds, using DuckLake row inlining to avoid small Parquet files.

This meetup brings together data leaders to discuss scaling open-source data infrastructure, focusing on technologies like Kafka, PostgreSQL, ClickHouse, and OpenSearch. Discussions center on simplifying operational management, cloud cost optimization, and architecting modern data stacks for growth.

The Kafka User Group session covers event-driven architectures with Apache Kafka, focusing on publishing database changes to downstream services. It also explores using Kafka as the core messaging backbone for multi-service superapps, alongside Atlas Search indexes for handling near-match queries.

Real-time data engineering patterns center on streaming architectures and change data capture. Delivery Hero routes Postgres CDC events into Kafka to feed its live ML feature store, while Zalando abstracts Kafka via Nakadi to swap topics seamlessly.

Snowflake launched Data Connectivity Proxy (DCP) for Openflow, allowing secure ingestion from private databases and Kafka brokers. Delivered as a Docker agent, it uses outbound TLS on port 443 with mTLS tunnels to bypass inbound firewall rules without inspecting data or storing source credentials.

Diskless Kafka decouples compute from storage by shifting durable logs to cloud object stores via KIP-1150. While retaining Kafka APIs and offset ordering, it allows brokers to scale independently without managing local disk partitions or moving datasets.

Data engineering developments highlight Diskless Kafka architectures that separate storage from brokers. Other focal points include DuckDB 2.0 performance, executing multi-day PostgreSQL CDC backfills during writes, and optimizing Spark resource waste across the stack.

Kestra 2.0 decouples its control and data planes, allowing workers to run anywhere via outbound-only gRPC streams without direct database access. The engine also decouples the queue from the repository, unifying the architecture while supporting backends like Postgres, Kafka, Redis, or AMQP.

Kestra 2.0 introduces a unified orchestration engine with decoupled queues and repositories. Workers now run anywhere using outbound gRPC with no database access. Data engineers can keep Postgres and switch queues to Kafka or Redis without touching flows, keeping execution next to the data.

Apache Iggy and Kafka both use partitioned, append-only logs with offsets and consumer groups. However, while Kafka runs on the JVM, Iggy uses Rust with a thread-per-core architecture on io_uring. The key architectural divergence lies in how each engine defines and guarantees successful writes.

Key data engineering topics highlight optimized data pipelines and storage architectures, including direct S3-to-GPU data loading for machine learning, database proxy patterns via ZGateway for ZippyDB, and distributed processing updates from Apache Kafka versus Iggy and Polars 2.0.

Kestra 2.0 decouples queue and repository backends under a unified engine. Data engineers can run Postgres for both, add AMQP/Redis for lower latency, or use Kafka for horizontal scale paired with Postgres or Elasticsearch, eliminating the operational complexity of Kafka Streams.

Kestra 2.0 rewrote its orchestration engine to decouple workers from the database via a new Controller service over gRPC. It unbundles the queue and repository, letting engineers choose backends like Kafka, Redis, or JDBC independently to eliminate database bottlenecks.

Kafka architectures often sink real-time events to S3 for long-term retention. However, raw files fail to behave like database tables for updates and consistent reads. Open table formats like Apache Iceberg bridge this gap by adding schema, snapshot, and partition metadata over the data.

Recent data engineering updates highlight the release of Spark 4.1.0 and Apache DataFusion 55.0.0. Technical deep dives focus on computing time in queue for Apache Hudi pipelines at petabyte scale, streaming table patterns with Kafka and Fluss, and replacing Fivetran for Snowflake ingestion.

Key updates cover stream architecture and storage engineering: Khaos provides tools for Kafka load testing and chaos engineering, while DuckLabs joins AWS with its open source projects intact. Other deep dives evaluate RocksDB amplification factors and shared log composition via LogDrive.

Uber’s payments platform uses immutable money orders and double-entry bookkeeping to process transactions. It relies on Amazon DynamoDB for strongly consistent ledger balances across 1.2B+ entities and Apache Kafka as an async messaging bus between stateless microservices.

Kestra released a DigitalOcean plugin enabling declarative orchestration for Droplets, DOKS, and managed databases like Kafka and Postgres. It automates ephemeral compute pipelines and teardowns, letting engineers orchestrate batch infrastructure with secure secret handling and state polling.

Rednote migrated its real-time indexing pipeline from Kafka to Apache Fluss to solve read amplification from wide tables. Fluss uses columnar streaming via Apache Arrow and bypasses brokers for cold-data replays, isolating historical scans from real-time traffic to reduce bandwidth and compute

This architecture detects streaming price anomalies using Kafka, Flink, and Spark. Flink processes Kafka event streams, calculates rolling Z-scores using historical baselines built by Spark, and routes anomalies to Claude to generate human-readable explanations displayed in Streamlit.

This end-to-end recommendation platform processes e-commerce events via PostgreSQL, MinIO, Debezium, and Kafka. Spark builds Iceberg tables while Flink processes real-time streams. Airflow orchestrates pipelines, feeding Feast with PostgreSQL offline and Redis online feature stores.

Recent engineering highlights focus on scaling stream architectures from Kafka and Flink into Cassandra cold storage, scaling Polars queries to 16 billion rows, and integrating DuckLake with DataFusion. Tooling updates also spotlight pyhctsa for time-series analysis and graph-based data engines.

The streaming cold tier processes up to 1,250 reads per second globally, delivering a p50 latency of ~150 ms and a p99 of ~500 ms. Once a time slice crosses the user-configured cold-tier boundary, TimeSeries stops accepting writes for that time slice making it immutable.

OpenData Log is an open-source, object-native log built in Rust for routing rather than funneling data. Unlike Kafka, it replaces topic-partitions with key-oriented indexing via an LSM tree to manage millions of individual logs with low read amplification.

Airflow 3.0 introduces Data Assets to optimize resource utilization by over 30%. Key technical coverage includes DoorDash's gateway for AI agent-tool access, disaggregated system architectures, asynchronous I/O in DuckDB, and evaluating Postgres index types like B-tree, GIN, and BRIN.