dataaaaaa!

Apache Flink Agents 0.4.0 brings AI agents into streaming pipelines, providing distributed execution and fault tolerance. The update adds sub-agents with durable execution, dynamic model routing, and the unified Decision Model API for scoring against business logic.

Key data engineering releases include Apache Kafka 4.4, DuckDB 2.0, Flink CDC 3.7, and Confluent's official Rust client for Kafka. Engineering highlights cover stream processing patterns like deduplication in Flink SQL, session-ordered Go pipelines, and scaling shuffle with Apache Celeborn.

Apache Fluss enables crash-safe retries for side-effecting workers by combining an append-only log table for replay with a primary key table for state lookups. Workers persist intent before external calls, allowing restarted jobs to reconcile pending actions against external service ledgers.

Netflix rebuilt its Ads Event Publisher around stateful stream processing with Apache Flink. To handle massive live traffic spikes, it replaced synchronous key-value store writes with a real-time streaming join of Kafka streams using a custom KeyedCoProcessFunction over a 60-minute window.

Apache Flink CDC 3.7.0 adds AI-powered stream processing with OpenAI-compatible model transforms and ordered async execution. It adds inline Python UDFs, SQL Server and Fluss pipeline sources, plus schema expansion for Paimon and Fluss sinks.

Iceberg delete file types are determined by the engine implementation, not user configs. Spark batch writes only position deletes, while Flink writes both. Because equality deletes cannot be directly compacted, maintain them using rewritedatafiles with delete-file-threshold to purge stale rows.

Implement the streamhouse pattern locally using Apache Fluss, Apache Flink, and Apache Iceberg to enable direct querying of streaming data. This architecture integrates Fluss for real-time storage with Flink processing and Iceberg lakehouse tables.

Apache Iceberg 1.12.0 expands semi-structured and geospatial data capabilities. Flink and Kafka Connect can now write shredded Variant columns alongside Spark. The release also adds a Flink maintenance task to convert equality deletes into positional deletion vectors for faster streaming reads.

The edition highlights Kafka and stream processing developments, focusing on broker architectures and Flink SQL patterns. Notable releases include Apache Fluss v1.0 and Khaos for failure testing, while new alternatives like blob-stream target cost over latency.

Delivery Hero replaced its batch ad measurement pipeline with Amazon Managed Service for Apache Flink. The streaming architecture resolved processing-time skew, synchronous API enrichment bottlenecks, and costly document database mutations, cutting infrastructure expenses by over half.

Data engineering is shifting from rigid ETL toward autonomous pipelines and open lakehouses. Teams are adopting formats like Apache Iceberg and Delta Lake alongside Kafka and Flink for unified streaming, reducing manual SQL scripting in favor of data reliability engineering and FinOps.

Build a local real-time pipeline by publishing raw events to Kafka, deduplicating them with Flink SQL, and streaming the results into DuckLake. A consumer writes batches every 50 records or 5 seconds, using DuckLake row inlining to avoid small Parquet files.

Apache Fluss 1.0 delivers real-time storage for streaming and AI workloads. It adds a Rust-based HTTP Gateway for REST operations and lakehouse ingestion, native Python clients for feature lookups, and support for Flink, Spark, Paimon, and Apache Hudi.

🎙️ Kai Lentit — An interview satire explores big data engineering and architecture concepts projected into 2026. The discussion highlights ecosystem tools and patterns including Apache Iceberg, Hive tables, Flink, and Airflow graphs, alongside Kubernetes, Docker, Databricks, and cloud observability.

Flink Kubernetes Operator 1.16.0 updates autoscaling with a new BALANCED default parallelism alignment, avoiding blocked rescales and over-provisioning. It also adds pluggable autoscaler SPIs and supports native Kubernetes pod ResourceRequirements for TaskManagers and JobManagers.

A technical meetup focused on real-time stream processing covers Apache Flink architectures and the streaming ecosystem. Sessions explore managing Flink state schema evolutions for bot detection, running real-time ad pacing workloads, and using Apache Fluss as a modern streaming analytics layer.

Lyft migrated its streaming fleet from an in-house Kubernetes operator to the open-source Apache Flink Kubernetes Operator to eliminate custom code debt. The shift unlocked native autoscaling and robust reconciliation loops without disrupting existing stateful job workflows.

Technical deep dives highlight parallel Apache Spark tuning, stream processing migrations via the Apache Flink Operator at Lyft, and caching storage optimization using Zstandard and Pingora. New tool releases include dbt doctor and the MLPerf Storage v3.0 benchmarks for data workloads.

Kafka architectures often sink real-time events to S3 for long-term retention. However, raw files fail to behave like database tables for updates and consistent reads. Open table formats like Apache Iceberg bridge this gap by adding schema, snapshot, and partition metadata over the data.

Key updates highlight optimizing SQL index design to cut query times from 17ms to 0.04ms, comparing Flink autoscalers, and handling automated Apache Kafka migrations. Tool launches spotlight DuckDB v2.0 parser improvements and DataFusion integrations with Cloudflare R2 SQL.

August data engineering roundups highlight streaming, query engines, and platform patterns. Apache Fluss became a top-level project, DuckDB 2.0 was previewed, and Netflix detailed streaming joins in Flink alongside its real-time distributed graph architecture.

Netflix is transitioning from an in-house Flink autoscaler to the Apache Flink Autoscaler to handle complex, stateful DAGs. Unlike coarse cluster metrics, the open-source engine estimates each operator's true processing rate to tune vertex-level parallelism, orchestrated via Temporal workflows.

Rednote migrated its real-time indexing pipeline from Kafka to Apache Fluss to solve read amplification from wide tables. Fluss uses columnar streaming via Apache Arrow and bypasses brokers for cold-data replays, isolating historical scans from real-time traffic to reduce bandwidth and compute

Netflix evolved its ads event architecture for live streaming traffic surges by replacing a synchronous database write on the ad-serving critical path with stateful stream processing.

CDC does not eliminate dual-writes; it merely shifts them downstream. Updating targets and committing pipeline offsets are independent, non-transactional acts. Without idempotent receivers, dual-write failure modes persist.