dataaaaa!

Snowflake launched Data Connectivity Proxy (DCP) for Openflow, allowing secure ingestion from private databases and Kafka brokers. Delivered as a Docker agent, it uses outbound TLS on port 443 with mTLS tunnels to bypass inbound firewall rules without inspecting data or storing source credentials.

Diskless Kafka decouples compute from storage by shifting durable logs to cloud object stores via KIP-1150. While retaining Kafka APIs and offset ordering, it allows brokers to scale independently without managing local disk partitions or moving datasets.

Kestra 2.0 decouples its control and data planes, allowing workers to run anywhere via outbound-only gRPC streams without direct database access. The engine also decouples the queue from the repository, unifying the architecture while supporting backends like Postgres, Kafka, Redis, or AMQP.

Kestra 2.0 introduces a unified orchestration engine with decoupled queues and repositories. Workers now run anywhere using outbound gRPC with no database access. Data engineers can keep Postgres and switch queues to Kafka or Redis without touching flows, keeping execution next to the data.

Apache Iggy and Kafka both use partitioned, append-only logs with offsets and consumer groups. However, while Kafka runs on the JVM, Iggy uses Rust with a thread-per-core architecture on io_uring. The key architectural divergence lies in how each engine defines and guarantees successful writes.

Kestra 2.0 decouples queue and repository backends under a unified engine. Data engineers can run Postgres for both, add AMQP/Redis for lower latency, or use Kafka for horizontal scale paired with Postgres or Elasticsearch, eliminating the operational complexity of Kafka Streams.

Kestra 2.0 rewrote its orchestration engine to decouple workers from the database via a new Controller service over gRPC. It unbundles the queue and repository, letting engineers choose backends like Kafka, Redis, or JDBC independently to eliminate database bottlenecks.

Kafka architectures often sink real-time events to S3 for long-term retention. However, raw files fail to behave like database tables for updates and consistent reads. Open table formats like Apache Iceberg bridge this gap by adding schema, snapshot, and partition metadata over the data.

Uber’s payments platform uses immutable money orders and double-entry bookkeeping to process transactions. It relies on Amazon DynamoDB for strongly consistent ledger balances across 1.2B+ entities and Apache Kafka as an async messaging bus between stateless microservices.

Kestra released a DigitalOcean plugin enabling declarative orchestration for Droplets, DOKS, and managed databases like Kafka and Postgres. It automates ephemeral compute pipelines and teardowns, letting engineers orchestrate batch infrastructure with secure secret handling and state polling.

Rednote migrated its real-time indexing pipeline from Kafka to Apache Fluss to solve read amplification from wide tables. Fluss uses columnar streaming via Apache Arrow and bypasses brokers for cold-data replays, isolating historical scans from real-time traffic to reduce bandwidth and compute

This architecture detects streaming price anomalies using Kafka, Flink, and Spark. Flink processes Kafka event streams, calculates rolling Z-scores using historical baselines built by Spark, and routes anomalies to Claude to generate human-readable explanations displayed in Streamlit.

This end-to-end recommendation platform processes e-commerce events via PostgreSQL, MinIO, Debezium, and Kafka. Spark builds Iceberg tables while Flink processes real-time streams. Airflow orchestrates pipelines, feeding Feast with PostgreSQL offline and Redis online feature stores.

The streaming cold tier processes up to 1,250 reads per second globally, delivering a p50 latency of ~150 ms and a p99 of ~500 ms. Once a time slice crosses the user-configured cold-tier boundary, TimeSeries stops accepting writes for that time slice making it immutable.

OpenData Log is an open-source, object-native log built in Rust for routing rather than funneling data. Unlike Kafka, it replaces topic-partitions with key-oriented indexing via an LSM tree to manage millions of individual logs with low read amplification.

clickhousectl v0.4.0 introduces horizontal autoscaling for Cloud services and ClickPipe schema discovery. Engineers can configure replica ranges and probe Kafka or Kinesis sources to infer schemas before creating ingestion pipes. "clickhousectl is the CLI for ClickHouse, local and cloud"

🎙️ DATANOSCO — Duckle is an open-source ETL engine using DuckDB to run fast pipelines locally, on VPS, or in browsers. It connects sources like PostgreSQL, Parquet, and Kafka, offering cost savings and binary deployments. Duckle features "AI-assisted pipeline design and seamless Git integration."

Streaming log tech enters a third wave by abandoning Kafka protocol compatibility. Apache Iggy uses Rust and thread-per-core io_uring architecture. S2 provides a low-level HTTP API without partitions. OpenData Log optimizes object-native storage for funnelling and routing workloads.

To handle 145 billion daily events, Atlassian migrated StreamHub from Amazon Kinesis to AWS Managed Streaming for Apache Kafka. Using Kafka Tiered Storage offloaded historical data to S3, reducing EBS costs while supporting multi-day retention.

The July 2026 data ecosystem round-up highlights key event streaming developments. Atlassian detailed transitioning from Kinesis to Kafka for 145 billion daily events, while Confluent covered cutting Flink OOMKills by 91.2%. Other topics include Apache Fluss graduating to an ASF top-level project.

Aiven acquired Flow AI to integrate analytical AI agent infrastructure into its managed open-source data platform. "Agents are only as good as the infrastructure serving them" so combining runtime and evaluation tools with Kafka and PostgreSQL helps teams deploy secure AI next to production data.

Snowflake announced Postgres CDC mirroring to automate ingestion and landed Snowflake DataStream, a Kafka-compatible endpoint streaming directly into native or Iceberg tables. OpenFlow also went headless, letting data engineers build ingestion pipelines programmatically via agentic code.

Apache Airflow has evolved beyond time-based batch ETL into an industry-standard orchestrator for AI, infrastructure, and event-driven data platforms. Modern features like asset dependencies, asset partitioning, and SQS or Kafka event triggers allow pipelines to execute dynamically based on

To combat schema drift, you can build an AI Impact Analyst that monitors Apache Iceberg catalog changes. By integrating Lakekeeper, a Rust-based REST catalog, DDL changes are pushed directly to Redpanda (Kafka).

Verihubs migrated its data warehouse from Postgres to ClickHouse to handle 50 million monthly API calls. By implementing Kafka and Debezium for streaming ingestion, they achieved 98% faster queries and 50% lower costs. Postgres was fast at first, but at some point it started to get slow.