Open infrastructure resources for data engineers

🎙️ DataTalksClub ⬛ — Sahil Walia explains how open lakehouses and Apache Iceberg decouple storage from compute to transform modern data platforms. The session highlights how Iceberg metadata simplifies tasks like GDPR compliance, data deletions, and file compactions across different compute engines.

Flink Kubernetes Operator 1.16.0 updates autoscaling with a new BALANCED default parallelism alignment, avoiding blocked rescales and over-provisioning. It also adds pluggable autoscaler SPIs and supports native Kubernetes pod ResourceRequirements for TaskManagers and JobManagers.

Declarative data pipelines can map nouns to sources and verbs to transformations using xorq expressions. Reusable metrics are registered in a content-addressed, versioned catalog. The setup enables decoupling transformations from execution engines across DataFusion, DuckDB, and Snowflake.

As Apache Ossie standardizes semantic models, differentiation shifts to query-time behavior. Agent workloads require dynamic SQL generation for ad-hoc transforms without config changes, plus hybrid search over entities and rules, moving beyond static caching and simple keyword lookups.

This guide explores Parquet internals, engineering trade-offs, and file storage formats including JSON, Avro, and ORC. It analyzes row versus columnar storage architectures to explain why Parquet remains the dominant choice for modern Spark pipelines and analytical data processing.

Diskless Kafka decouples compute from storage by shifting durable logs to cloud object stores via KIP-1150. While retaining Kafka APIs and offset ordering, it allows brokers to scale independently without managing local disk partitions or moving datasets.

Kestra 2.0 expands plugins beyond integrations to platform internals, adding custom policy rules, file renderers, and log data stores. Notably, flow-level pluginDefaults is removed, and open source now supports external log data stores to streamline migrations.

Data platforms outlive vendors when engineers isolate business logic from engines. Rent commoditized storage, compute, and runtimes, but own your pipeline code, metric definitions, and data contracts.

RustFS launched open-source S3 Tables, embedding an Apache Iceberg REST Catalog into its storage kernel. One process on port 9000 serves S3 and Iceberg REST protocols.

Knap is an open-source templating engine that transforms structured data into Markdown. It supports variables, logic, array iteration, and filters to format outputs or generate YAML frontmatter. Data pipelines can use its CLI to render files directly from JSON sources using simple templates.

Kestra 2.0 orchestrates workflows using declarative YAML over existing code. Its new Controller decouples workers from the database, letting them run in isolated networks via outbound gRPC without inbound ports. This architecture avoids direct database credentials on workers.

Filament is an Apache 2.0 open-source data movement engine engineered to reliably transfer data across databases, SaaS platforms, and object storage. Built to overcome rigid, row-based managed ETL pricing, it manages extraction, batching, retries, checkpoints, and fine-grained schema handling.

Postgres uses a process-per-connection architecture to isolate query execution and memory, maximizing code simplicity and crash resistance. While shared buffers handle base tables to ensure correctness, local session buffers manage temporary tables without locking overhead.

Kestra 2.0 decouples its control and data planes, allowing workers to run anywhere via outbound-only gRPC streams without direct database access. The engine also decouples the queue from the repository, unifying the architecture while supporting backends like Postgres, Kafka, Redis, or AMQP.

Filament provides pluggable data replication across sources and sinks via full, incremental, or CDC pipelines. It features bounded batching, verification on both sides of a write, and durable checkpoints using PostgreSQL and NATS JetStream. It can run embedded in Go or deploy to Kubernetes via Helm.

Kestra 2.0 introduces a unified orchestration engine with decoupled queues and repositories. Workers now run anywhere using outbound gRPC with no database access. Data engineers can keep Postgres and switch queues to Kafka or Redis without touching flows, keeping execution next to the data.

Narwhal is an agent-native terminal database client providing schema diffing, streaming cancellable queries, and an embedded MCP server. It supports six engines including DuckDB, ClickHouse, and Postgres via a single binary.

ADBC improves on ODBC by eliminating complex driver installs and system-level managers. While ODBC requires OS-level driver managers and vendor-specific setups, ADBC packages its driver manager directly into language client libraries while preserving cross-language database portability.

Apache Arrow Go v18.8.0 adds SIMD-accelerated compute kernels and Parquet optimizations. It brings direct bitmap decoding, memory reuse across page compressors, and native GeoArrow support, drastically cutting allocations and latency across vectorized data pipelines.

Sail 0.7 introduces blocking shuffle and checkpointing by delegating intermediate job state to object storage. An auxiliary stage merges mapper outputs into compressed Arrow data in storage, isolating task regions to prevent cascading retries and allow execution on smaller clusters.

Meta introduced ZGateway, a stateless proxy layer for ZippyDB that unifies client traffic and prevents connection storms. Handling over one billion operations per second, it decouples millions of clients from database shards, collapsing a dense TLS mesh to improve reliability and connection pooling.

🎙️ It's About Data — The episode discusses Nvidia acquiring open-source model repository Hugging Face. Guests Miriah Peterson and Matt Sharp explore the implications of this acquisition for machine learning infrastructure and deploying LLMs in production environments.

Emerging AI architectures shift data requirements beyond standard LLM pipelines. Accelerated Understanding relies on neural operators instead of transformers, claiming it can process up to 5 trillion data points in a single prompt to model complex physical systems at massive scale.

Astral endorsed four candidates for the inaugural Python Packaging Council, established by PEP 772 to standardize packaging governance. The body aims to modernize infrastructure like PyPI and tools like pip and uv, addressing complex dependency resolution, security, and scientific Python workflows.

MLPerf Storage v3.0 expands AI pipeline benchmarking by adding support for S3 object storage alongside POSIX layers. It introduces new benchmarks for KV cache I/O in LLM inference and vector database indexing, helping data engineers evaluate storage architectures against critical AI workloads.

AWS acquired DuckLabs to influence the engine powering modern data pipelines. Acting as a cost-efficient alternative to Spark with native Parquet support, DuckDB pairs with DuckLake and the Quack protocol to anchor an S3-based, serverless lakehouse stack that competes with Snowflake and Databricks.

Lyft migrated its streaming fleet from an in-house Kubernetes operator to the open-source Apache Flink Kubernetes Operator to eliminate custom code debt. The shift unlocked native autoscaling and robust reconciliation loops without disrupting existing stateful job workflows.

Kestra 2.0 decouples queue and repository backends under a unified engine. Data engineers can run Postgres for both, add AMQP/Redis for lower latency, or use Kafka for horizontal scale paired with Postgres or Elasticsearch, eliminating the operational complexity of Kafka Streams.

LakeSail co-founder Shehab Amin discusses rebuilding Apache Spark compatibility using Rust, Apache Arrow, and DataFusion. The project aims to turn existing data pipelines into foundations for AI workloads. "Spark isn't going anywhere."

DuckDB has branched v2.0-cyanoptera into feature freeze for an October release, entering alpha testing for Python and CLI clients. Extensions like quack 1.0, httpfs, ducklake, and iceberg are available in alpha, boosting query throughput and enabling community extension testing against ref_next