Scalability resources for data engineers

🎙️ DataTalksClub ⬛ — Building production AI platforms requires managing evaluation datasets, LLM memory, and tool orchestration. Key technical patterns include mitigating context window limits, optimizing tool calling, and overcoming retrieval degradation like the lost in the middle problem in RAG pipelines.

DuckDB on an iPhone outperformed Databricks Serverless SQL on TPC-H workloads up to 100 GB. Single-node execution engines avoid the query planning and network shuffle overhead inherent to distributed clusters, making them far more cost-effective for typical real-world business datasets.

Snowflake cost optimization requires granular tracking across compute pools and Cortex AI functions. Simple fixes like right-sizing warehouses and reducing job frequencies yield the highest ROI, while observability views help detect token spikes before limits are breached.

DuckDB database files can store only view definitions pointing to remote Parquet files on object storage instead of raw data. Acting as a lightweight catalog under a few hundred kilobytes, it creates a shareable semantic layer that abstracts file layouts while querying via the httpfs extension.

AI agents still require Kubernetes to run, scale, and recover reliably in production. However, because agents spend up to 90% of their time idle, assigning dedicated pods wastes resources; new patterns like Agent Substrate decouple agent sessions from pods to efficiently pack workloads.

Infino launched a unified agent retrieval layer that queries Parquet in object storage directly via SQL. By combining vector search, keyword search, joins, and aggregates into one query over a single copy of data, it eliminates fragmented ETL pipelines, sync jobs, and multi-database sprawl.

Right-sizing analytics pipelines prevents early technical debt while avoiding premature distributed complexity. Starting with single-node tools like Polars, Postgres, and DuckDB enables high performance before migrating to Databricks or Snowflake.

San Diego Data Foundations focuses on engineering data platforms ready for AI. Engineers will explore managing production Cassandra clusters with AI-driven operations for faster incident triage.

Turbopuffer v3 redesigns its storage engine around a unified columnar layout. It introduces batched reads and vectorized execution across full-text search and order-by queries, yielding a 9x speedup in full-text indexing while avoiding row materialization overhead.

Google Cloud details how Cloud Spanner powers its managed Apache Iceberg Lakehouse runtime catalog at scale, providing a robust architecture for high-concurrency metadata operations and distributed data lake analytics.

Parseable is a Rust-based, stateless columnar data lake for telemetry data. Built on open Parquet and object storage native architecture, it unifies logs, metrics, and traces while managing high-cardinality workloads at scale with SQL querying.

Python Polars 2.0.0 delivers major data engine updates, adding streaming out-of-core sorting, caching Iceberg manifest files across scans, and pushing down filters to Lance scans. Breaking changes include reading Parquet ENUM as String and evaluating SQL QUALIFY before projection.

ClickHouse Cloud outperformed Databricks Serverless SQL on continuous ingestion benchmarks, achieving 752x higher performance per dollar while streaming 113.2 billion quotes. Its efficient data preparation and real-time pre-aggregations reduce query cost and eliminate pipeline lag.

Polars 2.0 introduces out-of-core processing and makes its streaming engine default on LazyFrame.collect, spilling to disk at ~80% RAM. It adds native Arrow MapType support, join reordering, dynamic bloom filters, and first-class SQL coverage outperforming DuckDB in benchmarks.

dbt State avoids unnecessary warehouse rebuilds by tracking changes to compiled SQL logic and upstream data freshness. Following a cheapest valid action pattern, it reuses existing tables, clones across schemas, or rebuilds only when needed, with fine-grained controls like lag_tolerance.

Cousteau is a lakehouse built to simplify fragmented data stacks, unifying warehouse performance, storage economics, live data federation, and agent-ready context into a single foundation.

Databricks Serverless allows running lightweight tools like DuckDB and Polars alongside Spark for medium workloads with Unity Catalog support. Benchmarks evaluate how DuckDB and Polars handle joins and window functions on Databricks Serverless compared to Spark broadcast join patterns.

Amazon Aurora PostgreSQL now supports direct querying of Apache Iceberg and Parquet data lake files. Other data engineering highlights include extending Apache DataFusion across machines, using ClickHouse for metric discovery at LinkedIn, and DuckDB 2.0 adding dynamic ATTACH statements.

To replace slow MongoDB analytics, DuckDB was deployed on the same host to handle sub-second queries. After testing JDBC batching and the community Mongo extension, custom Java table functions were chosen for ingestion to avoid C++ maintenance while streaming data without intermediate disk files.

Dexaflow is a container-native workflow orchestrator compatible with the Apache Airflow UI and REST API. It replaces XCom plumbing with native map-reduce syntax. Built with a Go control plane, it supports isolated per-DAG virtual environments and pod-per-task execution.

To scale SQL Server pipelines, avoid SELECT * to reduce I/O and preserve covering indexes. Prevent table scans by writing sargable queries that keep indexed columns bare of functions, and join on indexed keys. Replacing IN with EXISTS speeds up subqueries by short-circuiting on matches.

Terminal multiplexers detach processes from SSH sessions using PTYs so remote workloads keep running after disconnects. tmux manages these via client-server sockets, but complex session CLI commands make managing continuous background processes cumbersome, prompting custom solutions like aplexer.

ClickHouse Managed Postgres uses Direct I/O for backups to bypass the Linux page cache, preventing cache eviction of warm database pages during query execution. By aligning reads across striped NVMe drives, it reduced query latency impact by two-thirds and avoided cache pollution entirely.

Tabby is hosting a technical meetup covering pipelines, warehousing, feature stores, and operating multi-region infrastructure under payment load. Sessions focus on aligning AI strategy with data platforms, embedding governance directly into pipelines via metadata and lineage, and the Tabby DWH

Leoflow is a container-native workflow orchestrator compatible with the Apache Airflow UI and REST API. Powered by a Go control plane and Postgres, it offers native map-reduce fan-out without XCom plumbing or brokers, plus isolated per-DAG virtual environments.

Speed up string aggregations by moving repeated strings into dimension tables with narrow integer keys. Hash aggregation on fixed-width integers reduces CPU overhead and memory footprint compared to variable-length text. The strings are joined back only after grouping.

Turbopuffer is overhauling its storage engine to make ANN search a secondary index rather than the primary layout. Centering storage around vector addresses caused write and storage amplification during multi-vector and full-text queries, prompting a new architecture for generalized search.

PayPal details its adoption of Google Cloud Managed Service for Apache Spark to accelerate data analytics workloads. The architecture focuses on optimizing large-scale Spark processing and enterprise data pipelines.

LinkedIn expanded ClickHouse from distributed tracing to metric discovery and analytics. The consolidated metadata index handles 150k+ queries per minute across 13B+ metrics with 68ms average latency, slashing memory use to one-fifth and compute to two-thirds.

Data engineering updates highlight DuckDB 1.5.6, Parquet X-ray, and billion-row record linkage with Splink 5. Key analyses cover streaming pipeline cost efficiency, PostgreSQL row-level security performance, handling hot shards, and building analytical MySQL replicas with Parquet and DuckDB.