dataaaaa!

LogHouse scales to 1.59 quadrillion rows using geosharding across 30+ regions to handle 190 million rows per second. It reconciles distributed writes with a unified read layer. Writes stay local to the region where they arrive, avoiding cross-region egress costs. This enables sub-second queries.

dbt Wizard is a CLI for agentic data work that automates migrations and semantic layer builds. It uses a validation loop to iterate on implementations until they pass project constraints. I didn't have to babysit Wizard or build any additional workflows on top of it.

Loading multi-GB datasets on memory-constrained 4 GB pods requires bypassing the default dlt pipeline state growth. By initializing a fresh pipeline object per source and using streaming two-pass CSV splitting, peak memory remains proportional to small chunks rather than the total file size.

Edwin Weber built a metadata-driven pipeline using Dagster, dlt, dbt, and DuckDB to process Danish parliament data. He utilized Dagster assets to manage infrastructure, specifically stopping Metabase containers to prevent DuckDB locking conflicts.

Orchestra Runtime is a control plane and serverless infrastructure for AI agents to build, run, and monitor autonomous data workflows. It enables agentic control loops by providing sandboxed execution and a context graph. Autonomous AI needs two things: a control plane and a runtime.

Snowflake now supports using SQL queries as logical tables within semantic views. This allows engineers to define tables via joins or transformations directly in the TABLES clause.

Snowflake launched Analytical search in Public Preview, an orchestration capability for Cortex Agents. It enables complex queries like counts and trends across large document sets. Analytical search addresses this by orchestrating multi-index Cortex Search queries with configurable result limits.

BigQuery's AI.AGG function enables data engineers to synthesize large datasets using LLMs directly within SQL. It simplifies trend analysis by summarizing unstructured data at scale. AI.AGG allows you to perform complex reasoning over groups of data without moving it out of BigQuery.

Data engineers are shifting toward agentic workflows and AI-assisted refactoring. Key developments include Databricks open-sourcing Omnigent for multi-agent orchestration and ktx providing a self-improving context layer for warehouse queries. AI enthusiasts are in a race against time.

Building for agents requires moving beyond human-centric developer experience. While agents favor text-based interfaces like SDKs, true agentic experience (AX) demands lower friction than traditional DX. Building a beautiful DX for a human can translate into an AX that is merely functional.

Managing 12TB of AI coding agent logs requires robust data engineering to handle high-volume telemetry and trace data. Engineers must focus on scalable storage architectures and efficient indexing to enable rapid retrieval. "We need to be able to see what the agent was thinking at every step."

Hardwood 1.0 is a dependency-free Parquet reader for Java 21+ that uses multi-threading to decode pages in parallel. It offers a row API for ergonomics and a batch-style column API for high-throughput analytical workloads. Hardwood fans out the decoding of the individual pages to multiple threads.

This project replicates dbt Cloud features using dbt Core, FastAPI, and React. By integrating the self-hosted Prefect API for orchestration, it enables job scheduling and real-time monitoring without a CLI. APIs are built for programs to communicate with software and cli for humans. Lesson learnt.

ClickHouse argues that benchmarks must be transparent and reproducible to avoid deceptive benchmarketing. They critique recent comparisons for lacking open methodologies. To be useful, benchmarks need to be reproducible and run with a clear methodology.

Operationalizing data orchestration requires separating infrastructure from business logic through GitOps. Using tools like Dagster on Kubernetes allows for scalable, state-aware deployments. DevOps is what data engineering was a decade ago when the focus shifted to stability and automation.

Cognee 1.0 provides open-source memory for AI agents using a Rust core and single-Postgres deployment. It integrates dlt to load structured data, enabling self-improving graph-based pipelines. cognee 1.0 is live: open-source memory for AI agents, now self-improving.

Using AI models like Claude and Gemini accelerates vulnerability research in large C++ codebases by generating security hypotheses and validating local environments. This approach identifies subtle flaws in professional-grade code.

Flink 2.3 introduces a native S3 filesystem that removes the Hadoop dependency tree and AWS SDK v1. This plugin unifies checkpointing and exactly-once sinks into a single 13MB JAR. The native plugin removes the trade-off entirely between performance and functionality found in older plugins.

Notion Custom Agents now feature a native ClickHouse connection via OAuth, enabling plain-language data exploration and read-only SQL queries. The 14 tools cover what an agent needs to explore and report on your data, including table listing and cost analysis without deploying MCP servers.

Apache Flink 2.3.0 expands SQL capabilities with FROMCHANGELOG and TOCHANGELOG operators for dynamic table conversion. It introduces a high-performance native S3 filesystem and SinkUpsertMaterializer improvements. Flink 2.3.0 brings together contributors from around the globe.

ClickHouse introduced Silk, a C++ fiber library and NUMA-aware scheduler designed for high-concurrency I/O. It uses iouring and avoids heap allocations to achieve nanosecond-scale yields.

Data engineering shifts toward open standards like ODCS for data contracts and ODPS for data products to enable automation. By defining output ports and semantic layers in standard-conformant files, engineers can use coding agents to build architectural units. A data contract exists to build trust.

SQLRooms is an open-source React toolkit for building data-centric apps using client-side DuckDB. It enables fast in browser data processing without a backend by leveraging OPFS for persistent storage. Engineers can build modular, privacy-focused analytics tools with AI agent integration.

dbt State optimizes workflows by parsing SQL into syntax trees to detect logic changes rather than simple file hashes. This allows it to skip rebuilds for whitespace or comment updates. It reuses existing objects across schemas and uses lag tolerance to avoid unnecessary data refreshes.

ClickHouse has launched the Managed ClickStack MCP server to connect AI agents to observability data. It uses specialized investigation primitives for logs and traces rather than raw SQL.