a platform to stack them all
GizmoSQL is an open-source data engineering tool that leverages columnar databases such as DuckDB and the Apache Arrow ecosystem to enhance SQL query performance.
Application of visual language models to structure product image data at marketplace scale, highlighting data engineering approaches for large-scale processing.
Technical demonstration of internal usage data analysis using SQL and MotherDuck, featuring queries for percentiles, consecutive streaks, and filtering out service accounts.
Benchmark of DuckDB processing 1TB of Parquet files on a single node, assessing read and aggregation performance.
rustocr is a Rust crate for OCR, providing data engineering tools for text extraction from images, with potential integration into data processing pipelines.
The article examines why traditional data catalogs fail at enterprise scale and presents a three-part framework for implementing them as products, featuring decoupled architecture, metadata contracts in CI/CD workflows, and ruthless curation through tiering.
The article discusses emerging skills for data engineers in 2026, focusing on AI integration and open table formats like Iceberg to enhance data pipelines and automation.
Introduction to 8 fundamental principles for data engineering, covering best practices, architectures, and essential tools for building robust and scalable data systems.
Explains building a semantic search engine with PartyKit, using Vectorize as a vector database and an embedding model to index and query documents through vectors.
The article explores DataOps methodology, applying DevOps principles to data engineering to enhance collaboration, CI/CD automation, data quality, and pipeline observability.
Introduces a local-first reversible PII scrubber tool using ONNX and regex for AI workflows, emphasizing privacy and efficiency in data engineering processes.
Risk-based data quality approach to optimize testing, focusing on critical aspects and pipeline automation for data engineers.
The article examines current arguments that AI makes data modeling obsolete, including context windows as schemas, denormalized tables for LLM queries, and agentic modeling, while highlighting technical pitfalls of these approaches.
Vercel simplified its text-to-SQL agent d0 by replacing specialized tools with a single bash tool, allowing Claude direct file system access for faster and more reliable execution.
The article examines the impact of LLMs on formal verification in data engineering, highlighting how AI assists in generating formal specifications and proofs, while discussing limitations such as the trusted computing base.
PostgreSQL 18 enables instant database clones using the FILE_COPY strategy with file_copy_method=clone, leveraging copy-on-write features of modern filesystems like XFS for zero-copy duplication and efficient storage management.
Technical comparison between cloud and on-premise data transformations, analyzing differences in scalability, costs, deployment, maintenance, and security to help data engineers select the appropriate approach.
Visual explanation of the Transformer architecture, a deep learning model using attention for parallel data processing, detailing encoding and decoding layers.
Framework for architecting a scalable data platform using Dagster as a central orchestrator, emphasizing composability and managing fragmentation in modern data stacks.
The article compares CDPs and reverse ETL, highlighting that reverse ETL activates data from the data warehouse to operational tools without creating silos, providing better governance and reduced latency.
Practical guide to building a DuckLake by combining DuckDB with open table formats like Iceberg, focusing on open-source architecture and analytical performance.
The article outlines seven essential pillars for building robust production-grade AI agent architectures, covering governance, orchestration, and observability for agentic systems.
The article discusses alert fatigue in data quality strategies, offering solutions such as building monitors that learn from data patterns and prioritizing alerts to enhance observability.
A Snowflake software update caused a 13-hour outage across 10 regions due to a backwards-incompatible database schema change, disrupting SQL queries and data ingestion via Snowpipe and Snowpipe Streaming.
Legacy BI tools like Tableau and Power BI face structural pressures from data platforms, AI-native analytics tools, and semantic layers, eroding their value proposition and leading to price increases.
