ClickHouse resources for data engineers

ClickHouse integrates with Postgres to unite OLTP and OLAP workloads. Tools include pgclickhouse to query ClickHouse via native Postgres SQL with query pushdowns, and chdbhook to import and export data lake files through Postgres COPY commands across cloud object storage.

ClickHouse Cloud outperformed Databricks Serverless SQL on continuous ingestion benchmarks, achieving 752x higher performance per dollar while streaming 113.2 billion quotes. Its efficient data preparation and real-time pre-aggregations reduce query cost and eliminate pipeline lag.

Executable UDFs are now generally available on ClickHouse Cloud across AWS, GCP, and Azure. Data engineers can deploy precompiled binaries in Rust, Go, C++, or JavaScript alongside Python, manage resources via Terraform, and leverage outbound network access and per-query execution metrics.

Amazon Aurora PostgreSQL now supports direct querying of Apache Iceberg and Parquet data lake files. Other data engineering highlights include extending Apache DataFusion across machines, using ClickHouse for metric discovery at LinkedIn, and DuckDB 2.0 adding dynamic ATTACH statements.

LinkedIn expanded ClickHouse from distributed tracing to metric discovery and analytics. The consolidated metadata index handles 150k+ queries per minute across 13B+ metrics with 68ms average latency, slashing memory use to one-fifth and compute to two-thirds.

Updates to pgclickhouse and chdb improve Postgres-ClickHouse integration with a new checkencoding option. It handles invalid byte sequences across foreign tables and chdb queries using fail, remove, replace, or truncate handlers.

Integrating C++ into Postgres extensions causes severe memory safety issues because Postgres uses MemoryContext arena allocation and setjmp/longjmp error handling. Jumps bypass C++ destructors, and uncaught C++ exceptions abort the database process. To fix this, pg_clickhouse replaced C++ with C.

🎙️ DataGen — ClickHouse evolves into a broader data and AI platform, acquiring Langfuse to build an Agentic Data Stack for AI observability. The real-time database expands its core data engineering capabilities by integrating PostgreSQL support, Change Data Capture (CDC), and native streaming queries.

Agent telemetry—traces, tool calls, costs, and outcomes—shares keys like trace_id. Joining these datasets via SQL enables tracking true cost per resolved task and detecting retry loops. Doing this requires full-fidelity storage, long retention, and unified columnar databases over sampled spans.

ClickHouse now writes Apache Iceberg tables directly to Microsoft OneLake via Iceberg REST-based Table APIs. This integration automatically registers tables in the OneLake catalog under native access controls, allowing downstream engines like Fabric Spark and Power BI to query ClickHouse output.

ClickHouse expanded its Microsoft collaboration with a native Fabric workload for sub-second queries directly on OneLake data. It also launched GA for OneLake read, public preview for OneLake write, and Bring Your Own Cloud availability in the Azure Marketplace.

ClickHouse is now a native workload in Microsoft Fabric, acting as an acceleration layer for OneLake. It syncs OneLake tables into ClickHouse Cloud without custom ETL pipelines, enabling sub-second analytical queries on billions of rows at high concurrency without consuming Fabric capacity.

The chDB Durable Layer introduces a durability layer around embedded chDB to persist state in object storage. It avoids remote database server latency by keeping analytical queries local while allowing MergeTree state to survive ephemeral hosts like CI environments.

ClickHouse 26.9 introduces key data engineering optimizations: incremental refreshes for append-only materialized views, disk spilling for high-cardinality DISTINCT queries, conditional boundaries for LIMIT, and accelerated min, max, and count queries alongside expanded PromQL support.

ClickHouse costs drop by decoupling compute from storage. Storing a single copy on object storage eliminates duplicate replica data, while localizing merge placements cuts inter-node bandwidth fourfold.

Fountain rebuilt its data plane on ClickHouse Cloud, replacing a BigQuery and S3/Iceberg batch stack. Using ClickPipes CDC to stream Postgres and MongoDB data into 2,000+ materialized views, it cut latency from three hours to under two minutes while reducing costs by 66%.

ClickHouse appointed former Snowflake CFO Mike Scarpelli to its board as run-rate revenue topped $350M. The real-time OLAP database scales analytical ingestion and query workloads across cloud architectures for over 4,000 customers balancing speed and cost.

DatI is a semantic gateway connecting AI agents to enterprise databases like MySQL, PostgreSQL, ClickHouse, and Doris. It enriches semantic metadata with business terms and enum dictionaries, exposing parameterized SQL tools over standard MCP (Streamable HTTP) for natural language querying.

Postgres bottlenecks often stem from storage latency when working sets exceed RAM. Fast NVMe drives resolve I/O limits for transactions, autovacuum, and replication lag. However, hardware cannot bypass row-store architecture limits for analytical queries.

The ClickHouse and PyData Seattle meetup features technical sessions on data engineering architectures for automation. Topics include building "Data Systems for Machine Consumers" with ClickHouse, designing agent-ready data pipelines, and operating continuous feedback loops across pipeline

Modern data architectures power production AI through specialized data layers and observability. Engineers use ClickHouse as a unified backend for tracing agentic multi-step tool calls via Jaeger, enabling granular latency analysis and cost tracking alongside vector search implementations.

The Stockholm ClickHouse Meetup focuses on real-time analytics at scale. Data engineers can explore updates in the ClickHouse ecosystem, hear technical presentations from practitioners in the field, and discuss high-performance analytical architectures and patterns during interactive sessions.

The ClickHouse Singapore Meetup covers production LLM engineering, tracing, and observability. Presentations focus on storing and analyzing telemetry data using ClickHouse and Langfuse to evaluate and debug AI applications at scale, highlighting practical observability frameworks for enterprise

The Dubai ClickHouse Meetup focuses on real-time analytics at scale, gathering data professionals to explore architecture updates and operational patterns in the ClickHouse analytical database. The technical sessions cover data pipelines, high-throughput queries, and distributed analytics.

The ClickHouse South Bay Meetup brings data engineers together to explore analytical database solutions and examine "how folks are using us to solve their big data challenges." Sessions include user presentations and technical discussions highlighting modern high-performance data architectures.

This session covers engineering data platforms for AI workloads at scale. ClickHouse addresses architecture patterns when automated AI agents drive primary analytical query traffic, while White Circle details transitioning past simple Postgres replication into a dedicated ClickHouse serving layer.

Data engineers face a shifting landscape where OLTP and OLAP systems collide. Key topics include using Apache Iceberg as an interoperability layer, scaling CDC pipelines to process three billion monthly updates, and rethinking application reporting architectures beyond standard query tuning.

This meetup brings together data leaders to discuss scaling open-source data infrastructure, focusing on technologies like Kafka, PostgreSQL, ClickHouse, and OpenSearch. Discussions center on simplifying operational management, cloud cost optimization, and architecting modern data stacks for growth.

SereneDB merges Elasticsearch retrieval with ClickHouse analytics into a Postgres-compatible engine. Powered by the C++ IResearch library, it enables zero-ETL querying over Parquet, Iceberg, and S3 data with BM25, vector search, and column-wise updates.

The official ClickHouse provider for Apache Airflow, apache-airflow-providers-clickhousedb, is now available. It uses clickhouse-connect over HTTP(S), integrates with standard common SQL operators, and adds a dedicated hook for bulk queries and client-specific database operations.