Transactional resources for data engineers

To migrate 2.55 billion rows from Aurora MySQL 8.0 to Outposts RDS MySQL 8.4, engineers bypassed complex Glue ETL by using MySQL Shell streaming utilities. Querying metadata excluded 63 archive tables, allowing a 105 GB zstd-compressed export to complete in under 27 minutes.

ClickHouse Managed Postgres expanded alerts to target CDC pipeline issues, including inactive logical replication slots retaining unconfirmed WAL. It warns when inactive slots hold at least 10 GiB of WAL over two hours, preventing unbounded disk growth from detached downstream consumers.

Shipsidekick migrated from Neon to ClickHouse Managed Postgres, using ClickPipes to replicate 500 GB in hours. The move dropped CPU utilization from 100% to under 10%, increased the cache hit rate to 99%, and delivered 10x faster query execution across their transaction and analytics workloads.

duckdb-differential is a C++ extension adding versioning to DuckDB 2.0 databases via a differential filesystem. It enables snapshots, time travel, branching, and S3 replication while sharing unchanged data blocks across versions without altering native catalogs, SQL, or transactions.

Delta Lake serves as a standard for ACID lakehouse storage, but coordinating diverse ecosystem tools presents challenges. Delta Kernel addresses this integration layer, enabling developers to interact directly with Delta tables using Rust for custom data engineering tasks.

Query Insights for ClickHouse Managed Postgres now tracks wait event metrics. Engineers can monitor average active sessions grouped by wait types like IO and Lock to pinpoint contention bottlenecks alongside CPU, read, and write usage.

Snowflake announced the general availability of data mirroring for Snowflake Postgres. This release expands database replication tooling, alongside the v5.0.0 Python Connector. Recent platform additions also include Elastic Channels for Snowpipe Streaming and partitioned Apache Iceberg tables.

Offloading data from OLTP databases like Postgres to ClickHouse yields order-of-magnitude cost savings as data scales. Scaling transactional engines leads to linear infrastructure cost growth, but moving non-transactional workloads to an OLAP engine avoids paying high ACID compliance overhead.

Snowflake announced general availability for Postgres data mirroring. Using the open-source pg_lake extension and logical replication, it streams changes directly into Snowflake or Apache Iceberg tables without external CDC pipelines, providing transactional consistency and a live view.

Recent data engineering updates highlight zero-downtime Postgres upgrades via logical replication, high-speed Python JSON parsing at tens of gigabytes per second, and SQLBuild for safe warehouse SQL refactoring. Teams also optimize star history queries and tap opportunistic AI inference.

ClickHouse integrates Postgres for OLTP and ClickHouse for OLAP into a single managed stack. To accelerate analytics pipelines, it launched WalShadow in private preview and introduced the chdb Postgres extension alongside Managed Postgres on Google Cloud to handle heavy write bursts.

Responsive is joining Databricks to advance Lakebase and online databases built on object storage. Their architecture disaggregates compute and storage via SlateDB, an embedded LSM engine, enabling transactional and analytical engines to share open data lake storage.

Clera migrated its 500 GB production database across 500+ tables overnight to ClickHouse Managed Postgres. Selected for NVMe-backed performance over alternatives in sysbench TPC-C tests, the live cross-region migration slashed database CPU utilization from continuous 100% spikes down to 10-20%.

ClickHouse integrates with Postgres to unite OLTP and OLAP workloads. Tools include pgclickhouse to query ClickHouse via native Postgres SQL with query pushdowns, and chdbhook to import and export data lake files through Postgres COPY commands across cloud object storage.

Databricks Serverless Micro Apps enforce zero external egress, restricting compute to internal workspace tools like Unity Catalog and Lakebase. While blocking external scraping, this sandbox excels at internal tooling by scaling compute to zero and cutting idle costs despite a 30-second cold start.

ClickHouse Managed Postgres uses Direct I/O for backups to bypass the Linux page cache, preventing cache eviction of warm database pages during query execution. By aligning reads across striped NVMe drives, it reduced query latency impact by two-thirds and avoided cache pollution entirely.

Aurora PostgreSQL now embeds DuckDB to directly query Apache Iceberg and Parquet in S3 without ETL pipelines. Using the aurora_analytics extension, it supports foreign tables with predicate pushdown and column pruning to join operational and lake data.

Snowflake released general availability for inline stored procedures on hybrid tables, expanding transactional capabilities. Recent updates also added CHECK constraints for hybrid tables, Snowpipe Streaming support for partitioned Apache Iceberg tables, and dynamic table incremental refresh.

🎙️ Practical Data Modeling — Data modeling is fractured across five isolated camps: relational, analytics, application, ML, and knowledge. Stacking modern workloads requires bridging these silos, as uncoordinated application schemas break downstream pipelines and corrupt analytical models.

PostgreSQL 19 adds built-in online table rewrites via REPACK CONCURRENTLY, utilizing logical decoding and a temporary replication slot instead of triggers. While avoiding downtime, its cluster-wide slot holds back autovacuum dead tuple cleanup across all databases until the final swap commits.

Databricks launched Lakebase, combining serverless Postgres with open lakehouse storage to blur OLTP and OLAP. By enabling instant branching, scale-to-zero compute, and native zero-ETL access for Spark and Delta Lake, it removes brittle CDC pipelines and offers unified governance via Unity Catalog.

Change Data Capture extends beyond updating transactional databases to encompass streaming data, API calls, and write-ahead logs. The discussion explores CDC architecture, use cases, and tools like Estuary built for streaming, updating, and materializing pipeline data efficiently.

Postgres bottlenecks often stem from storage latency when working sets exceed RAM. Fast NVMe drives resolve I/O limits for transactions, autovacuum, and replication lag. However, hardware cannot bypass row-store architecture limits for analytical queries.

Adding an index can slow queries when the Postgres planner misjudges scan costs. A createdat index slowed a filtered query from 16.5ms to 241ms by discarding 495,944 rows. A composite index on (userid, created_at DESC) fixed the plan to execute in 0.067ms.

Postgres hypertable queries stalled due to excessive chunk metadata planning overhead. Cold planners walked 11,520 chunks across 188,000 catalog blocks. Increasing the chunk interval to six hours reduced catalog reads by 99%, proving chunk over-partitioning severely degrades query planning.

TIN is a Postgres full-text search index extension designed for fast query execution and continuous updates. It supports BM25 scoring, boolean and phrase queries, and transaction visibility while handling complex joins and concurrent writes without requiring large memory footprints.

pgrawparse is a Rust library providing direct access to the PostgreSQL query parser. By mapping Rust structs onto PostgreSQL arena allocators rather than using Protobuf serialization, it enables AST parsing, deparsing, and traversal while operating 20-60x faster with 90% less memory.

DuckDB-Wasm now supports persistent storage using the browser's Origin Private File System (OPFS). Using an opfs:// URI pattern creates standard DuckDB and Parquet files with WAL support that persist across reloads, eliminating manual serialization to IndexedDB.

ClickHouse integrates with PostgreSQL via open-source tools like pg_clickhouse, an Apache 2.0 foreign data wrapper. By mapping ClickHouse tables into a Postgres schema, engineers keep Postgres as the operational front door while offloading analytical heavy lifting directly to ClickHouse.

PostgreSQL 19 adds pgplanadvice and pgstashadvice to lock query plans across ANALYZE runs. Advice captures join orders, methods, and scans without costs or timings, surviving stats changes. Tools like sqlfmt can extract this structure from standard EXPLAIN output for deterministic plan diffs.