Change capture resources for data engineers

Hands-on workshops detail how to wire up data pipelines and run agents in production using DuckDB and MotherDuck. Engineers build conversational architectures to profile raw files, generate schemas, execute transformations, run data quality checks, and host MCP servers for automated workloads.

Snowflake dynamic tables now support incremental refresh after failover, maintaining continuous data pipeline processing without requiring full refreshes. The release also adds general availability for Optimized Refresh and RPO Assurance for failover groups to improve disaster recovery.

Embrasure Flow is a Rust service that streams PostgreSQL changes into Apache Iceberg. It captures initial COPY and logical replication, writing Parquet data with v2 position deletes or v3 deletion vectors to S3 storage via an Iceberg REST catalog, while handling local compaction.

Building a Postgres CDC connector relies on reading the Write-Ahead Log via logical decoding and replication slots. If a consumer halts and fails to acknowledge its slot, Postgres retains unbounded WAL files, filling disk space and halting writes across the production database.

Dropping a table in dlt fails to backfill because cursor timestamps persist in dltpipeline_state. To selectively re-ingest without wiping the dataset, sync remote state and run dlt pipeline drop with the targeted --state-path to reset the incremental cursor for that resource alone.

PostgreSQL 19 adds built-in online table rewrites via REPACK CONCURRENTLY, utilizing logical decoding and a temporary replication slot instead of triggers. While avoiding downtime, its cluster-wide slot holds back autovacuum dead tuple cleanup across all databases until the final swap commits.

Real-time ETL spans batch, micro-batch, log-based CDC, and event streaming via tools like Kafka. Selection hinges on required latency rather than raw speed: log-based CDC efficiently reads database transaction logs, while true streaming continuously processes events for sub-second workloads.

Fountain rebuilt its data plane on ClickHouse Cloud, replacing a BigQuery and S3/Iceberg batch stack. Using ClickPipes CDC to stream Postgres and MongoDB data into 2,000+ materialized views, it cut latency from three hours to under two minutes while reducing costs by 66%.

Change Data Capture extends beyond updating transactional databases to encompass streaming data, API calls, and write-ahead logs. The discussion explores CDC architecture, use cases, and tools like Estuary built for streaming, updating, and materializing pipeline data efficiently.

Choosing a data migration tool depends on recovery semantics and whether you need dump-and-restore, batch cursors, or log-based CDC. CLI tools like ingestr offer scriptable batch and CDC runs without servers, while managed platforms suit broader connector needs. State durability is critical.

Evaluate ingestion tools by failure modes, not connector counts. Use CDC for large mutable tables needing delete capture, and incremental batching for append-only data. Destination write semantics, rate limits, schema evolution, and merge keys dictate pipeline stability.

Real-time data engineering patterns center on streaming architectures and change data capture. Delivery Hero routes Postgres CDC events into Kafka to feed its live ML feature store, while Zalando abstracts Kafka via Nakadi to swap topics seamlessly.

Offline-first sync mirrors change-data-capture challenges. Reserving sequence numbers before committing rows creates race windows where cursors advance past uncommitted data. Because cursor increments assume older data is processed, missed rows are skipped forever during incremental sync.

DuckDB v2.0-alpha introduces client/server mode via Quack, async S3 I/O, and an efficient VARIANT type. Additionally, the MSSQL extension v0.2.3 optimizes bulk loading via parallel writers and schema tuning, cutting load times for 38M rows from 933s to 96s.

Replicate DynamoDB to Apache Iceberg on Amazon S3 Tables via Glue zero-ETL without impacting operational traffic. An AWS Lambda function embedding DuckDB executes fast vectorized SQL directly against S3 Tables metadata and Parquet files, returning JSON query results serverlessly.

Snowflake introduced data lineage for streams, enhancing visibility into CDC pipelines. This update helps data engineers track pipeline dependencies and schema transformations across change tracking objects directly in the data platform.

Bruin orchestrates near-real-time fraud monitoring using an append-only ClickHouse MergeTree log fed by PostgreSQL change capture. It aggregates events into minute, hour, and daily rollups, applying interval modifiers to cleanly restate late-arriving payment statuses without full recomputations.

WalShadow enables sub-second Postgres replication to ClickHouse by directly consuming physical WAL streams instead of using logical replication slots. It decodes WAL off-host and writes native blocks to ClickHouse, hitting 200 ms latency and 289K rows/sec in benchmarks.

Selecting CDC tools requires evaluating capture methods, schema evolution, and recovery. Log-based CDC reads transaction logs like WAL or binlog, reducing source database load and latency compared to polling tables.

Filament provides pluggable data replication across sources and sinks via full, incremental, or CDC pipelines. It features bounded batching, verification on both sides of a write, and durable checkpoints using PostgreSQL and NATS JetStream. It can run embedded in Go or deploy to Kubernetes via Helm.

ClickHouse Managed Postgres is now in Private Preview on GCP. The service pairs an NVMe-backed OLTP engine with ClickHouse OLAP, featuring native CDC replication and a unified query layer via pg_clickhouse to prevent cross-cloud egress and simplify pipelines.

Databricks Lakebase integrates managed, serverless Postgres to bridge OLAP warehouses and OLTP applications. Built on Neon technology, it separates compute from storage for autoscaling and removes custom CDC pipelines by syncing operational data directly with Delta tables.

ClickHouse Managed Postgres pairs NVMe-backed Postgres for OLTP with ClickHouse for real-time analytics. Its onboarding streamlines loading data, configuring CDC ClickPipes for managed replication, and querying synchronized tables across both engines.

QuintoAndar unified its CDP on ClickHouse Cloud to process 900M monthly events and serve 300M API requests. By replacing a Lambda architecture and using ClickHouse Managed Postgres as an interface, they made real-time analytical data accessible to tools requiring Postgres compatibility.

DuckDB v2.0 introduces four JSON functions to optimize state reconciliation and CDC pipelines. Tools like jsonmergepatchdiff and jsondeep_merge compute minimal RFC 7396 diffs and merge payloads in SQL without external scripts.

CDC does not eliminate dual-writes; it merely shifts them downstream. Updating targets and committing pipeline offsets are independent, non-transactional acts. Without idempotent receivers, dual-write failure modes persist.

Snowflake announced the general availability of the Openflow Connector for SQL Server (CDC). This update enables automated change data capture pipelines from SQL Server into Snowflake for real-time data engineering workflows.

Healthcare AI requires real-time, complete data. Legacy ETL delays AI adoption with rigid pipelines. Fivetran enables modern ELT by providing automated schema handling and continuous change data capture. Fivetran pipelines are configured in minutes and run themselves.

Data replication tools automate continuous data movement from sources like SQL databases or SaaS apps to target warehouses. Methods include CDC-based transactional replication and low-latency streaming.