Change capture resources for data engineers

apisip is a Rust CLI that ingests OpenAPI list endpoints directly into JSONL or Parquet. It automates pagination, rate limits, and incremental loads. Instead of runtime inference, it enforces schema types from the spec, safely casting lossless values and sending unparseable data to quarantine.

ClickHouse Managed Postgres expanded alerts to target CDC pipeline issues, including inactive logical replication slots retaining unconfirmed WAL. It warns when inactive slots hold at least 10 GiB of WAL over two hours, preventing unbounded disk growth from detached downstream consumers.

Standard dbt snapshots fail on out-of-order batches, corrupting SCD Type 2 history. The open-source dbtiterativesnapshot materialization for Databricks repairs this by ordering batches via sequence_by and automatically replaying timeline updates when late data lands.

Snowflake announced the general availability of data mirroring for Snowflake Postgres. This release expands database replication tooling, alongside the v5.0.0 Python Connector. Recent platform additions also include Elastic Channels for Snowpipe Streaming and partitioned Apache Iceberg tables.

Snowflake announced general availability for Postgres data mirroring. Using the open-source pg_lake extension and logical replication, it streams changes directly into Snowflake or Apache Iceberg tables without external CDC pipelines, providing transactional consistency and a live view.

Replicating PostgreSQL to Snowflake via incremental queries misses deletes and adds polling overhead. Using CDC with PostgreSQL logical replication streams WAL changes directly. Setting wal_level to logical enables row-level change capture into Estuary, which materializes data directly to Snowflake.

In Databricks, an insert-only MERGE INTO uses a Left Anti Join with the small source on the left, preventing Spark from broadcasting it. Replacing it with an INSERT INTO using an Inner Join and a right-broadcasted Left Anti Join optimizes performance and avoids shuffling large target tables.

ClickHouse integrates Postgres for OLTP and ClickHouse for OLAP into a single managed stack. To accelerate analytics pipelines, it launched WalShadow in private preview and introduced the chdb Postgres extension alongside Managed Postgres on Google Cloud to handle heavy write bursts.

BigQuery Data Transfer Service now generally supports incremental data ingestion from PostgreSQL and ServiceNow. Data engineers can automate continuous pipeline updates without running full table reloads, optimizing ingestion workloads.

Apache Flink CDC 3.7.0 adds AI-powered stream processing with OpenAI-compatible model transforms and ordered async execution. It adds inline Python UDFs, SQL Server and Fluss pipeline sources, plus schema expansion for Paimon and Fluss sinks.

Hands-on workshops detail how to wire up data pipelines and run agents in production using DuckDB and MotherDuck. Engineers build conversational architectures to profile raw files, generate schemas, execute transformations, run data quality checks, and host MCP servers for automated workloads.

Snowflake dynamic tables now support incremental refresh after failover, maintaining continuous data pipeline processing without requiring full refreshes. The release also adds general availability for Optimized Refresh and RPO Assurance for failover groups to improve disaster recovery.

Embrasure Flow is a Rust service that streams PostgreSQL changes into Apache Iceberg. It captures initial COPY and logical replication, writing Parquet data with v2 position deletes or v3 deletion vectors to S3 storage via an Iceberg REST catalog, while handling local compaction.

Building a Postgres CDC connector relies on reading the Write-Ahead Log via logical decoding and replication slots. If a consumer halts and fails to acknowledge its slot, Postgres retains unbounded WAL files, filling disk space and halting writes across the production database.

Dropping a table in dlt fails to backfill because cursor timestamps persist in dltpipeline_state. To selectively re-ingest without wiping the dataset, sync remote state and run dlt pipeline drop with the targeted --state-path to reset the incremental cursor for that resource alone.

Aurora PostgreSQL zero-ETL integration with Amazon SageMaker uses enhanced logical replication CDC to stream transactional changes into an AWS Glue catalog as Apache Iceberg tables. This enables near real-time SQL, Spark, and ML queries without complex pipelines.

PostgreSQL 19 adds built-in online table rewrites via REPACK CONCURRENTLY, utilizing logical decoding and a temporary replication slot instead of triggers. While avoiding downtime, its cluster-wide slot holds back autovacuum dead tuple cleanup across all databases until the final swap commits.

Real-time ETL spans batch, micro-batch, log-based CDC, and event streaming via tools like Kafka. Selection hinges on required latency rather than raw speed: log-based CDC efficiently reads database transaction logs, while true streaming continuously processes events for sub-second workloads.

Fountain rebuilt its data plane on ClickHouse Cloud, replacing a BigQuery and S3/Iceberg batch stack. Using ClickPipes CDC to stream Postgres and MongoDB data into 2,000+ materialized views, it cut latency from three hours to under two minutes while reducing costs by 66%.

Change Data Capture extends beyond updating transactional databases to encompass streaming data, API calls, and write-ahead logs. The discussion explores CDC architecture, use cases, and tools like Estuary built for streaming, updating, and materializing pipeline data efficiently.

Choosing a data migration tool depends on recovery semantics and whether you need dump-and-restore, batch cursors, or log-based CDC. CLI tools like ingestr offer scriptable batch and CDC runs without servers, while managed platforms suit broader connector needs. State durability is critical.

Evaluate ingestion tools by failure modes, not connector counts. Use CDC for large mutable tables needing delete capture, and incremental batching for append-only data. Destination write semantics, rate limits, schema evolution, and merge keys dictate pipeline stability.

Offline-first sync mirrors change-data-capture challenges. Reserving sequence numbers before committing rows creates race windows where cursors advance past uncommitted data. Because cursor increments assume older data is processed, missed rows are skipped forever during incremental sync.

DuckDB v2.0-alpha introduces client/server mode via Quack, async S3 I/O, and an efficient VARIANT type. Additionally, the MSSQL extension v0.2.3 optimizes bulk loading via parallel writers and schema tuning, cutting load times for 38M rows from 933s to 96s.

Replicate DynamoDB to Apache Iceberg on Amazon S3 Tables via Glue zero-ETL without impacting operational traffic. An AWS Lambda function embedding DuckDB executes fast vectorized SQL directly against S3 Tables metadata and Parquet files, returning JSON query results serverlessly.

Snowflake introduced data lineage for streams, enhancing visibility into CDC pipelines. This update helps data engineers track pipeline dependencies and schema transformations across change tracking objects directly in the data platform.

Bruin orchestrates near-real-time fraud monitoring using an append-only ClickHouse MergeTree log fed by PostgreSQL change capture. It aggregates events into minute, hour, and daily rollups, applying interval modifiers to cleanly restate late-arriving payment statuses without full recomputations.

WalShadow enables sub-second Postgres replication to ClickHouse by directly consuming physical WAL streams instead of using logical replication slots. It decodes WAL off-host and writes native blocks to ClickHouse, hitting 200 ms latency and 289K rows/sec in benchmarks.

Selecting CDC tools requires evaluating capture methods, schema evolution, and recovery. Log-based CDC reads transaction logs like WAL or binlog, reducing source database load and latency compared to polling tables.

Filament provides pluggable data replication across sources and sinks via full, incremental, or CDC pipelines. It features bounded batching, verification on both sides of a write, and durable checkpoints using PostgreSQL and NATS JetStream. It can run embedded in Go or deploy to Kubernetes via Helm.