Ingestion resources for data engineers

dltHub updated its REST API, breaking pipelines by renaming scripts to jobs, runs to job-runs, and returning 404s on legacy trigger endpoints. Inspecting the dlthub-client wheel reveals payload and SDK changes like jobs.getbyref, requiring updates to Snowflake stored procedures.

To build reliable healthcare AI, data engineers pair automated ingestion via Fivetran with governed transformations and lineage using dbt Labs. Integrated with Snowflake, Tuva Health, and Omni, this architecture establishes shared data definitions and strict governance for conversational analytics.

OrchardFlow provides an observable, deterministic orchestration layer for cloud-native ELT pipelines. It models data ingestion, transformation, and delivery as graftable branches with predictable yields, documented failure modes, and clear lineage to eliminate brittle, chained commands.

Snowflake released zero-copy integration for Workday Data Cloud into general availability. This feature allows direct data access without manual ETL or data duplication pipelines. Snowflake also added lineage preservation across temporary tables and views.

ClickHouse integrates with Postgres to unite OLTP and OLAP workloads. Tools include pgclickhouse to query ClickHouse via native Postgres SQL with query pushdowns, and chdbhook to import and export data lake files through Postgres COPY commands across cloud object storage.

Databricks Serverless Micro Apps enforce zero external egress, restricting compute to internal workspace tools like Unity Catalog and Lakebase. While blocking external scraping, this sandbox excels at internal tooling by scaling compute to zero and cutting idle costs despite a 30-second cold start.

Databricks Lakeflow complements Fivetran and dbt within a unified architecture. Fivetran automates SaaS and database ingestion into open formats like Delta Lake, Lakeflow processes Kafka event streams with Spark Declarative Pipelines, and dbt models SQL business logic with built-in testing.

Production DuckLake architectures require deliberate maintenance. Because the catalog tracks data files and inlines inserts under 10 rows in a database, engineers must periodically flush inlined data to Parquet and run file compaction to prevent performance bottlenecks.

To replace slow MongoDB analytics, DuckDB was deployed on the same host to handle sub-second queries. After testing JDBC batching and the community Mongo extension, custom Java table functions were chosen for ingestion to avoid C++ maintenance while streaming data without intermediate disk files.

Engineers ported Doom directly into SQL, running the game loop and renderer inside a database. Game state is modeled relationally from WAD data, executing logic via in-database procedures at 35 Hz. A separate query pipeline renders the frame buffer directly from state tables.

Hands-on workshops detail how to wire up data pipelines and run agents in production using DuckDB and MotherDuck. Engineers build conversational architectures to profile raw files, generate schemas, execute transformations, run data quality checks, and host MCP servers for automated workloads.

Sqmeow.nvim integrates database management into Neovim, letting engineers browse schemas, execute queries, and stage edits directly in the editor. Powered by a Rust engine that pages large results, it supports PostgreSQL, DuckDB, ClickHouse, and exports datasets to CSV, JSON, or SQL INSERTs.

Even with S3 read-after-write consistency, Spark can silently drop CSV rows during in-place overwrites. Spark resolves file lengths at planning time; if an overwrite increases file size before tasks execute, Spark reads only up to the initial planned byte boundary.

redb.Route is a .NET integration framework acting like Apache Camel. It uses a C# DSL to build From-Process-To pipelines connecting Kafka, RabbitMQ, Redis, SQL, and S3 without message-bus lock-in. It supports 29 external transports, hot-reloading, and token streaming.

The Data Berlin Meetup highlights practical geospatial analytics using SQLite alongside discussions on autonomous data platforms with dltHub. Technical topics center on modern open-source data engineering patterns, including ELT pipelines built with dlt and dbt, plus orchestration via Airflow and

Embrasure Flow is a Rust service that streams PostgreSQL changes into Apache Iceberg. It captures initial COPY and logical replication, writing Parquet data with v2 position deletes or v3 deletion vectors to S3 storage via an Iceberg REST catalog, while handling local compaction.

Dbc 0.3.1 updates the CLI tool for managing ADBC drivers, fixing Windows output and unsupported lockfiles. Recent driver additions include Apache Spark, DataFusion, and Presto, streamlining Arrow-native database connectivity for data engineers. The tool is also now available via npm.

Snowflake has reached general availability for the migration of gen 1 Openflow deployments and runtimes to gen 2. This milestone follows the retirement of gen 1 deployment creation and expands data pipeline infrastructure alongside recent additions like Iceberg tables and Snowpipe Streaming.

Strata runs locally via Docker to deliver a semantic layer over existing warehouses. Engineers connect sources like Snowflake or ClickHouse, while AI coding agents generate YAML semantic models using Git, enforcing join rules and grain-safe blending. Models are deployed via the Strata CLI.

Updates to pgclickhouse and chdb improve Postgres-ClickHouse integration with a new checkencoding option. It handles invalid byte sequences across foreign tables and chdb queries using fail, remove, replace, or truncate handlers.

Building a Postgres CDC connector relies on reading the Write-Ahead Log via logical decoding and replication slots. If a consumer halts and fails to acknowledge its slot, Postgres retains unbounded WAL files, filling disk space and halting writes across the production database.

The edition highlights Kafka and stream processing developments, focusing on broker architectures and Flink SQL patterns. Notable releases include Apache Fluss v1.0 and Khaos for failure testing, while new alternatives like blob-stream target cost over latency.

Dropping a table in dlt fails to backfill because cursor timestamps persist in dltpipeline_state. To selectively re-ingest without wiping the dataset, sync remote state and run dlt pipeline drop with the targeted --state-path to reset the incremental cursor for that resource alone.

TypeSafe AI's Jev outputs typed decisions as JSON. Representing these outputs via Apache Arrow schemas enables zero-copy ingestion into engines like DuckDB and Polars, avoiding serialization overhead. However, analytical batch evaluation remains bottlenecked by the API's single-state request design.

Fivetran leads Snowflake ingestion tools at 12.7% of surveyed companies, followed by Azure Data Factory at 7.9% and Informatica at 6.5%. Most teams bypass standard ETL vendors, relying instead on native ingestion like COPY INTO, Snowpipe, or custom code.

Event-driven S3 to BigQuery pipelines break when spreadsheets shift schemas. Standard Airflow setups rely on human alerts because retries cannot fix structural file changes.

🎙️ Orchestra — Build a self-healing ELT pipeline using Orchestra, S3, and Claude. When runs fail, the agent accesses a production S3 staging layer to reformat and reschematize the bad data instead of opening PRs, then triggers an automated pipeline rerun without requiring code changes.

ClickHouse is now a native workload in Microsoft Fabric, acting as an acceleration layer for OneLake. It syncs OneLake tables into ClickHouse Cloud without custom ETL pipelines, enabling sub-second analytical queries on billions of rows at high concurrency without consuming Fabric capacity.

BigQuery continuous queries now support writing output directly into Apache Iceberg managed tables using INSERT DML statements, enabling real-time streaming into open-format lakehouses. Additionally, the BigQuery Data Transfer Service MCP server is generally available.

Databricks introduces a FILE data type to integrate unstructured multimodal data like audio and video directly into the Lakehouse. For each row, the FILE type stores metadata and a governed link to the file in storage.