dltHub resources for data engineers

dltHub launched job-inspector in public preview to diagnose failed pipeline jobs using read-only access. It parses runs, logs, and definitions to classify root causes, evaluate confidence, and post fix recommendations to Slack, reducing manual triage for data engineers.

dltHub updated its REST API, breaking pipelines by renaming scripts to jobs, runs to job-runs, and returning 404s on legacy trigger endpoints. Inspecting the dlthub-client wheel reveals payload and SDK changes like jobs.getbyref, requiring updates to Snowflake stored procedures.

dltHub launched job-inspector in public preview to diagnose failed pipeline jobs. Running with read-only telemetry access, the background agent analyzes logs and definitions to post the root cause and evidence to Slack. It logs findings to a Context Graph to improve future failure analysis.

The Data Berlin Meetup highlights practical geospatial analytics using SQLite alongside discussions on autonomous data platforms with dltHub. Technical topics center on modern open-source data engineering patterns, including ELT pipelines built with dlt and dbt, plus orchestration via Airflow and

Dropping a table in dlt fails to backfill because cursor timestamps persist in dltpipeline_state. To selectively re-ingest without wiping the dataset, sync remote state and run dlt pipeline drop with the targeted --state-path to reset the incremental cursor for that resource alone.

A Snowflake Cortex Agent inspects failed dltHub ingestion jobs using external network egress rules and read-only Python tool functions. By parsing logs and traces within the Snowflake security perimeter, it classifies pipeline errors and outputs structured JSON for alerting.

dltHub's AI harness builds ingestion pipelines and syncs Iceberg tables, while Snowflake's Cortex Code manages downstream data engineering. Once data lands, CoCo reads schemas and builds models using native tool calls to automate catalog labeling, dynamic tables, and semantic views.

dlt functions as a stateful engine that tracks schemas and incremental watermarks across runs. State lives in both a local cache and the destination warehouse system table dltpipeline_state. CLI inspections show stale data unless reconciled with dlt pipeline sync to fetch the remote truth.

dltHub frames the data platform like a power grid: routine faults cleared automatically within a bounded scope, anything risky escalated to an engineer. A context graph and four-tier risk model let pipeline health agents resolve low-impact bugs autonomously while keeping humans in the loop.

Declarative data pipelines can map nouns to sources and verbs to transformations using xorq expressions. Reusable metrics are registered in a content-addressed, versioned catalog. The setup enables decoupling transformations from execution engines across DataFusion, DuckDB, and Snowflake.

dlt functions as a stateful engine tracking watermarks and schemas locally and remotely. Local CLI runs read from ephemeral cache, causing stale metadata. To reconcile state, run dlt pipeline sync to fetch the destination table dltpipeline_state and overwrite your local JSON cache.

Tycoon-cli is a CLI tool that automates a local analytics pipeline, connecting dlt, DuckDB, dbt, and Rill without Docker or cloud requirements. It supports modular components, letting engineers swap in tools like Snowflake, BigQuery, or Airbyte through an adaptable initialization workflow.

dlt enables custom reverse ETL pipelines using its @dlt.destination decorator. A data pipeline extracts REST API hiring signals into DuckDB, enriches them via People Data Labs, and syncs records directly to the Attio CRM. It gives data engineers code-level control over destination upsert logic.

BigQuery enables native property graphs directly over clean canonical tables using GQL with zero data movement. Ingest data with dlt into canonical views, then use CREATE PROPERTY GRAPH to replace complex recursive SQL joins with simple multi-hop MATCH queries.

dltHub job decorators miss platform crashes that fail before Python executes. To fix this blind spot, a local Python watchdog queries the dltHub CLI for recent runs, triggering a single retry upon failure and alerting via Teams only if the retry also fails.

To document raw tables across 70 dlt pipelines in Snowflake, an agent inspects pipeline sources and live INFORMATION_SCHEMA schemas to generate metadata JSON files. A deterministic runtime function reads the JSON to execute COMMENT ON TABLE and COMMENT ON COLUMN statements after each load.

While AI agents can write pipelines fast, data engineering remains bottle-necked by business logic decisions. Data stacks require accountability to explain metrics and align with operational reality, as models lack the long-term context needed to maintain defensible data transformations.

Migrating HubSpot pipelines from Fivetran to dlt requires reverse-engineering vendor schemas and mapping APIs. Declarative REST sources handle pagination, while custom @dlt.resource functions manage batch associations and fan-outs, with SQL transformations reproducing downstream mart models.

Small companies do not need Big Data to benefit from a data platform. Centralizing sources using open tools like Dagster, dltHub, dbt, and self-hosted ClickHouse on a $20 VPS reveals critical operational cross-source insights.

Modern orchestrators unify disjointed data stacks into a cohesive control plane. As data platforms grow complex, an orchestrator serves as "the heart of any data engineering project" to manage ingestion, transformation, and lineage.

Enterprise data pipelines combine Kestra, dlt, and Snowflake to separate orchestration from business logic. Using Kestra's SyncNamespaceFiles plugin, engineers pull code bases from GitHub into namespace files. The project demonstrates a real-world data pipeline using Kestra as the orchestrator.

DuckDB handles analytics through embedded query capabilities and virtualized data access. Teams use it for local-first pipelines, processing trillions of records, and replacing heavy Spark clusters. FinQore reduced pipeline times from 8hr to 8min while Okta queried massive datasets efficiently.

dltHub provides managed infrastructure and an AI Harness to make AI-built data pipelines production-ready. Agents handle schema evolution, pagination, and incremental loading. Writing the code was never the hard part. Everything around it is.

A live demonstration shows building an end-to-end data pipeline using a single unified schema. An AI agent uses dltHub to generate and deploy ingestion pipelines from a REST API into MotherDuck serverless warehouse.

To send dltHub failure alerts to Microsoft Teams instead of Slack or email, custom Power Automate webhooks can be integrated using Adaptive Cards. By shadowing the job decorator, alerts scale across pipelines.

dltHub transforms data ingestion into validated, testable code running on dlt. It enables data teams to land pipeline outputs in Snowflake alongside the lineage, automated tests, and audit evidence required by regulators in life sciences.

dlthub offers two methods for running data transformations: native dlthub transformations and custom dbt projects. The post details how these options differ for data engineers and decision makers.

Using shared Python modules in a dlt workspace reduces code duplication across pipelines. Pulling reflection logic, staging, and execution into helper files allows pipelines to act as pure configuration. Pipeline files are treated as opaque Python scripts.

dlt is an open-source Python library designed to build, deploy, monitor, and scale data pipelines. It extracts data from REST APIs into standardized formats, supporting agentic data consulting and forward-deployed engineering patterns for modern data architectures.

To load LinkedIn Ads data using 3-legged OAuth in ephemeral containers, store the long-lived refresh token directly in Snowflake. As long as the refresh token is alive, no human is needed. The scheduled dlt pipeline reads the token, mints a fresh access token, and updates Snowflake upon rotation.