Lineage resources for data engineers

This project implements an automated ETL pipeline orchestrated with Apache Airflow. Running on Docker Compose, the DAG extracts weather metrics from the OpenWeatherMap REST API, ingests them into a PostgreSQL database, and feeds analytics dashboards built in Metabase.

Duckpad is an open-source data transformation tool and SQL workspace powered by DuckDB. Data engineers can visually compose pipelines, integrate Git versioning, and trace cell-level data lineage back to contributing source values. SQL execution runs locally on modern Linux via Landlock.

Data catalogs organize structural metadata and schemas across warehouses and lakes. Technical catalogs track schemas, access policies, and column-level lineage, enabling tools like Iceberg and Glue to manage state and evaluate the impact of schema changes across pipelines.

dbt Wizard is an analytics engineering AI agent that validates changes in development schemas and visualizes downstream DAG impacts. It separates cheap compilation from expensive materializations, using an isolated sub-agent for validation to prevent inherited assumptions before generating PRs.

dbt v2 converges the ecosystem onto one engine with faster parsing and unified binaries. Built on the ADBC standard, shared adapters eliminate dependency conflicts across warehouses like Snowflake and Databricks. Docs v2 also replaces manifest.json with queryable Parquet files for lineage analysis.

To debug data errors across BI and warehouse layers, an AI agent integrates three MCP servers: Looker, Databricks SQL, and dbt. It inspects compiled SQL and lineage to trace a dashboard metric anomaly down to underlying transformation logic, attaching QA validation queries in minutes.

AlphaSense paired Euno context graphs with dbt to rebuild legacy data models into a governed star schema. Using Cursor, they refactored a 287-field table into 13 tested dbt models in hours instead of weeks, shifting business logic from BI back into the data layer.

PetScreening connected an in-house pet tag printer as a dbt exposure using a medallion architecture. Because bad joins physically waste materials, the team uses strict dbt tests for validation. The same gold-layer model powers BI dashboards and prints over 1,000 tags daily across two machines.

Build audit-ready regulatory pipelines using S3, Glue, and dbt on Athena or Redshift. Store immutable raw files via S3 Object Lock, enforce dbt model contracts, and archive dbt run artifacts alongside report exports to a restricted evidence bucket to ensure reproducibility.

Snowflake previewed Horizon Context, adding metadata connectors across 18 sources to track lineage and reuse business definitions for AI. Column description propagation carries documentation downstream across views when values pass through unchanged, reducing pipeline maintenance.

🎙️ Data Engineering Podcast — Cross-platform metadata architectures are critical to reconcile business logic across databases, pipelines, and BI tools. MetaKarta provides a Semantic Hub offering deep lineage and configuration management across systems to prevent semantic drift in downstream analytics and agentic AI.

Amazon OpenSearch Service traces multi-agent pipeline threats like prompt injection by indexing span data with Bedrock vector embeddings, provenance tags, and PII markers. An index with a kNN vector field enables similarity search to trace lineage across hops from input to tool execution.

Automate slide generation by turning governed Databricks gold tables into editable presentations. Every reported metric traces directly to its underlying query, and queries run securely under the requesting user's identity to ensure data governance.

Clinical AI requires trust architected directly into the data pipeline. Pipelines must prioritize FHIR normalization, lineage tracking, and explicit consent binding. Models also need uncertainty calibration and abstention mechanisms so they avoid surfacing misleading predictions to clinicians.

OLCF is an open-source lakehouse platform creating an Apache Spark Standalone cluster across custom machines, cloud VMs, or Kubernetes. It extends Microsoft Fabric by using olcf.run_remote() to offload notebook compute to owned Spark clusters while directly reading and writing OneLake.

Snowflake announced Horizon Context in preview along with named workload identity management. Recent data engineering updates also include Snowpipe Streaming SDK v1.9.0 and data lineage support for storage integrations, external tables, and Cortex Search services.

To analyze cross-language news coverage without shared identifiers, an auditable Python pipeline transformed Arabic bulletins into location-day records matched against English media queries. Intermediate transformations materialized as plain CSV files to enable deterministic manual auditing.

🎙️ The Data Flowcast by Astronomer — CSET leverages Apache Airflow to orchestrate complex ETL pipelines and streamline large-scale ingestion. The data engineering team employs DAG Factory, dynamic task mapping, and automated DAG generation to simplify adoption, while tracking data lineage to manage dependencies across workflows.

Snowflake expanded data lineage support across key architectural components, adding tracking for storage integrations, external tables, model monitors, and Cortex Search services.

Platform engineering perfected infrastructure like Kafka and Snowflake, but data payloads still break downstream. Supercargo enforces shift-left data contracts and CI/CD gates to inspect source data models before pull requests merge, protecting pipelines from schema drift and unencrypted PII.

AI agents require explicit metadata because, unlike humans, they lack ingrained business knowledge. Using raw schemas yields shallow descriptions, so engineering context pipelines requires feeding lineage, upstream queries, and glossary terms left-to-right across transformations.

AML detection pipelines fail when upstream schema mutations and silent data drift distort customer profiles without triggering pipeline crashes. Ingestion jobs succeed, but unmonitored changes like dropped timezone offsets break rolling time-window joins and corrupt detection logic.

Software-defined assets introduce declarative data engineering to manage tables, models, and dashboards as code. Dagster infers asset graphs from function dependencies rather than manual DAGs, shifting orchestration from isolated tasks directly to data products.

AI data engineering is shifting from collection to downstream processing pipelines. Useful data is the output of continuous transformation, validation, and sanitization rather than raw input.

Snowflake announced general availability for lineage preservation through temporary tables and views. This update maintains end-to-end data lineage across intermediate pipeline transformation stages, preventing gaps caused by ephemeral session objects.

Zero data loss requires semantic integrity, not just matching row counts. Schema migrations need layered validation, mapping artifacts, and relationship checks. Pipeline success requires auditability: field-level reconciliation, foreign-key tracing, and tracking orphaned records.

Data pipelines deriving metrics from single raster sources introduce predictable errors. Processing chains that bin modeled values must expose provenance and methodology alongside estimates so downstream users do not treat synthetic outputs as verified facts.

At extreme scale, teams avoid managed warehouses and build on primitives like S3, Kubernetes, and open engines such as Spark and Trino. OpenAI even rewrote Airflow's scheduler and executor in Rust to handle 70,000+ concurrent tasks, relying on exact DAG state metadata to safely drive AI agents.

Data platforms split into physical infrastructure and derived assets like metrics, features, and AI models. Disconnected tools break end-to-end lineage across catalogs, semantic layers, and models.

AI agent governance requires runtime technical enforcement over static approvals. Teams use Policy Enforcement Points to intercept tool calls and Policy Decision Points to evaluate context, returning decisions like allow, mask, or require approval before execution.