Migration resources for data engineers

Skipping data engineering for AI is a disaster. Pointing LLMs at messy source tables fails; solid modeling remains essential. Meanwhile, AI tools like Cortex Code accelerate SQL Server to Snowflake migrations, converting code to dbt projects or stored procedures in hours rather than weeks.

🎙️ The Data Flowcast by Astronomer — HCL Tech relies on Apache Airflow for cloud data migrations, managing on-prem extraction, watermarking, and loading into Snowflake and Databricks. The team generates dynamic DAGs from JSON configs for micro-batching and uses blue-green deployment patterns across environments.

DuckDB eliminates big data operational complexity by querying S3, Parquet, and Iceberg via standard SQL without JVM clusters. DuckLake optimizes catalogs by storing metadata in a SQL database rather than nested files, cutting planning latency to under a millisecond with minimal configuration.

Waiting for a fully modernized data platform delays AI value. Teams should run AI delivery and infrastructure modernization in parallel, using a unified policy layer to govern data across legacy systems in place while migration pipelines continue independently.

To migrate 2.55 billion rows from Aurora MySQL 8.0 to Outposts RDS MySQL 8.4, engineers bypassed complex Glue ETL by using MySQL Shell streaming utilities. Querying metadata excluded 63 archive tables, allowing a 105 GB zstd-compressed export to complete in under 27 minutes.

Frequent Zapier polling keeps Snowflake warehouses running continuously, billing 60-second minimums for millisecond lookups. Offloading these queries to a lightweight engine like DuckDB cut compute costs by 93% by routing repetitive row checks away from analytical compute.

Infrastructure automation often fails across tool handoffs during Day 2 operations like patching. While Terraform or Ansible automate isolated tasks, cross-tool orchestration is missing. Providing a unified control plane coordinates execution, passes outputs, and creates a single audit log.

Twilio replaced Kafka, Spark Streaming, and Elasticsearch with a consolidated ClickHouse pipeline managed by dbt. Using the dbt-clickhouse adapter, materialized views handle streaming transformations, while GitOps and Argo Workflows enable automated testing and multi-region deployment.

Ecolab used Datafold's migration agent to replace fragile Snowflake stored procedures and tasks with over 40 modular dbt Mesh projects. Automated data diffing verified parity against production data, cutting code 6x and data connections 4x across the pipeline.

Sweetwater scaled its governed dbt platform by integrating Datadog for proactive pipeline observability. The data team monitors Kafka inserts, warehouse compute, and table freshness to catch pipeline issues like mis-scheduled legacy scripts before reporting breaks.

Guitar Center rebuilt its data stack using Snowflake and managed dbt Core, replacing legacy stored procedures with CI/CD and automated tests. Scoped builds and smaller warehouses cut Snowflake compute costs by 40% and Airflow usage by 50%, speeding up pipeline delivery from months to days.

Lake Compute lets data teams run low-complexity dbt models on cheaper lake engines instead of expensive data warehouses. Built on Apache Iceberg and open REST catalogs, this architecture cuts compute costs model by model without disrupting downstream DAGs or BI tools.

Data consultancy Modeo has partnered with Kestra, adopting it over Airflow and Dagster to orchestrate end-to-end data pipelines. Modeo deploys Git-versioned YAML flows for ingestion and dbt SQL transformations, citing that Kestra is simpler to operate, maintain, and scale in production.

Delivery Hero scaled dbt across 45 domain teams processing 60 petabytes. Using Astronomer Cosmos and custom CI scripts, the platform converts dbt manifests into Airflow DAGs with priority weights, enforcing model ownership tags and central policy macros directly in CI pipelines.

dbt v2 converges the ecosystem onto one engine with faster parsing and unified binaries. Built on the ADBC standard, shared adapters eliminate dependency conflicts across warehouses like Snowflake and Databricks. Docs v2 also replaces manifest.json with queryable Parquet files for lineage analysis.

Apache Airflow CI moved canary builds and committer PRs to ephemeral AWS self-hosted runners using RunsOn, freeing shared ASF GitHub runners. GitHub Actions still orchestrates runs. RunsOn bypassed the overhead of managing an EKS cluster while keeping CI setups consistent.

dbt demonstrated that pre-compressing Gong transcripts with warehouse-native AI functions and serving them via dbt MCP cut cost 48.5% and run time 43.2%. Moving context engineering upstream into SQL pipelines avoids vendor API limits and eliminates duplicate LLM token consumption.

Netflix built Pensive AI to triage failed data workflows across engines like Spark and Trino. By returning only log line numbers, the model extracts root causes without expensive token generation. Diagnostic runs are cached per failed step attempt to slash duplicate inference costs and latency.

Obie Insurance rebuilt its data stack into a monorepo dbt project using Apache Iceberg lakehouse architecture and Snowflake catalog integration. Utilizing dbt State for state-aware orchestration and model reuse drove 40% cost savings and reduced month-end close from 25 hours to 2 hours.

Verisk migrated billions of rows of legacy mainframe data to Snowflake and dbt, replacing brittle notebooks and uncoordinated tasks with modular staging, intermediate, and mart layers. Automated tests, macros, and versioned lineage now ensure pipeline integrity and support AI workflows.

Virgin Media O2 reduced BigQuery compute costs by roughly 30% and build times by 35% using dbt v2 and dbt State. By enabling state-aware builds, their pipeline skipped unchanged models, reducing slot consumption from 4,500 to 3,400 per hour on a massive 1,400-model DAG.

Data platform engineers replaced costly Looker instances with an internal service hosting LLM-generated HTML in S3 on Kubernetes. The architecture queries Snowflake directly via user OAuth tokens, delegating data governance and access controls to existing Snowflake roles and upstream dbt models.

Shipsidekick migrated from Neon to ClickHouse Managed Postgres, using ClickPipes to replicate 500 GB in hours. The move dropped CPU utilization from 100% to under 10%, increased the cache hit rate to 99%, and delivered 10x faster query execution across their transaction and analytics workloads.

Public sector AI readiness requires modernizing legacy architectures into unified, governed data platforms. Multnomah County migrated a legacy SQL Server footprint in five months using automated code conversion, dynamic SQL translation, and test generation to eliminate cross-agency data silos.

Migrate from VMware vRA to Kestra without a complete rewrite. Kestra's Migration Assistant parses vRA/vRO exports to build dependency inventories and migration waves. It replaces boilerplate JavaScript and retry handling with declarative tasks and plugins while keeping your existing execution tools.

Snowflake AIM is a desktop app automating end-to-end migrations to Snowflake. It pairs deterministic SQL conversion with an agentic test-fix loop to validate DDL, procedures, and ETL pipelines like SSIS or Informatica across dependency-aware waves.

OpenReplay scales front-end session replay by using ClickHouse for real-time analytics. Its largest deployments ingest billions of events and ~6 TB daily on a single vertically scaled instance with sub-second queries, leveraging built-in TTLs and materialized views to streamline data pipelines.

ClickHouse released an experimental .NET native TCP client using native columnar format. Built with zero-allocation and zero-copy designs, it is 1.5x-2x faster than HTTP, streaming up to 1.1M rows/s and cutting GC allocations by over 99%.

Spotify uses generative LLMs for podcast recommendations via beam search over semantic IDs. To scale real-time inference and handle latency budgets, they project user embeddings into prompt soft tokens and use SGLang candidate masking to score multiple items per request.

🎙️ vutr — Open table formats like Apache Iceberg, Delta Lake, and Apache Hudi power modern lakehouse architectures. Tracing a sales table from warehouses through data lakes and Apache Hive highlights what each storage layer offered and cost.