Migration resources for data engineers

Data platforms outlive vendors when engineers isolate business logic from engines. Rent commoditized storage, compute, and runtimes, but own your pipeline code, metric definitions, and data contracts.

Lift-and-shift rewrites to Databricks spike DBU costs by turning unoptimized queries into metered waste. Avoid line-by-line translation. Instead, audit portfolios, kill zombie jobs, and consolidate redundant filter variants into unified Gold tables to migrate business capabilities, not technical

me&u migrated four BI tools to Omni to unify their regionalized Snowflake data stack. By implementing YAML-based semantic modeling in Git alongside dbt, the team established governed joins and definitions that support direct warehouse querying, self-service dashboards, and Claude via MCP.

Migrating unstructured support attachments between CRMs requires building an explicit identity manifest rather than relying on destination APIs. Resolving relational lineage early maps physical files to target records and ensures cross-system traceability across platforms.

PostgreSQL 19 brings major pipeline updates: JIT is now disabled by default for queries, and default TOAST compression shifts to lz4. SQL gains temporal table updates via FOR PORTION OF to automatically split ranges, plus INSERT ON CONFLICT DO SELECT to fetch conflicting rows in one step.

Snowflake released a preview for SAS to Snowflake migration, targeting pipeline conversions. Recent data engineering updates also include external lineage support, dbt project file limit increases, and Apache Iceberg integrations with Snowflake Horizon Catalog and Amazon S3 Tables.

Lyft migrated its streaming fleet from an in-house Kubernetes operator to the open-source Apache Flink Kubernetes Operator to eliminate custom code debt. The shift unlocked native autoscaling and robust reconciliation loops without disrupting existing stateful job workflows.

Teams are migrating production databases from Neon to ClickHouse Managed Postgres to solve reliability and latency issues under load. Using ClickPipes, teams completed migrations in hours, achieving consistent query performance and a unified stack for transactional and analytics workloads.

Suprema Gaming migrated its analytics stack from Snowflake to ClickHouse Cloud to enable real-time agentic workflows. The new OLAP platform slashed warehouse costs by 62%, dropped query latencies from minutes to milliseconds, and boosted data freshness from a four-hour lag to real time.

The Paris ClickHouse Meetup focuses on real-time analytics at scale. Technical sessions highlight Equativ's ClickHouse Cloud migration journey, stream-enrichment pipeline optimization to eliminate redundant SQL workflows, and building self-improving agents with Langfuse for real-time systems.

Handshake migrated its 10-year-old BI estate to Omni in 8 weeks. Using Claude Code and Omni APIs, a single analytics engineer rebuilt 50 data models into Omni Topics and migrated 50+ dashboards, closing the feedback loop with Omni's Content Validator to automate code conversion and validation.

Google Cloud introduces its Lakehouse runtime catalog to help data engineers modernize Apache Hive deployments. This analytics solution enables modernization of legacy Hive architectures within cloud data lakehouse environments.

Rednote migrated its real-time indexing pipeline from Kafka to Apache Fluss to solve read amplification from wide tables. Fluss uses columnar streaming via Apache Arrow and bypasses brokers for cold-data replays, isolating historical scans from real-time traffic to reduce bandwidth and compute

Migrating HubSpot pipelines from Fivetran to dlt requires reverse-engineering vendor schemas and mapping APIs. Declarative REST sources handle pagination, while custom @dlt.resource functions manage batch associations and fan-outs, with SQL transformations reproducing downstream mart models.

Vercel migrated their build warm pool state from Redis to DynamoDB to move billing mappings and container statuses to durable storage. Lose a mapping and the build is never billed, because nothing else records which deployment it belonged to.

Migrating from WordPress to the static site generator GoHugo enables a git-based workflow by treating content as markdown. Using the wordpress-to-hugo-exporter plugin, posts convert to local markdown files, improving rendering speeds and eliminating dynamic database overhead.

Postgres handles basic workloads, but struggles with large-scale analytics. Organizations scale by moving to cloud data warehouses, yet these separate storage and compute, prioritizing massive capacity over speed. For low-latency queries, specialized real-time analytical databases are required.

Mid- to large-sized enterprises struggle to adopt the Modern Data Stack due to legacy systems and technical debt. Working with data within a big organization is complicated and unpredictable. Instead, teams build custom scripts or rely on cloud providers, keeping MDS tools in early adoption.

Migrating views from Oracle to SQL Server requires performance and content testing. Performance is measured using SELECT COUNT queries with STATISTICS TIME ON after clearing cache. Content parity is validated using linked servers, OPENQUERY, and generated EXCEPT queries to compare datasets directly.

Migrating Oracle recursive queries using SYSCONNECTBY_PATH to SQL Server CTEs causes severe performance issues on nested views. Replacing CTEs with Inline User-Defined Functions restored speed.

Migrating Oracle SQL views to SQL Server requires rewriting functions like TOCHAR and TONUMBER using CONVERT or CAST. Precision differs significantly: the built-in TO_NUMBER() function rounds at the 16th decimal place, whereas SQL Server's numeric or float types handle precision differently.

Starting September 30, 2026, all new Databricks workspaces will be provisioned as Unity Catalog-only without access to the legacy Hive metastore. Even on Unity Catalog, clusters connect to Hive metastore by default. To secure data, run spark.databricks.unityCatalogOnlyMode True on clusters.

Physical Intelligence unified its robotics data stack by migrating from AWS RDS Postgres to Postgres managed by ClickHouse. The team needed to handle both OLTP workloads and high-cardinality OLAP analytics as their dataset expanded from 10 to 100 billion rows for robotics foundation models.

Migrating high-MAR database replication from Fivetran to Bruin and ingestr replaces per-row pricing with open-source ingestion. Pipelines remain version-controlled in your repository alongside transformations and data checks.

Migrating Apache Spark pipelines to Snowflake and Snowpark Connect using the Snowflake CoCo spark-migration skill helps data engineers improve performance and cut costs. Transitioning legacy workloads directly optimizes execution.

🎙️ The Data Flowcast by Astronomer — Trading Technologies consolidated two decades of legacy C#/.NET scheduling onto Apache Airflow and Snowflake. By modernizing their architecture, the team collapsed thousands of legacy jobs into just 15-20 Airflow DAGs.

To handle 145 billion daily events, Atlassian migrated StreamHub from Amazon Kinesis to AWS Managed Streaming for Apache Kafka. Using Kafka Tiered Storage offloaded historical data to S3, reducing EBS costs while supporting multi-day retention.

Database consolidation simplifies production AI by replacing separate transactional, caching, and vector databases with a unified platform. Managing a fragmented data stack creates risk because every additional database increases operational complexity and expands the failure surface for AI agents.

Designing pipelines for both incremental loads and historical backfills prevents fragile migration scripts. Using Dagster and DuckDB, the same transform loop handles single entity refreshes, incremental updates, and full backfills.

Database choices exist on a ladder of irreversibility. While tuning session parameters or adding indexes is easy, changing column types or partition keys causes massive table rewrites and outages. Code is ephemeral; schema is permanent. Prevention through guardrails is critical.