Fivetran resources for data engineers

Salesforce cannot ingest granular event telemetry at scale or model complex predictive insights due to strict API limits and compute constraints. Reverse ETL pipelines bridge this by syncing warehouse-modeled data and external IDs back to CRMs through governed, automated workflows.

AI integration requires strong data engineering foundations. Fivetran advocates for open formats, portable storage, and reliable data pipelines to power AI agents securely, noting that data should not be trapped in closed systems. Reliable, well-governed data pipelines make AI accountable.

Enterprise AI bottlenecks have shifted from model reasoning to reliable data architectures. Instead of siloed vector databases, teams need unified platforms pairing SQL with native vector search and open formats like Apache Iceberg to deliver minimal, high-quality context directly to agents.

Data catalogs organize structural metadata and schemas across warehouses and lakes. Technical catalogs track schemas, access policies, and column-level lineage, enabling tools like Iceberg and Glue to manage state and evaluate the impact of schema changes across pipelines.

AI agents require shared context across structured and unstructured data to act reliably. Fivetran Context Layer addresses this by indexing docs and data sources into warehouse tables called Agents Schema, letting agents query unified ontologies directly via MCP instead of proprietary APIs.

Lake Compute lets data teams run low-complexity dbt models on cheaper lake engines instead of expensive data warehouses. Built on Apache Iceberg and open REST catalogs, this architecture cuts compute costs model by model without disrupting downstream DAGs or BI tools.

Integrating NetSuite with Snowflake decouples operational processing from analytical reporting. Ingestion pipelines replicate ERP records into Snowflake for cross-domain modeling, while reverse ETL writes approved values back into NetSuite via web services using explicit object permissions.

Open formats like Apache Iceberg decouple storage from compute, letting teams match engines like DuckDB or Snowflake to specific workloads. True interoperability requires unified governance through standards like the Iceberg REST protocol so policies travel with data across engines.

Fivetran Skills provides Claude with schema context and dbt models to query synced warehouse data directly. Every answer comes from an executable SQL query against Snowflake, BigQuery, or Databricks, enabling natural language troubleshooting without altering warehouse architecture.

Databricks ingestion should prioritize ownership over tool names. Teams risk custom pipeline chaos by source forty. Evaluate options sequentially: query in place via Lakehouse Federation or Delta Sharing, leverage existing tools like Fivetran, or use native serverless Lakeflow Connect connectors.

Fivetran introduced LakeCompute, letting dbt users run transformations on Iceberg tables via DuckDB compute instead of Snowflake. By managing the landing layer and compute, Fivetran disintermediates the data warehouse to cut query costs, shifting workloads away from Snowflake.

Fivetran automated AI usage analytics using ELT connectors for OpenAI and Anthropic. Ingested raw usage logs are enriched with HR data and transformed using pre-built data models to track cost and activity by department in near real time.

Fivetran Activations enables reverse ETL to sync warehouse data into downstream tools. Product updates add source-based sync grouping, global cloud region deployments, PrivateLink support, and redesigned APIs with webhooks to automate data activation pipelines at scale.

DuckDB on an iPhone outperformed Databricks Serverless SQL on TPC-H workloads up to 100 GB. Single-node execution engines avoid the query planning and network shuffle overhead inherent to distributed clusters, making them far more cost-effective for typical real-world business datasets.

Databricks Lakeflow complements Fivetran and dbt within a unified architecture. Fivetran automates SaaS and database ingestion into open formats like Delta Lake, Lakeflow processes Kafka event streams with Spark Declarative Pipelines, and dbt models SQL business logic with built-in testing.

Fivetran leads Snowflake ingestion tools at 12.7% of surveyed companies, followed by Azure Data Factory at 7.9% and Informatica at 6.5%. Most teams bypass standard ETL vendors, relying instead on native ingestion like COPY INTO, Snowpipe, or custom code.

Agentic AI requires reliable business context over raw compute. About 80% of enterprises cite data quality as their main obstacle. Engineering open, governed data pipelines prevents critical failures, as context is lost when trapped inside proprietary stacks.

New data tools cannot fix fundamental business misalignment. Throwing data into central warehouses, semantic layers, or context graphs without resolving domain definitions simply creates a semantic swamp. Real progress requires defining data ownership and building cross-functional alignment first.

dbt v2 marks dbt's evolution from a SQL tool into a broader data platform. The release consolidates Fusion into a single Rust-based engine with free and open-source distributions, unifying transformation and metadata to support semantic metrics and AI agent context alongside Fivetran.

Choosing a data migration tool depends on recovery semantics and whether you need dump-and-restore, batch cursors, or log-based CDC. CLI tools like ingestr offer scriptable batch and CDC runs without servers, while managed platforms suit broader connector needs. State durability is critical.

dbt v2 is now GA, replacing the Python engine with Rust to parse projects up to 10x faster. Fivetran and dbt Labs also introduced Lake Compute, a DuckDB SQL engine querying managed Apache Iceberg tables, letting teams offload model testing from expensive warehouses like Snowflake or BigQuery.

dbt v2 rewrites the engine in Rust, parsing models up to 10x faster with pre-run lineage checks. Paired with dbt State, it optimizes pipeline orchestration by selectively building or deferring runs via warehouse metadata. DuckDB-powered Lake Compute also enables running dbt on Iceberg tables.

Fivetran Context Layer is a managed service that builds a governed context layer directly in your data warehouse or data lake. Operating on data moved by Fivetran and modeled by dbt, it publishes semantic metadata into SQL tables to reliably feed AI agents without extra pipelines.

Selecting CDC tools requires evaluating capture methods, schema evolution, and recovery. Log-based CDC reads transaction logs like WAL or binlog, reducing source database load and latency compared to polling tables.

Centralizing data is not enough; retail and CPG pipelines require dbt to validate data integrity. Automated dbt tests catch broken feeds, while the dbt Semantic Layer defines shared metrics. Lineage graphs via dbt Explorer trace downstream numbers back to source systems for reliable automation.

Manual reporting breaks as ecommerce scales because transactional data stays isolated from ad platforms. Engineering automated ELT pipelines into a cloud data warehouse unifies Shopify, web, and ad sources, enabling reliable multi-touch attribution and full customer lifecycle analysis.

🎙️ DataFramed — Data engineering is shifting from serving dashboards to powering autonomous agents. This transition requires teams to clean gold-layer tables and formalize metrics within the semantic layer so AI tools retrieve reliable business logic. Open modular stacks and semantic layers ensure agent trust.

Open Data Infrastructure decouples compute and storage using open table formats like Apache Iceberg and Delta Lake on object storage. Storing data once provides transactional lake reliability, letting teams swap engines, avoid duplicate pipelines, and support AI without vendor lock-in.

dbt brings software engineering practices to data transformation by organizing SQL select statements into a DAG. It persists cleaned, modeled tables in the warehouse, while its semantic layer provides consistent metric definitions required for downstream BI tools and AI agents.

Recent data engineering updates highlight the release of Spark 4.1.0 and Apache DataFusion 55.0.0. Technical deep dives focus on computing time in queue for Apache Hudi pipelines at petabyte scale, streaming table patterns with Kafka and Fluss, and replacing Fivetran for Snowflake ingestion.