Sharing resources for data engineers

Salesforce cannot ingest granular event telemetry at scale or model complex predictive insights due to strict API limits and compute constraints. Reverse ETL pipelines bridge this by syncing warehouse-modeled data and external IDs back to CRMs through governed, automated workflows.

Snowflake release notes detail platform updates, including Apache Iceberg table enhancements like Time Travel preview, secondary indexes, and DCM DEFINE TASK GRAPH for workflow orchestration. Other features include declarative feature stores and bulk file loading to optimize data pipelines.

Choose SAP OData patterns by workload: use CDS views for read-heavy analytics, and custom RAP services for complex transactional logic. Crossing hybrid boundaries via Cloud Connector adds latency, making metadata caching and tuning expand payloads critical for remote BTP performance.

A catalogs.yml v2 spec decouples dbt Iceberg configurations from engine-specific storage, enabling cross-platform dbt Mesh across Snowflake and Databricks. Catalog federation lets engines query shared Iceberg tables without data duplication, while DuckDB routes specific models.

dbt Charts Cloud supports public dashboards defined in versioned YAML, modeled via dbt. Access requires dual-level opt-in controls across organization and project settings, letting external users view curated analytics without logins while admins retain kill-switch controls.

Kramp scaled dbt Mesh on Google BigQuery using Data Project Factory, a Terraform module that provisions GCP projects, dbt repositories, and access controls for domain teams. This self-serve architecture cut onboarding from days to minutes across 25 projects while enforcing CI governance.

Mitratech turned dbt into an orchestrator to automate customer data onboarding. Using macro packages and Snowflake stored procedures via "dbt run-operation", they build secure views and shares with tenant filtering and PII masking, cutting provisioning time from days to hours.

Greenpeace used a 3-layer dbt package architecture to unify transformations across 25 autonomous entities on BigQuery and Fivetran. Shared dbt packages hold 75% of the logic, standardizing metrics and semantic definitions while allowing automated CI rollouts to 170 projects.

🎙️ DataFramed — AI agents depend heavily on accessible enterprise data across hybrid environments. Managing access is challenging, making governance, permissions, and audit trails the primary bottleneck for production deployments across architectures like data fabrics and meshes.

Snowflake released updates including general availability for Declarative Feature Store, Online Feature Store, and Apache Iceberg v3 table upgrades. Data engineers also gain support for recompacting Iceberg tables after target file size increases and schema variables.

AWS enables cross-Region data access using Lake Formation and IAM Identity Center Trusted Identity Propagation. Queries across S3 and Redshift are authorized per individual user instead of shared IAM roles, letting engines like Athena trace every cross-Region request directly in AWS CloudTrail.

Public sector AI readiness requires modernizing legacy architectures into unified, governed data platforms. Multnomah County migrated a legacy SQL Server footprint in five months using automated code conversion, dynamic SQL translation, and test generation to eliminate cross-agency data silos.

Brilliant Earth deployed Posit Team as a Snowflake Native App on Snowpark Container Services, unifying governance and user permissions directly in Snowflake.

Snowflake release updates detail recent data engineering features, including Iceberg table compaction enhancements, declarative feature stores, and incremental refresh support for dynamic tables using the VALUES clause and non-deterministic aggregates.

Multiplayer AI architectures face systemic handoff gaps across autonomous agents. Reliable pipelines require explicit shared context, scoped metadata, and ownership protocols, ensuring that downstream systems receive granular governance context rather than isolated outputs lacking lineage.

Snowflake announced the general availability of centralized event sharing for Snowflake Native Apps. This release streamlines telemetry and log aggregation across consumer accounts, enabling data engineers to monitor distributed app performance, usage, and errors from a single destination.

Snowflake released general availability support for sharing Streamlit in Snowflake apps directly with individual users. The update enables data engineers to grant granular access to data applications alongside existing role-based access control.

Snowflake published updates for Snowflake Data Clean Rooms alongside general availability for Snowflake Postgres data mirroring. Supporting tooling includes ODBC Driver v4.1.0 and Python Connector v5.0.0 releases.

Amazon Redshift Data Sharing enables cross-cluster querying without copying data between clusters. Producers grant datashare access, allowing consumers to read live data directly. Teams can then run dbt Cloud on consumer clusters to build and materialize transformed models via SQL pipelines.

Data engineers operationalize data governance by treating it as code. Within a Data Mesh architecture, teams define domains, tribes, and squads in versioned YAML files alongside platform code. This allows CI pipelines and data catalogs to automate ownership checks and enforce standards.

Life sciences architectures require moving beyond static repositories toward continuously refreshed, governed pipelines. Integrating multimodal data with lineage, tokenization, and semantic context allows teams to run agentic evidence workflows directly where governed data rests.

Workday and Snowflake now offer a bidirectional, zero-copy integration via Apache Iceberg. Engineers can query Workday objects in place or mount Snowflake tables into Workday without ETL pipelines, preserving native governance and semantic models while eliminating data replication lag.

Fivetran Activations enables reverse ETL to sync warehouse data into downstream tools. Product updates add source-based sync grouping, global cloud region deployments, PrivateLink support, and redesigned APIs with webhooks to automate data activation pipelines at scale.

Snowflake has announced general availability for LISTINGACCESSHISTORY improvements, enhancing governance tracking for data listings. The release is accompanied by tools like Snowpipe Streaming SDK v1.9.0 and features like lineage preservation and partition evolution for Apache Iceberg tables.

🎙️ The Data Flowcast by Astronomer — Reddit manages petabytes of daily data that must be processed, stored, and analyzed by backend services. The team leverages Airflow for pipeline orchestration and scales out a self-service data architecture across multiple teams to support warehousing and engineering functions.

🎙️ The Data Flowcast by Astronomer — Covestro uses Apache Airflow and Kubernetes to run scientific computing pipelines. By mounting persistent volumes, tasks share large datasets, while custom operators isolate business logic from infrastructure to handle varied workloads.

Build cross-account Kinesis pipelines by isolating producer, stream, and consumer into separate AWS accounts. KPL assumes a cross-account IAM role to write records, while KCL reads via stream resource policies and manages DynamoDB leases locally. Encrypt streams using customer-managed KMS keys.

Listing object metadata in an unpartitioned prefix caused ingestion pipelines to run out of memory as files accumulated. To eliminate broad discovery overhead and isolate backfills, the team collaborated with the vendor to implement date-partitioned storage prefixes.

Turn agent run transcripts and tool calls into customer data products like live audit logs and real-time metering APIs. Supporting external query loads requires isolating tenant workloads and enabling sub-second ingest so unstructured session logs become queryable tabular data.

dbt now natively supports Amazon Redshift data sharing reads and cross-database writes via the datasharing: true flag. By adopting Redshift SVV system views instead of legacy Postgres APIs, dbt can resolve cross-database metadata, enabling isolated multi-account development and dbt Mesh