Security resources for data engineers

Snowflake has released general availability for Tri-Secret Secure on Snowflake storage for Apache Iceberg tables. Data engineers can now leverage enhanced customer-managed encryption key architectures for Iceberg table storage directly within Snowflake, improving security and compliance.

Amazon SageMaker Unified Studio introduces designations, named permission sets backed by the Cedar policy language to enforce least privilege within shared projects. Administrators can isolate scheduled ETL pipelines, notebook compute, and catalog stewardship across personas in a single project.

Snowflake has announced the general availability of External OAuth for Snowflake Postgres. This security update allows data engineers to integrate external identity providers for role-based authentication and database access control across relational pipelines.

Snowflake added support to carry over comments, tags, and privileges using the COPY = ( ... ) syntax. This enhances DDL workflows by automatically retaining critical object metadata and access controls during table replication.

Data roles are shifting from core reporting to operational automation and data ingestion. Engineers must close data action loops using domain data, virtual knowledge graphs, and LLM-driven agents. Understanding pipeline trade-offs remains essential as automation absorbs routine tasks.

This edition covers data engineering architectures like JD.com building a lakestream with Apache Fluss and Apache Hudi, Azure SQL Database Hyperscale internals, DuckDB 2.0.0 features, and how Flink's Kafka connector commits offsets to Kafka.

Snowflake release notes detail platform updates, including Apache Iceberg table enhancements like Time Travel preview, secondary indexes, and DCM DEFINE TASK GRAPH for workflow orchestration. Other features include declarative feature stores and bulk file loading to optimize data pipelines.

A new community DuckDB extension queries Snowflake directly via an ADBC driver without custom scripts. Engineers can attach Snowflake databases and enable filter pushdown to cut data transfer, joining warehouse tables directly with local Parquet files.

Snowflake release updates highlight the general availability of Request Access in Workspaces and secondary indexes. Data engineers also gain Time Travel for Apache Iceberg tables in preview, DCM Projects task graph definitions, and continuous or bulk file loading via CoCo.

Choose SAP OData patterns by workload: use CDS views for read-heavy analytics, and custom RAP services for complex transactional logic. Crossing hybrid boundaries via Cloud Connector adds latency, making metadata caching and tuning expand payloads critical for remote BTP performance.

Infrastructure automation often fails across tool handoffs during Day 2 operations like patching. While Terraform or Ansible automate isolated tasks, cross-tool orchestration is missing. Providing a unified control plane coordinates execution, passes outputs, and creates a single audit log.

Google Cloud provides an architecture pattern to securely connect Gemini Enterprise with Snowflake data pipelines. Using the Google Application Integration Development Kit, data engineers can bridge enterprise analytics and external warehouse workloads seamlessly.

Recent security disclosures do not impact Metabase. The stack contains no PHP, avoids affected npm dependencies, and removed the compromised eMudhra root CA in recent updates. Metabase Cloud instances are already patched, but self-hosted deployments must upgrade to trusted releases.

TELIXO is a real-time M-Pesa transaction streaming pipeline designed to ingest, process, and analyze fintech data via event-driven architecture. It leverages Docker Compose across multi-language runtimes, including Python and Java, to maintain scalable stream processing and data pipelines.

dbt Charts Cloud supports public dashboards defined in versioned YAML, modeled via dbt. Access requires dual-level opt-in controls across organization and project settings, letting external users view curated analytics without logins while admins retain kill-switch controls.

Snowflake introduced ingress policies in preview alongside server release 10.38. The security feature provides data engineers and administrators with inbound network traffic governance to restrict and control incoming connections to data platform architectures.

Snowflake announced the general availability of the Trust Center Data Security scanner package. Recent platform updates also include Snowpipe Streaming SDK v1.9.1, Apache Iceberg v3 table upgrades, Parquet Direct, and dynamic tables continuing incremental refreshes after failovers.

Mitratech turned dbt into an orchestrator to automate customer data onboarding. Using macro packages and Snowflake stored procedures via "dbt run-operation", they build secure views and shares with tenant filtering and PII masking, cutting provisioning time from days to hours.

🎙️ DataFramed — AI agents depend heavily on accessible enterprise data across hybrid environments. Managing access is challenging, making governance, permissions, and audit trails the primary bottleneck for production deployments across architectures like data fabrics and meshes.

Minilake is a free, local Databricks emulator enabling offline testing of databricks-sdk and Terraform code against Delta Lake, real SQL, and jobs without cloud costs. Baked-in DuckDB delta extensions and Unity Catalog jars support air-gapped development, though it lacks access-control enforcement.

AWS enables cross-Region data access using Lake Formation and IAM Identity Center Trusted Identity Propagation. Queries across S3 and Redshift are authorized per individual user instead of shared IAM roles, letting engines like Athena trace every cross-Region request directly in AWS CloudTrail.

Data platform engineers replaced costly Looker instances with an internal service hosting LLM-generated HTML in S3 on Kubernetes. The architecture queries Snowflake directly via user OAuth tokens, delegating data governance and access controls to existing Snowflake roles and upstream dbt models.

Enterprise AI workflows require unified orchestration to manage cross-domain agents. Moving data is solved, but governance demands keeping execution IDs outside agents. Running workers near data platforms like Snowflake or Databricks ensures auditability and end-to-end control.

Enterprise AI architectures require an automated governance layer across policy, data, model, runtime, and assurance planes. Data pipelines and retrieval indexes need lineage, classification, and immutable audit logs to track inputs, model versions, and guardrail decisions systematically.

Hardcharger is an open-source server for DuckDB 2.0 that replaces gRPC Flight SQL with Warp SQL, transporting SQL over standard HTTP with Arrow IPC streams. It adds native RBAC, SSO, and mTLS, running 22 TPC-H queries on 1 TB in 42.4s on Graviton5.

Snowflake released the Data Security scanner package for Trust Center to general availability, enhancing governance monitoring. It follows pipeline tools like incremental refreshes for dynamic tables and Iceberg partitioning. Data engineers can evaluate account security posture natively.

Build audit-ready regulatory pipelines using S3, Glue, and dbt on Athena or Redshift. Store immutable raw files via S3 Object Lock, enforce dbt model contracts, and archive dbt run artifacts alongside report exports to a restricted evidence bucket to ensure reproducibility.

Cube Core v1.8 introduces safer defaults and faster SQL API performance. It enables log redaction by default and makes development mode strictly opt-in via CUBEJSDEVMODE. Engine updates deliver pushdown support for UNIONs and up to 1.76x faster SQL API result streaming.

DeepSQL is a self-hosted database agent for PostgreSQL and MySQL that enables schema exploration, query generation, and slow query analysis via natural language. It supports custom LLM backends like LiteLLM while keeping database credentials encrypted locally. The backend JVM requires 3 GB heap.

ObsessionDB launched a BYOC model to run ClickHouse on your own cloud object storage. It decouples stateless compute from storage using a distributed NVMe cache, isolating ingestion and query workloads over a single bucket without re-sharding.