dataaaaaa!

Snowflake announced Cortex Training in preview, alongside hybrid table support for Google Cloud and External OAuth for Snowflake Postgres. Recent updates also expand data engineering tooling, including DCM project task graphs, Time Travel for Apache Iceberg tables, and secondary indexes.

Split data reconciliation into two steps. First, use scheduled deterministic checks like row counts, measure sums, and join-key coverage to detect raw gaps. Then, apply an AI layer over historical resolution logs to triage expected noise and explain cross-platform pipeline discrepancies.

Data engineering hiring is recovering, but employers now expect more than traditional SQL, Python, and Airflow pipelines. Engineers must build AI-enabled systems and master tools like Databricks, dbt, and AWS, even as junior opportunities remain limited to just 5.5% of openings.

Decision models like Jev prioritize fast answers paired with calibrated probabilities rather than generating text. Data engineers can integrate these models into data analytics pipelines using the dbt-decisions package, linking AI decision outputs directly with a semantic layer inside dbt workflows.

Decision models reshape ETL by offering fast, typed transformations instead of text generation. OpenAI's Decisions API evaluates multimodal inputs like images via predicates, choices, and scores with zero output tokens, enabling flexible, low-latency automated pipeline actions.

Run open-weight LLMs locally to cut token costs and protect data privacy. Use quantization like 4-bit to shrink tensor memory footprints for consumer GPUs, and orchestrate automated pipelines using tools like Kestra for scheduled batch inference and continuous tasks.

BigQuery lets data teams call LLMs in SQL queries via AI functions instead of Python APIs. Integrating calls like AI.GENERATE into dbt incremental models prevents expensive full-refresh recomputation. Output schemas force typed JSON, enabling validation with standard dbt tests.

Jev evaluates agentic analytics pipelines by classifying run failures and ranking warehouse tables. It scores tables at nDCG@10 0.951 in 0.4 seconds without fine-tuning, identifying that data selection and entity resolution cause most agent query errors.

Snowflake launched Pipeline Builder in preview, expanding pipeline development features alongside task graph improvements like atomic deployments and advanced branching. Updates also bring schema variables and Iceberg v3 upgrades to streamline data architectures.

Netflix built an automated video pipeline that generates 9:16, 4:5, and 1:1 crops in a single pass. The system combines neural shot boundary detection, active speaker tracking, and saliency maps, exporting transform metadata via XML to external editors to streamline mobile asset workflows.

Netflix built Pensive AI to triage failed data workflows across engines like Spark and Trino. By returning only log line numbers, the model extracts root causes without expensive token generation. Diagnostic runs are cached per failed step attempt to slash duplicate inference costs and latency.

Apache Airflow now officially supports LLM calls and agentic actions via six new AI operators and six AI task decorators. When orchestrating these pipelines, data engineers must balance deterministic and non-deterministic workflows, since identical inputs to AI tasks may yield varying outputs.

Recent engineering deep dives cover scaling online storage for over one billion ChatGPT users and rebuilding LinkedIn tracking infrastructure. Tool updates highlight the DuckDB hf extension to query Hugging Face directly and diagnosing Oracle GoldenGate replication lag queue by queue.

Enterprise AI architectures require an automated governance layer across policy, data, model, runtime, and assurance planes. Data pipelines and retrieval indexes need lineage, classification, and immutable audit logs to track inputs, model versions, and guardrail decisions systematically.

🎙️ The Analytics Engineering Podcast (dbt) — AI data agents struggle with raw data, often pulling everything into pandas instead of pushing down compute. Tested on messy benchmarks, top agents solve only 61% of queries on the first try. Data engineers must provide gold-layer tables and robust semantic layers to enable reliable queries.

Snowflake announced the general availability of custom runtime images for ML Jobs, enabling tailored container environments for machine learning workloads. The release coincided with updates including general availability for Iceberg table re-compaction and the declarative feature store.

BigQuery now offers the generally available AI.TRANSLATE function to translate text strings between languages directly in SQL pipelines. It supports standard Neural Machine Translation workloads and Translation LLM capabilities for data processing.

Snowflake updated Cortex Fine-tuning: new model training is no longer available in AWS US East 1. Data engineers managing ML pipelines must adjust region targets. Concurrently, Snowflake introduced features like Declarative Feature Store and Iceberg table data recompaction.

DuckDB released version 2.0.0 LTS Cinnamon Teal. Key data engineering topics highlight Apache Arrow for columnar memory architectures, change-data-capture patterns via PyDebeziumAI, and handling late-arriving data in dbt snapshots alongside data governance and lineage in Snowflake.

Snowflake announced general availability for its Declarative Feature Store and Online Feature Store, expanding ML engineering tools. Related platform updates introduce incremental refreshes for dynamic tables and support for partitioned Apache Iceberg tables in Snowpipe Streaming.

Vivix built an executable digital twin by deploying lightweight Siemens reduced-order physics models natively into Snowflake. Integrating IT and OT sensor data into a unified namespace allows near-real-time ingestion to predict furnace states alongside NVIDIA AI models.

BigQuery has made AI.CHUNK_DOCUMENT generally available. The table-valued function splits unstructured documents into semantically coherent text chunks optimized for vector embeddings and Retrieval-Augmented Generation workflows directly within SQL.

NVIDIA Kumo integrates with Snowflake ML, allowing teams to train graph neural networks directly on relational tables without flattening data. Running via Snowflake Notebooks on GPU container runtimes, "data never has to leave it," orchestrating reads, training, and scoring securely in place.

Snowflake ML unifies stream ingestion, feature stores, and inference to power real-time fraud detection. Built on Postgres, the Online Feature Store serves fresh signals in under 10 ms directly to containerized model endpoints, delivering predictions under 100 ms while avoiding data egress.

Deep customization pairs continued pre-training on curated domain tokens with multi-stage post-training. Teams build specialized pipelines mixing proprietary corpora with general data. Thomson Reuters used a 200B token mixture, proving data engineering and evaluation infra matter most.