Machine Learning resources for data engineers

🎙️ DataTalksClub ⬛ — Building production AI platforms requires managing evaluation datasets, LLM memory, and tool orchestration. Key technical patterns include mitigating context window limits, optimizing tool calling, and overcoming retrieval degradation like the lost in the middle problem in RAG pipelines.

AI systems require computer vision as core data infrastructure to process massive uncurated visual streams. Edge architectures must run low-latency vision-language-action pipelines locally because shipping high-throughput sensor frames to cloud datacenters introduces critical latency.

BigQuery now offers TimesFM 3.0 in preview for built-in time series forecasting. Data engineers can use SQL functions like AI.FORECAST for multivariate forecasting with covariates, AI.EVALUATE for univariate model evaluation, and AI.DETECT_ANOMALIES to run univariate anomaly detection pipelines

Snowflake previewed the Kimi K3 model alongside pipeline enhancements, including data lineage for storage integrations, external tables, and Cortex Search. Recent updates also expand dynamic table incremental refresh support and Snowpipe Streaming for partitioned Apache Iceberg tables.

Deploying clinical AI requires robust human-in-the-loop pipeline architectures. Systems need escalation routing logic to triage interruptions, mitigate alert fatigue, and capture clinician override feedback data to monitor distribution shift and model drift.

Engineering teams must calibrate AI judges against human labels and strict rubrics to eliminate bias. Evaluating RAG architectures stage by stage allows data engineers to pinpoint retrieval versus generation failures and replace fuzzy scores with targeted, deterministic checks in production

Deploying machine learning requires robust engineering and surrounding data infrastructure to handle real-world workloads. Focus areas include constructing resilient pipelines, deploying recommender engines, and supporting complex agentic workflows to ensure production systems remain reliable.

Conversational analytics in BigQuery now supports the AI.CAUSAL_EFFECT function in preview. This feature allows data teams to quantify the impact of specific interventions on time series data directly using SQL workflows.

SAP is acquiring TechWolf to feed its three-layer context graph for skills and work into HR AI agents. The platform ingests business application data, infers skills, and writes them back to SAP SuccessFactors, optimizing LLM token efficiency and agent query grounding.

MotherDuck integrates Jev via prompt_jev() to execute fast decision-model evaluations directly in SQL queries on every row. It classifies text into closed sets like scores and labels alongside telemetry spans, separating plain SQL counts from subjective judgment calls.

Redacting PII with an LLM still leaks sensitive data outside your privacy boundary. Instead of AI, build deterministic data pipelines combining candidate detection with structural and checksum validation, such as the Luhn algorithm for cards, ensuring reproducible, testable sanitization.

BigQuery now supports flexible column names by default on Iceberg external and managed tables. Additionally, the gemini-embedding-2 model is generally available for AI.EMBED, AI.SIMILARITY, and AI.GENERATE_EMBEDDING functions, enabling multimodal embeddings across text, audio, and video.

Amazon Aurora PostgreSQL now supports direct querying of Apache Iceberg and Parquet data lake files. Other data engineering highlights include extending Apache DataFusion across machines, using ClickHouse for metric discovery at LinkedIn, and DuckDB 2.0 adding dynamic ATTACH statements.

jev4pg adds natural language to SQL and semantic operators to PostgreSQL. It features 41 JEV operators for filtering, extraction, and verification. PostgreSQL manages joins and windows while JEV uses parallel execution, input deduplication, and evidence caching for efficient query planning.

In entity matching benchmarks, GPT-4o-mini tied Jev with an F1 score around 0.81 at roughly the same batch cost ($22.57 vs $20.56 per million pairs). While Jev requires no prompt tuning and is twice as cheap for live matching pipelines, both zero-shot models reliably handle unseen data at scale.

🎙️ The Data Flowcast by Astronomer — Airflow orchestrates local LLM pipelines for sensitive data, running GPU inference via KubernetesPodOperator into feature stores. DAGs manage ingest, Pydantic validation to stop silent failures, and PEFT fine-tuning. Production requires strict pinning and tracing adapter provenance per row.

🎙️ DataTalksClub ⬛ — Building an enterprise AI platform prevents fragmented engineering and eliminates AI sprawl. Production infrastructure requires architecting scalable data and model services, robust request tracing for observability, session memory management, and durable execution for complex agent workflows.

Snowflake updated its platform features, previewing the Anthropic Claude Sonnet 5.5 model and Spark Monitoring UI. Data pipeline enhancements include GA for dynamic table incremental refreshes with VALUES and non-deterministic aggregates, plus Snowpipe Streaming support for partitioned Iceberg

AI agents struggle in production when data architectures fail to provide fresh, governed business context across disparate sources. Beyond standard vector search or RAG, engineers need a contextual data layer to prevent stale data pollution, avoid high latency, and maintain unified context.

🎙️ DATANOSCO — Expleo deploys ML architectures across critical industries, relying on classical statistical models for predictive aircraft maintenance and computer vision on assembly lines. In rail systems, RAG pipelines query historical technical docs, where document chunking and indexing quality drive success.

Production AI engineering is shifting toward LLMOps, requiring stronger observability, tracing, and governance with tools like Databricks, MLflow, and Unity Catalog. Evaluating agentic systems is harder than traditional ML, making testing and architecture critical for reliable data pipelines.

To classify data job postings, a two-step ingestion pipeline uses regex filtering at the gate to drop irrelevant roles cheaply before database storage.

Snowflake Expedition features technical workshops for data engineers and developers to build data pipelines, AI models, and agentic workflows. Focus areas include using Snowflake CoCo to accelerate data engineering with coding agents, utilizing Snowflake CoWork, and governing enterprise data at

To infer missing CRM salutations from first names, a Python script queried Jev and local Laya models. Jev achieved 18/20 accuracy with usable confidence scores, while local Unsloth runs processed 1,000 records in 5-10 seconds using parallel calls.

OSA Con features technical sessions on open-source data infrastructure and AI workloads. Data engineers explore AI agents generating Apache Spark pipelines, serverless eventing for data flows, and the Presto C++ engine, alongside architectures like shared data lakes and production ClickHouse on

Jev acts as a fast SQL decision function to classify unstructured text into typed choices without token-by-token LLM generation. MotherDuck's prompt_jev() processed 100,000 text rows in 82.2 seconds for $0.50, outputting a struct with the chosen label and confidence scores directly in the query.

TypeSafe AI's Jev outputs typed decisions as JSON. Representing these outputs via Apache Arrow schemas enables zero-copy ingestion into engines like DuckDB and Polars, avoiding serialization overhead. However, analytical batch evaluation remains bottlenecked by the API's single-state request design.

Jevstiller acts as a drop-in proxy, routing inference to a 15 ms local CPU model instead of a 300 ms vendor API call. Using Clopper-Pearson bounds and fixed-sequence testing on calibration data, it guarantees a strict target agreement rate with the vendor model across requests.

AI data engineering is shifting from collection to downstream processing pipelines. Useful data is the output of continuous transformation, validation, and sanitization rather than raw input.

TypeSafe AI released Jev, enabling DuckDB extensions to filter, classify, and score Parquet and SQL tables via natural language. Extensions process data in 2,048-row chunks and push down standard SQL predicates first, preventing unnecessary third-party API calls on rejected rows.