dataaaaa!

🎙️ It's About Data® — Snowflake's Chief Security and Trust Officer Mayank Upadhyay discusses security and trust architectures for autonomous agents. The session focuses on securing AI agent frameworks and managing data trust.

AI agents excel at analytical tasks, but require modeled data and open infrastructure to work. Instead of wiring agents directly into SaaS APIs via protocols like MCP, teams should centralize business data in open table formats like Iceberg to ensure control, clean metadata, and composability.

🎙️ The Data Flowcast by Astronomer — Insignis runs a cost-effective platform using Airflow as the orchestrator, not the executor. They keep compute lean via CI/CD enforcement, failure callbacks, metadata purging, and custom tools like Tariff to surface SQL costs and Maestro to optimize worker sizing.

🎙️ DataTalksClub ⬛ — The session details financial data pipelines, focusing on ingesting and cleaning market data via APIs. It covers processing time series with Pandas and Polars, designing custom backtesting engines, and integrating machine learning workflows for algorithmic trading systems.

🎙️ DataFramed — Data career targets are shifting from data scientist to AI engineer as enterprise training pivots toward prompt engineering, agents, and AI literacy. Learner analytics show technical professionals rely on short study habits and certifications to validate skills alongside automated code generation.

🎙️ The Analytics Engineering Podcast (dbt) — Modern data platforms serve as the foundation for AI agents extracting meaning from business data. Industry leaders explore why an open, composable data architecture remains essential during rapid technological shifts, adapting modern engineering pipelines directly for autonomous systems.

🎙️ DataTalksClub ⬛ — Reliable production LLM systems require moving beyond vibe checking to automated evaluation architectures. Data pipelines must curate golden datasets and leverage an LLM-as-a-judge pattern to benchmark nondeterministic agent outputs against strict human quality standards.

🎙️ blef — Mistral released a new model alongside updates related to OpenAI dots, expanding options for AI integration pipelines. Data engineers can evaluate these emerging model endpoints for pipeline architectures.

🎙️ DataGen — Modern data management architectures leverage Federated Knowledge Graph technology to optimize data catalog performance. Integrated platforms apply AI directly to governance workflows, automating descriptions and tagging to shift search paradigms toward conversational interfaces.

🎙️ DataTalksClub ⬛ — Production ML engineering requires robust architectures over raw model complexity. Teams reduce inference costs and latency using semantic routing, while securing workflows by building trace redaction pipelines to protect sensitive customer data across deployed AI systems.

🎙️ Practical Data Modeling — The Mixed Model Arts book club introduces technical discussions on real-world data modeling practices and architectures. Engineers evaluate schema design decisions, recurring modeling tradeoffs, and requirements gathering before applying automated tools or AI to pipeline and schema design.

🎙️ The Data Flowcast by Astronomer — A podcast series explores Apache Airflow through discussions with data practitioners across industry and government, including leaders from Astronomer, ARGO Labs, the City of San Diego, and ING. The teaser previews real-world orchestration insights.

🎙️ The Data Flowcast by Astronomer — Airflow creator Maxime Beauchemin discusses the core problems the workflow orchestrator was designed to solve. The discussion covers orchestration architecture, his definition of Airflow, and the key design principles governing hook and operator use in data engineering pipelines.

🎙️ The Data Flowcast by Astronomer — This episode highlights real-world Apache Airflow implementations across data engineering teams, exploring six production use cases from organizations including Zapier, WePay, ING, and Astronomer. Technical leaders discuss practical workflow orchestration and pipeline architecture patterns.

🎙️ The Data Flowcast by Astronomer — This episode explores Apache Airflow best practices for data orchestration. It features technical perspectives from data leaders at Tails.com, WePay, and ING, detailing practical insights on workflow management and analytics engineering.

🎙️ The Data Flowcast by Astronomer — Luigi creator Erik Bernhardsson discusses workflow orchestration, examining the engineering motivations behind building Luigi and sharing his perspective on Apache Airflow for data pipeline management.

🎙️ The Data Flowcast by Astronomer — Data engineers examine Apache Airflow pain points across ETL pipelines. While maintaining that Airflow is the future of ETL, the discussion explores operational hurdles within incubating orchestration workflows and highlights community plugins to shape platform architecture.

🎙️ The Data Flowcast by Astronomer — This episode explores the integration of Apache Airflow with Kubernetes, focusing on the development of the Kubernetes executor. Engineers Daniel Imberman and Greg Neiheisel discuss building and advancing this executor architecture to run Airflow task instances dynamically in isolated pods.

🎙️ The Data Flowcast by Astronomer — WePay's Joy Gao discusses implementing role-based access control (RBAC) in Apache Airflow 1.10. The episode details her engineering work on Airflow's RBAC architecture and security components, alongside her experience contributing to open-source software.

🎙️ The Data Flowcast by Astronomer — The episode covers Apache Airflow's 1.10 release with release manager Ash Berlin-Taylor. It highlights Airflow's bid to graduate from incubating status, managing open-source orchestrators, and the next development phase for data engineering pipelines.

🎙️ The Data Flowcast by Astronomer — Google Cloud Composer Product Manager James Malone discusses open source at Google and future contributions to Apache Airflow. The episode explores the platform's trajectory and orchestration roadmap within managed cloud environments.

🎙️ The Data Flowcast by Astronomer — Airflow Breeze optimizes the developer experience for Apache Airflow contributors. Created by committer Jarek Potiuk, it simplifies local environments to fix slow PR workflows where unit tests and CI builds previously took hours, streamlining open-source development for data engineers.

🎙️ The Data Flowcast by Astronomer — Airflow 2.0 targets performance, autoscaling, and usability improvements for data engineering pipelines. Key updates include a production-grade API, the Knative Executor, optimized scheduler and webserver performance, and an official production Docker image.

🎙️ The Data Flowcast by Astronomer — Netlify adopted Apache Airflow to decouple orchestration logic from a complex collection of Spark jobs. The data engineering team is expanding their Airflow footprint to manage and scale pipeline architectures for a broader set of use cases across their platform.

🎙️ The Data Flowcast by Astronomer — Typeform leverages Apache Airflow to automate GDPR compliance and build self-service data platforms. By developing custom infrastructure tooling and adopting frameworks like Dag Factory, their data operations team evolved into a platform engineering unit supporting internal data workflows.