Python resources for data engineers

The new BigQuery Accessor integrates directly with Pandas to enable zero-shot predictions on data analytics workflows. This integration allows data engineers to run AI functions natively on BigQuery datasets through standard Python data manipulation tooling.

Apache Airflow 3.4.0 adds native Task Loops for bounded iteration at runtime via decorated task groups. It also enables multi-language DAG authoring without Python, supporting TypeScript, Go, Java, and YAML via pluggable DAG importers, and matures common.ai integrations for LLM workflows.

Generating Excel workbooks with openpyxl omits cached formula values, causing downstream tools like pandas to read nulls. Running headless LibreOffice computes these values, though parallel runs require isolated user profiles. Modern Excel functions like XLOOKUP also require an _xlfn. prefix.

Databricks Runtime 16 introduces Apache Spark 4.0, integrating ETL, batch, and streaming data orchestration within the Lakehouse architecture. The platform supports serverless SQL warehousing and unified governance across analytics and AI pipelines to streamline large-scale data engineering.

DataSearcher is an analytical tool integrating DuckDB, FastAPI, and an MCP server. It connects to PostgreSQL, MySQL, and ClickHouse, executing translated SQL queries alongside automated PII masking, read-only query guards, and metadata ingestion from dbt manifests and DataHub.

GitHub Actions cron jobs can drop silently under load. Data pipelines avoid failures by scheduling off-peak, enabling workflow_dispatch for manual reruns, setting concurrency groups, and validating data freshness and volume before publishing results.

MotherDuck introduced Agent Chat to run end-to-end data workflows directly in its UI. Using natural language, the native agent generates SQL and executes Python-based data pipelines called Flights on managed infrastructure, automating raw data ingestion, transformation, and interactive app delivery.

Unshackle migrates BI metrics, dashboards, and permissions into Git repositories. It translates measures into Apache Ossie semantic models, dashboards into Jupyter SQL notebooks, and row-level security into Cedar policies, while verifying query outputs against warehouse data.

Apache Arrow 26.0.1 is a security bugfix release resolving 20 issues across core components. Key updates patch a Brotli CVE in PyArrow and R binaries, fix Parquet treating unknown compression codecs as uncompressed, and resolve out-of-bounds reads in IPC streams and Gandiva.

ohi LinkML is a fast, cross-platform implementation of LinkML for validating and generating data schemas across JSON Schema, SHACL, and OWL. Benchmarked at up to 38.5x faster than Python, the tool runs as a self-contained binary or library across the JVM, JS, and Python.

Apache Flink Agents 0.4.0 brings AI agents into streaming pipelines, providing distributed execution and fault tolerance. The update adds sub-agents with durable execution, dynamic model routing, and the unified Decision Model API for scoring against business logic.

MiniLake is a local Databricks API emulator that executes real SQL on DuckDB, writes genuine Delta files, and runs containerized spark-submit jobs. Designed for CI and local development without cloud costs, it natively supports Unity Catalog CRUD, DAG job scheduling, and official SDK clients.

To migrate 2.55 billion rows from Aurora MySQL 8.0 to Outposts RDS MySQL 8.4, engineers bypassed complex Glue ETL by using MySQL Shell streaming utilities. Querying metadata excluded 63 archive tables, allowing a 105 GB zstd-compressed export to complete in under 27 minutes.

Using pyIceberg with AWS Glue streamlines data lake management on S3. Configuring client properties allows shared credentials across Glue and S3 to define schemas, create tables, and query Iceberg metadata directly using DuckDB.

Data teams can query Snowflake tables locally in DuckDB using Arrow Database Connectivity (ADBC) without managing Iceberg tables. By avoiding row-column serialization found in ODBC, ADBC keeps data columnar throughout transfer, speeding up data movement by up to 38x.

Greybeam transpiles Snowflake queries to DuckDB via SQLGlot, but anonymous bind variables lose their lexical order in the AST. To preserve ordering, they intercept the token stream before AST parsing, rewriting anonymous placeholders into prefixed, named bind variables for DuckDB execution.

AI agents querying revenue data often fail on fan-out joins, time zones, and test accounts despite valid SQL. Test them against frozen golden datasets, comparing output values within tolerances rather than syntax. Build an exam on reconciled numbers to catch semantic modeling errors early.

SignalWeave normalizes heterogeneous AI telemetry from tools like LangChain and OpenTelemetry into canonical signals. Using standard Python, it processes JSON/JSONL via CLI pipes to aggregate reliability, quality, and cost metrics by run ID.

DuckDB 2.0.0 adds a Python appender that buffers rows generated on the fly and flushes them in a single batch operation for faster ingestion. The release also adds secure views to prevent data leakage via query plans, asynchronous I/O, and a first-class VARIANT type for semi-structured data.

TELIXO is a real-time M-Pesa transaction streaming pipeline designed to ingest, process, and analyze fintech data via event-driven architecture. It leverages Docker Compose across multi-language runtimes, including Python and Java, to maintain scalable stream processing and data pipelines.

BigQuery Python user-defined functions now support volume mounts in preview. This feature allows data pipelines to access dependency files stored directly in Cloud Storage within Python UDF code.

DuckDB 2.0.0 adds a Python appender that collects rows and writes them together in a single operation to optimize ingestion pipelines. Other data engineering updates include PostgreSQL-style SHOW for settings, secure views to prevent plan leakage, and setval to reset sequences after bulk loading.

A successful pipeline status does not guarantee data correctness. Total record counts only show net changes and mask discrepancies. Engineers should use bidirectional set comparisons with MINUS or EXCEPT on unique identifiers to detect both unexpected additions and missing records.

DuckDB and Polars outperform pandas on 1.14 billion rows. DuckDB was the only engine that finished every query, capping memory at 0.45 GB by querying Parquet directly.

Databricks ingestion should prioritize ownership over tool names. Teams risk custom pipeline chaos by source forty. Evaluate options sequentially: query in place via Lakehouse Federation or Delta Sharing, leverage existing tools like Fivetran, or use native serverless Lakeflow Connect connectors.

Apache Airflow released Common AI provider 1.0, compatible with Airflow 2.11+ and 3.3. It enables LLM agents in DAGs via the @task.agent decorator, integrating toolsets like SQLToolset for read-only warehouse queries with table-level guardrails, sandboxed execution, and human-in-the-loop approvals.

Snowpipe Streaming achieved 41.8 GB/s and 133.5M rows/sec using Elastic Channels and 455 EKS worker pods. Bypassing Python serialization via a native Rust NDJSON encoder kept ingest-to-visible latency under 30 seconds.

BigQuery supports Python UDF logging to improve data pipeline observability. Engineers can capture runtime logs directly within user-defined functions, simplifying monitoring and debugging across analytics workloads.

Snowpipe Streaming with Elastic Channels achieved 41.8 GB/s (133.5M rows/s) into a Snowflake FDN table using 455 Kubernetes workers. Leveraging Rust-based NDJSON encoding and server-side PIPE objects offloads client compute, enabling linear scaling for high-throughput streaming ingestion pipelines.

Microsoft sqlsimtools provides open-source load testing and benchmarking tools for SQL Server, Azure SQL, and Fabric databases. It includes native and Python workload engines (sqlsim, sqlpysim) and a local browser studio for repeatable performance experiments requiring ODBC Driver 18.