BigQuery resources for data engineers

BigQuery now supports retrieving query results directly in the Apache Arrow format in preview. This improves data pipelines by offering faster retrieval speeds and enhanced serialization efficiency when extracting analytical query outputs.

BigQuery measures has reached general availability to unify business intelligence metrics and knowledge graphs. This capability enhances data analytics architectures by standardizing metric definitions and integrating graph structures to boost accuracy for AI workloads.

Poorly written SQL queries drive up BigQuery costs. The open-source BigQuery FinOps Agent uses a multi-agent workflow to collect table metadata, refactor code, and validate output schemas. Using dry-runs, it cuts processed bytes while ensuring the behavior remains identical.

BigQuery lets data teams call LLMs in SQL queries via AI functions instead of Python APIs. Integrating calls like AI.GENERATE into dbt incremental models prevents expensive full-refresh recomputation. Output schemas force typed JSON, enabling validation with standard dbt tests.

ADBC standardizes database access by returning Apache Arrow columnar data to bypass row-oriented protocol bottlenecks. While native engines like DuckDB enable zero-copy transfers, drivers face severe type-mapping mismatches across engines like Postgres, BigQuery, and Spark. Test schemas early.

BigQuery's Data Engineering Agent now integrates with Google Drive. Generally available, it lets you supply Google Sheets, CSV, or JSONL files directly as input for data engineering pipelines.

BigQuery Python user-defined functions now support volume mounts in preview. This feature allows data pipelines to access dependency files stored directly in Cloud Storage within Python UDF code.

Virgin Media O2 reduced BigQuery compute costs by roughly 30% and build times by 35% using dbt v2 and dbt State. By enabling state-aware builds, their pipeline skipped unchanged models, reducing slot consumption from 4,500 to 3,400 per hour on a massive 1,400-model DAG.

BigQuery now offers the generally available AI.TRANSLATE function to translate text strings between languages directly in SQL pipelines. It supports standard Neural Machine Translation workloads and Translation LLM capabilities for data processing.

BigQuery has made AI.CHUNK_DOCUMENT generally available. The table-valued function splits unstructured documents into semantically coherent text chunks optimized for vector embeddings and Retrieval-Augmented Generation workflows directly within SQL.

Google Cloud announced the general availability of BigQuery ObjectRef. It lets data engineers analyze unstructured data directly within the warehouse, enabling multimodal data platforms and conversational analytics workflows implemented entirely using SQL.

BigQuery supports Python UDF logging to improve data pipeline observability. Engineers can capture runtime logs directly within user-defined functions, simplifying monitoring and debugging across analytics workloads.

Google BigQuery has expanded availability for conversational analytics. Data engineers can now deploy conversational analytics workloads in Canadian cloud regions, specifically northamerica-northeast1 in Montréal and northamerica-northeast2 in Toronto.

BigQuery now supports cross-region table copies in GA. GoogleSQL also adds GA support for column aliases, allowing queries to reference aliases directly in the same SELECT or WHERE clause to streamline migrations from Snowflake and Teradata.

ClickHouse 26.9 introduces AFTER and UNTIL boundary conditions to the LIMIT clause to simplify log analysis pipelines. The release also adds incremental refresh for append-only materialized views, disk spilling for DISTINCT operations, and faster reads for the TimeSeries engine.

BigQuery now generally supports publishing Conversational Analytics agents in Gemini Enterprise. Additionally, the AI.AGG function reached GA with an output_schema argument, allowing data engineers to explicitly define function output schemas via comma-separated field lists.

BigQuery introduces cost management features for generative AI functions, enabling data engineers to maintain predictable AI spend within their data analytics workflows.

BigQuery has made the AI.AGG function generally available. Data engineers can use this BigQuery ML function to semantically aggregate unstructured input data using natural language instructions.

🎙️ The Data Flowcast by Astronomer — Credit Karma uses Apache Airflow and BigQuery to manage credit data ingestion and PII governance. Using DAG Factory for YAML-based pipeline generation, the platform standardizes BigQuery quality checks across thousands of DAGs, halting ETL runs and alerting on failures.

🎙️ The Data Flowcast by Astronomer — Skimlinks orchestrates its BigQuery-to-Druid pipeline using Airflow. To eliminate sensor polling overhead, the team migrated from external task sensors to event-driven Assets. They also built a YAML-based DAG factory validated by Pydantic to catch configuration errors at parse time.

BigQuery Data Transfer Service now generally supports incremental data ingestion from PostgreSQL and ServiceNow. Data engineers can automate continuous pipeline updates without running full table reloads, optimizing ingestion workloads.

Google BigQuery introduces TimesFM 3.0, enabling multivariate time-series forecasting directly within the data warehouse. This release expands SQL-driven predictive modeling capabilities for complex analytics pipelines.

BigQuery Data Engineering Agent is generally available in the asia-northeast1 (Tokyo) regional endpoint. The release supports the gemini-3.5-flash model for regional data processing tasks.

Alteryx integrates with Google Cloud BigQuery to modernize unstructured data workflows using Alteryx Live Query. This architecture enables data engineers to process analytics workloads directly within BigQuery environments.

Gemini assistance in the BigQuery Studio SQL editor is now generally available. Data engineers can access AI features directly via an inline action button after selecting query text to streamline SQL authoring and data transformation workflows.

ClickHouse Cloud delivered over 400x better performance per dollar than BigQuery during concurrent real-time ingestion and querying. Benchmarking 113.2 billion rows showed its columnar storage, ordering, and native pre-aggregations minimized data preparation overhead and reduced query runtime.

BigQuery now offers TimesFM 3.0 in preview for built-in time series forecasting. Data engineers can use SQL functions like AI.FORECAST for multivariate forecasting with covariates, AI.EVALUATE for univariate model evaluation, and AI.DETECT_ANOMALIES to run univariate anomaly detection pipelines

Conversational analytics in BigQuery now supports the AI.CAUSAL_EFFECT function in preview. This feature allows data teams to quantify the impact of specific interventions on time series data directly using SQL workflows.

BigQuery now supports flexible column names by default on Iceberg external and managed tables. Additionally, the gemini-embedding-2 model is generally available for AI.EMBED, AI.SIMILARITY, and AI.GENERATE_EMBEDDING functions, enabling multimodal embeddings across text, audio, and video.

The Rust SDK for BigQuery is now generally available. Additionally, the Data Engineering Agent supports the gemini-3.7-flash model across us, eu, and global multi-regional endpoints.