BigQuery resources for data engineers

BigQuery now offers TimesFM 3.0 in preview for built-in time series forecasting. Data engineers can use SQL functions like AI.FORECAST for multivariate forecasting with covariates, AI.EVALUATE for univariate model evaluation, and AI.DETECT_ANOMALIES to run univariate anomaly detection pipelines

Conversational analytics in BigQuery now supports the AI.CAUSAL_EFFECT function in preview. This feature allows data teams to quantify the impact of specific interventions on time series data directly using SQL workflows.

BigQuery now supports flexible column names by default on Iceberg external and managed tables. Additionally, the gemini-embedding-2 model is generally available for AI.EMBED, AI.SIMILARITY, and AI.GENERATE_EMBEDDING functions, enabling multimodal embeddings across text, audio, and video.

The Rust SDK for BigQuery is now generally available. Additionally, the Data Engineering Agent supports the gemini-3.7-flash model across us, eu, and global multi-regional endpoints.

To connect AI agents to QuickBooks, split reads from writes. Bruin and ingestr load QuickBooks data into BigQuery or DuckDB to model metrics via SQL, while Intuit's MCP server handles writes. This architecture avoids API rate limits and ensures agents query reliable, pre-reconciled data.

BigQuery pipelines now support automated metadata enrichment and Knowledge Catalog data quality scorecard integration. Additionally, the Data Engineering Agent can proactively generate semantic metadata for pipeline assets. These features are generally available.

BigQuery launched the Security center in GA to manage row- and column-level security policies and governance tags. Additionally, query execution plans now display steps for UPDATE, DELETE, MERGE, and EXPORT operations, improving performance optimization for DML and data export pipelines.

BigQuery released in Preview the INFORMATIONSCHEMA.FAILOVERHISTORY view. Data engineers can query it to retrieve a near real-time list of failover events for reservations within an administration project that use managed disaster recovery.

Event-driven S3 to BigQuery pipelines break when spreadsheets shift schemas. Standard Airflow setups rely on human alerts because retries cannot fix structural file changes.

BigQuery continuous queries now support writing output directly into Apache Iceberg managed tables using INSERT DML statements, enabling real-time streaming into open-format lakehouses. Additionally, the BigQuery Data Transfer Service MCP server is generally available.

BigQuery Studio query editor now keeps row numbers visible on the left when scrolling horizontally through the Query results table. Now generally available, this UI update helps data engineers track records and maintain context when inspecting wide schema tables.

BigQuery FinOps requires aligning query performance with pricing models. Compute costs stem from query volume, unit pricing, slot reservations, and contracts. For data engineers, table partitioning and clustering provide essential query-level optimization by pruning data before burning slots.

BigQuery now allows publishing data agents to Gemini Enterprise via Agent Registry using default Google-managed credentials. Colocating BigQuery and Gemini Enterprise in the same project and matching Agent Gateway region removes the need to manually copy A2A JSON cards or configure OAuth

While open table formats like Iceberg decouple compute and storage, connecting client apps to diverse engines remains fractured. Apache Arrow and ADBC solve this by providing a standardized columnar API, eliminating costly row-to-column conversions and yielding up to 38x transfer speedups over ODBC.

BigQuery has expanded its built-in generative AI functions to support the gemini-3.8-flash model. This update enables data engineers to integrate the faster Gemini model directly into SQL pipelines, streamlining in-database text generation, enrichment, and LLM inference workflows.

Evaluate ingestion tools by failure modes, not connector counts. Use CDC for large mutable tables needing delete capture, and incremental batching for append-only data. Destination write semantics, rate limits, schema evolution, and merge keys dictate pipeline stability.

This session covers modern data engineering architectures on Google Cloud Platform, focusing on scalable analytics, data lake design, and real-time data streaming.

Google Cloud announced the preview of cross-cloud caching for BigQuery to advance borderless lakehouse architectures. The feature targets improved multi-cloud query performance and data analytics workflows across distributed cloud environments.

BigQuery now supports generally available unit tests for Dataform pipelines. Data engineers can validate SQL transformation logic directly against mock datasets to improve pipeline reliability and test data transformations before production runs.

ChatGPT connects to warehouses like Snowflake, BigQuery, Databricks, and Redshift to generate dashboards. It reads metrics via dbt or semantic layers and respects warehouse permissions. It requires solid engineering foundations, like unified data definitions and access controls, to avoid errors.

dbt State is generally available, shifting pipelines from rigid schedules to running on change. It inspects model SQL and warehouse metadata to build, skip, clone, or defer nodes automatically. By eliminating redundant builds, early adopters cut warehouse compute costs by 15-30%.

BigQuery introduced a migration lineage service in Preview. It helps data engineers visualize data flows and dependencies in source databases to plan warehouse migrations.

BigQuery Graph metadata is now automatically ingested and searchable in Knowledge Catalog. This metadata cataloging feature is currently available in preview, improving data discovery and governance across graph architectures.

BigQuery introduces agent-ready augmented analytics utilizing table-valued functions (TVFs). This feature enables data engineering workflows to integrate analytical insights and automated data processing directly within BigQuery data pipelines.

Ingestr extracts Chargebee records to BigQuery raw staging, while Bruin transforms them into typed SQL models and insert-only daily snapshots. This pipeline normalizes mutable subscription states, unrolls items into numeric minor units, and tracks MRR retention movements.

Google BigQuery now integrates its Data Engineering Agent with BigQuery Graph, which is generally available. The integration adds context between source and destination schemas, improving automated schema mapping accuracy across data pipelines.

BigQuery generative AI functions now support new Gemini models: gemini-3.5-flash-lite, gemini-3.6-flash, and gemini-3.7-flash. This update expands built-in SQL AI capabilities for analytical workflows.

BigQuery conversational analytics now supports predictive modeling questions using the AI.PREDICT function. Currently in Preview, this allows data engineers to integrate BigQuery ML predictions directly through conversational queries.

Manual reporting breaks as ecommerce scales because transactional data stays isolated from ad platforms. Engineering automated ELT pipelines into a cloud data warehouse unifies Shopify, web, and ad sources, enabling reliable multi-touch attribution and full customer lifecycle analysis.

CostBench benchmarked real-time performance per dollar across ClickHouse Cloud, Snowflake, BigQuery, and Redshift Serverless under sustained loads. Streaming 100 billion rows at 1M rows/s, ClickHouse Cloud achieved the lowest ingestion and query serving costs, outperforming alternatives by