Analytical databases resources for data engineers

Snowflake DCM Projects provide native declarative infrastructure-as-code using SQL syntax. Users define objects using DEFINE instead of CREATE, following a plan-and-deploy loop where state is tracked in Snowflake. It pairs with inherited grants to replace complex future grant configurations.

Replicate DynamoDB to Apache Iceberg on Amazon S3 Tables via Glue zero-ETL without impacting operational traffic. An AWS Lambda function embedding DuckDB executes fast vectorized SQL directly against S3 Tables metadata and Parquet files, returning JSON query results serverlessly.

ClickHouse Cloud introduced a TimeSeries table engine to serve as a Prometheus replacement. Data engineers can ingest metrics via Prometheus remote write and run native PromQL directly in ClickHouse, Grafana, or ClickStack without rewriting queries into SQL.

ClickHouse Cloud introduced replica-aware routing to direct requests to the same replica. This ensures access to temporary tables and enables read-after-write consistency. Clients route traffic by setting an HTTP header or overriding the TLS SNI value to target the identical host.

Choosing the right database depends on access patterns and scale. While Postgres handles typical OLTP workloads, OLAP engines like DuckDB use columnar disk layouts to cut latency on heavy aggregations across millions of rows. SQLite fits single-file apps, but lacks concurrent write scaling.

Snowflake announced general availability for inherited grants and container-level MANAGE GRANTS privileges. This update simplifies access control architectures by streamlining permission inheritance across data containers. Other features include dbt project Slim CI and Iceberg Scan Plan API.

BigQuery Graph metadata is now automatically ingested and searchable in Knowledge Catalog. This metadata cataloging feature is currently available in preview, improving data discovery and governance across graph architectures.

Snowflake launched Data Connectivity Proxy (DCP) for Openflow, allowing secure ingestion from private databases and Kafka brokers. Delivered as a Docker agent, it uses outbound TLS on port 443 with mTLS tunnels to bypass inbound firewall rules without inspecting data or storing source credentials.

BigQuery introduces agent-ready augmented analytics utilizing table-valued functions (TVFs). This feature enables data engineering workflows to integrate analytical insights and automated data processing directly within BigQuery data pipelines.

Snowflake introduced multimodal AI_SUMMARIZE for automatic theme summarization in public preview. Concurrently, interactive warehouses gained general availability for zero-copy support across all table formats, streamlining data pipeline operations.

Snowflake announced the general availability of zero-copy support for all table formats in interactive warehouses. Data engineers can leverage metadata-driven cloning across formats, while additional platform updates include the Iceberg Scan Plan API and online constraint management for hybrid

Using DuckDB CLI with AI agents streamlines analytics by treating terminal tools as localized MCP servers. With clean upstream gold layer tables, semantic layers add marginal gains. DuckDB provides a lightweight, local analytical engine that AI models interact with natively to query data quickly.

Predicate pushdown cannot be taken for granted across modern data platforms. While Snowflake pushes filters past deterministic aggregates like MAX, it fails on non-deterministic expressions such as ARRAY_AGG.

Build full data infrastructure to capture, store, transform, and serve analytical data directly in apps without external dependencies.

Ingestr extracts Chargebee records to BigQuery raw staging, while Bruin transforms them into typed SQL models and insert-only daily snapshots. This pipeline normalizes mutable subscription states, unrolls items into numeric minor units, and tracks MRR retention movements.

Snowflake has expanded second generation Openflow deployments and runtimes to GCP following general availability. This architectural update modernizes pipeline execution after Openflow gen 1 deployment creation was retired.

ClickHouse introduced native AI SQL functions, enabling direct LLM and embedding calls within the analytical engine. By eliminating external data movement for tasks like classification and extraction, engineers can run vector and RAG pipelines directly inside their database queries.

Snowflake released the Iceberg Scan Plan API in Horizon Catalog in public preview. This enhances data lakehouse integration by exposing Iceberg table scan planning directly through the catalog.

Data architecture remains the primary barrier to industrial AI: 67% of data is trapped in siloed

Google BigQuery now integrates its Data Engineering Agent with BigQuery Graph, which is generally available. The integration adds context between source and destination schemas, improving automated schema mapping accuracy across data pipelines.

Bruin orchestrates near-real-time fraud monitoring using an append-only ClickHouse MergeTree log fed by PostgreSQL change capture. It aggregates events into minute, hour, and daily rollups, applying interval modifiers to cleanly restate late-arriving payment statuses without full recomputations.

Snowflake built InvoiceIQ to run accounts payable natively inside Snowpark Container Services. The pipeline uses Cortex AI functions, AIPARSEDOCUMENT, and AICOMPLETE alongside JAROWINKLERSIMILARITY to parse, enrich, and match raw invoice documents directly against enterprise tables.

Cube Store is a distributed columnar engine written in Rust for serving pre-aggregations. Using Apache Arrow and DataFusion, it maintains indexes as sorted copies of Parquet data to skip non-matching rows and run streaming aggregations directly over partitioned worker caches.

WalShadow enables sub-second Postgres replication to ClickHouse by directly consuming physical WAL streams instead of using logical replication slots. It decodes WAL off-host and writes native blocks to ClickHouse, hitting 200 ms latency and 289K rows/sec in benchmarks.

ClickHouse launched an integration for the Data agent in ChatGPT Work via a Remote MCP server. Authenticated with OAuth, the managed server exposes tools that allow agents to inspect schemas, execute SELECT queries, and construct dashboards directly from ClickHouse Cloud data.

DuckDB 2.0 boosts S3 queries via async I/O: a thread pool prefetches Parquet row groups so workers decode continuously without idle CPU time. It also overhauls recursive CTEs, indexing deep parent-child hierarchies once instead of re-reading tables each round.

ClickHouse Cloud launched On-Demand Compute to execute queries on shared worker pools outside a cluster. Powered by a cost-based optimizer and multistage query execution, it isolates heavy ad hoc queries from production workloads without waiting for cluster autoscaling.

xlDuckDb is an add-in enabling DuckDB SQL execution directly inside 64-bit Excel 365. It queries dynamic ranges and named tables using the DuckDbQuery function, letting analytical SQL run on spreadsheet data and returning results as native cells via dynamic arrays.

Databricks Photon executes Spark plans in native C++ using SIMD instructions. Because it doubles DBU consumption on classic clusters, workloads must achieve a 2x speedup to break even. Heavy scans and joins benefit, but Python UDFs and shuffles fall back to the JVM, increasing costs.

ClickHouse 26.8 LTS introduces background queries that execute independently of client connections, benefiting long-running ingest and export pipelines. The release also adds pipelined SQL, expanded data lake integrations, and query optimizations for Parquet, joins, and GROUP BY operations.