Visualization resources for data engineers

dbt Charts introduces an open-source, dashboards-as-code spec designed for AI agents to write and engineers to audit. Aiming to bypass complex BI layers, it lets teams rely more on relational data architectures instead of pre-joining models solely to satisfy traditional BI tools.

🎙️ The Analytics Engineering Podcast (dbt) — dbt Charts introduces a human-readable spec for dashboards that AI can generate and data teams can audit. Dave Fowler calls it "the Markdown of dashboards" to tackle BI bottlenecks and unreviewable generated code, while exploring how AI may eventually reduce reliance on dimensional modeling.

Streamlit in Snowflake container runtime is now generally available in China. The release expands native application hosting within local Snowflake deployments, enabling data engineers to build and run containerized data apps directly alongside regional data pipelines and storage.

Nao launched a cloud UI for its open-source agentic BI builder. It syncs warehouse metadata across systems like Snowflake and ClickHouse to build a governed context layer of schemas, metrics, and rules. It features offline testing and MCP integration to serve governed data directly to AI agents.

This project implements an automated ETL pipeline orchestrated with Apache Airflow. Running on Docker Compose, the DAG extracts weather metrics from the OpenWeatherMap REST API, ingests them into a PostgreSQL database, and feeds analytics dashboards built in Metabase.

Silent pipeline bugs often stem from boundary mismatches and unwritten definitions rather than crashes. A healthy job run does not guarantee correctness. Robust platforms centralize business rules across modular models and use external reconciliation checks to catch silent join drops.

Greysight is an open-source cost dashboard for Snowflake that queries ACCOUNTUSAGE and ORGANIZATIONUSAGE metadata. It visualizes warehouse spend, tracks idle compute via QUERYATTRIBUTIONHISTORY, consolidates Cortex AI usage, and forecasts capacity balance depletion.

To cut embedded analytics costs, Headset used Greybeam as a wire-compatible proxy in front of Snowflake. The proxy evaluated Looker queries, routing eligible reads to managed DuckDB clusters and the rest to Snowflake, reducing overall compute spend by 92% without migrations.

Omni introduced viz layers for X/Y charts, letting engineers configure independent mark types, axes, and aggregations per measure. Smart aggregation automatically enables required query totals to handle varying levels of detail without custom code.

Duckpad is an open-source data transformation tool and SQL workspace powered by DuckDB. Data engineers can visually compose pipelines, integrate Git versioning, and trace cell-level data lineage back to contributing source values. SQL execution runs locally on modern Linux via Landlock.

DoorDash consolidated its analytics architecture by decoupling the semantic layer from consumption. Data stored in Apache Iceberg and Snowflake is modeled via dbt metrics, serving ThoughtSpot and AI agents centrally. DoorDash treats semantic definitions as version-controlled code.

dbt Labs launched dbt Charts, an open-source charting language configured via YAML that lives directly in repositories. Designed around an underlying grammar, it automatically adjusts axis label density and flags charts that are too dense to render legibly before they reach production.

dbt Labs launched dbt Charts, an open-source YAML language for defining dashboards alongside dbt models. Stored in a charts directory, configurations run through standard git workflows and tests. It operates via the dct CLI, integrating analytics engineering and BI into unified deployments.

As AI agents write more SQL, data bottlenecks shift from query creation to verification. Robust pipelines require test-driven approaches with automated linting, whole-tree compiler checks across data lineage, and iterative test suites to validate complex metrics before deployment.

dbt Charts Cloud supports public dashboards defined in versioned YAML, modeled via dbt. Access requires dual-level opt-in controls across organization and project settings, letting external users view curated analytics without logins while admins retain kill-switch controls.

To debug data errors across BI and warehouse layers, an AI agent integrates three MCP servers: Looker, Databricks SQL, and dbt. It inspects compiled SQL and lineage to trace a dashboard metric anomaly down to underlying transformation logic, attaching QA validation queries in minutes.

Compare the Market shifted from a 4,000-model dbt monorepo to a multi-repo dbt Mesh pattern. Transitioning to domain ownership and tested data models cut report delivery from six weeks to under one week, raised test coverage past 75%, and increased model reuse to 3.0x.

Data platform engineers replaced costly Looker instances with an internal service hosting LLM-generated HTML in S3 on Kubernetes. The architecture queries Snowflake directly via user OAuth tokens, delegating data governance and access controls to existing Snowflake roles and upstream dbt models.

AI data agents struggle with complex analytics across disparate databases and unstructured text. In the Data Agent Bench, top agents achieved only a 61% pass@1 rate. Adding a semantic layer significantly improves query accuracy, reinforcing the need for curated data modeling over raw text-to-SQL.

DuckDB and Polars outperform pandas on 1.14 billion rows. DuckDB was the only engine that finished every query, capping memory at 0.45 GB by querying Parquet directly.

Daylight saving time shifts local midnight, turning 24/7 days into 23- or 25-hour windows that distort metrics and anomaly models. Avoid hardcoded offsets and truncate only after converting to local time. To normalize comparisons, join against a calendar table tracking each day's exact duration.

Brilliant Earth deployed Posit Team as a Snowflake Native App on Snowpark Container Services, unifying governance and user permissions directly in Snowflake.

To build a dynamic SaaS demo pipeline, Cursor was used to generate synthetic data with Python, build staging and mart models via dbt to keep historical tables fresh, and configure data modeling in Omni. Cursor streamlined context switching across data generation, dbt transformations, and UI code.

pm-watcher aggregates cross-platform pricing pipelines across five prediction markets for real-time monitoring. The read-only system ingests data from Polymarket, Kalshi, and others, tracking divergence and archiving historical state data for post-event probability calibration analysis.

Langfuse traces Cursor agent model calls and tool executions using ClickHouse as its analytical backend. In Langfuse v4, a wide ClickHouse observations table stores session context with each record, avoiding costly table joins when querying aggregates for latency, cost, and usage metrics.

Fivetran Skills provides Claude with schema context and dbt models to query synced warehouse data directly. Every answer comes from an executable SQL query against Snowflake, BigQuery, or Databricks, enabling natural language troubleshooting without altering warehouse architecture.

Metabase replaced multi-query pivot tables with SQL GROUPING SETS, executing subtotals and details in a single query. Instead of dispatching up to 32 separate queries, it computes entire pivots in one pass, cutting round trips and boosting benchmark performance by 2-3x while ensuring consistent

Metabase optimized pivot tables by replacing multi-query generation with SQL GROUPING SETS. Instead of running up to 32 separate queries to compute subtotals, a single pushdown query calculates the entire pivot directly in the warehouse, cutting latency 2-3x and guaranteeing consistency.

This Claude Code skill automates dashboard development for analytics and metrics UI. Distilled from 127 production systems, it guides agents to build KPI tiles, sparklines, and date filters, finishing with a screenshot QA pass to validate metrics and catch visual data discrepancies.

Optimize Databricks Serverless SQL Warehouse costs by targeting the right control: auto-stop cuts idle time, warehouse size boosts single-query compute, and max clusters handles concurrency.