dataaaaa!

Databricks Agent Bricks manages production AI agents directly on Unity Catalog. It embeds model access, context, and governance into existing data infrastructure, solving security and deployment debt by enforcing data access via the user permissions and lineage already governing production tables.

Modern end-to-end data platforms aim to replace multi-vendor stacks of Fivetran, dbt, and Airflow. Tools like Bruin run on existing warehouses, combining ingestion, SQL and Python transformations, quality checks, and orchestration into one dependency graph to reduce pipeline points of failure.

Optimize Databricks costs by matching workloads to compute. Use serverless for bursty tasks and classic compute with spot instances for steady ETL. Disable Photon on UDF-heavy jobs since it will not accelerate them, and tune auto-termination timeouts to eliminate expensive cluster idle time.

OpenAI launched the Data agent in ChatGPT Work, connecting directly to Snowflake, Databricks, BigQuery, and Redshift. It integrates with dbt and semantic layers to enforce data models, while respecting existing row- and column-level access controls.

Databricks Photon executes Spark plans in native C++ using SIMD instructions. Because it doubles DBU consumption on classic clusters, workloads must achieve a 2x speedup to break even. Heavy scans and joins benefit, but Python UDFs and shuffles fall back to the JVM, increasing costs.

Lift-and-shift rewrites to Databricks spike DBU costs by turning unoptimized queries into metered waste. Avoid line-by-line translation. Instead, audit portfolios, kill zombie jobs, and consolidate redundant filter variants into unified Gold tables to migrate business capabilities, not technical

Orchestra raised $3.3m to build an agentic control plane for data. Key engineering updates include Glue jobs fetching logs, task retries creating separate runs while preserving history, dbt lineage source support, and automated agents for reducing Databricks costs.

To ingest volatile Google Sheets data into Databricks, engineers built a daily pipeline using OAuth2 and tab regex filtering. It extracts full sheets into bronze string tables and typed silver Delta Lake tables using SHA-256 row fingerprints and full overwrites to handle in-place row edits.

Databricks Serverless automates infrastructure management and scales on demand, billing per DBU-second. It suits spiky, unpredictable workloads like ad hoc SQL and incremental ETL pipelines, provisioning compute in seconds with autoscaling and Photon engine optimizations enabled by default.

AWS acquired DuckLabs to influence the engine powering modern data pipelines. Acting as a cost-efficient alternative to Spark with native Parquet support, DuckDB pairs with DuckLake and the Quack protocol to anchor an S3-based, serverless lakehouse stack that competes with Snowflake and Databricks.

AI increases data pipeline complexity and volume without adding headcount. Engineers risk burnout by manually managing alerts and dependencies. To scale without human bottlenecks, teams should adopt a unified agentic control plane and serverless orchestration across compute and harness engines.

To stand out, build production data projects addressing real business problems rather than toy pipelines. One strong pattern uses a Databricks Medallion Lakehouse ingesting live REST APIs, partitioning by county FIPS, and executing vectorized spatial joins to prevent memory shuffle and compute

Orchestra added integrations for MongoDB, Dremio, MySQL, and AWS Step Functions. Power BI now supports partition-specific dataset refreshes, while anomaly detection identifies runs finishing faster than baseline. New Databricks Cost Agents help mitigate spend. Glue jobs can now fetch logs.

Databricks Genie splits costs into LLM usage and triggered compute. While LLM calls offer free allowances, triggered SQL warehouse and serverless compute lack explicit tags in billing system tables, driving hidden infrastructure spend.

QuintoAndar unified its CDP on ClickHouse Cloud to process 900M monthly events and serve 300M API requests. By replacing a Lambda architecture and using ClickHouse Managed Postgres as an interface, they made real-time analytical data accessible to tools requiring Postgres compatibility.

Databricks tags Genie LLM usage directly, but triggered compute lacks native attribution in billing logs. To track costs without complex log correlation, data engineers should isolate workloads onto dedicated SQL warehouses with upfront resource tags.

Databricks split Genie into four products. Genie Code builds pipelines, debugs, and integrates with Unity Catalog. Genie Ontology automates knowledge graphs to ground metric definitions across tables. Genie Agents store trusted business context, while Genie One serves as the user query interface.

August data engineering roundups highlight streaming, query engines, and platform patterns. Apache Fluss became a top-level project, DuckDB 2.0 was previewed, and Netflix detailed streaming joins in Flink alongside its real-time distributed graph architecture.

Databricks compute costs split into two bills: platform DBU charges and cloud infrastructure fees. Classic compute separates VM costs, while serverless bundles VM infrastructure directly into the DBU rate. Total spending requires tracking both DBUs and underlying cloud hardware.

Databricks raised new capital while Snowflake committed $6B to AWS compute spend. Orchestra added data engineering features including AWS Step Functions, SFTP loaders, dbt live alerting logs, and an EKS operator to trigger and monitor flows on Kubernetes.

Choosing Databricks for compute differs from letting it own transformation logic. Using dbt keeps transformations in open, version-controlled SQL, ensuring portable metrics and preventing vendor lock-in across warehouses while reducing compute via state-aware runs.

dbt Core v1.12 is GA, introducing an opt-in Rust parser that speeds up DAG compilation by 5-10x for large projects. It adds an on_error config to continue downstream models after upstream failures, dedicated vars.yml files, and expanded JavaScript and Python UDF support across data warehouses.

Databricks engineer Scott Haines explores transitioning from corporate IC roles to independent consulting. The session covers how autonomous coding agents are actively reshaping technical workflows, alongside the practical value of in-person community networking over cold outreach.

Mastering Databricks requires core data engineering foundations: workspaces with Git integration, serverless compute, and Unity Catalog governance. Engineers should leverage Delta tables with liquid clustering, CDC patterns, and MERGE INTO commands to build robust, idempotent pipelines.

Partnering with Databricks requires solving scale limits in native tools like Databricks Workflows. Orchestra integrates as a control plane, managing end-to-end pipelines across dbt, Lakeflow, and BI tools to orchestrate compute without forcing teams to maintain separate Airflow infrastructure.