dbt on Databricks: Best Practices for Production Data Pipelines
Databricks Events
780 views
Databricks is one of the best platforms for building data pipelines with dbt because it unifies ingestion, transformation, orchestration, and governance on a single open lakehouse — eliminating the hand-offs and brittle toolchains that stall analytics engineering in production.
In this video, Srilekha Dornadula and Ramiz Bozai cover the full dbt lifecycle on Databricks: building a medallion architecture in Databricks SQL (which offers up to 5x lower TCO than alternatives), extending dbt ingestion with Lakeflow Connect for streaming tables, orchestrating end-to-end pipelines with Lakeflow Jobs, and governing data with Unity Catalog tags, lineage, and attribute-based access control. You will see how materialized views cut incremental refresh overhead, how dbt semantic layer metrics integrate with AI/BI Genie, and how cost attribution ties dbt model spend back to the teams running it. For teams evaluating data platform migration to Databricks, the video closes with a two-step migration pattern and partner accelerators.
Talk By: Ramiz Bozai, Solutions Architect, Databricks; Srilekha Dornadula, Product Manager, Databricks
00:00 - Introduction: getting the most from dbt on Databricks
01:00 - Goals: fast, cost-efficient, and unified dbt pipelines
02:05 - dbt in the full data pipeline: ingest, transform, and serve
04:05 - Why Databricks SQL? Price-performance and open format governance
06:45 - The status quo problem: brittle orchestration and siloed metrics
08:03 - Building a dbt medallion architecture end to end
11:10 - Extending dbt for ingestion with Lakeflow Connect
17:05 - Orchestrating dbt with Lakeflow Jobs: unified pipelines
20:01 - Demo: creating and deploying dbt jobs in Databricks
24:02 - Materialized views and performance optimization strategies
27:06 - Governance with Unity Catalog: tags, lineage, and ABAC
33:02 - Migrating dbt to Databricks: a two-step approach
🤝 Stay connected:
Website: https://databricks.com
Twitter: https://x.com/databricks/
LinkedIn: LinkedIn: databricks
Instagram: Instagram: databricksinc
Facebook: Facebook: databricksinc
dbt on Databricks: Best Practices for Production Data Pipelines
Databricks Events
780 views
Databricks is one of the best platforms for building data pipelines with dbt because it unifies ingestion, transformation, orchestration, and governance on a single open lakehouse — eliminating the hand-offs and brittle toolchains that stall analytics engineering in production.
In this video, Srilekha Dornadula and Ramiz Bozai cover the full dbt lifecycle on Databricks: building a medallion architecture in Databricks SQL (which offers up to 5x lower TCO than alternatives), extending dbt ingestion with Lakeflow Connect for streaming tables, orchestrating end-to-end pipelines with Lakeflow Jobs, and governing data with Unity Catalog tags, lineage, and attribute-based access control. You will see how materialized views cut incremental refresh overhead, how dbt semantic layer metrics integrate with AI/BI Genie, and how cost attribution ties dbt model spend back to the teams running it. For teams evaluating data platform migration to Databricks, the video closes with a two-step migration pattern and partner accelerators.
Talk By: Ramiz Bozai, Solutions Architect, Databricks; Srilekha Dornadula, Product Manager, Databricks
00:00 - Introduction: getting the most from dbt on Databricks
01:00 - Goals: fast, cost-efficient, and unified dbt pipelines
02:05 - dbt in the full data pipeline: ingest, transform, and serve
04:05 - Why Databricks SQL? Price-performance and open format governance
06:45 - The status quo problem: brittle orchestration and siloed metrics
08:03 - Building a dbt medallion architecture end to end
11:10 - Extending dbt for ingestion with Lakeflow Connect
17:05 - Orchestrating dbt with Lakeflow Jobs: unified pipelines
20:01 - Demo: creating and deploying dbt jobs in Databricks
24:02 - Materialized views and performance optimization strategies
27:06 - Governance with Unity Catalog: tags, lineage, and ABAC
33:02 - Migrating dbt to Databricks: a two-step approach
🤝 Stay connected:
Website: https://databricks.com
Twitter: https://x.com/databricks/
LinkedIn: LinkedIn: databricks
Instagram: Instagram: databricksinc
Facebook: Facebook: databricksinc