End to End Data Engineering Project using Databricks Free Edition | Spark Declarative Pipelines
codebasics
86,015 views
In this Databricks tutorial, you will learn Lakeflow Spark Declarative Pipelines and build a real end-to-end project in the transportation domain that you can confidently put on your resume. This Databricks micro Course explains everything in a simple, intuitive way!
Signup to Databricks Free Edition: https://bit.ly/4nK0NTN
Code, Dataset, other resources: https://resources.codebasics.io/AvKXOB
The Certification link: https://resources.codebasics.io/RuTWFY
⭐️ Timestamps ⭐️
00:00:00 Intro
00:01:20 Project Introduction
00:03:30 Stakeholder Discussion
00:05:32 Technical Architecture & Data Understanding
00:08:25 Databricks Free Edition
00:09:10 Create Catalog and Schemas
00:11:41 S3 Bucket Setup Connecting with Databricks
00:16:41 Declarative vs Imperative Programming
00:18:19 City Table: Bronze Layer
00:31:20 City Table: Silver Layer
00:35:53 SDP Key Concepts
00:43:25 Calendar Table: Silver Layer
00:51:03 Trips Table: Bronze Layer
01:10:53 Generate BI Ready Data in Gold Layer
01:19:46 Incremental Load
01:25:20 Access Management
01:29:58 LDSP Benefits
01:32:42 Stakeholder Overview
01:34:35 What's Next
Do you want to learn technology from me? Check https://codebasics.io/?utm_source=des... for my affordable video courses.
Need help building software or data analytics/AI solutions? My company https://www.atliq.com/ can help. Click on the Contact button on that website.
🎥 Codebasics Hindi channel: @codebasicshindi
#️⃣ Social Media #️⃣
🧑🤝🧑 Discord for Community Support: Discord: discord
📸 Codebasics' Instagram: Instagram: codebasicshub
📝 Codebasics' Linkedin : LinkedIn: codebasics
------
📝 Dhaval's Linkedin : LinkedIn: dhavalsays
📝 Hem's Linkedin: LinkedIn: hemvad
📽️ Hem's Instagram for daily tips: Instagram: hemvadivel
📸 Dhaval's Personal Instagram: Instagram: dhavalsays
🔗 Patreon: https://www.patreon.com/codebasics?fa...
20 Comments
End to End Data Engineering Project using Databricks Free Edition | Spark Declarative Pipelines
codebasics
86,015 views
In this Databricks tutorial, you will learn Lakeflow Spark Declarative Pipelines and build a real end-to-end project in the transportation domain that you can confidently put on your resume. This Databricks micro Course explains everything in a simple, intuitive way!
Signup to Databricks Free Edition: https://bit.ly/4nK0NTN
Code, Dataset, other resources: https://resources.codebasics.io/AvKXOB
The Certification link: https://resources.codebasics.io/RuTWFY
⭐️ Timestamps ⭐️
00:00:00 Intro
00:01:20 Project Introduction
00:03:30 Stakeholder Discussion
00:05:32 Technical Architecture & Data Understanding
00:08:25 Databricks Free Edition
00:09:10 Create Catalog and Schemas
00:11:41 S3 Bucket Setup Connecting with Databricks
00:16:41 Declarative vs Imperative Programming
00:18:19 City Table: Bronze Layer
00:31:20 City Table: Silver Layer
00:35:53 SDP Key Concepts
00:43:25 Calendar Table: Silver Layer
00:51:03 Trips Table: Bronze Layer
01:10:53 Generate BI Ready Data in Gold Layer
01:19:46 Incremental Load
01:25:20 Access Management
01:29:58 LDSP Benefits
01:32:42 Stakeholder Overview
01:34:35 What's Next
Do you want to learn technology from me? Check https://codebasics.io/?utm_source=des... for my affordable video courses.
Need help building software or data analytics/AI solutions? My company https://www.atliq.com/ can help. Click on the Contact button on that website.
🎥 Codebasics Hindi channel: @codebasicshindi
#️⃣ Social Media #️⃣
🧑🤝🧑 Discord for Community Support: Discord: discord
📸 Codebasics' Instagram: Instagram: codebasicshub
📝 Codebasics' Linkedin : LinkedIn: codebasics
------
📝 Dhaval's Linkedin : LinkedIn: dhavalsays
📝 Hem's Linkedin: LinkedIn: hemvad
📽️ Hem's Instagram for daily tips: Instagram: hemvadivel
📸 Dhaval's Personal Instagram: Instagram: dhavalsays
🔗 Patreon: https://www.patreon.com/codebasics?fa...
after extensive market research with live problem-solving webinars and
a job-ready engine. Check it out 👇
https://codebasics.io/bootcamps/data-engineering-bootcamp-with-virtual-internship