End to End Data Engineering Project using Databricks Free Edition | FMCG Domain
codebasics
196,698 views
This is an end end to data engineering project using databricks free edition that is perfect for beginners as well as advanced users. In FMCG domain, we are building an ETL pipeline for a real life industry use case where a big retail company acquires a small one and we will build the data pipeline to consolidate data from both the companies in a single lakehouse architecture.
The tech stack data engineering project is: Python, SQL, Amazon S3, Medallion architecture, Spark, BI dashboard and Genie.
Signup to Databricks Free Edition: https://bit.ly/4nK0NTN
Databricks foundations video: Databricks Tutorial | Databricks Free Edit...
Code, dataset and other resources: https://resources.codebasics.io/pD0lLR
🚀 Ready to go from Data Analyst to AI-Enabled Data Engineer? Our live bootcamp takes you from dashboards to production pipelines: https://resources.codebasics.io/5yaUOR
🎮 Want to learn by doing? Datakata is a free, super fun game where you build data pipelines hands-on: https://resources.codebasics.io/Ahkp3O
Get Certified by Codebasics with this Databricks Mini Course : https://resources.codebasics.io/dgbaAX
⭐️ Timestamps ⭐️
00:00:00 Intro
00:00:57 Project Introduction
00:02:57 Stakeholder Discussion
00:04:22 2 Learning Paths
00:04:40 Tech Architecture
00:09:47 Data Model
00:16:42 Signup to Databricks free edition
00:17:26 Setup Catalog. Data Import
00:27:36 AWS Account Setup
00:30:48 Upload Data to S3
00:32:50 Databricks S3 Connection
00:34:34 Dimension Data Processing
01:40:23 Fact Processing - Historical Load
02:02:11 Fact Processing - Incremental Load
02:25:51 Orchestration
02:38:07 Denormalized Table and Genie
02:43:24 Dashboarding
02:54:46 Stakeholder Review Meeting
02:55:55 Conclusion
Do you want to learn technology from me? Check https://codebasics.io/?utm_source=des... for my affordable video courses.
Need help building software or data analytics/AI solutions? My company https://www.atliq.com/ can help. Click on the Contact button on that website.
🎥 Codebasics Hindi channel: @codebasicshindi
#️⃣ Social Media #️⃣
🧑🤝🧑 Discord for Community Support: Discord: discord
📸 Codebasics' Instagram: Instagram: codebasicshub
📝 Codebasics' Linkedin : LinkedIn: codebasics
------
📝 Dhaval's Linkedin : LinkedIn: dhavalsays
📝 Hem's Linkedin: LinkedIn: hemvad
📽️ Hem's Instagram for daily tips: Instagram: hemvadivel
📸 Dhaval's Personal Instagram: Instagram: dhavalsays
🔗 Patreon: https://www.patreon.com/codebasics?fa...
20 Comments
practice it manually and learn through llm and gain knowledge overall a good project video(take your time it took me around 1 week to completely understand everything )
Bro, your course feels like AWS billing—it starts free and ends up costing us patience, sanity, and respect.
End to End Data Engineering Project using Databricks Free Edition | FMCG Domain
codebasics
196,698 views
This is an end end to data engineering project using databricks free edition that is perfect for beginners as well as advanced users. In FMCG domain, we are building an ETL pipeline for a real life industry use case where a big retail company acquires a small one and we will build the data pipeline to consolidate data from both the companies in a single lakehouse architecture.
The tech stack data engineering project is: Python, SQL, Amazon S3, Medallion architecture, Spark, BI dashboard and Genie.
Signup to Databricks Free Edition: https://bit.ly/4nK0NTN
Databricks foundations video: Databricks Tutorial | Databricks Free Edit...
Code, dataset and other resources: https://resources.codebasics.io/pD0lLR
🚀 Ready to go from Data Analyst to AI-Enabled Data Engineer? Our live bootcamp takes you from dashboards to production pipelines: https://resources.codebasics.io/5yaUOR
🎮 Want to learn by doing? Datakata is a free, super fun game where you build data pipelines hands-on: https://resources.codebasics.io/Ahkp3O
Get Certified by Codebasics with this Databricks Mini Course : https://resources.codebasics.io/dgbaAX
⭐️ Timestamps ⭐️
00:00:00 Intro
00:00:57 Project Introduction
00:02:57 Stakeholder Discussion
00:04:22 2 Learning Paths
00:04:40 Tech Architecture
00:09:47 Data Model
00:16:42 Signup to Databricks free edition
00:17:26 Setup Catalog. Data Import
00:27:36 AWS Account Setup
00:30:48 Upload Data to S3
00:32:50 Databricks S3 Connection
00:34:34 Dimension Data Processing
01:40:23 Fact Processing - Historical Load
02:02:11 Fact Processing - Incremental Load
02:25:51 Orchestration
02:38:07 Denormalized Table and Genie
02:43:24 Dashboarding
02:54:46 Stakeholder Review Meeting
02:55:55 Conclusion
Do you want to learn technology from me? Check https://codebasics.io/?utm_source=des... for my affordable video courses.
Need help building software or data analytics/AI solutions? My company https://www.atliq.com/ can help. Click on the Contact button on that website.
🎥 Codebasics Hindi channel: @codebasicshindi
#️⃣ Social Media #️⃣
🧑🤝🧑 Discord for Community Support: Discord: discord
📸 Codebasics' Instagram: Instagram: codebasicshub
📝 Codebasics' Linkedin : LinkedIn: codebasics
------
📝 Dhaval's Linkedin : LinkedIn: dhavalsays
📝 Hem's Linkedin: LinkedIn: hemvad
📽️ Hem's Instagram for daily tips: Instagram: hemvadivel
📸 Dhaval's Personal Instagram: Instagram: dhavalsays
🔗 Patreon: https://www.patreon.com/codebasics?fa...