Don’t forget to hit the LIKE and hope you find this valuable 😄

Do let me know your thoughts in comments

Detailed timestamp :
1. Introduction & Overview
[00:00] – Project & Video Introduction
[01:59] – End-to-End Architecture Walkthrough (Medallion Architecture)
[04:30] – Comparison: Old Architecture vs. New Architecture
[09:18] – Problem Statement & Business Goals
[11:26] – AWS vs. GCP vs. Azure Comparison for Data Engineers

2. AWS Core Fundamentals
[15:34] – Cloud Computing Basics (IaaS, PaaS, SaaS)
[23:24] – AWS Global Infrastructure (Regions, Availability Zones & Edge Locations)
[30:40] – AWS Account Structure, Root User, Organizations & Free Tier
[34:53] – Key AWS Services for Data Engineers Overview

3. Amazon S3 & Local Environment Setup
[35:43] – Amazon S3 Concepts, Storage Classes & Lifecycle Rules
[38:28] – Setting up Project Folder & Downloading Kaggle Dataset
[42:52] – Exploring CSV and JSON YouTube Data Structures
[45:51] – Creating S3 Buckets (Bronze, Silver, Gold, Scripts)
[53:10] – Configuring S3 Lifecycle Rules for Glacier Archiving

4. AWS CLI & Data Ingestion (Bronze Layer)
[58:45] – Installing and Configuring AWS CLI Locally
[01:02:58] – Creating IAM User & Generating Access Keys for CLI
[01:05:38] – Uploading Kaggle CSV & JSON Files to S3 Bronze Layer using Bash/CLI
[01:10:47] – Overview of AWS Storage Services: EBS vs. EFS vs. S3

5. AWS IAM Setup & Permissions
[01:15:04] – IAM Roles vs. IAM Users Explained
[01:16:43] – Creating IAM Execution Role & Inline Policies for AWS Lambda
[01:34:02] – Creating IAM Execution Role & Policies for AWS Glue
[01:36:40] – Setting up AWS SNS Topic and Email Subscription for Alerts

6. AWS Glue Data Catalog & Amazon Athena
[01:41:20] – Core Concepts: AWS Glue vs. AWS EMR
[01:45:07] – Exploring AWS Glue Console & Creating Glue Databases
[01:48:46] – Configuring and Running AWS Glue Crawler on Bronze Layer
[01:52:13] – Amazon Athena Querying Overview & Setting Query Result Location
[01:56:41] – Querying Bronze Data in Athena and Identifying Data Quality Issues

7. Real-Time API Ingestion via Lambda
[02:00:54] – Creating AWS Lambda Function for YouTube Data API v3 Ingestion
[02:07:44] – Generating YouTube Data API Key from Google Cloud Console
[02:09:12] – Configuring Lambda Environment Variables, Memory & Timeouts
[02:11:04] – Testing YouTube API Lambda Ingestion to S3 Bronze Bucket

8. Silver Layer: Cleaning & Transforming to Parquet
[02:16:55] – Creating Lambda Function for JSON-to-Parquet Transformation
[02:21:06] – Configuring S3 Trigger on Bronze Bucket & Adding Pandas Lambda Layer
[02:24:36] – Testing Lambda JSON-to-Parquet Pipeline & Checking S3 Output
[02:30:43] – Writing AWS Glue PySpark ETL Script (Bronze to Silver Statistics)
[02:36:07] – Configuring Glue Job Parameters & Running PySpark Job
[02:45:23] – Verifying Silver Layer Parquet Data in Athena

9. Data Quality Gate & Gold Aggregation Layer
[02:48:04] – Creating Data Quality Lambda Function with Validation Checks & SNS Alerts
[02:52:53] – Writing Glue PySpark Job for Silver-to-Gold Aggregations
[02:57:18] – Configuring & Executing Silver-to-Gold Glue Job
[02:59:42] – Overview of QuickSight, Kinesis, RDS, DynamoDB & Redshift
[03:03:27] – Running Analytical SQL Queries in Athena on Gold Layer Tables

10. Pipeline Orchestration with AWS Step Functions
[03:08:19] – AWS Step Functions Concepts & State Machine Design
[03:10:09] – Setting up IAM Role for Step Functions Orchestration
[03:12:03] – Defining Step Function JSON Definition (Tasks, Parallel Runs, Catch & Retries)
[03:21:38] – Executing End-to-End Orchestration State Machine
[03:25:50] – Debugging Lambda Layers, Permissions & Athena Query Settings
[03:47:51] – Successful Full Pipeline Execution Run & SNS Success Notification

11. Summary & Clean-up
[03:50:46] – Architecture Summary & Full Workflow Recap
[03:52:09] – AWS Resource Clean-up Guidelines & Next Steps

I am running my first glue job.

Project Introduction and Architecture Overview: (0:00 - 38:07)
Project Setup and Local Development: (38:07 - 55:30)
AWS S3 Configuration and Life Cycle Rules: (55:30 - 57:23)
Ingesting Static Data from Kaggle to S3: (57:23 - 1:07:12)
Overview of Other AWS Storage Services (EBS/EFS): (1:10:37 - 1:11:36)
Setting up AWS Athena and Exploring Raw Data: (1:55:57 - 2:01:00)
YouTube Data API Integration (Lambda Setup): (2:01:00 - 2:16:36)
Writing Lambda Transformation Code (JSON to Parquet): (2:16:36 - 2:26:32)
Data Quality Lambda Function Setup: (2:28:12 - 2:29:16)
AWS Glue ETL Job Transformation (Spark): (2:29:16 - 2:44:21)
Data Analysis in Athena (Gold Layer): (3:03:06 - 3:07:56)
AWS Step Functions Orchestration: (3:07:56 - 3:14:07)

Great stuff, thanks Darshil! Gonna challenge myself and add Terraform to this project for some extra practice. 🔥

Stuck at Step orchestration, will get it eventually today.

thank you so much, the project and explanations are top notch! I was even thinking about watching a video at x0.75, because you included so much stuff into these 4 hours😂 I’ve managed to solve all my problems and finish the project successfully!🔥🎉

Thank you so much for this insightful video. It really made some AWS DE concepts relatable. Thank you Darshil

Very good explanation on each topis today completed with hand on!!! Thanks waiting for part 2

Kudos for the efforts 🙌🙌

Best video I found so far!

Most awaited video

Thanks, it was really insightful and great

Thank you sir

that leading UUID was real killer ! anyway wonderful video !Thanks buddy : )💚

I learned a lot from you, amazing

Thanks a ton darshil for making this video. Its super amazing!!

The project has been great so far! One suggestion that would make it even more valuable would be to include a section on creating an orchestration using AWS Step Functions with the Visual Workflow Designer.
🙌🧡

Very well explained next on redshift

Much needed course Darshil

Thanks Darshil 👍