OpenTube

Data Modeling in the Modern Data Stack

Kahan Data Solutions

168,904 views

Get all of my free resources to help you learn: https://skool.com/moderndata/about
Work with me to modernize your data architecture: https://bit.ly/kdsconsulting

-----
Data Modeling is arguably the most impactful decision for a data team.

It determines your architecture and the path that the whole team will follow.

While this is not a new topic, the new tools and tech over the last decade has caused many to reconsider what's best in a modern landscape.

So in today's video I want to break down this topic.

We'll discuss:
1. Why is Data Modeling (still) important?
2. What are the common approaches?
3. What things should you consider?

Timestamps:
0:00 - Intro
0:44 - Why is Data Modeling Important?
2:33 - Common Approaches
7:39 - Things to Consider

Title & Tags:
Data Modeling in the Modern Data Stack
#kahandatasolutions #dataengineering #datamodeling

20 Comments

@KahanDataSolutions Creator 4 years ago (edited) Pinned

- Get all of my free resources to help you learn: https://skool.com/moderndata/about
- Work with me to modernize your data architecture: https://bit.ly/kdsconsulting

@firefoxmetzger9063 3 years ago

Just because storage is cheap doesn't mean it's unimportant. I/O is the often slowest part of OLAP and ETL queries and directly dictates compute cost. The two current OLAP pricing models (pay by minute or pay by bytes "scanned") both depend on how much data you load per query. The former because you pay for an idle cluster that is waiting for data, the latter because it literally bills you for how much data you load. Data modeling helps you get the compression and partitioning you need to control and minimize that cost.

@davemeech 2 years ago

Oh man, it's so nice to get something substantial on YouTube for data modeling! Awesome awesome stuff.

What do you think of unistore? I have yet to dive in, and I'm also very fresh into the industry, but the prospect of combining oltp and olap capabilities is certainly compelling!

@amitsaha7756 3 years ago

In some cases, we observe another pattern whether industry standard data model is used after raw layer.

@nlopedebarrios 3 years ago

Great summary, thank you! Personally, I'm more comfortable with the Kimball approach, but the "modern data stack" part in the title drove catched my attention, and I think it's important to know what trends align better withe the modern tech. Also, I couldn't agree more than design is a key part of the process, and if done correctly is going to prevent a lot of headaches. It would be nice to get deeper into the hybrid model you presented.

@SnowFlake-h4y 3 years ago

Could you please explain the differences between different data models(Inmon,Kimball,3NF,Dimension Modelling,Data Vault).

@Rex_793 3 years ago

great stuff - when are you going on Joe Reis and Matt Housley's podcast?

@okj4521 3 years ago

Next: How to date a model!

@nlopedebarrios 3 years ago

Hi Michael, interesting approach the hybrid model. What could be used to transition from the star schema to the OBT data marts? for example, views, materialized views? or are they separate schemas? Also, in what scenario this would make sense?

@Joeymbryan 3 years ago

Wow this is spot on. I've been using the modern data stack for the last ~8 years or so. Starting out, I was definitely focusing on a star schema/denormalized approach with the MPP databases but as I started to learn they do best with fewer joins and can handle wide tables I strive for the OBT approach. In practice, the hybrid is what typically happens, there are so many dimensional tables which very in need from team to team so especially in a larger enterprise, the hybrid is almost a guaranteed.

Great video! Love the work you did here.

@johncosgrove7727 3 years ago

Hey Mike, this video was straight 🔥🔥🔥 - This is such an important topic for all businesses taking the journey right now and it really blows my mind how much understanding of modelling has been lost. Your overview of the pros and cons of each was a well balanced and to-the-point summary - I especially love that you correctly call out the source data and scale risks of OBT. I think too many peeps in the MDS are working for SaaS companies where they don't realise that 1) source systems are way less static in enterprise and 2) most enterprises outside of SaaS aren't just okay with massive pipeline code spaghettis. Great video!

@IbnShahid 1 year ago

I love the plasmic nature of your data modelling.

@theukulelegod 2 years ago

What are the downsides of doing all three in one? Pull all source systems raw data (inmon) then modeling fact and dim tables (kimball) then making data marts? If storage is getting cheaper wouldn’t this be the best way?

@mrcool4uall 3 years ago (edited)

Good one Michael. Well summarized and to the point. Even I think Hybrid approach is the best for the MDS.

@sunil-de 2 years ago

hey, videos was top tier, can you suggest a any good course to get the in depth understandin of the DM

@wingnut29 3 years ago

Great overview! Thanks for educating us on the newer approaches. We currently are using Denormalized Modeling. This is due to the fact that all our current needs are around our ERP. The Marketing Dept wants to start collecting more website and estore analytics, which I believe will lead us to a Hybrid model.

@theconfusedchannel6365 3 years ago

Good one. I think taking an actual data and flowing through model would be great.

@zulkhaireesulaiman8575 4 years ago

thank you, incredibly helpful.

@bradleymiller437 3 years ago

I literally hit this video so fast sincerely thinking I was going to learn "How to date a model." My brain went faster than my eyes and lost the game.