OpenTube

Create on premise Data Lakehouse with Apache Iceberg | Nessie | MinIO | Lakehouse

BI Insights Inc

14,414 views

In this video cover the data lakehouse. A data lake house is a concept that combines elements of both data lakes and data warehouses to bring us the best of both worlds. It aims to provide a unified platform for storing, managing, and analyzing both unstructured data and structured data.

What is Data Lake? https://aws.amazon.com/big-data/datal...

Link to GitHub repo: https://github.com/hnawaz007/pythonda...

Link to Data Lake Video:
On-premis: How to build on-premise Data Lake? | Build...
AWS: How to create an AWS S3 Data Lake? | AWS S...

💥Subscribe to our channel:
haqnawaz

📌 Links
-----------------------------------------
#️⃣ Follow me on social media! #️⃣

🔗 GitHub: https://github.com/hnawaz007
📸 Instagram: Instagram: bi_insights_inc
📝 LinkedIn: LinkedIn: haq-nawaz
🔗 Medium: hnawaz100

-----------------------------------------

#dataanalytics #datalakehouse #opensource

Topics covered in this video:
==================================
0:00 - Introduction to Data Lakehouse
0:53 - Data Lakehouse prominent Features
1:50 - Data Lake from Previouse session
2:31 - Data Lakehouse Overview
3:34 - Tech Stack of on-premise Data Lakehouse
3:44 - Start Docker Containers
4:02 - MinIO (S3) Buckets, File(s) & Keys
4:56 - Configure Dremio
5:07 - Add MinIO (S3) Source
5:57 - Add Nessie Catalog
6:38 - Format File
7:33 - Create Iceberg Table
7:59 - Copy Data to Table
8:35 - SQL DML Operations
9:47 - Table History and Time Travel
10:29 - Coming Soon

18 Comments

@BiInsightsInc Creator 2 years ago Pinned

Link to to Data Lake Videos On-premis and AWS:
https://www.youtube.com/watch?v=DLRiUs1EvhM&t
https://www.youtube.com/watch?v=KvtxdF7b_l8

@AndréCavallaro-q9s 2 years ago

Amazing!

@andriifadieiev9757 2 years ago

Great video, thank you!

@SonuKumar-fn1gn 2 years ago

Great video 👍 Thanks for sharing the video 💝

@hungnguyenthanh4101 2 years ago

very good!

@orafaelgf 2 years ago

great video, congrats.
If possible, bring an end-to-end architecture with streaming data ingested directly into the lakehouse.
also something related to the integration of datalake and datalakehouse.

@jeanchindeko5477 2 years ago

3:18 Apache Iceberg has ACID transactions out of the box, and it’s not Nessie which brings ACID transactions to Iceberg. In Iceberg specification the catalog only has knowledge of the list of snapshots, and the catalog doesn’t track the list of individual files part of commit or snapshots.

@AndréCavallaro-q9s 2 years ago

Today I use apache Nifi to retrieve data from APIs, DBs and mariadb is my main DW. I've been testing dremio/nessie/minIO using docker-compose and I still have doubts about the best way to ingest data in Dremio. There are databases and APIs that cannot be connected directly to it. I tested sending parquet files directly to the storage, but the upsert/merge is very complicated and the jdbc connection with Nifi didn't help me either. What would you recommend for these cases?

@arifalbastomi2303 1 year ago (edited)

can i connect to in a published minio (not in the same docker with dremio)?

@BananaBiscuit25 1 year ago

Great Video!
Although i have set this up in a server, how would i be able to get data from tables and write to using pyiceberg or any other library. I'm trying fetch data from the iceberg using API. I have tried a lot of methods they are not working. Please help. thanks

@waheedrazzaq6482 1 year ago

hi is this possible if i installed this setup in linux OS or edge device should it work or not?

@Xairfox 1 year ago

Great video! What hardware do you need to make it work for educational purpose? I have a Raspberry pi 4 4gb laying around, would this run on this kind of hardware?

@MarcoMikuš 2 years ago

This is so insane. Is it also possible to query data from a specific versionstate directly instead of only the metadata? I am wondering if this would be suitable for bigger Datasets? Have you ever benchmarked this stack with a big Dataset? If the versioncontrol is scalable with bigger datasets and higher change frequency, this would be a crazy good solution to implement.

@dipakchandnani4310 1 year ago

It is giving me "A fatal error has been detected by the Java Runtime Environment" error, it was working fine 3-4 months back but is failing now. I am using Mac and there is no openjdk installed in

"cd /opt/java/openjdk
cd: no such file or directory: /opt/java/openjdk"

appreciate this video and your help here

@luisurena1770 1 year ago

is it possible to replace dremio with Trino?

@Muno_edits06 2 years ago

I want to create Iceberg table with rest catalog using pyiceberg. Does this setup works for it?

@nicky_rads 2 years ago

nice video! Data lakehouses offer a lot of functionality at an affordable price. It seems like dremio is the platform that allows you to aggregate all of these services together ? Could you go a little more in depth on some of the services.