making apache spark™ better with delta lake€¦ · making apache spark™ better with delta lake...
TRANSCRIPT
![Page 1: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/1.jpg)
Making Apache Spark™ Better with Delta Lake
Michael Armbrust@michaelarmbrust
![Page 2: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/2.jpg)
1. Collect Everything
• Recommendation Engines• Risk, Fraud Detection• IoT & Predictive Maintenance• Genomics & DNA Sequencing
3. Data Science & Machine Learning
2. Store it all in the Data Lake
The Promise of the Data Lake
Garbage In Garbage Stored Garbage Out
🔥
🔥
🔥
🔥🔥
🔥
🔥
![Page 3: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/3.jpg)
What does a typical data lake project look like?
![Page 4: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/4.jpg)
Evolution of a Cutting-Edge Data Lake
Events
?AI & Reporting
StreamingAnalytics
Data Lake
![Page 5: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/5.jpg)
Evolution of a Cutting-Edge Data Lake
Events
AI & Reporting
StreamingAnalytics
Data Lake
![Page 6: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/6.jpg)
Challenge #1: Historical Queries?
Data Lake
λ-arch
λ-arch
StreamingAnalytics
AI & Reporting
Events
λ-arch1
1
1
![Page 7: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/7.jpg)
Challenge #2: Messy Data?
Data Lake
λ-arch
λ-arch
StreamingAnalytics
AI & Reporting
Events
Validation
λ-arch
Validation
1
21
1
2
![Page 8: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/8.jpg)
Reprocessing
Challenge #3: Mistakes and Failures?
Data Lake
λ-arch
λ-arch
StreamingAnalytics
AI & Reporting
Events
Validation
λ-arch
Validation
Reprocessing
Partitioned
1
2
3
1
1
3
2
![Page 9: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/9.jpg)
Reprocessing
Challenge #4: Updates?
Data Lake
λ-arch
λ-arch
StreamingAnalytics
AI & Reporting
Events
Validation
λ-arch
Validation
Reprocessing
Updates
Partitioned
UPDATE &MERGE
Scheduled to Avoid Modifications
1
2
3
1
1
3
4
4
4
2
![Page 10: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/10.jpg)
Wasting Time & Money
Solving Systems Problems
Instead of Extracting Value From Data
![Page 11: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/11.jpg)
Data Lake Distractions
No atomicity means failed production jobs leave data in corrupt state requiring tedious recovery
✗
No quality enforcement creates inconsistent and unusable data
No consistency / isolation makes it almost impossible to mix appends and reads, batch and streaming
![Page 12: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/12.jpg)
Let’s try it instead with
![Page 13: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/13.jpg)
Reprocessing
Challenges of the Data Lake
Data Lake
λ-arch
λ-arch
StreamingAnalytics
AI & Reporting
Events
Validation
λ-arch
Validation
Reprocessing
Updates
Partitioned
UPDATE &MERGE
Scheduled to Avoid Modifications
1
2
3
1
1
3
4
4
4
2
![Page 14: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/14.jpg)
AI & Reporting
StreamingAnalytics
The Architecture
Data Lake
CSV,JSON, TXT…
Kinesis
![Page 15: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/15.jpg)
AI & Reporting
StreamingAnalytics
The Architecture
Data Lake
CSV,JSON, TXT…
Kinesis
Full ACID Transaction
Focus on your data flow, instead of worrying about failures.
![Page 16: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/16.jpg)
AI & Reporting
StreamingAnalytics
The Architecture
Data Lake
CSV,JSON, TXT…
Kinesis
Open Standards, Open Source (Apache License)
Store petabytes of data without worries of lock-in. Growing community including Presto, Spark and more.
![Page 17: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/17.jpg)
AI & Reporting
StreamingAnalytics
The Architecture
Data Lake
CSV,JSON, TXT…
Kinesis
Powered by
Unifies Streaming / Batch. Convert existing jobs with minimal modifications.
![Page 18: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/18.jpg)
Data Lake
AI & Reporting
StreamingAnalytics
Business-level Aggregates
Filtered, CleanedAugmented
Raw Ingestion
The
Bronze Silver Gold
CSV,JSON, TXT…
Kinesis
Quality
Delta Lake allows you to incrementally improve the quality of your data until it is ready for consumption.
*Data Quality Levels *
![Page 19: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/19.jpg)
Data Lake
AI & Reporting
StreamingAnalytics
Business-level Aggregates
Filtered, CleanedAugmented
Raw Ingestion
The
Bronze Silver Gold
CSV,JSON, TXT…
Kinesis
• Dumping ground for raw data
• Often with long retention (years)
• Avoid error-prone parsing
🔥
![Page 20: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/20.jpg)
Data Lake
AI & Reporting
StreamingAnalytics
Business-level Aggregates
Filtered, CleanedAugmented
Raw Ingestion
The
Bronze Silver Gold
CSV,JSON, TXT…
Kinesis
Intermediate data with some cleanup applied.
Queryable for easy debugging!
![Page 21: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/21.jpg)
Data Lake
AI & Reporting
StreamingAnalytics
Business-level Aggregates
Filtered, CleanedAugmented
Raw Ingestion
The
Bronze Silver Gold
CSV,JSON, TXT…
Kinesis
Clean data, ready for consumption.
Read with Spark or Presto**Coming Soon
![Page 22: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/22.jpg)
Data Lake
AI & Reporting
StreamingAnalytics
Business-level Aggregates
Filtered, CleanedAugmented
Raw Ingestion
The
Bronze Silver Gold
CSV,JSON, TXT…
Kinesis
Streams move data through the Delta Lake
• Low-latency or manually triggered
• Eliminates management of schedules and jobs
![Page 23: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/23.jpg)
Data Lake
AI & Reporting
StreamingAnalytics
Business-level Aggregates
Filtered, CleanedAugmented
Raw Ingestion
The
Bronze Silver Gold
CSV,JSON, TXT…
Kinesis
Delta Lake also supports batch jobs and standard DML
UPDATE
DELETEMERGE
OVERWRITE
• Retention
• Corrections
• GDPR
• UPSERTS
INSERT
*DML Coming in 0.3.0
![Page 24: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/24.jpg)
Data Lake
AI & Reporting
StreamingAnalytics
Business-level Aggregates
Filtered, CleanedAugmented
Raw Ingestion
The
Bronze Silver Gold
CSV,JSON, TXT…
Kinesis
Easy to recompute when business logic changes:
• Clear tables
• Restart streams
DELETE DELETE
![Page 25: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/25.jpg)
Who is using ?
![Page 26: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/26.jpg)
Used by 1000s of organizations world wide
> 1 exabyte processed last month alone
![Page 27: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/27.jpg)
27
Improved reliability:
Petabyte-scale jobs
10x lower compute:
640 instances to 64!
Simpler, faster ETL:
84 jobs → 3 jobs
halved data latency
![Page 28: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/28.jpg)
How do I use ?
![Page 29: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/29.jpg)
dataframe
.write
.format("delta")
.save("/data")
Get Started with Delta using Spark APIs
dataframe
.write
.format("parquet")
.save("/data")
Instead of parquet... … simply say delta
Add Spark Package
pyspark --packages io.delta:delta-core_2.12:0.1.0
bin/spark-shell --packages io.delta:delta-core_2.12:0.1.0
<dependency><groupId>io.delta</groupId> <artifactId>delta-core_2.12</artifactId> <version>0.1.0</version>
</dependency>
Maven
![Page 30: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/30.jpg)
Data Quality
Enforce metadata, storage, and quality declaratively.
table("warehouse")
.location(…) // Location on DBFS
.schema(…) // Optional strict schema checking
.metastoreName(…) // Registration in Hive Metastore
.description(…) // Human readable description for users
.expect("validTimestamp", // Expectations on data quality
"timestamp > 2012-01-01 AND …",
"fail / alert / quarantine")
*Coming Soon
![Page 31: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/31.jpg)
Data Quality
Enforce metadata, storage, and quality declaratively.
table("warehouse")
.location(…) // Location on DBFS
.schema(…) // Optional strict schema checking
.metastoreName(…) // Registration in Hive Metastore
.description(…) // Human readable description for users
.expect("validTimestamp", // Expectations on data quality
"timestamp > 2012-01-01 AND …",
"fail / alert / quarantine")
*Coming Soon
![Page 32: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/32.jpg)
How does work?
![Page 33: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/33.jpg)
Delta On Disk
my_table/
_delta_log/
00000.json
00001.json
date=2019-01-01/
file-1.parquet
Transaction Log
Table Versions
(Optional) Partition Directories
Data Files
![Page 34: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/34.jpg)
Table = result of a set of actions
Change Metadata – name, schema, partitioning, etc
Add File – adds a file (with optional statistics)
Remove File – removes a file
Result: Current Metadata, List of Files, List of Txns, Version
![Page 35: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/35.jpg)
Implementing Atomicity
Changes to the table are stored as ordered, atomic units called commits
Add 1.parquet
Add 2.parquet
Remove 1.parquet
Remove 2.parquet
Add 3.parquet
000000.json
000001.json
…
![Page 36: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/36.jpg)
Ensuring Serializablity
Need to agree on the order of changes, even when there are multiple writers. 000000.json
000001.json
000002.json
User 1 User 2
![Page 37: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/37.jpg)
Solving Conflicts Optimistically
1. Record start version2. Record reads/writes3. Attempt commit4. If someone else wins,
check if anything you read has changed.
5. Try again.
000000.json
000001.json
000002.json
User 1 User 2
Write: Append
Read: Schema
Write: Append
Read: Schema
![Page 38: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/38.jpg)
Handling Massive Metadata
Large tables can have millions of files in them! How do we scale the metadata? Use Spark for scaling!
Add 1.parquet
Add 2.parquet
Remove 1.parquet
Remove 2.parquet
Add 3.parquet
Checkpoint
![Page 39: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/39.jpg)
Road Map
• 0.2.0 – Released!• S3 Support
• Azure Blob Store and ADLS Support
• 0.3.0 (~July)• UPDATE (Scala)
• DELETE (Scala)
• MERGE (Scala)
• VACUUM (Scala)
• Rest of Q3• DDL Support / Hive Metastore
• SQL DML Support
![Page 40: Making Apache Spark™ Better with Delta Lake€¦ · Making Apache Spark™ Better with Delta Lake ... Full ACID Transaction Focus on your data flow, instead of worrying about failures](https://reader034.vdocuments.mx/reader034/viewer/2022042810/5f9936b4ce4d300cd90deff2/html5/thumbnails/40.jpg)
Build your own Delta Lakeat https://delta.io