![Page 1: Deconstructiong Recommendations on Spark-(Ilya Ganelin, Capital One)](https://reader031.vdocuments.mx/reader031/viewer/2022032305/55d0d396bb61eb290e8b47b6/html5/thumbnails/1.jpg)
(or: How I learned to stop worrying and love the shuffle)
By: Ilya Ganelin
Lessons from Spark
![Page 2: Deconstructiong Recommendations on Spark-(Ilya Ganelin, Capital One)](https://reader031.vdocuments.mx/reader031/viewer/2022032305/55d0d396bb61eb290e8b47b6/html5/thumbnails/2.jpg)
• Recommendations (ML Lib) – Collaborative Filtering (CF) – 65 million X 10 million – 2.5 TB, ~12 PFlops – All recs (long tail)
• You’re mad! – Don’t usually generate all possible
recs – 6 data nodes (~40 GB RAM) – Shared cluster – Spark stability
What:
![Page 3: Deconstructiong Recommendations on Spark-(Ilya Ganelin, Capital One)](https://reader031.vdocuments.mx/reader031/viewer/2022032305/55d0d396bb61eb290e8b47b6/html5/thumbnails/3.jpg)
• Goal: – Understand how Spark internals drive design and configuration
• Contents: – Background
• Partitions • Caching • Serialization • Shuffle
– Lessons 1-4
Overview
![Page 4: Deconstructiong Recommendations on Spark-(Ilya Ganelin, Capital One)](https://reader031.vdocuments.mx/reader031/viewer/2022032305/55d0d396bb61eb290e8b47b6/html5/thumbnails/4.jpg)
Background
![Page 5: Deconstructiong Recommendations on Spark-(Ilya Ganelin, Capital One)](https://reader031.vdocuments.mx/reader031/viewer/2022032305/55d0d396bb61eb290e8b47b6/html5/thumbnails/5.jpg)
• Partitions – How data is split on disk – Affects memory usage, shuffle size – Count ~ speed, Count ~ 1/memory
• Caching – Persist RDDs in distributed memory – Major speedup for repeated operations
• Serialization – Efficient movement of data – Java vs. Kryo
Partitions, Caching, and Serialization
![Page 6: Deconstructiong Recommendations on Spark-(Ilya Ganelin, Capital One)](https://reader031.vdocuments.mx/reader031/viewer/2022032305/55d0d396bb61eb290e8b47b6/html5/thumbnails/6.jpg)
Shuffle?
![Page 7: Deconstructiong Recommendations on Spark-(Ilya Ganelin, Capital One)](https://reader031.vdocuments.mx/reader031/viewer/2022032305/55d0d396bb61eb290e8b47b6/html5/thumbnails/7.jpg)
Shuffle! • All-all operations
– reduceByKey, groupByKey • Data movement
– Serialization – Akka
• Memory overhead – Dumps to disk when OOM – Garbage collection
• EXPENSIVE!
Map Reduce
![Page 8: Deconstructiong Recommendations on Spark-(Ilya Ganelin, Capital One)](https://reader031.vdocuments.mx/reader031/viewer/2022032305/55d0d396bb61eb290e8b47b6/html5/thumbnails/8.jpg)
Lessons
![Page 9: Deconstructiong Recommendations on Spark-(Ilya Ganelin, Capital One)](https://reader031.vdocuments.mx/reader031/viewer/2022032305/55d0d396bb61eb290e8b47b6/html5/thumbnails/9.jpg)
• Memory – You’re using more than you think
• JVM overhead • Spark metadata (shuffles, long-running jobs) • Scala vs. Java
– Shuffle & heap • Debugging is hard
– Distributed logs – Hundreds of tasks
Lesson 1: Spark is a problem child!
![Page 10: Deconstructiong Recommendations on Spark-(Ilya Ganelin, Capital One)](https://reader031.vdocuments.mx/reader031/viewer/2022032305/55d0d396bb61eb290e8b47b6/html5/thumbnails/10.jpg)
• Tame the beast (memory) – Partition wisely – Know your data!
• Size, Types, Distribution – Kryo Serialization
• Cleanup – Long-term jobs consume memory indefinitely – Spark context cleanup fails in production environment – Solution: YARN!
• Separate spark-submits per batch • Stable Spark-based job that runs for weeks
Lesson 1: Discipline
![Page 11: Deconstructiong Recommendations on Spark-(Ilya Ganelin, Capital One)](https://reader031.vdocuments.mx/reader031/viewer/2022032305/55d0d396bb61eb290e8b47b6/html5/thumbnails/11.jpg)
• Why? – Speed up execution – Increase stability – ???? – Profit!
• How? – Use the driver!
• Collect • Broadcast • Accumulators
Lesson 2: Avoid shuffles!
![Page 12: Deconstructiong Recommendations on Spark-(Ilya Ganelin, Capital One)](https://reader031.vdocuments.mx/reader031/viewer/2022032305/55d0d396bb61eb290e8b47b6/html5/thumbnails/12.jpg)
• Limited memory – Collected RDDs – Metadata – Results (Accumulators)
• Akka messaging – 10e6 x (120 bytes) ~ 1.2GB; 20 partitions – Read ~60 MB per partition – (Default is 10MB) – Solution: Partition & set akka.frameSize - know your data!
• Big data – Solution: Batch process
• Problem: Cleanup and long-term stability
Lesson 3: Using the driver is hard!
![Page 13: Deconstructiong Recommendations on Spark-(Ilya Ganelin, Capital One)](https://reader031.vdocuments.mx/reader031/viewer/2022032305/55d0d396bb61eb290e8b47b6/html5/thumbnails/13.jpg)
• Cache, but cache wisely – If you use it twice, cache it
• Broadcast variables – Visible to all executors – Only serialized once – Blazing-fast lookups!
• Threading – Thread pool on driver – Fast operations, many tasks
• 75x speedup over ML Lib ALS predict() – Start: 1 rec / 1.5 seconds – End: 50 recs / second
Lesson 4: Speed!
![Page 14: Deconstructiong Recommendations on Spark-(Ilya Ganelin, Capital One)](https://reader031.vdocuments.mx/reader031/viewer/2022032305/55d0d396bb61eb290e8b47b6/html5/thumbnails/14.jpg)
Questions?