Download - How Spotify scales Apache Storm Pipelines
![Page 1: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/1.jpg)
![Page 2: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/2.jpg)
![Page 3: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/3.jpg)
![Page 4: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/4.jpg)
![Page 5: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/5.jpg)
○○○
○○
![Page 6: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/6.jpg)
![Page 7: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/7.jpg)
○
○
○
○
○
○
○
○
![Page 8: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/8.jpg)
![Page 9: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/9.jpg)
Log EventsApache Kafka
Real-time Personalization Pipeline
Apache Storm
User Profile Store
Apache Cassandra
Entity Metadata Store
Apache Cassandra
![Page 10: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/10.jpg)
○
○
○
○
○
○
○
○
![Page 11: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/11.jpg)
![Page 12: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/12.jpg)
![Page 13: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/13.jpg)
![Page 14: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/14.jpg)
![Page 15: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/15.jpg)
○
○
○
○
○
○
○
○
![Page 16: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/16.jpg)
![Page 17: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/17.jpg)
![Page 18: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/18.jpg)
○
○
○
○
○
○
○
○
![Page 19: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/19.jpg)
![Page 20: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/20.jpg)
![Page 21: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/21.jpg)
○
○
○
○
○
○
○
○
![Page 22: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/22.jpg)
![Page 23: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/23.jpg)
○
○
○
○
○
○
○
○
![Page 24: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/24.jpg)
Build v2
Storm Cluster
Running v1
t1 t2
Storm Cluster
t3 t4
Running v1
Running v2
Deactivate v1
Submit v2
Check v2
graphs
Kill v1
Storm Cluster
Running v2
t5 t6 t7 t8
![Page 25: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/25.jpg)
○
○
○
○
○
○
○
○
![Page 26: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/26.jpg)
![Page 27: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/27.jpg)
Pagerduty Inhouse Solution
![Page 28: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/28.jpg)
○
○
○
○
○
○
○
○
![Page 29: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/29.jpg)
○○○○
![Page 30: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/30.jpg)
○
○
○
○
○
○
○
○
![Page 31: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/31.jpg)
Log EventsApache Kafka
Real-time Personalization Pipeline
Apache Storm
User Profile Store
Apache Cassandra
Entity Metadata Store
Apache Cassandra
![Page 32: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/32.jpg)
![Page 33: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/33.jpg)
![Page 34: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/34.jpg)
● Different tables for different TTLs and set gc_grace_period=0. Read repairs are disabled.
● Used DateTieredCompactionStrategy for short lived data.
● Control the number of open connections from Storm topology to Cassandra
● Configure Snitch to ensure proper call routing
![Page 35: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/35.jpg)
![Page 36: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/36.jpg)
● 1 worker per node per topology● 1 executor per core for CPU bound tasks● 1-10 executors per core for IO bound tasks● Compute total parallelism possible and distribute it
amongst slow and fast tasks. High parallelism for slow tasks, low for fast tasks.
* Parallelism tuning inspired by P Taylor Goetz’s Strata 2014 talk
![Page 37: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/37.jpg)
○
○
○
○
○
○
○
○
![Page 38: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/38.jpg)
● Think about constraints in external vs in-process caching○ External Caching
■ Network IO■ Latency■ Another point of failure
○ In-process■ Limited memory■ No persistence
![Page 39: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/39.jpg)
![Page 40: How Spotify scales Apache Storm Pipelines](https://reader033.vdocuments.mx/reader033/viewer/2022042602/55a201161a28ab47268b458c/html5/thumbnails/40.jpg)