resource aware scheduling in storm (hadoop summit 2016)
TRANSCRIPT
![Page 1: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/1.jpg)
RESOURCE AWARE
SCHEDULING IN APACHE STORM
Presented by Boyang Jerry Peng
![Page 2: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/2.jpg)
2
ABOUT ME• Apache Storm Committer and PMC member• Member of the Yahoo’s low latency Team
Data processing solutions with low latency • Graduate student @ University of Illinois, Urbana-Champaign
Research emphasis in distributed systems and stream processing
• Contact: [email protected]
![Page 3: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/3.jpg)
3
AGENDA•Overview of Apache Storm•Problems and Challenges •Introduction of Resource Aware Scheduler
•Results
![Page 4: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/4.jpg)
4
OVERVIEW• Apache Storm is an open source distributed real-time data stream
processing platform Real-time analytics Online machine learning Continuous computation Distributed RPC ETL
![Page 5: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/5.jpg)
5
STORM TOPOLOGY
• Processing can be represented as a directed graph• Spouts are sources of information• Bolts are operators that process data
![Page 6: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/6.jpg)
6
DEFINITIONS OF STORM TERMS• Stream
an unbounded sequence of tuples.• Component
A processing operator in a Storm topology that is either a Bolt or Spout
• Executors Threads that are spawned in
worker processes that execute the logic of components
• Worker Process A process spawned by Storm that
may run one or more executors.
![Page 7: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/7.jpg)
7
STORM ARCHITECTURE
Master Node
Cluster Coordinati
onWorker
processes
Worker
Nimbus
Zookeeper
Zookeeper
Zookeeper
Supervisor
Supervisor
Supervisor
Supervisor Worker
Worker
Worker
Launches workers
![Page 8: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/8.jpg)
8
LOGICAL VS PHYSICAL CONNECTION IN STORM
![Page 9: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/9.jpg)
9
OVERVIEW OF SCHEDULING IN STORM• Default Scheduling Strategy
Naïve round robin schedulerNaïve load limiter (Worker Slots)
• Multitenant SchedulerDefault Scheduler with multitenant capabilities (supported by security)
Can allocate a set of isolated nodes for topology (Soft Partitioning)
Resource Aware
![Page 10: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/10.jpg)
10
RUNNING STORM AT YAHOO - CHALLENGES• Increasing heterogeneous clusters
Isolation Scheduler – handing out dedicated machines• Low cluster overall resource utilization
Users not utilizing their isolated allocation very well• Unbalanced resource usage
Some machines not used, others over used• Per topology scheduling strategy
Different topologies have different scheduling needs (e.g. constraint based scheduling)
![Page 11: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/11.jpg)
11
RUNNING STORM AT YAHOO – SCALE
2012 2013 2014 2015 20160
5001000150020002500300035004000
0100200300400500600700800
600
2300
3500
120
300
680
Total Nodes Running Storm at YahooTotal Nodes Largest Cluster Size
Year
Nod
es
![Page 12: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/12.jpg)
12
RESOURCE AWARE SCHEDULING IN STORM• Scheduling in Storm that takes into account resource availability on machines and resource requirement of workloads when scheduling the topology Fine grain resource control Resource Aware Scheduler (RAS) implements this function
- Includes many nice multi-tenant features• Built on top of:
Peng, Boyang, Mohammad Hosseini, Zhihao Hong, Reza Farivar, and Roy Campbell. "R-storm: Resource-aware scheduling in storm." In Proceedings of the 16th Annual Middleware Conference, pp. 149-161. ACM, 2015
![Page 13: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/13.jpg)
13
RAS API• Fine grain resource control
Allows users to specify resources requirement for each component (Spout or Bolt) in a Storm Topology:
API to set component memory requirement:
API to set component CPU requirement:
Example of Usage:
public T setMemoryLoad(Number onHeap, Number offHeap)
public T setCPULoad(Number amount)
SpoutDeclarer s1 = builder.setSpout("word", new TestWordSpout(), 10);s1.setMemoryLoad(1024.0, 512.0);builder.setBolt("exclaim1", new ExclamationBolt(), 3) .shuffleGrouping("word").setCPULoad(100.0);
![Page 14: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/14.jpg)
14
CLUSTER CONFIGURATIONSconf/storm.yaml
.
.
.supervisor.memory.capacity.mb: 20480.0supervisor.cpu.capacity: 400.0
.
.
.
![Page 15: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/15.jpg)
15
RAS FEATURES – PLUGGABLE PER TOPOLOGY SCHEDULING STRATEGIES• Allows users to specify which scheduling strategy to use
• Default Strategy- Based on:
• Peng, Boyang, Mohammad Hosseini, Zhihao Hong, Reza Farivar, and Roy Campbell. "R-storm: Resource-aware scheduling in storm." In Proceedings of the 16th Annual Middleware Conference, pp. 149-161. ACM, 2015.
- Enhancements have been made (e.g. limiting max heap size per worker, better rack selection algorithm, etc)- Aims to pack topology as tightly as possible on machines to reduce communication latency and increase
utilization- Collocating components that communication with each other (operator chaining)
• Constraint Based Scheduling Strategy CSP problem solver
conf.setTopologyStrategy(DefaultResourceAwareStrategy.class);
![Page 16: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/16.jpg)
16
RAS FEATURES – RESOURCE ISOLATION VIA CGROUPS (LINUX PLATFORMS ONLY*)• Replaces resource isolation via isolated nodes• Resource quotas enforced on a per worker basis• Each worker should not go over its allocated resource quota• Guarantee QOS and topology isolation• Documentation:
https://storm.apache.org/releases/2.0.0-SNAPSHOT/cgroups_in_storm.html
*RHEL 7 or higher. Potential critical bugs in older RHEL versions.
![Page 17: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/17.jpg)
17
RAS FEATURES – PER USER RESOURCE GUARANTEES• Configurable per user resource guarantees
![Page 18: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/18.jpg)
18
RAS FEATURE – TOPOLOGY PRIORITY• Users can set the priority of a topology to indicate its importance
• The range of topology priorities can range form 0-29. The topologies priorities will be partitioned into several priority levels that may contain a range of priorities
conf.setTopologyPriority(int priority)
PRODUCTION => 0 – 9STAGING => 10 – 19DEV => 20 – 29
![Page 19: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/19.jpg)
19
RAS FEATURES – PLUGGABLE TOPOLOGY PRIORITY• Topology Priority Strategy
Which topology should be scheduled first? Cluster wide configuration set in storm.yaml Default Topology Priority Strategy
- Takes into account resource guarantees and topology priority- Schedules topologies from users who is the most under his or her
resource guarantee. - Topologies of each user is sorted by priority- More details:
https://storm.apache.org/releases/2.0.0-SNAPSHOT/Resource_Aware_Scheduler_overview.html
![Page 20: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/20.jpg)
20
RAS FEATURES – PLUGGABLE TOPOLOGY EVICTION STRATEGIES• Topology Eviction Strategy
When there is not enough resource which topology from which user to evict?
Cluster wide configuration set in storm.yaml Default Eviction Strategy
- Based on how much a user’s guarantee has been satisfied- Priority of the topology
FIFO Eviction Strategy- Used on our staging clusters. - Ad hoc use
More details:https://storm.apache.org/releases/2.0.0-SNAPSHOT/Resource_Aware_Scheduler_overview.html
![Page 21: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/21.jpg)
21
SELECTED RESULTS (THROUGHPUT) FROM PAPER [1] – YAHOO TOPOLOGIES
47% improvement!
50% improvement!
* Figures used [1]
![Page 22: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/22.jpg)
22
SELECTED RESULTS (THROUGHPUT) FROM PAPER [1] – YAHOO TOPOLOGIES
![Page 23: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/23.jpg)
23
PRELIMINARY RESULTS IN YAHOO STORM CLUSTERS
![Page 24: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/24.jpg)
24
PRELIMINARY RESULTS IN YAHOO STORM CLUSTERS
![Page 25: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/25.jpg)
25
CONCLUDING REMARKS AND FUTURE WORK• In Summary
Built resource aware scheduler• Migration Process
In the Progress from migrating from MultitenantScheduler to RAS
Working through bugs with Cgroups, Java, and Linux kernel• Future Work
Improved Scheduling Strategies Real-time resource monitoring Elasticity
![Page 26: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/26.jpg)
26
QUESTIONS
![Page 27: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/27.jpg)
27
REFERENCES• [1] Peng, Boyang, Mohammad Hosseini, Zhihao Hong, Reza Farivar, and Roy Campbell.
"R-storm: Resource-aware scheduling in Storm." In Proceedings of the 16th Annual Middleware Conference, pp. 149-161. ACM, 2015.
http://web.engr.illinois.edu/~bpeng/files/r-storm.pdf• [2] Official Resource Aware Scheduler Documentation
https://storm.apache.org/releases/2.0.0-SNAPSHOT/Resource_Aware_Scheduler_overview.htm
• [3] Umbrella Jira for Resource Aware Scheduling in Storm https://issues.apache.org/jira/browse/STORM-893
![Page 28: Resource Aware Scheduling in Storm (Hadoop Summit 2016)](https://reader035.vdocuments.mx/reader035/viewer/2022070602/587813ba1a28ab971e8b785f/html5/thumbnails/28.jpg)
28
EXTRA SLIDES