automatic configuration of internet services wei zheng, ricardo bianchini, and thu nguyen department...
TRANSCRIPT
![Page 1: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/1.jpg)
Automatic Configuration of Internet Services
Wei Zheng, Ricardo Bianchini, and Thu Nguyen
Department of Computer ScienceRutgers University
![Page 2: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/2.jpg)
Motivation Downtime and management are expensive for services
• Downtime can cost millions of dollars per hour [IW00]
• Management can account for 50—80% of IT budgets [Brown05]
Operator mistakes are a major source of downtime
• Cause up to 36% of service failures [Oppenheimer03]
Misconfiguration is the most common type of mistake
• More than 50% of service failures [Oppenheimer03]
• 24 out of 42 mistakes in a multi-tier service [Nagaraja04]
![Page 3: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/3.jpg)
Understanding Misconfigurations Misconfigurations occur frequently for two main
reasons:
• Servers and services are becoming increasingly complex
• Operators modify servers’ configs often, as service evolves via software and hardware upgrades, node additions and removals
Example: Apache
• One config file has 240+ lines relating to performance, structure, modules, and others
• Config files must change every time a node is added or removed in the 2nd tier
![Page 4: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/4.jpg)
Our Work Idea: Automatically generate server config files as
the system and its workload evolve
Goals: Reduce config complexity and operator effort, and generate optimized config files efficiently
Contributions:
• Infrastructure for automatic configuration of services
• Heuristics for efficient parameter tuning
• Quantitative evaluation for a realistic service
![Page 5: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/5.jpg)
Outline Motivation Approach and Contributions Automatic Configuration Infrastructure
(ACI) Tuning Parameters Evaluation Conclusions
![Page 6: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/6.jpg)
Key Observations Behind ACI Most of each server’s config doesn’t change
Node additions and removals require tedious and mistake-prone reconfigurations
Changes to a config parameter affect only a few other parameters
![Page 7: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/7.jpg)
Key Observations Behind ACI Most of each server’s config doesn’t change
• Config file templates and generation scripts
Node additions and removals require tedious and mistake-prone reconfigurations
Changes to a config parameter affect only a few other parameters
![Page 8: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/8.jpg)
Key Observations Behind ACI Most of each server’s config doesn’t change
• Config file templates and generation scripts
Node additions and removals require tedious and mistake-prone reconfigurations
• Network of membership daemons
Changes to a config parameter affect only a few other parameters
![Page 9: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/9.jpg)
Key Observations Behind ACI Most of each server’s config doesn’t change
• Config file templates and generation scripts
Node additions and removals require tedious and mistake-prone reconfigurations
• Network of membership daemons
Changes to a config parameter affect only a few other parameters
• Parameter dependency graph and parameter-tuning heuristics
![Page 10: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/10.jpg)
Global View of ACI (3-tier Service)
![Page 11: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/11.jpg)
Local View of ACI
Daemon
Scripts + templates
Config
files
Node i
Global membership info + tuned parameter values
![Page 12: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/12.jpg)
Outline Motivation Approach and Contributions Automatic Configuration Infrastructure
(ACI) Tuning Parameters Evaluation Conclusions
![Page 13: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/13.jpg)
Tuning Parameters Tune parameters for some metric, as service
evolves
Evolution may cause current config to behave poorly• Ex: when throughput is the metric, more memory may
allow more threads and higher throughput
Problem• Hard to know what parameters are affected• Hard to select new values for the parameters
![Page 14: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/14.jpg)
Parameter Dependency Graph Idea: Constrain the search space with a parameter
dependency graph and value ranges
Vertex = parameter, directed edge = dependency
After each change, ACI traverses the graph to find the affected parameters and tune them experimentally• Ex: if Apache node is upgraded, make Apache
parameters “sources” and tune reachable parameters
Overhead of generating graph can be amortized
![Page 15: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/15.jpg)
Generating the Graph 1st step: find important parameters using CART
2nd step: find dependencies between important parametersPair-wise test: parameter A depends on B iff different settings
of B lead to different best values for A; check min, mid, max values
0
1
2
3
4
5
6
7
1 3Parameter A
Met
ric R
espo
nse
Parameter B=1
Parameter B=3
0
1
2
3
4
5
6
7
1 3Parameter B
Met
ric R
espo
nse
Parameter A=1
Parameter A=3
![Page 16: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/16.jpg)
Generating the Graph 1st step: find important parameters using CART
2nd step: find dependencies between important parametersPair-wise test: parameter A depends on B iff different settings
of B lead to different best values for A; check min, mid, max values
0
1
2
3
4
5
6
7
1 3Parameter A
Met
ric R
espo
nse
Parameter B=1
Parameter B=3
0
1
2
3
4
5
6
7
1 3Parameter B
Met
ric R
espo
nse
Parameter A=1
Parameter A=3
A depends on B
![Page 17: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/17.jpg)
Generating the Graph 1st step: find important parameters using CART
2nd step: find dependencies between important parametersPair-wise test: parameter A depends on B iff different settings
of B lead to different best values for A; check min, mid, max values
0
1
2
3
4
5
6
7
1 3Parameter A
Met
ric R
espo
nse
Parameter B=1
Parameter B=3
0
1
2
3
4
5
6
7
1 3Parameter B
Met
ric R
espo
nse
Parameter A=1
Parameter A=3
A depends on B B doesn’t depend on A
![Page 18: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/18.jpg)
Generating the Graph (cont.) An evolution may change the bottleneck tier
Dependencies may be affected by provisioning of tiers• Ex: if throughput is the metric, different bottleneck tiers
may lead to different dependencies
Run dependency checking once per tier, each time forcing the tier to become the bottleneck
Dependency graph is the union of these results
![Page 19: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/19.jpg)
Tuning = Traversing the Graph In a cycle, tune all member vertices at the same
time
In an acyclic chain, tune parents first assuming best values for ancestors and current values for descendants
A singleton is tuned in isolation
Select values using Simplex algorithm and value sets
# experiments increases with connectivity, value set size
![Page 20: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/20.jpg)
Summary
Find important parameters
Find dependencies
Tune parameters
Normal execution
Evolution
![Page 21: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/21.jpg)
Outline Motivation Approach and Contributions Automatic Configuration Infrastructure
(ACI) Tuning Parameters Evaluation Conclusions
![Page 22: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/22.jpg)
Evaluation: Performance Tuning 3-tier auction (Apache, Tomcat, MySQL) on 9
nodes; 24 performance parameters
Management complexity and mistake-elimination results in the paper
Efficiency: Number of throughput-tuning experiments after DB machine upgrade, 2 application server removal
![Page 23: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/23.jpg)
Online Auction Dependencies
![Page 24: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/24.jpg)
Performance Tuning Approaches Exhaustive: O(RN) runs; R = parameter range and
N = # parameters = 24; infeasible
Dependency graph + exhaustive: N = 9; infeasible
Simplex
ACI = Dependency graph + Simplex
![Page 25: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/25.jpg)
Performance Tuning Efficiency
ACI needs 2 service changes to amortize the graph overhead
EvolutionTuning Approach
Throughput (reqs/sec)
Number of Experiments
DB node upgrade
Best prior values
343 -
Simplex search
372 299
ACI 377 821 + 73
Removal of 2 application
servers
Best prior values
91 -
Simplex search
114 220
ACI 110 0 + 35
![Page 26: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/26.jpg)
Amortizing the Graph Overhead Does dependency graph change when service
evolves?
DB node upgrade• Graph does not change
Removal of 2 application server nodes• Graph does not change
Upgrade MySQL server version• Graph is a subset of original one
![Page 27: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/27.jpg)
Conclusions Proposed automatic configuration infrastructure
Proposed notion of parameter dependency graph
Quantified benefits experimentally for realistic service
ACI reduces config complexity, eliminates misconfigs, and produces high-performance services efficiently
![Page 28: Automatic Configuration of Internet Services Wei Zheng, Ricardo Bianchini, and Thu Nguyen Department of Computer Science Rutgers University](https://reader036.vdocuments.mx/reader036/viewer/2022062417/5518a136550346b31f8b4890/html5/thumbnails/28.jpg)
Questions?
http://vivo.cs.rutgers.edu/