automated performance testing and analysis€¦ · automated performance testing and analysis armin...
TRANSCRIPT
![Page 1: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/1.jpg)
Automated Performance Testing and Analysis
Armin PruessnerGAMS Development Corporation
INFORMS Annual Meeting, San JoseOct 17-20, 2002
![Page 2: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/2.jpg)
2
Agenda
1. Performance World! Brief overview
2. Main Focus: Performance Tools! Performance Tools and Metrics! PAVER Web Server (Automation)! Examples! Conclusions
![Page 3: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/3.jpg)
3
Performance World
![Page 4: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/4.jpg)
4
Motivation for Tools
Performance Tools driven by user needs:
! Finding the most reliable way to solve a proprietary model
! Testing a new algorithm against a set of existing test problems and competing approaches
! Reproducibility of performance results
![Page 5: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/5.jpg)
5
Performance World
Applications-Oriented User
PerformanceWorld
Algorithm-Oriented User
CustomSolver Options
DefaultSolver Options
Performance Analysis &
Visualization
![Page 6: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/6.jpg)
6
Tools: Performance Analysis
! Different objectives:! Solver robustness and correctness! Solver efficiency! Quality of solution (nonconvex and discrete
models )
"Tools are GAMS independent"Results in HTML format: platform independent
![Page 7: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/7.jpg)
7
Open Testing ArchitectureCan use Performance World tools
Translate: GAMS/Convert
Solve with“other” systems
Web
PAVERServer
GAMSModels
Web
Solve withGAMS
III. Analysis &Visualization
II. Data CollectionI. Models
![Page 8: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/8.jpg)
8
PAVER Server
! PAVER server (Performance Analysis and Visualization for Effortless Reproducibility)
www.gamsworld.org/performance/paver
! Online server to facilitate performance testing and analysis/visualization
! Results sent via e-mail in HTML format! System independent
![Page 9: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/9.jpg)
9
Tools: Robustness
Solver Square Utility:
! Cross comparison of solver outcomes of two solvers:! Optimal, integer, infeasible, unbounded, fail
! Compact tabular form for results
! Shows resource time and objective value information
" Can use online using PAVER
![Page 10: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/10.jpg)
10
PAVER: Solver Square
![Page 11: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/11.jpg)
11
PAVER: Square (cont.)
![Page 12: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/12.jpg)
12
Tools: Efficiency
Resource Time Utility:
! Cross comparison of solver resource times of two solvers
! Further disaggregation by objective function
! Ratios of resource times
" Can use online using PAVER
![Page 13: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/13.jpg)
13
PAVER: Solver Resource Time
![Page 14: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/14.jpg)
14
PAVER: Resource Time (cont.)
![Page 15: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/15.jpg)
15
Tools: Visualization
Performance Profiles (Dolan and More, 2002):
! Cumulative distribution function for a performance metric
! Performance metric: ratio of current solver time over best time of all solvers
! Intuitively: probability of success if given τtimes fastest time (τ=ratio)
![Page 16: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/16.jpg)
16
Tools: Performance Profiles
Interpretation (for τ=ratio, P=profile):
! Efficiency:
! Probability of success:
! Compact graphs summarize all information
1for )( =ττP
∞→ττ as )(lim P
![Page 17: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/17.jpg)
17
Profiles (best resource time)
![Page 18: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/18.jpg)
18
Tools: Visualization
Performance Profiles: considers both
! Further disaggregation by objective function:
! Success only if best solution (over all solvers) found
Quality of Efficiency
Solution
![Page 19: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/19.jpg)
19
Profiles (best objective)
![Page 20: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/20.jpg)
20
Benchmarking process
Two components:
! Subjective component:! Choice of models! Choice of solvers! Choice of solver options
! Non-subjective component: ! Obtaining performance data! Performance analysis and visualization
" reproducible
![Page 21: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/21.jpg)
21
Subjectivity in Benchmarking
Performance Tools:! Takes care of non-subjective component
Choices:! Models?! Solver and solver options?
Example:! Can choose a set of models where each
solver is best:
![Page 22: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/22.jpg)
22
Profile: MINLP/Filter (best obj.)
![Page 23: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/23.jpg)
23
Profile: SBB (best obj.)
![Page 24: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/24.jpg)
24
Profile: DICOPT (best time)
![Page 25: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/25.jpg)
25
Issues
Choice of models, solvers, options is subjective!
! Models used can skew data! several models of same structure may exist with different
data
! Default or custom solver options! Platform dependence:
! Different resource times on different platforms
![Page 26: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/26.jpg)
26
Other Issues
Timings:
! How is resource time measured (dependence on solver)?
! How are resource time limits enforced?
! Intermediate results if resource time limit reached
![Page 27: Automated Performance Testing and Analysis€¦ · Automated Performance Testing and Analysis Armin Pruessner GAMS Development Corporation INFORMS Annual Meeting, San Jose Oct 17-20,](https://reader035.vdocuments.mx/reader035/viewer/2022081514/5f62035e1b9ac83f4227b1bd/html5/thumbnails/27.jpg)
27
Conclusions! Automation tools for collecting performance
measurements
! Tools for analyzing and visualizing test results ! Solver efficiency, robustness! Profiles and profile plots
! Enable users to reproduce performance results
! Automated performance analysis using the PAVER Server:
www.gamsworld.org/performance/paver