accomplishments and challenges - york...

Performance Regression AnalysisPerformance Regression Analysis

Cor-Paul Bezemer

Accomplishments and ChallengesAccomplishments and Challenges

““A burnt child dreads fire”A burnt child dreads fire”

Regression testing makes sure we do Regression testing makes sure we do not make the same mistake twicenot make the same mistake twice

Write codeExecute regressiontest

Deploy or fix changes

Regression testing also helps us to make Regression testing also helps us to make sure we are not re-introducing more bugs sure we are not re-introducing more bugs

Performance regression testing helps ensure Performance regression testing helps ensure we are not breaking performancewe are not breaking performance

Write codeExecute performance test

Deploy if performance is OK or fix changes

Performance regression testing Performance regression testing can/should be done on several metricscan/should be done on several metrics

Performance regression testing Performance regression testing can/should be done on several metricscan/should be done on several metrics

Performance regression testing Performance regression testing can/should be done on several metricscan/should be done on several metrics

Performance regression testing Performance regression testing can/should be done on several metricscan/should be done on several metrics

Directions in research on Directions in research on performance regression analysisperformance regression analysis

Directions in performance Directions in performance regression analysis researchregression analysis research

1. Execution profile comparison

2. Source code analysis

3. Visualization

Execution profile comparisonExecution profile comparison

Execution profile comparisonExecution profile comparison

Disadvantages of execution Disadvantages of execution profile comparisonprofile comparison

- Need to run performance tests several times because of variation

- Small changes may be difficult to interpret

Source code analysisSource code analysis

Identify risky commits and recommend those for testing (Huang et al. @ ICSE 2014)

Disadvantages of source Disadvantages of source code analysiscode analysis

- Can not replace 'real' performance measurements (yet)

Visualization of performance Visualization of performance regression analysis resultsregression analysis results

Control charts Control charts (Nguyen et al. @ MSR 2014)(Nguyen et al. @ MSR 2014)

Differential flame graphsDifferential flame graphs(Bezemer et al. @ SANER 2015)(Bezemer et al. @ SANER 2015)

Deciding on the baseline Deciding on the baseline performanceperformance

Deciding on the acceptable Deciding on the acceptable performance deviationperformance deviation

Reporting results to the Reporting results to the developerdeveloper

Availability of performance Availability of performance teststests

Execution time of testsExecution time of tests

Performance regression testing helps ensure Performance regression testing helps ensure we are not breaking performancewe are not breaking performance

Write codeExecute performance test

Deploy if performance is OK or fix changes

Directions in performance Directions in performance regression analysis researchregression analysis research

1. Execution profile comparison

2. Source code analysis

3. Visualization

Availability of performance Availability of performance teststests

Execution time of testsExecution time of tests