msr2016 tarek

1

Studying the Effectiveness of Application Performance Management (APM) Tools for

Detecting Performance Regressions for Web Applications

Tse-Hsun Chen Ahmed E. Hassan Weiyi ShangTarek M. Ahmed Cor-Paul Bezemer

An Experience Report

2

3

$1.6 billion loss for a one-second slowdown

4

What is a performance regression?

Version 1 Version 2

Does version 2 have worse performance than the version 1?

5

Performance regression:changed source code + unchanged workload

change in performance?

Version 1

Version 2

requests

requests

requests

Performance metrics

Performance metrics

Unchanged workload Testing environment

Testing environment

Changed source code Change in

performance?

6

MSR techniques are widely used to detect performance regressions

7

MSR techniques are widely used to detect performance regressions

Building statistical models Mining data associationsComparing distributions

8

An example of using MSR techniques to detect performance regressions

Version 1 Version 2

Build model

Verify model

Model Prediction error

< threshold

> threshold

Shang et al. ICPE 2015

9

The extensive performance regression detection research is hardly adopted in practice

Too complex to implement by practitionersNot easily accessible

10

Application performance management (APM) tools are well adopted in practice

11

APM tools work with:unchanged source code + changing workload

requests

requests

requests

Real users

Field environment

Slow transactionsChanging workload

Unchanged source code

12

An example report from an APM tool

Most time consuming transactions

Time breakdown for the transaction

13

Performance regressions detection research

+ unchanged workload

changed source code

Not well adopted

unchanged source code +

changing workload

Widely adopted

14

Research Questions:

Can APM tools detect performance regressions?

Can APM tools be used as a vehicle to deploy performance regression detection research in practice?

15

Case study: subject systems and injected performance regressions

CPU overhead

Memory overhead

Inefficient use of database

• Unreleased resources• Inefficient use of data streams

• Unnecessary loop• Loop with a missing break

condition

• One-by-one processing• Excessive data

16

We leverage APM tools to detect injected performance regressions

Slow transactions in new version

Slow transactions in old version

Version 1

Version 2

requests

requests

requests

Unchanged workload Testing environment

Testing environment

Changed source code

17

Series10

1

2

3

4

5

6 Series1 Series2

APM tools can detect performance regressions

# of

det

ecte

d iss

ues d

etec

ted

Commercial Open source

Total 6 issues injected

18

APM tools have rooms of improvement

APM tools are a great vehicle to push performance regression detection research into practice.

Large amounts of reports for manual exploration Basic analysis

Lack of actionable suggestion

19

$1.6 billion loss for a one-second slowdown

20

21

unchanged source code +

changing workload

Widely adopted

Performance regressions detection research

changed source code +

unchanged workload

Not well adopted

22

23

Case study: subject systems and injected performance regressions

CPU overhead

Memory overhead

Inefficient use of database

• Unreleased resources• Inefficient use of data streams

• Unnecessary loop• Loop with a missing break

condition

• One-by-one processing• Excessive data

24

25

Series10

1

2

3

4

5

6 Series1 Series2

APM tools can detect performance regressions

# of

det

ecte

d iss

ues d

etec

ted

Commercial Open source

Total 6 issues injected

26

Weiyi Shang shang@.encs.concordia.ca