making numbers count: metrics that matter
DESCRIPTION
As testers and test managers, we are frequently asked to report on the progress and results of our testing. The question “How is testing going?” may seem simple enough, but our answer is ultimately based on our ability to extract useful metrics from our work and present them in a meaningful way. This is particularly important in agile environments, where clear, concise, and up-to-date metrics are potentially needed multiple times per day. Mike Trites identifies a number of ways metrics can be used to measure progress during a test cycle and, ultimately, to determine when to consider testing complete. Learn the common pitfalls that metrics misuse can lead to and how you can avoid them by giving proper context when communicating metrics to your stakeholders. Take back key metrics for measuring the effectiveness of your testing and discover how to use what is learned on one project to improve your testing process on future projects.TRANSCRIPT
W18
Special Topics
5/7/2014 3:00:00 PM
Making Numbers Count: Metrics
That Matter
Presented by:
Mike Trites
Professional Quality Assurance, Ltd.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Mike Trites
Professional Quality Assurance, Ltd.
Senior test consultant with Professional Quality Assurance, Ltd. Mike Trites has nearly a decade of experience in the software testing industry and holds multiple certifications, including an Advanced-Level Test Manager Certification from the ISTQB. As a consultant, Mike has worked with clients, both small and large, in a wide range of environments. He has helped deliver projects from web applications and database management software to financial software and video lottery systems. As a test manager, Mike has managed all aspects of the testing lifecycle and works with clients to help define, implement, and improve their internal testing processes. Find Mike on LinkedIn at ca.linkedin.com/in/miketrites.
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 1
STAREAST 2014
May 4 – 9, 2014
Orlando, Florida
Mike Trites
Making Numbers Count
Metrics that Matter
About Me
� Mike Trites
‣ Senior Test Consultant
‣ Nearly 10 years of experience in software testing, working
with several clients in multiple industry verticals
� Certifications
‣ Black Box Software Testing Foundations – AST
‣ Certified Tester, Foundation Level – ISTQB
‣ Certified Tester, Advanced Level Test Manager – ISTQB
2©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 2
Introduction
3©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
How is testing going?
“95% of test cases so far
have passed.”
“We found 15 new defects yesterday.”
“We’re on-track.”
What I’m here forA
� Illustrate the important role that metrics play in the software
testing process.
� Explain how to give context when communicating metrics.
� Describe how to use metrics to improve the testing process
for future projects.
� Identify some common pitfalls and how to avoid them.
� Identify a selection of metrics to:
‣ Measure progress during a test cycle.
‣ Determine when to stop testing.
‣ Measure the effectiveness of testing.
4©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 3
What I’m NOT here forA
� Provide a “one size fits all” solution for you or your
organization.
� Why not?
‣ Metrics are situation / context specific.
‣ There is no silver bullet.
‣ There is no “right” answer.
‣ They will not solve all your problems.
5©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Agenda
� Why Metrics are Important
� Defining and Capturing Metrics
� Reporting on Metrics
� Using Metrics for Process
Improvement
� Misuse of Metrics
6©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 4
Why Metrics are Important
8©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 5
Why Metrics are Important
9©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Metrics play an important role in:
• Estimation
• Evaluating suspension criteria
• Evaluating exit criteria
• Status reporting
• Process improvement
Estimation
� Metrics captured on one project can help us estimate
various activities for future projects.
� With enough historical data, we can begin to estimate:
‣ Number of test cases
‣ Execution rate
‣ Number of defects
‣ Overall effort
10©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 6
Suspension Criteria
� Occasionally, it may be necessary to halt testing of a
particular version / build if the quality is found to be too low.
� How do we make this determination?
‣ Number of defects?
‣ Number of high-severity defects?
‣ Defect detection rate?
‣ Test failure rate?
11©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Exit Criteria
� How do we know when to stop testing?
‣ The allotted effort (schedule and / or budget) is expended.
‣ All planned tests have been executed.
‣ All planned tests pass.
‣ The defect detection rate drops below a certain threshold.
� There are pros and cons for each approach.
12©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 7
Exit Criteria
� The best approach is a combination that takes into account
multiple criteria, such as:
‣ Effort (schedule / budget)
‣ Defect detection rate
‣ Number of open defects
‣ Completion percentage
‣ Pass rate
13©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Status Reporting
� Simple and clear method for communicating the current
status of testing to stakeholders
� Allows stakeholders to:
‣ Draw conclusions
‣ Make decisions
� Detailed status reports provide a transparent view into the
testing effort and show progress toward defined exit criteria.
14©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 8
Defining and Capturing Metrics
Defining Metrics
Every phase of the testing process can be measured.
� Schedule / Budget
� Test Coverage:
‣ Requirements
‣ Configurations
‣ Code
‣ Automation
� Test Design
� Test Execution
� Defects
16©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 9
Capturing Metrics
17©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Expensive commercial tools can be
useful, but are not always necessary.
• Examples:
‣ Test Case Management Tool
‣ Defect Management Tool
‣ Time Tracking System
There are less expensive tools that can
be used to achieve the same results:
• Examples:
‣ Bugzilla
‣ JIRA
‣ Excel
Reporting on Metrics
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 10
Metrics Need Context
� When metrics are used in isolation, their significance is left
open to interpretation.
� Stakeholders can draw the wrong conclusions if no context
is provided and they are instead left to interpret the data on
their own.
� Textual summary can provide context, but so can a good
report.
19©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
What makes a good report?
� Accurate – Data is not purposely skewed or omitted.
� Clear – Information is presented in a way that is
unambiguous and easy to interpret at a glance.
� Concise – Report is not cluttered with unneeded or
distracting data.
� Includes any necessary:
‣ Titles
‣ Labels
‣ Data values
‣ Trend lines
‣ Annotations
20©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 11
21©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Medium68%
Low 18%
High 14%
Medium68%
Low 18%
High 14%
Colour Conventions
Defects by Severity
More meaningful with
appropriate colours
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Example: Test Progress
22©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 12
Example: Requirements Coverage
23©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Example: Configuration Coverage
24©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
WindowsXP
WindowsVista
Windows7
Windows8
MacOSX
IE 7 ✓✓✓✓ ✓✓✓✓
IE 8 ✓✓✓✓ ✓✓✓✓
IE 9 ✓✓✓✓ ✓✓✓✓
IE 10 ✓✓✓✓
Chrome ✓✓✓✓ ✓✓✓✓ ✓✓✓✓ ✓✓✓✓ ✓✓✓✓
Firefox ✓✓✓✓ ✓✓✓✓ ✓✓✓✓ ✓✓✓✓ ✓✓✓✓
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 13
Example: Automation Coverage
25©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Total Number
of Test Cases
Cycle 2
(40%)
Cycle 1
(15%)
0
5
10
15
20
25
30
Example: Open vs. Closed Defects
26©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
0
5
10
15
20
25
30
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 14
0
2
4
6
8
10
12
14
16
18
0
2
4
6
8
10
12
14
16
18
Example: Open Defects
27©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Example: Defects by Release
28©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
= 63 (49%)
= 56 (44%)
= 9 (7%)
Fixed / Planned
v0.1 v0.2 v0.3 v0.4 v0.5 v0.6 v0.7
Discovered
v0.1 1 9 2
v0.2 7 10 15 1 1 3
v0.3 11 17 1 3
v0.4 1 31 10 3
v0.5 1 1
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 15
Example: Scorecard
29©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Execution
Progress
Pass
Rate
Blocked
TestsMust-Fix Other
Component 1 100 % 100 % 0 0 0
Component 2 100 % 78 % 0 0 4
Component 3 87 % 80 % 0 1 1
Component 4 95 % 100 % 0 0 4
Component 5 75 % 92 % 1 1 0
Component 6 98 % 79 % 2 1 4
Overall 93 % 88 % 3 3 13
Defects Test Execution
Execution
Progress
Pass
Rate
Blocked
TestsMust-Fix Other
Component 1 100 % 100 % 0 0 0
Component 2 100 % 78 % 0 0 4
Component 3 87 % 80 % 0 1 1
Component 4 95 % 100 % 0 0 4
Component 5 75 % 92 % 1 1 0
Component 6 98 % 79 % 2 1 4
Overall 93 % 88 % 3 3 13
Defects Test Execution
http://dilbert.com/strips/comic/2007-05-16/
Dashboards
30©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 16
Process Improvement
Process Improvement
"Those who cannot remember the past are condemned to
repeat it.”
- George Santayana
� What is “better”? How do we know if we’re better now than
we were?
� Estimates:
‣ Comparing planned effort and actual effort
‣ Estimate variance and what factors affected it
32©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 17
Metrics for Process Improvement
33©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Closure Period / Turn-around Time
• Measures how long it takes for a reported defect to be rectified
Defect Ratio
• Can be used to predict how many defects may be found based on the number of test cases that remain to be executed
Metrics for Process Improvement
34©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Root Cause Analysis
• Allows you to proactively avoid anticipated defects in the future
Closed Defects by Resolution
• Identifies instances of duplicate or invalid defects
Defect Removal Efficiency
• Identifies defects that are uncovered after the application is released
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 18
Trends
� One measurement captures status at a point in time.
� When a series of data points are examined, trends in that
data start to emerge.
� Trends allow us to
‣ Differentiate between systemic behaviour and temporary
anomalies.
‣ Make predictions about the future.
� Accuracy increases with more historical data.
35©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Misuse of Metrics
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 19
Intentional Manipulation
“There are three kinds of lies: lies, damned lies, and
statistics.”
- Benjamin Disraeli
� Metrics can be manipulated to make
any situation look good or bad.
37©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Misleading Graphs
An example of a good graph:
38©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
0
100
200
300
400
500
A B C D
430455 448 465
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 20
Misleading Graphs
Inappropriate scale:
39©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
0
200
400
600
800
1000
A B C D
Misleading Graphs
Scale does not start at 0:
40©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
420
425
430
435
440
445
450
455
460
465
A B C D
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 21
Tester 1Tester 2
Tester 3
Tester 4
41©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Misleading Graphs
Defects by Tester
11%42%
5%42%
Tester 1
Tester 2
Tester 3
Tester 4
Defects by
Tester
42©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Misleading Graphs
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 22
Poor Choice of Format
43©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
12%
15%
14%
8%11%
17%
13%
10% Tester 1
Tester 2
Tester 3
Tester 4
Tester 5
Tester 6
Tester 7
Tester 8
Defects by Tester
Better Choice of Format
44©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
12%
15%14%
8%
11%
16%
13%
10%
Defects by Tester
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 23
Overloading
� Most managers are interested in
defect statistics.
� Does defect count measure:
‣ Tester effectiveness?
‣ Developer skill?
‣ Product quality?
‣ Effort remaining?
� One metric does not tell the entire story.
45©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Surrogate Measures
� Sometimes, we don’t have a way to easily measure the
attribute we are really interested in.
� Instead, we use another, more convenient metric in its
place.
� But, does the metric we have chosen actually give us the
information we are looking for?
‣ To find out, we can use a “five-why” analysis
� Examples:
‣ Code Coverage
‣ Velocity
46©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 24
Performance Assessment
� People optimize what we measure them against, at the
expense of what we don’t.
� For example, if we assess testers based on their defect
counts, how much time will testers spend:
‣ Documenting their tests?
‣ Coaching other testers?
‣ Researching and polishing defect reports to make their
defect easier to understand, assess and replicate?
‣ Logging variants of the same failure to increase defect
count?
47©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 25
Key Points to Remember
� Metrics can be captured for just about any testing activity.
� Metrics play an important role in:
‣ Estimation
‣ Status reporting
‣ Evaluating suspension and exit criteria
‣ Process improvement
� Metrics, like anything, should be planned in advance.
� Reporting needs should be identified before setting up
tools.
� To avoid misinterpretation, metrics need context.
49©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Key Points to Remember
� Even when properly presented, too much information can
be overwhelming.
� It is best to limit the number of metrics and reports that are
presented to stakeholders to only what is essential.
� Capturing, analysing and reporting on metrics can be a
drain or waste of time if those metrics are not being used to
provide some value in return.
� Metrics influence tester performance, but not always in the
way you might think.
� Metrics should be used as a starting point for discussion.
� One metric does not tell the entire story.
50©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
PQA Company Overview 4/26/2014
©2012 Professional Quality Assurance Ltd. (PQA) --
http://www.pqatesting.com 26
Questions or Comments?
51©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Thanks for coming!
Contact Information:
52©2014 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
ca.linkedin.com/in/miketrites
@mike_trites
www.pqatesting.com