15 16-chapter 17-metrics

1

No part of this presentation may be reproduced without prior written permission from authors, copyright owner and publisher of the book

Metrics & MeasurementsChapter 17

04/10/23 Metrics & Measurements Slide : 2

Agenda

0.1 What is metrics?0.2 Why Metrics?0.3 Steps for metrics0.4 Types of metrics0.5 Overview slide

1. Project Metrics2. Progress Metrics3. Productivity Metrics4. Development Metrics5. Release Metrics


What is Metrics

This is the period we noticed excellent profits in the organization …and…

Boss, you were on vacation that period!


Terminology1. Set of data is called information and set of information

combined to provide a perspective is called Metrics.

2. A quantitative measure to explain at what degree an attribute of testing or product quality or process has performed is called Metrics.

3. Effort is the actual time that is spent on a particular activity or a phase. “Elapsed days” is the difference between start of an activity to completion of the activity.

4. Measurement is an unit used by metrics (e.g Effort, elapsed days, number of defects …etc). A metric typically uses one of more measurements


Why Metrics?

1. How do you determine quality and progress of testing?

2. How much testing is completed?

3. How much more time is needed for release?

4. How much time needed to fix defects?

5. How many Days needed for release?

6. How many defects that will be reported by customers?

7. Do you know how to prevent defects rather than finding and fixing them?

Do you have answers?


Why Metrics for QA?

1. Testing is penultimate cycle of product release --- Determining quality and progress of testing thus is very important

2. How much testing is completed can be measured if you know how much total testing is needed

3. How much more time is needed for release (e.g) Days needed to complete testing = total test cases yet to be executed / test case execution productivity

4. How much time needed to fix defects (e.g) The defect trend gives a rough estimate of defects that will come in future. Metrics helps in predicting the number of defects that can be found in future test cycles (e.g) Total days needed for defect fixes = (Outstanding defects yet to be fixed + Defects that can be found in future test cycles) / defect fixing capability

5. Days needed for release = Max of (days needed for testing, days needed for defect fixes)


Why Metrics for QA?

Days needed for release = Max of (Days needed for testing, (Days needed for defect fixes + Days needed for regressing outstanding defect fixes))

In Summary When to make the release What to release – Based on defect density across modules, their

importance to customers and impact analysis of those defects, scope of the product can be decided to release the product on time. Metrics help in making this decision.

Are we releasing the product with known quality? – The idea of metrics is not only for meeting the date but also to know the quality of product and ascertaining the decision on whether we are releasing the product with the known quality and whether it will function in predictable way in the field.


Steps for metrics

Step 1: Identify what measurements are important

Step 2: Define granularity of measurements ; Granularity depends on data drilling. Example

Tester: We found 100 more defects in this test pass compared to the previous one

Manager: What aspect of the product testing produced more defects?

Tester: Functionality aspect produced 60 defects out of 100

Manager: Good, what are the components in the product that produced more functional defects?

Tester: “Installation” component produced 40 out of those 60

Manager: What particular feature produced that many defects?

Tester: The data migration involving different schema produced 35 out of those 40 defects…….


Steps for metrics

Step 3: Decide on periodicity of metrics

Step 4: Analyze metrics and take action items for both positives and improvement areas

Step 5-n: Track action items from metrics


Types of metrics

Project metrics: The set of metrics which indicate how the project is planned and executed

Progress metrics: The set of metrics to indicate how different activities of the project are progressing. The activities include both development and testing activities. Since the focus of this training is testing, only those metrics applicable to testing are discussed.

Productivity metrics: The set of metrics that takes into account various productivity numbers that can be collected and used for planning and tracking the testing activities.


OverviewProcess Metrics

Product Metrics

Project Metrics

Progress Metrics

Productivity Metrics

Effort distribution

Schedule Variance

Effort Variance

Effort distribution

Schedule Variance

Effort Variance

Defect cause distribution

Weighted defects trend

Defect classification trend

Defects trend

Priority outstanding rate

Outstanding defects rate

Defect fix rate

Defect find rate

Defect cause distribution



Defects trend

Priority outstanding rate

Outstanding defects rate

Defect fix rate

Defect find rate

Introduced and reopened defects rate

Age analysis of outstanding defects

Defect density and defect removal rate

Component-wise defect distribution

Introduced and reopened defects rate



Component-wise defect distribution

Closed defects distribution

Test phase effectiveness

Defects per 100 failed test cases

Defects per 100 test cases

Test cases developed per 100 hours

Test cases executed per 100 hrs of testing

Defects per 100 hrs of testing



Defects per 100 failed test cases

Defects per 100 test cases

Test cases developed per 100 hours

Test cases executed per 100 hrs of testing

Defects per 100 hrs of testing.

.

.

.

Development defect metrics

Testing defect metrics


1. Project Metrics – Effort Variance (Planned Vs Actual)

Phase Wise Effort Variation

0.00

10.00

20.00

30.00

40.00

Req Design Coding Testing Doc Defectfixing

Pe

rso

n D

ay

s

Baselined Estimate Revised Estimate Actual

Knowledge Check

• What is baselined, revised estimates?.

• How person days is calculated?.

• What is the purpose of this metric?

• What is allowed variance %


1. Project Metrics – Schedule Variance (Planned Vs Actual)

Knowledge Check

• What is baselined, revised estimates?.

• How no of days is calculated?.

• What is the purpose of this metric?

• What is allowed variance % for both effort & schedule

Schedule Variance

126.00 136.00110.00

56.00

0.00

50.00

100.00

150.00

200.00

Baseline Estimated Actual/Remaining

No

. o

f D

ay

s

Estimated Remaining


1. Project Metrics – Effort & Schedule Variance

Over estimation & over schedule;Both effort and schedule Estimation needs improvement

Negative varianceNegative variance

Over estimation & schedule;Both effort and schedule Estimation needs improvement

Zero or acceptable variance

Negative variance

Under estimation of both effort and schedule

Unacceptable variance


Under estimation (People get burnt);Needs further analysis



Need slight improvement in effort / schedule estimation

Acceptable variance


A well executed projectZero varianceZero or Acceptable variance

Probable causes / ResultSchedule VarianceEffort Variance

Over estimation & over schedule;Both effort and schedule Estimation needs improvement

Negative varianceNegative variance

Over estimation & schedule;Both effort and schedule Estimation needs improvement


Negative variance

Under estimation of both effort and schedule



Under estimation (People get burnt);Needs further analysis



Need slight improvement in effort / schedule estimation

Acceptable variance


A well executed projectZero varianceZero or Acceptable variance

Probable causes / ResultSchedule VarianceEffort Variance


1. Project Metrics – Effort distributionActual effort distribution

23%

18%

15%

22%

0%

5%17%

Req Design Coding Testing Doc bug f ixing

1. Matured orgn spend atleast 10-15% in requirements 10-15% in design and 40-50% in testing (This data normally comes from time sheets)

2. Adequate effort needs to be spent in each of the SDLC phase for a quality product release (both more testing and less testing are issues)


2. Progress Metrics – Testing progress

0%

20%

40%

60%

80%

100%

1 2 3 4 5 6 7 8

Week

Test cases executed Blocked

Not Run

Fail

Pass

• Increase in pass % indicate, quality of product improving

• Decrease in Blocked % indicate, tests can progress well

• Reduced % in fail, is requirement for a release

• Not run % should be Zero for the release ; final week should have only Pass and Fail %


2. Progress Metrics – Defect find rate

Objective: The purpose of testing is to find defects early in the test cycle

Knowledge Check

1. Why bell curve (as above) happens in almost all releases if objective is to find all defects early?

Defect find rate

time->

Nu

mb

er

of

de

fect

s


2. Progress Metrics – Defect fix rate

Objective: The purpose of development is to fix defects as soon as they are identified

Knowledge Check

1. Why bell curve (as above) happens here?

Defect find rate

time->

Nu

mb

er

of

de

fect

sDefect fix rate


2. Progress Metrics – Outstanding defects

Objective: A well-executed project has the number of outstanding defects which is very close to zero all the time during test cycle

Knowledge Check

1. Why bell curve (as above) happens then?

Defect find rate

time->

Nu

mb

er

of

de

fect

sOutstanding defects


2. Progress Metrics – Priority Outstanding (P0, P1) defects

Objective: Provide additional focus for those defects that matters to the release

Defect find rate

time->

Nu

mb

er

of

de

fect

s


2. Progress Metrics – Defect trend

Objective: Effectiveness of analysis increases when several perspectives of find rate, fix rate, outstanding and priority outstanding defects are combined

Defect Trend

0

50

100

150

200

250

300

350

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

Week

Def

ects

Defect find rate

Defect fix rateOutstanding defects

Priority outstanding

Knowledge check:

What is your analysis about the trend above?


2. Progress Metrics – Defect Distribution & Trend

Defect distribution

P011%

P118%

P218%

P335%

P418%


020406080

100120140

1 2 3 4 5 6 7 8 9 10

week

P4

P3

P2

P1

P0

Objective:

Providing the perspective of defect classification in the chart helps in finding out on how the defects are distributed

Knowledge Check:Analyze week5 and week9 data


2. Progress Metrics – Weighted Defects

Weighted defects = (P0* 5 + P1 * 4 + P2 *3 + P3 *2+ P4)

Both “large defects” and “large number of small defects” affect the product release0

50

100

150

200

250

300

350

400

1 2 3 4 5 6 7 8 9 10Week

Weighted

Knowledge check:

What do you understand from week 9 data?

Can we make a release at the end of Week10?


3. Development Metrics – Defect Cause

Knowing the causes (Why a defect happened) of defects help in finding more defects and also in preventing such defects early

Knowledge check:

How do you prevent defects?

What is the diff between Change Request (not defects) and Feature request?

Requirement15%

Design10%

Code37%

Feature request

4%

Change request

20%

Third party8%

Others6% Requirement

Design

Code

Featurerequest

Changerequest

Third party

Others


3. Development Metrics – Module-wise defects

Knowing the components producing more defects help in defect fix plan and in deciding what to release

0 10 20 30 40

Defect

Install

Reports

admin

login

GUI

Client

Server

Database

Media

API

Modulewise Defect DistributionP0

P1

P2

P3

P4

Knowledge Check:

How do you decide what to release?


3. Development Metrics – Defects/KLOC & defect removal rate

Defects per KLOC = (Total defects found in the product) / (Total executable AMD lines of the code in KLOC)

Note: AMD=added/modified/deleted

Defect removal % = (Defects found thru verification activities + defects found by DEV team) / (Defects found by test teams)* 100

Defects/KLOC & Defect removal %

0

10

20

30

40

50

60

70

1 2 3 4 5 6 7 8Releases

Val

ue

Defects/KLOC

Defect removal %


3. Development Metrics – Age analysis of outstanding defects


0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5 6 7 8 9 10

Week

Cu

mu

lati

ve

ag

e

P4

P3

P2

P1

P0The time needed

to fix a defect may be

proportional to its age

Knowledge check:

What do you observe based on week-4 data?

What is happening from week-3 to week-7?


3. Development Metrics – Reopened and introduced defects

Testing is not there to find same

defects again; Release readiness should consider quality of defect

fixes

Knowledge check:

What do you observe based on week-4 data?

What is happening from week-3 to week-7?

0

10

20

30

40

50

60

Defects

1 2 3 4 5 6 7 8 9 10

Week

Introduced & Reopened Defects Reopened defects

Introduced defects


4. Productivity metrics

Time for a break! – how to communicate productivity loss

A manager to his Secretary: I have plenty of work to do in the afternoon; how about you taking afternoon off…

Manager to his sub-ordinate: We are not firing you! We are just fixing expense limits and you have already exceeded yours!


4. Productivity metrics – Defects per 100 hrs of testing

Defects per 100 hours of testing

0

20

40

60

80

100

120

1 2 3 4 5 6 7 8 9 10

Week

Defec

ts

Cosmetic

Minor

Important

Critical

Extreme

Defects per 100 hours of testing = (Total defects found in the product for a period / Total hours spent to get those defects) * 100

Normalizing the defects with effort spent indicates another perspective for release quality


4. Productivity metrics – Test productivity

Productivity Metrics

0

20

40

60

80

100

120

140

160

180

200

1 2 3 4 5 6 7 8 9 10Week

Test cases executed per 100 hours Test cases developed per 100 hours

Defects per 100 test cases Defects per 100 failed test cases


4. Productivity metrics – Test Phase effectiveness


CT32%

IT17%

ST12%

UT39%

Testing is the responsibility of everyone and multiple teams does testing

Hence it is important to analyze which phase (not teams) found more defects

Knowledge check:

What is the right ratio for unit, Feature/component, Integration and system test?


4. Productivity metrics – Closed defect distribution

Fixed in closed is a good metric to have for both DEV & test teams

Duplicate to be avoided (<5%)

Not reproducable defects may reappear again; need to be careful

Defects moving to next release needs to be with in certain band (3-6%)

Closed defect distribution

Will not fix32%

Next release1%

Others8%

Fixed28%

Duplicate19%

Not reproducable

11%

As per design7%


5. Release Metrics

1.Weighted defects trend showing “Bell curve”2.Close to zero weighted defects in the last few weeks prior to release

High priority defects as well as high number of low priority defects


Close to zero high-priority defects in the last few weeks prior to release

High Priority defectsPriority Outstanding defects trend

1.Outstanding defects trend showing “downward” trend2.Close to zero outstanding defects in the last few weeks prior to release

Outstanding defectsOutstanding defects trend

Defect fixing trend matching arrival trendDefect fix trendDefect fix rate

1.Defect arrival trend showing ‘bell curve”2.Incoming defects close to zero in the last week

Defect trendDefect find rate

15-20% effort spent each on Requirements, design and Testing phases

Adequate effort has been spent on all phases

Effort Distribution

1.All 100% of test cases to be executed2.Test cases passed should be minimum 98%

Execution %Pass %

Test cases executed

GuidelinesPerspectives to be considered

Name of the metric

1.Weighted defects trend showing “Bell curve”2.Close to zero weighted defects in the last few weeks prior to release

High priority defects as well as high number of low priority defects


Close to zero high-priority defects in the last few weeks prior to release

High Priority defectsPriority Outstanding defects trend

1.Outstanding defects trend showing “downward” trend2.Close to zero outstanding defects in the last few weeks prior to release

Outstanding defectsOutstanding defects trend

Defect fixing trend matching arrival trendDefect fix trendDefect fix rate

1.Defect arrival trend showing ‘bell curve”2.Incoming defects close to zero in the last week

Defect trendDefect find rate

15-20% effort spent each on Requirements, design and Testing phases

Adequate effort has been spent on all phases

Effort Distribution

1.All 100% of test cases to be executed2.Test cases passed should be minimum 98%

Execution %Pass %

Test cases executed


Name of the metric


5. Release Metrics

Test cases executed showing an upward trend

Whether improved quality in product allowing more test cases being executed

Whether test cases executed is proportional to effort spent

Test cases executed per 100 hours of testing

1. Defects per 100 hours of testing should be less than 5

2. Defects per 100 hours of testing trend showing downward trend

Whether defect arrival is proportional to effort spent


1. Combined number of outstanding & reopened defects showing downward trend

2. Introduced & reopened defects are less than 5% of defect arrival rate

Quality of defect fixSame defects reappearing

again

Introduced and reopened defects

Age of defects showing downward trendAge of defectsAge analysis of outstanding defects

1. Defects / KLOC less than 72. Defects / KLOC less than last release3. Defect removal is 50% of more4. Defect removal % better than last

release

Defects /KLOCDefect removal %



Name of the metric

Test cases executed showing an upward trend

Whether improved quality in product allowing more test cases being executed

Whether test cases executed is proportional to effort spent

Test cases executed per 100 hours of testing

1. Defects per 100 hours of testing should be less than 5

2. Defects per 100 hours of testing trend showing downward trend

Whether defect arrival is proportional to effort spent


1. Combined number of outstanding & reopened defects showing downward trend

2. Introduced & reopened defects are less than 5% of defect arrival rate

Quality of defect fixSame defects reappearing

again

Introduced and reopened defects

Age of defects showing downward trendAge of defectsAge analysis of outstanding defects

1. Defects / KLOC less than 72. Defects / KLOC less than last release3. Defect removal is 50% of more4. Defect removal % better than last

release

Defects /KLOCDefect removal %



Name of the metric


5. Release Metrics

1.At least 70% of closed defects are fixed2.Non-reproducible defects are less than 5%3.Defects moved to next release should be less than 10%

Whether good proportion of defects found by testing are fixed


1.Very low percentage of defects found in system and acceptance test phase (say less than 12%)2.A distribution of defects and reduction in defects % compared to next test phase3.A distribution of UT=50%, CT=30%, IT=13% and ST=7% would be ideal

Defects found in each of the test phase


GuidelinesPerspectives to be consideredName of the metric

1.At least 70% of closed defects are fixed2.Non-reproducible defects are less than 5%3.Defects moved to next release should be less than 10%

Whether good proportion of defects found by testing are fixed


1.Very low percentage of defects found in system and acceptance test phase (say less than 12%)2.A distribution of defects and reduction in defects % compared to next test phase3.A distribution of UT=50%, CT=30%, IT=13% and ST=7% would be ideal

Defects found in each of the test phase


GuidelinesPerspectives to be consideredName of the metric


Agenda - Recap

0.1 What is metrics?0.2 Why Metrics?0.3 Steps for metrics0.4 Types of metrics0.5 Overview slide

1. Project Metrics2. Progress Metrics3. Productivity Metrics4. Development Metrics5. Release Metrics

15 16-chapter 17-metrics

Software

release metrics

set of metrics

development metrics

metrics applicable

idea of metrics

metrics measurements

periodicity of metrics

product testing