current defect density statistics - softrel, llc defect density statistics.pdf · copyright...

You may not reprint or copy any part of this presentation withouYou may not reprint or copy any part of this presentation without express and written t express and written consent from Ann Marie consent from Ann Marie NeufelderNeufelder

Current Defect Density Current Defect Density StatisticsStatistics

Ann Marie Ann Marie NeufelderNeufelderCopyright SoftRel, LLC 2007Copyright SoftRel, LLC 2007

Copyright SoftRel, LLC 2007Copyright SoftRel, LLC 2007

Actual fielded defect density from 90+ projects Actual fielded defect density from 90+ projects spanning nearly every industryspanning nearly every industry

Defect density clusters

00.5

11.5

22.5

3

0 0.5 1

Percentile group

Ave

rage

del

iver

ed

norm

aliz

ed d

efec

t de

nsity

World classVery good

Good

Average

FairPoor

Ugly

This data is in terms of fielded (escaped) defects per 1000 lines of effective code normalized to assembler.

Seven “clusters” are visible. A method to predict which cluster your project will fall into was developed from this data.


How to determine normalized effective sizeHow to determine normalized effective size

Predict/count new and modified lines of codePredict/count new and modified lines of codePredict/count deleted lines Predict/count deleted lines Multiply existing but unchanged code by 10%Multiply existing but unchanged code by 10%Entire functions deleted reduce existing sizeEntire functions deleted reduce existing size

Effective size = Modified + New + Lines subtracted + (10% of exiEffective size = Modified + New + Lines subtracted + (10% of existing code)sting code)

Multiply effective size by conversion ratio to Multiply effective size by conversion ratio to assembler using industry tables as summarized assembler using industry tables as summarized belowbelow Second generation (C, Fortran) Second generation (C, Fortran) –– 33 Object oriented (Java, C++, Object oriented (Java, C++, AdaAda 9x) 9x) –– 66 Visual Basic Visual Basic –– 1010


Use the SoftRel Survey to predict the best Use the SoftRel Survey to predict the best clustercluster

Appropriate Appropriate ““ClusterCluster”” determined by these thingsdetermined by these things Inherent stability of existing design and codeInherent stability of existing design and code Methods and techniques use to prevent defects and develop Methods and techniques use to prevent defects and develop

softwaresoftware Application typeApplication type Existence of major obstacles (new technology, new Existence of major obstacles (new technology, new

environments, etc.)environments, etc.) Existence of major opportunities (end user domain experts Existence of major opportunities (end user domain experts

available to project, etc.)available to project, etc.) Inherent stability of development processInherent stability of development process

Process alone will not guarantee a world class cluster!Process alone will not guarantee a world class cluster! An SEI CMM level 1 organization can be world class An SEI CMM level 1 organization can be world class An SEI CMM level 4 or 5 does not guarantee world class An SEI CMM level 4 or 5 does not guarantee world class


How the survey score maps to the clusters How the survey score maps to the clusters

SoftRel survey score versus defect density clusters

0

20

40

60

0 0.5 1

Percentile group

Sof

tRel

Sur

vey

scor

e

World classVery goodGoodAverageFairPoorUgly

The most variation exists in the world class cluster, however, this cluster is easily predictable because of void of major obstacles and presence of major opportunities as shown next.


The World Class Cluster had no major The World Class Cluster had no major obstaclesobstacles

Major project obstacles versus defect density clusters

0123456

0 0.5 1

Percentile group

Num

ber o

f maj

or

proj

ect o

bsta

cles World class

Very good

Good

Average

FairPoor

Ugly

Obstacles are defined specifically as – new technology, new operating system, new development environment, new compiler, new target hardware


The The ““UglyUgly”” group had no opportunities group had no opportunities

Major project opportunities versus defect density clusters

0

2

4

6

8

0 0.5 1

Percentile group

Num

ber

of m

ajor

pr

ojec

t op

port

uniti

es

World classVery goodGoodAverageFairPoorUgly

Opportunity – Explicitly defined as the degree to which end user domain experts are available to the software engineers for this project


Average defect density by system typeAverage defect density by system type

5.12.1040.414Total/average3.60.4480.123No special target hardware

n/a 1.0925Power systemsn/a 0.134GPS

23.74.7870.202Small devices3.82.4950.649

Large stationery capitol equipment

4.10.3580.087Satelliten/a0.106Military ground vehicle

33.90.3660.011Command, control and

communications

1.70.1800.106Command and control

Ratio of test to field

Average testing defect density

Average fielded defect densitySystem application type


Average defect density by software typeAverage defect density by software type

0.70.2780.378Application process evolving4.02.5880.642Target HW is new or evolving

11.10.0910.008Web based3.51.5130.430Mathematically intensive3.21.4590.456DB interfaces4.72.1040.449Multi-tasking4.62.1720.476Real time4.00.4340.108Client server

n/a n/a0.068Domain knowledge can be acquired via public

domain in short period of time

3.21.2900.400Biometrics18.73.0920.165Wireless capabilities



Average fielded defect densitySoftware application type

This is the same set of data sliced a different way.


Average defect density by risk levelAverage defect density by risk level

22.53.1650.141Government regulated

13.73.1580.230Recall risk

3.21.5350.476Monetary risks (loss of product with monetary

value)

26.94.5390.169Legal risks (banking, etc)

5.12.6080.509Safety risk (occupational, regional, national or

global)



Average fielded defect densityRisk level

This is the same set of data sliced a different way.


You can also predict the risk of a late deliveryYou can also predict the risk of a late delivery

Normalized Fielded Defect Density

Percentile

Ratio of testing to fielded defect density Ave Min Max

Stddev

Probability of a late delivery

(%)

Margin of error when

delivery is late (%)

World Class 8.5 .011 0.0055 0.0180 .006 10 17.5 Very Good 12.4 .060 0.0396 0.0756 .0172 20 25

Good 10.7 .112 0.0888 0.135 .0169 25 25 Average 10.6 .250 0.180 0.366 .0590 36 41

Fair 2.1 .618 0.400 0.835 .177 85 125 Poor 16.1 1.111 1.0357 1.224 .081 100 100 Ugly .5 2.069 1.743 2.674 .524 83 75

Probability of late delivery – If your organization makes 10 releases and the probability of being late is 10% then 1 out of 10 will be late

Margin of error – Measured as a percentage of the original schedule prediction


How to predict your clusterHow to predict your clusterAnswer a survey based onAnswer a survey based on RisksRisks Product characteristics Product characteristics Application typeApplication type ResourcesResources Practices, techniques and methodsPractices, techniques and methods Process stabilityProcess stability

Determine a baseline cluster for your Determine a baseline cluster for your ““typicaltypical””projectprojectEach project specific additional obstacle lowers Each project specific additional obstacle lowers the cluster while adding domain expertise raises the cluster while adding domain expertise raises the clusterthe cluster


Lessons LearnedLessons LearnedRisks cannot be overcome by any of the followingRisks cannot be overcome by any of the following New expensive automated tools to theoretically speed up New expensive automated tools to theoretically speed up

development (this will actually increase the risk level for the development (this will actually increase the risk level for the first first time project)time project)

Wishful thinkingWishful thinkingRisks can be minimized byRisks can be minimized by More granular milestonesMore granular milestones Addressing high risk items before everything else in the Addressing high risk items before everything else in the

scheduleschedulesoftware engineers tend to work on the low risk tasks firstsoftware engineers tend to work on the low risk tasks first

Design prototyping when design is a riskDesign prototyping when design is a risk Requirements prototyping when end user requirements are Requirements prototyping when end user requirements are

volatilevolatile Defect prevention techniques such as formal unit testingDefect prevention techniques such as formal unit testing Increasing the end user domain knowledge of the team Increasing the end user domain knowledge of the team

This does not mean software experience This does not mean software experience –– this means application this means application experienceexperience

current defect density statistics - softrel, llc defect density statistics.pdf · copyright...

Documents