Software Defect Modeling at JPL

John N. Spagnuolo Jr. and John D. Powell19th International Forum on

COCOMO and Software Cost Modeling

10/27/2004

10/27/2004 John N. Spagnuolo Jr. &John D. Powell

2

Measurement & BenchmarkingProcess & Product Definition

Capture, define, and refine repeatable processes

and a set of engineering practices for project use

Provide measurement infrastructure for projects,

conduct empirical analysis, and package experiences for future use

Software Engineering Technology Infusion

Deployment

Identify, evaluate, and support software tools and techniques

to facilitate process and product improvement

Infuse practices into project use;provide training, products, mentoring

and consulting for projects

SQI Project EngineeringProvide overall technical infrastructure

and work element integration

SQI Project EngineeringProvide overall technical infrastructure

and work element integration

The JPL SQI Project

10/27/2004 John N. Spagnuolo Jr. &John D. Powell

3

SQI Measurement & Benchmarking• The objective of the SQI Measurement Program is to provide

the basis for a quantitatively based software management approach– Define models and measures– Create an infrastructure– Provide consulting and support– Produce Handbooks & Training

• Cost estimation and planning– Help develop total cost, schedule and plan for project activities and phases

• Quality planning and assessment– Help predict and assess the quality of products

• Management tracking – Help managers plan / monitor activities and assess risks during project execution

• Guiding improvement– Help JPL assess the overall effectiveness of software processes

10/27/2004 John N. Spagnuolo Jr. &John D. Powell

4

Quality Purpose and Goals

• The purpose of this work is to predict software quality via the development of Defect Prediction Models

– Predict defects and their effects on software projects early in project’s life cycle

• The current goal of this work is to make use of analysis of JPL defect data to support decision making.

– Determine and make use of existing software defect trends– Determine the driving forces behind exceptions to trends

• Critical Discriminators (CDs) are the distinguishing characteristics of a project(s) that capture/quantify the driving force behind a deviation from over all JPL defect trends

• Combine Critical discriminators along with trends to assess threats to software project goals

– Provide a means for managers to make use of Trends and CDs for • Planning• Prediction• Corrective Actions• Process Improvement

10/27/2004 John N. Spagnuolo Jr. &John D. Powell

5

Project-Defect-SLOC-ChartJ PL PROJECT Partial Defects All Defects Analyzed Defects Code Data Correlated Influence

1 Project F12 Project F23 Project F34 Project F45 Project F56 Project F67 Project F78 Project F89 Project F910 Project F1011 Project F1112 Project F1213 Project F1314 Project F1415 Project F1516 Project F1617 Project F1718 Project F18

19 Project G120 Project G221 Project G322 Project G423 Project G524 Project G625 Project G726 Project G827 Project G928 Project G1029 Project G1130 Project G1231 Project G1332 Project G1433 Project G1534 Project G1635 Project G1736 Project G1837 Project G1938 Project G2039 Project G2140 Project G2241 Project G2342 Project G2443 Project G2544 Project G26

Flight Software

Ground Software

10/27/2004 John N. Spagnuolo Jr. &John D. Powell

6

Basic Defect Prediction Approach

0

500

1000

1500

2000

2500

0 100 200 300 400 500 600 700 800 LKSLOC

Nu

mb

er

of

defe

cts

FSW

FSW & GSW

GSW

Early Defect Measurement

• Average Defect Density for– Flight Software (FSW)– Ground Software (GSW)– FSW & GSW

• Plot Lines with Slopes = Densities

10/27/2004 John N. Spagnuolo Jr. &John D. Powell

7

One Perspective of Data:

Average Work Hours per SDS vs SLOC • Formed Two Linear Trends• Philosophical Connection with Subsequent Charts

Average WH's/PFR vs LKSLOC

6.3

0

4.94

4.6042

11.4

8.85

2

4.293

5.539

14.4055

0

2

4

6

8

10

12

14

16

0 20 40 60 80 100 120 140 160 180 200

LKSLOC

Avera

ge W

H's

per

PFR

10/27/2004 John N. Spagnuolo Jr. &John D. Powell

8

Preview of Major Findings

• New Trends: Given the basic Defect prediction Approach (Previous slide), the actual JPL defect data shows defects trends that are:– Are Counter intuitive on first inspection– More complex on further inspection

• New CDs: Newly developed software size CDs simplify the complex and counterintuitive Defect versus Size trend.– Managers may quickly make use of simple valuable sub-trends that apply

to their projects:• Without dealing with overall trend complexities that may be based on factors

that do not apply to their project• Based on characteristics such as size that may be estimated early in the

software lifecycle and easily revised for accuracy as the project progresses.– The new CD provides a means for managers to make use of the trends for

planning, prediction, corrective actions and process improvement.

10/27/2004 John N. Spagnuolo Jr. &John D. Powell

9

The High Level Story

•Difference between Fault and Failure–Fault : error which may or may not have been discovered–Failure : Outward sign of an error (Discovered Fault)–In This presentation Failure = Defect

•Related Literature suggests that faults vs. code size may fit to an exponential curve

•Relation of exponential curve to Project G4 Data Plots–Linear fit is reasonable for Project G4 defects vs. size–Not necessarily a contradiction

•All faults may not have to be fixed•More testing resources and/or time

•Project G4 defect curve ~ exponential fault curve

•Expand and correlate Project G4 analysis to all FSW and GSW Data collected thus far

10/27/2004 John N. Spagnuolo Jr. &John D. Powell

10

Project G4 and JPL Projects

• Project G4 DATA : Number of Defects vs. SDS SLOC– Project G4 TEST (SDS’s) - Quadratic Fit the best – Project G4 OPS (SDS’s) - Quadratic Fit is the best – Project G4 TEST & OPS (SDS’s) - Quadratic Fit is the best– Project G4 TEST + OPS (SDS’s) - Quadratic Fit is the best

• JPL Projects: Number of Defects vs. Project Size– All FSW - Quadratic fit is the best– All GSW - Quadratic Fit is the best– All FSW & GSW - Quadratic Fit is the best

10/27/2004 John N. Spagnuolo Jr. &John D. Powell

11

All Collected FSW-TEST-June 30-2004

y = -0.0749x 2 + 28.781x - 363.52

R2 = 0.9981

0100200300400500600700800900

10001100120013001400150016001700180019002000210022002300240025002600

0 50 100 150 200 250 300 350

LKSLOC

Num

ber

of

Defe

cts

FSW

Poly. (FSW)

Interesting Fit for ALL JPL FSW

10/27/2004 John N. Spagnuolo Jr. &John D. Powell

12

JPL GSW -TEST June 30 2004

y = -0.0113x 2 + 8.7463x - 8.1403

R2 = 0.8374

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

1400

1500

1600

1700

1800

0 100 200 300 400 500 600 700 800

LKSLOC

Nu

mber

of

Defe

cts

All JPL FSW & GSW:TEST June 30 2004

y = -0.0104x 2 + 7.8956x + 135.87

R2 = 0.689

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

1400

1500

1600

1700

1800

0 100 200 300 400 500 600 700 800

LKSLOC

Num

ber o

f Def

ects

JASON1 GSW TEST

y = -0.0023x 2 + 0.7257x + 4.3511

R2 = 0.5132

0

5

10

15

20

25

30

35

40

45

50

55

60

65

70

75

80

0 50 100 150 200 250 300

LKSLOC

Num

ber

of D

efe

cts

JASON1 GSW : OPS

y = -0.0029x 2 + 0.5949x + 2.3269

R2 = 0.5848

0

2

4

6

8

10

12

14

16

18

20

22

24

26

28

30

32

34

36

38

40

0 20 40 60 80 100 120 140 160 180 200

LKSLOC

Num

ber

of D

efe

cts

Curve Extends to GSW, GSW&FSWProject G4 TEST, OPS, TEST & OPS,TEST+OPS

10/27/2004 John N. Spagnuolo Jr. &John D. Powell

13

Trends and Potential Explanation

• Upper bound on amount of testing resources & time– Testing resources / KSLOC is higher for smaller modules than for larger modules.– One possible explanation of defects / size ratios– Smaller Modules

• As code gets bigger more people tend to test more• Defect curve increases to a certain point

– Medium Modules• With further size increases, testing resource approach their upper limit• Defect curve begins to “level off”

– Larger Modules• Upper bound on testing resources for Project G4• Testing resources/KSLOC not big enough to maintain defect / size ratio

observed in smaller modules• Eventually defects / KSLOC trend curve begins to decrease

10/27/2004 John N. Spagnuolo Jr. &John D. Powell

14

Al FSW & GSW < 248 LKSLOC

y = 65.224e0.0317x

R2 = 0.7548

0100200300400500600700800900

10001100120013001400150016001700180019002000210022002300240025002600270028002900300031003200330034003500

0 20 40 60 80 100 120 140

FSW&GSW< 248 LKSLOCKExpon. (FSW&GSW< 248 LKSLOCK)

Where Does the Exponential come in? All FSW and GSW: Exponential Fit for LKSLOC < 247

All FSW & GSW

y = 220.32e 0.0005x

R2 = 0.0138

y = -0.0107x 2 + 8.0852x + 156.95

R2 = 0.7076

y = 0.0004x + 497.46

R2 = 6E-08

050

100150200250300350400450500550600650700750800850900950

10001050110011501200125013001350140014501500155016001650170017501800

0 100 200 300 400 500 600 700 800

All FSW & GSWVertical1Vertical2Expon. (All FSW & GSW)Poly. (All FSW & GSW)Linear (All FSW & GSW)Linear (Vertical1)Linear (Vertical2)

10/27/2004 John N. Spagnuolo Jr. &John D. Powell

15

Exponential Fits for JPL FSW, GSW Project G4 GSW TEST and Project G4

GSW OPSFSW < 160

LKSLOC

y = 19.295e 0.0661x

R2 = 0.9207

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

1400

1500

1600

1700

1800

1900

2000

0 10 20 30 40 50 60 70 80

GSW < 247 LKSLOC

y = 125.53e 0.0189x

R2 = 0.9157

0

50

100

150

200

250

300

350

400

450

500

550

600

650

700

750

800

850

0 10 20 30 40 50 60 70 80 90 100

JASON1 GSW: Exponential Fit for LKSLOC < 103

y = 1.6196e0.0992x

R2 = 0.874

0

5

10

15

20

25

30

35

40

45

50

55

60

65

70

75

80

0 5 10 15 20 25 30 35 40

LKSLOC

# o

f D

efe

cts

JASON1 OPS # of pfr's vs size <57K

y = 2.1255e 0.0777x

R2 = 0.8899

0

3

6

9

12

15

18

21

24

27

30

33

36

39

42

0 5 10 15 20 25 30 35 40

LKSLOC

# o

f pf

r's

10/27/2004 John N. Spagnuolo Jr. &John D. Powell

16

• Examination of Lower SLOC range: Project G4– Computed (max LKSLOC - min LKSLOC)/3– Project G4 TEST- Excellent fit to Exponential Curve < 103 LKSLOC– Project G4 OPS - Excellent fit to Exponential Curve < 57 LKSLOC– Project G4 TEST & OPS - Excellent fit to Exponential Curve < 103

LKSLOC– Project G4 TEST + OPS - Excellent fit to Exponential Curve < 103

LKSLOC

• Examination of Lower SLOC range: FSW, GSW, FSW & GSW

– Computed (max LKSLOC - min LKSLOC)/3– All FSW - Excellent fit to Exponential Curve < 118 LKSLOC– All GSW - Excellent fit to Exponential Curve < 247 LKSLOC– All FSW and GSW - Excellent fit to Exponential Curve < 247 LKSLOC

Relevance to Exponential Curve

10/27/2004 John N. Spagnuolo Jr. &John D. Powell

17

Observations at this Point

• The Empirical Evidence thus far seems to suggest:

– Number of defects seems to increase as code size increases, peaks out, then begins a decreasing trend.

– The exponential relationship between number of faults vs. code size, as indicated by the “Exponential curve”, appears to be most closely approximated by the relationship between the number of defects vs. code size for smaller sized Projects/SDS’s here at JPL

– Code does not require exponential discovery of faults to work appropriately for JPL purposes

10/27/2004 John N. Spagnuolo Jr. &John D. Powell

18

Summary• Defect Differences from Test to Operations by SDS

• The Size Critical Discriminator (CD) for Defect Rates– Size CD is a code size threshold

• Represents successful development of a new and valuable CD• Distinguishes between very different expected defects rate behaviors• Allows defect prediction superior to simple “defects per lines of code” models

– Significance of the Size CD• Intuitive results below Size CD threshold• Counterintuitive results above the Size CD threshold

– Plausible and Reasonable explanations – Valuable rules for projects

– Calculation of the Size CD and Differing Trends • Analysis of Size CD and Defect Trends gives reasonable confidence level• Very High Statistical Confidence Levels (R2, etc..) but more data is need to make

claims of high confidence overall• Actual Data driven results• Trends persist

– Across multiple analyses of the data– Across various subsets of the data

10/27/2004 John N. Spagnuolo Jr. &John D. Powell

19

• Relations between criticality 1 and criticality 2 defects• For each “Number of PFR’s vs.. size” chart, construct a chart

corresponding to defect density vs.. size – determine corresponding hump in trend curves

• Identify and Classify More Critical Discriminators– New individual CDs– Interaction of multiple CDs– For each CD associate modules, characteristics and data– Establish Trends for all modules corresponding to each CD

• Investigate interaction of multiple CDs at work in a given project / SDS

Future Work