spin 23 february 2006 norman fenton agena ltd and queen mary university of london

Slide 1

SPIN23 February 2006

Norman Fenton

Agena Ltd and

Queen Mary University of London

Improved Software Defect Prediction

Slide 2

delivery

Pre-release testing Post-release operation

Using fault data to predict reliability

Slide 3

Pre-release vs post-release faults?

0

0

Post-release faults

10

20

30

40 80 120 160

Pre-release faults

?

Slide 4

Pre-release vs post-release faults: actual

Post-release faults

0

10

20

30

0 40 80 120 160

Pre-release faults

Slide 5

Software metrics….?

Slide 6

Regression models….?

Slide 7

Solution: causal models (risk maps)

Slide 8

What’s special about this approach?

• Structured

• Visual

• Robust

Slide 9

The specific problem for Philips

time

Defectsfound

predicted

actual

Slide 10

Background to work with Philips

Fixed Risk Map (AID)

2000

MODIST

Agena Risk

2001 20032002 2004

Reliability models

Slide 11

Projects in the trial

MM&C (9)28%

TV (16)50%

DVD (7)22%

(actual number of projects shown in brackets)

Slide 12

Factors used at Philips

• Existing code base…

• Complexity and management of new requirements ...

• Development, testing, rework, process …

• Overall project management …

Slide 13

Actual versus predicted defects

0

200

400

600

800

1000

1200

1400

1600

1800

2000

0 500 1000 1500 2000 2500

Actual defects

pre

dic

ted

def

ects

Correlationcoefficient

95%

Slide 14

Validation Summary (Philips’ words)

“Bayesian Network approach is innovative and one of the most promising techniques for predicting defects in software projects”

“Our evaluation showed excellent results for project sizes between 10 and 90 KLOC”

“Initially, projects outside this range were not within the scope of the default model so predictions were less accurate, but accuracy was significantly improved after standard model tailoring”

“AgenaRisk is a valuable tool as is”

Slide 15

Specific benefits

• Accurate predictions of defects at different phases

• Know when to stop testing

• Identify where to allocate testing

• Minimise the cost of rework

• Highlight areas for improvement

• Use out of the box now if you have no data

• Approach fully customisable

• Models arbitrary project life-cycles

Slide 16

Summary

• Risk maps – the way forward

• Validation results excellent

Slide 17

…And

You can use the technology NOW

www.agenarisk.com