are bugs eating your software budget?

36
AdvantageLeadership.com 1 Are Bugs Eating Your Budget? Cost of Quality Analysis Results Rebecca Staton-Reinstein Irv Brownstein 1 Advantage Leadership, Inc.

Upload: rebecca-staton-reinstein

Post on 12-Jun-2015

652 views

Category:

Technology


1 download

DESCRIPTION

20% of software projects fail. Failed projects cost the world economy over $500 billion a year. The hidden costs of making, finding, and fixing software defects consume 30% or more of the IT budget. The facts are continuing to get worse. Software bugs are eating your lunch. You can reverse the trend as many companies have by applying some proven, practical methods – Good Software Engineering Practice. You can do much of this on your own if you analyze the Cost of Quality and use it as a fundamental metric to demonstrate the effectiveness and efficiency of everything you do. If you add Fagan Inspections (static testing) to your existing software testing you can start reducing the cost of quality immediately. In this study of one company’s results over several years, we show you step by step how they demonstrated over $6 million in cost savings and avoidance. You can do the same and more. We can help you realize your own dramatic improvements in bottom- and top-line results. Rebecca Staton-Reinstein, president, Advantage Leadership, Inc.

TRANSCRIPT

Page 1: Are bugs eating your software budget?

AdvantageLeadership.com 1

Are Bugs Eating Your Budget?

Cost of Quality Analysis Results

Rebecca Staton-Reinstein

Irv Brownstein

1Advantage Leadership, Inc.

Page 2: Are bugs eating your software budget?

AdvantageLeadership.com 2

2

Most large IT departments are currently using powerful methodologies, tools,

and metrics to produce business-sustaining and competitive-edge software

and services. Most have adopted elements of proven software quality

assurance strategies.

Yet Gartner Group just reported 20% of IT projects do not pay off. The world

economy loses over $500 Billion/year in FAILED projects:

• 30% of cost of new product development is rework. One in 3 software

engineers spends 100% of time fixing what other 2 broke!

• 17% of US software projects meet performance, cost, time, scope targets

• 50% of projects must change targets (over budget, reduce scope)• 33% of all projects are cancelled• Cost of projects constantly patched & maintained cannot be calculated

Even well-functioning IT departments with few production defects may not know their cost of rework or opportunities to reduce these costs.

Page 3: Are bugs eating your software budget?

AdvantageLeadership.com 3

Basis for Analysis: Using Cost of Quality to

improve business results and reduce rework

3

Joseph Juran

The true cost of producing software, new development or existing

system maintenance, is hidden in extensive, expensive rework that is

rarely accounted for and can be 30 – 70% of the IT budget. Rework is

usually a root cause of blown software project budgets and schedules.

Joseph Juran developed COST OF QUALITY analysis. COQ can be used to

discover these huge costs and demonstrate the impact of any remedy.

Cost of Quality = Cost of Appraisal (finding defects) + Cost of Prevention

(preventing & reducing defects) + Cost of Failure (rework, bugs in product,

etc.) COQ is added to Cost of Production (creating product or service.)

(Illustrated in next 3 slides)

Page 4: Are bugs eating your software budget?

AdvantageLeadership.com 4

PRODUCTION

APPRAISAL

PREVENTION

FAILURE

} Cost of Quality

COQ = Cost of Appraisal + Cost of Prevention + Cost of Failure

COQ is added to cost to produce the product or service

4

Page 5: Are bugs eating your software budget?

AdvantageLeadership.com 5

Reduce Cost of QualityTO REDUCE COQ:

• Shift the emphasis of Appraisal activities:

– Do more robust reviews earlier in the Life Cycle,

– Do more sophisticated testing

– Use test tools more effectively, completely

• Increase Prevention:

– Enhance: standards, training, Good Software Engineering Practice

– Improve: Process improvement based on defect root cause analysis

• Decrease Cost of Failure:

– Declines due to fewer defects in production (defects discovered at

lower cost to find and fix)

ROI: High (10:1 – 5:1)

OTHER BENEFITS: Quicker Time to Market; Higher Customer Delight 5

Page 6: Are bugs eating your software budget?

AdvantageLeadership.com 6

ROI

PRODUCTION

APPRAISAL

PREVENTION

FAILUREFAILURE

PREVENTION

APPRAISAL

PRODUCTION

} COQ

Reduce COQ: Shift Appraisal emphasis; Increase Prevention; Cost of Failure declines

6

Page 7: Are bugs eating your software budget?

AdvantageLeadership.com 7

Basis for Software Cost Of Quality Analysis

7

Two classic studies from the early 1980s have been verified many times

and are the basis for software COQ analysis today. [Barry Boehm, James

Martin]

Barry Boehm demonstrated the escalation cost to find and fix defects

from a relative cost of $1 - 2 in Requirements to $100 – 1000 in

Production. These are the relative costs, the actual costs are much

higher.

Current studies find the same relationships between finding and fixing

defects over the Life Cycle, independent of technology and methodology.

(Illustrated in next 6 slides)

Barry Boehm, Software Engineering Economics, Prentice Hall, 1981

Page 8: Are bugs eating your software budget?

AdvantageLeadership.com 8

Software Engineering

Economics Barry Boehm

8

Page 9: Are bugs eating your software budget?

AdvantageLeadership.com 9

Basis for Software Cost Of Quality Analysis

9

James Martin showed where Defects are caused:

Requirements = 56%

Design = 27%

Code & Test = 7%

Other = 10% (1 new defect is created for every 2 repaired)

Current studies show requirements defects are increasing.

About 50% of all errors are found during Coding/Testing and 23% in

Production. Only about 25% were found in Requirements and Design

combined if some sort of review is conducted.

This is almost a complete reversal of where they are caused.

James Martin, An Information Systems Manifesto, Prentice-Hall, Inc., 1984

Page 10: Are bugs eating your software budget?

AdvantageLeadership.com 10

0

10

20

30

40

50

60

Caused

REQ

Design

Code/Test

Production

James Martin - where

Defects are caused:

Requirements = 56%

Design = 27%

Code & Test = 7%

Other = 10%

10Advantage Leadership, Inc.

Page 11: Are bugs eating your software budget?

AdvantageLeadership.com 11

0

10

20

30

40

50

60

Caused

REQ

Design

Code/Test

Production

Found

Martin - Where

Defects found:

Req +Des = 25%*

Code/Test = 50%

Production = 23%

*IF reviewed:

Walkthrough

11Advantage Leadership, Inc.

Page 12: Are bugs eating your software budget?

AdvantageLeadership.com 12

0

20

40

60

80

100

Caused Cost

REQ

Design

Code/Test

Production

Found

Combining work of

Boehm & Martin

to show increasing

costs when defects

are not found

where they are

caused

12

Page 13: Are bugs eating your software budget?

AdvantageLeadership.com 13

Pay to

do it

wrong

Pay to

find it

Pay to

fix it

13

Hidden Costs for Rework

Page 14: Are bugs eating your software budget?

AdvantageLeadership.com 14

Case Study COQ Analysis - IT Client Results

14Advantage Leadership, Inc.

The results that follow are based on data from one long-term client

(Beta Company) where data were tracked over several years and show

the impact of:

• Improving processes: Requirements, testing, standards development,

training

• Implementing formal Fagan Inspections

• Measuring Cost of Quality

Methodology:

• Baseline & follow up assessment of process effectiveness

• Analyze all defects; calculation of actual defect costs (COQ)

• Implement process improvements

• Measure changes in COQ

Page 15: Are bugs eating your software budget?

AdvantageLeadership.com 15

Beta Co. Cost of Quality (defect discovery & rework) for 1 Year

Relying on inconsistent testing processes at beginning of Analysis

Life Cycle Phase

Number of

Defects

Found

Percent of

Defects

Found

Cost per

Defect*

Cost to

Find/Fix

Defects*

Requirements 46 1% $ 106 $ 4,876

Design 27 0.6% $ 265 $ 7,155

Code/Unit Test 89 2% $ 530 $ 47,170

Test 3954 86% $ 1,060 $ 4,191,240

Configuration 14 0.3% $ 2,650 $ 37,100

Warranty 37 0.8% $ 5,300 $ 196,100

Production 437 9.5% $ 10,600 $ 4,632,200

TOTAL 4604 100%$ 9,115,841

30% IT Budget

15*Calculations based on Boehm’s Formulas

Page 16: Are bugs eating your software budget?

AdvantageLeadership.com 16

Based on COQ and Testing Effectiveness Assessment

BETA developed IT Quality Framework Plan

Define Framework Principles

IT Steering Committee; COO/CIO, VPs

IT Policies: SDLC, Quality, Testing, Requirements

Goal: reduce number of defects being introduced, identify/resolve defects closer to point of origin

16Funded by finding more defects before production & reducing rework costs

Page 17: Are bugs eating your software budget?

AdvantageLeadership.com 17

17

Multi-year Initiative: Improve Testing Effectiveness/Efficiency

Testing Resource Center Team, Supported by IT Steering Committee

ESTABLISHED:

•Testing Resource Center

• Automation and

Performance Testing

• Stage Gate Assessments

• Business Acceptance Testing

• Testing and Quality Policy

• Universal use of Test Tools

• Definitions of Testing Roles,

Responsibilities

• Improved Testing Templates

• Early Involvement in Project

Control Boards

• IT Quality Plan Deployment

Advantage Leadership, Inc.

Page 18: Are bugs eating your software budget?

AdvantageLeadership.com 18

18Advantage Leadership, Inc.

Beta assessed perception of testing and process effectiveness

among all IT managers and staff with roles in software projects

on 8 parameters.

Although Beta still had a need to improve further, scores on

all parameters increased significantly. Improvement came from

improved processes and standards, education/awareness, and

management focus on Cost of Quality.

Note: Beta IT and the corporation experienced several major

organizational changes during the deployment but the team’s

persistence paid off.

(Illustrated in next 3 slides)

Page 19: Are bugs eating your software budget?

AdvantageLeadership.com 19

BCBS MN Testing & QA

3.01

3.19

2.64

3.08

3.07

2.38

2.50

2.71

0

1

2

3

4

5

1. Continuous Improvement

2. Team Skills &

Responsibilities

3. Configuration Management

& Standards

4. Testing Process

Definition

5. Testing Execution

6. Training & Support

7. Risk & Defect

Management

8. Measurement Analysis

& Reporting

Baseline Assessment of Testing Effectiveness

19

Input solicited from all managers and staff associated with software projects

Page 20: Are bugs eating your software budget?

AdvantageLeadership.com 20

BCBS MN Testing & QA

0

1

2

3

4

5

Continuous Improvement

Team Skills &

Responsibilities

Configuration Management

& Standards

Testing Process

Definition

Testing Execution

Training & Support

Risk & Defect

Management

Measurement Analysis

& Reporting

Survey Dimension 2008 Response 2006 Response

Baseline = Blue vs. Year 3 Change = Purple

20Year 3 Response Year 1 Response

Page 21: Are bugs eating your software budget?

AdvantageLeadership.com 21

21

Change in Staff Perceptions

Of Quality Processes

Test Baseline SCORE

Change 2 years

Test Progress SCORE

Assessment Parameters PERCENT PERCENT

Continuous Improvement 60.2% 62.2%

Test Team Skills & Responsibilities

63.8% 67.8%

Requirements Management Team Skills, Responsibilities

N/A 66.2%

Configuration Management & Standards

52.8% 65.6%

Testing Process Definition

61.6% 68.4%

Testing Execution 61.4% 68.6%

Training & Support 47.6% 59.4%

Risk & Defect Management

50.0% 60.8%

Measurement Analysis & Reporting

54.2% 63.6%

Page 22: Are bugs eating your software budget?

AdvantageLeadership.com 22

22Advantage Leadership, Inc.

Perceptions of quality improvement were validated by dramatic

changes in Beta’s ability to find significantly more defects through

more robust testing and a dramatic decline in defects escaping into

production.

Over a 3-year period, Beta demonstrated over $6 MM in

cost savings and avoidance.

(Illustrated on next 3 slides)

Page 23: Are bugs eating your software budget?

AdvantageLeadership.com 23

Improved defect detection from improved testing processes

100%

90%

80%

70%

Year 1 Year 2 Year 3

78.3%

86.0%

90.5%

Defects found in Testing

Page 24: Are bugs eating your software budget?

AdvantageLeadership.com 24

Fewer defects enter production due to improve testing processes

20%

15%

10%

5%

Year 1 Year 2 Year 3

15.1%

9.5%

6.2%

Defects found in Production

Page 25: Are bugs eating your software budget?

AdvantageLeadership.com 25

Defect Costs Comparison: As focus on testing increased,

testing costs increased 14% & Production defect costs deceased 39%

$-

$2,000,000

$4,000,000

$6,000,000

$8,000,000

$10,000,000

$12,000,000

Require

ment

s

Analysis

/Desig

n

Constru

ction

Test

Config

urat

ion

Warra

nty

Produc

tion

Y1 Cost

Y2 Cost

Y3 Cost

Ind Cost

25

Page 26: Are bugs eating your software budget?

AdvantageLeadership.com 26

26

BETA began implementing Fagan Inspections

• Trained Inspectors & Moderators; BAs served as moderators, team members

• Established Defect Analysis and Reporting

• Compared similar projects with/without Inspections (Same Project Manager,

similar scope & application, 75% same team)

• 5 pilot Requirements Inspections

• Total defects found = 408

• Costs of defects found in Requirements ($50,592) + Inspection costs

($36,855) = Total cost of Inspection ($87,447)

• Cost avoided in Testing phase = $5,059,200

• Stopped Pilot because of success and began Inspection Roll Out

Fagan Inspections track record: Experienced teams find 85% of defects at

point of origin; cheaper to find & fix. (Illustrated on next 6 slides)

Page 27: Are bugs eating your software budget?

AdvantageLeadership.com 27

27

# Defects Cost of Defects

Stage Found

Project A (Inspections)

Project B(No Inspections)

Project A Project B

REQ 220 -0- $22,000 -0-

DES 2 1 $500 $250

CODE -0- -0- -0- -0-

TEST 12 83 $12,000 $83,000

Config -0- -0- -0- -0-

Warranty -0- -0- -0- -0-

PROD -0- 16 -0- $160,000

Inspection/ Train Cost

$11,346

TOTAL 234 100 $45,846 $243,250

Cost Avoidance

$518,404 -0-

Similar Projects With (A) & Without (B) Inspections

Page 28: Are bugs eating your software budget?

AdvantageLeadership.com 28

28

Year Req Des Test # Insp

Req Des Test Sev 1 Sev 2 Sev 3 - 5 Insp Defects

Year 1 5 0 0 5 406 0 0 104 114 188 406

Year 2 17 1 11 29 369 10 211 6 96 488 590

Year 3 0 0 11 11 0 0 162 9 33 120 162

Totals 22 1 22 45 775 10 373 119 243 796 1158

Severity 1 defects = total failure in production;

Severity 2 defects = Major problems in production;

Both must be fixed quickly, before release

Defects found in Inspections of Requirements, Design, and Test Plans – by Severity

Page 29: Are bugs eating your software budget?

AdvantageLeadership.com 29

29

5

0 0

5

17

1

11

29

0 0

11 11

0

5

10

15

20

25

30

Requirements Design Testing Total # of Inspections

2010 Data as of 5/31

Number of Inspections by High-level Work Product Type

Year 1

Year 2

Year 3

Page 30: Are bugs eating your software budget?

AdvantageLeadership.com 30

30

406

0 0

406

369

10

211

590

0 0

162 162

0

100

200

300

400

500

600

700

Requirements Design Testing Total Inspection Defects

2010 Data as of 5/31

Number of Inspection Defects by High-level Work Product Type

Year 1

Year 2

Year 3

Page 31: Are bugs eating your software budget?

AdvantageLeadership.com 31

31

104 11493

5936

406

6

96

202

250

36

590

933

80

391

162

0

100

200

300

400

500

600

700

Severity 1 Severity 2 Severity 3 Severity 4 Severity 5 Total Inspection Defects

2010 Data as of 5/31

Number of Inspection Defects by Severity Level

Year 1

Year 2

Year 3

Page 32: Are bugs eating your software budget?

AdvantageLeadership.com 32

32

Year 1 – Year 3 = 45 Inspections

• Requirements = 22

• Design = 1

• Test Plans = 22

Discovered 1158 Defects

• Severity 1 & 2 = 362

• Severity 3 – 5 = 796

If Severity1 & 2 Defects Undiscovered

• Cost in production: $3,826,600*

Inspection Costs & Savings/Avoidance

• Cost to perform 45 Inspections = $47,000

• Cost of Rework using Inspections = $432,480

• Cost if Defects found in Testing & Production = $2.3 MM

• Cost Avoidance with Inspections = $1.8 MM

*Conservative estimates because salaries had risen over the 3 years.

Page 33: Are bugs eating your software budget?

AdvantageLeadership.com 33

33Advantage Leadership, Inc.

Beta proceeded slowly with its improvement efforts but achieved

major savings.

Each process and change was practical with a proven track record.

Implementation required modest training with “train the trainer”

and coaching to build new processes into the company infrastructure.

Beta did not use significant root cause analysis to improve targeted

software–related processes.

Page 34: Are bugs eating your software budget?

AdvantageLeadership.com 34

34Advantage Leadership, Inc.

To Reduce the Cost of Quality

Increase early defect

detection with

Inspections

Improve testing

processes, training, tools penetration

Implement COQ

reduction plan as part

of IT strategic plan

Page 35: Are bugs eating your software budget?

AdvantageLeadership.com 35

Your Next Steps

35

Review Beta Case Study: Cost of Quality Analysis

Consider Cost of Quality Analysis to Discover Hidden Rework Costs

Consider Assessment of Mission Critical Processes

Consider Opportunities to Improve Results Using Practical, Proven Approaches

Page 36: Are bugs eating your software budget?

AdvantageLeadership.com 36

36Advantage Leadership, Inc.

Rebecca Staton-Reinstein, Ph.D., CSQA, PresidentADVANTAGE LEADERSHIP, INC.

320 S. Flamingo Road, Suite 291

Pembroke Pines, Florida 33027 USA

Phone: 305-606-9312 Fax: 305-397-2343

Irv Brownstein, PresidentTHE PRODUCTIVITY GROUP, INC. (TPG)

[email protected]

[email protected]

www.AdvantageLeadership.com

(Note: Logo photo by Whitney Cranshaw, Colorado State University, Bugwood.org;

Mantis religiosa (praying mantis) chows down on a red-legged grasshopper)