are bugs eating your software budget?
DESCRIPTION
20% of software projects fail. Failed projects cost the world economy over $500 billion a year. The hidden costs of making, finding, and fixing software defects consume 30% or more of the IT budget. The facts are continuing to get worse. Software bugs are eating your lunch. You can reverse the trend as many companies have by applying some proven, practical methods – Good Software Engineering Practice. You can do much of this on your own if you analyze the Cost of Quality and use it as a fundamental metric to demonstrate the effectiveness and efficiency of everything you do. If you add Fagan Inspections (static testing) to your existing software testing you can start reducing the cost of quality immediately. In this study of one company’s results over several years, we show you step by step how they demonstrated over $6 million in cost savings and avoidance. You can do the same and more. We can help you realize your own dramatic improvements in bottom- and top-line results. Rebecca Staton-Reinstein, president, Advantage Leadership, Inc.TRANSCRIPT
AdvantageLeadership.com 1
Are Bugs Eating Your Budget?
Cost of Quality Analysis Results
Rebecca Staton-Reinstein
Irv Brownstein
1Advantage Leadership, Inc.
AdvantageLeadership.com 2
2
Most large IT departments are currently using powerful methodologies, tools,
and metrics to produce business-sustaining and competitive-edge software
and services. Most have adopted elements of proven software quality
assurance strategies.
Yet Gartner Group just reported 20% of IT projects do not pay off. The world
economy loses over $500 Billion/year in FAILED projects:
• 30% of cost of new product development is rework. One in 3 software
engineers spends 100% of time fixing what other 2 broke!
• 17% of US software projects meet performance, cost, time, scope targets
• 50% of projects must change targets (over budget, reduce scope)• 33% of all projects are cancelled• Cost of projects constantly patched & maintained cannot be calculated
Even well-functioning IT departments with few production defects may not know their cost of rework or opportunities to reduce these costs.
AdvantageLeadership.com 3
Basis for Analysis: Using Cost of Quality to
improve business results and reduce rework
3
Joseph Juran
The true cost of producing software, new development or existing
system maintenance, is hidden in extensive, expensive rework that is
rarely accounted for and can be 30 – 70% of the IT budget. Rework is
usually a root cause of blown software project budgets and schedules.
Joseph Juran developed COST OF QUALITY analysis. COQ can be used to
discover these huge costs and demonstrate the impact of any remedy.
Cost of Quality = Cost of Appraisal (finding defects) + Cost of Prevention
(preventing & reducing defects) + Cost of Failure (rework, bugs in product,
etc.) COQ is added to Cost of Production (creating product or service.)
(Illustrated in next 3 slides)
AdvantageLeadership.com 4
PRODUCTION
APPRAISAL
PREVENTION
FAILURE
} Cost of Quality
COQ = Cost of Appraisal + Cost of Prevention + Cost of Failure
COQ is added to cost to produce the product or service
4
AdvantageLeadership.com 5
Reduce Cost of QualityTO REDUCE COQ:
• Shift the emphasis of Appraisal activities:
– Do more robust reviews earlier in the Life Cycle,
– Do more sophisticated testing
– Use test tools more effectively, completely
• Increase Prevention:
– Enhance: standards, training, Good Software Engineering Practice
– Improve: Process improvement based on defect root cause analysis
• Decrease Cost of Failure:
– Declines due to fewer defects in production (defects discovered at
lower cost to find and fix)
ROI: High (10:1 – 5:1)
OTHER BENEFITS: Quicker Time to Market; Higher Customer Delight 5
AdvantageLeadership.com 6
ROI
PRODUCTION
APPRAISAL
PREVENTION
FAILUREFAILURE
PREVENTION
APPRAISAL
PRODUCTION
} COQ
Reduce COQ: Shift Appraisal emphasis; Increase Prevention; Cost of Failure declines
6
AdvantageLeadership.com 7
Basis for Software Cost Of Quality Analysis
7
Two classic studies from the early 1980s have been verified many times
and are the basis for software COQ analysis today. [Barry Boehm, James
Martin]
Barry Boehm demonstrated the escalation cost to find and fix defects
from a relative cost of $1 - 2 in Requirements to $100 – 1000 in
Production. These are the relative costs, the actual costs are much
higher.
Current studies find the same relationships between finding and fixing
defects over the Life Cycle, independent of technology and methodology.
(Illustrated in next 6 slides)
Barry Boehm, Software Engineering Economics, Prentice Hall, 1981
AdvantageLeadership.com 8
Software Engineering
Economics Barry Boehm
8
AdvantageLeadership.com 9
Basis for Software Cost Of Quality Analysis
9
James Martin showed where Defects are caused:
Requirements = 56%
Design = 27%
Code & Test = 7%
Other = 10% (1 new defect is created for every 2 repaired)
Current studies show requirements defects are increasing.
About 50% of all errors are found during Coding/Testing and 23% in
Production. Only about 25% were found in Requirements and Design
combined if some sort of review is conducted.
This is almost a complete reversal of where they are caused.
James Martin, An Information Systems Manifesto, Prentice-Hall, Inc., 1984
AdvantageLeadership.com 10
0
10
20
30
40
50
60
Caused
REQ
Design
Code/Test
Production
James Martin - where
Defects are caused:
Requirements = 56%
Design = 27%
Code & Test = 7%
Other = 10%
10Advantage Leadership, Inc.
AdvantageLeadership.com 11
0
10
20
30
40
50
60
Caused
REQ
Design
Code/Test
Production
Found
Martin - Where
Defects found:
Req +Des = 25%*
Code/Test = 50%
Production = 23%
*IF reviewed:
Walkthrough
11Advantage Leadership, Inc.
AdvantageLeadership.com 12
0
20
40
60
80
100
Caused Cost
REQ
Design
Code/Test
Production
Found
Combining work of
Boehm & Martin
to show increasing
costs when defects
are not found
where they are
caused
12
AdvantageLeadership.com 13
Pay to
do it
wrong
Pay to
find it
Pay to
fix it
13
Hidden Costs for Rework
AdvantageLeadership.com 14
Case Study COQ Analysis - IT Client Results
14Advantage Leadership, Inc.
The results that follow are based on data from one long-term client
(Beta Company) where data were tracked over several years and show
the impact of:
• Improving processes: Requirements, testing, standards development,
training
• Implementing formal Fagan Inspections
• Measuring Cost of Quality
Methodology:
• Baseline & follow up assessment of process effectiveness
• Analyze all defects; calculation of actual defect costs (COQ)
• Implement process improvements
• Measure changes in COQ
AdvantageLeadership.com 15
Beta Co. Cost of Quality (defect discovery & rework) for 1 Year
Relying on inconsistent testing processes at beginning of Analysis
Life Cycle Phase
Number of
Defects
Found
Percent of
Defects
Found
Cost per
Defect*
Cost to
Find/Fix
Defects*
Requirements 46 1% $ 106 $ 4,876
Design 27 0.6% $ 265 $ 7,155
Code/Unit Test 89 2% $ 530 $ 47,170
Test 3954 86% $ 1,060 $ 4,191,240
Configuration 14 0.3% $ 2,650 $ 37,100
Warranty 37 0.8% $ 5,300 $ 196,100
Production 437 9.5% $ 10,600 $ 4,632,200
TOTAL 4604 100%$ 9,115,841
30% IT Budget
15*Calculations based on Boehm’s Formulas
AdvantageLeadership.com 16
Based on COQ and Testing Effectiveness Assessment
BETA developed IT Quality Framework Plan
Define Framework Principles
IT Steering Committee; COO/CIO, VPs
IT Policies: SDLC, Quality, Testing, Requirements
Goal: reduce number of defects being introduced, identify/resolve defects closer to point of origin
16Funded by finding more defects before production & reducing rework costs
AdvantageLeadership.com 17
17
Multi-year Initiative: Improve Testing Effectiveness/Efficiency
Testing Resource Center Team, Supported by IT Steering Committee
ESTABLISHED:
•Testing Resource Center
• Automation and
Performance Testing
• Stage Gate Assessments
• Business Acceptance Testing
• Testing and Quality Policy
• Universal use of Test Tools
• Definitions of Testing Roles,
Responsibilities
• Improved Testing Templates
• Early Involvement in Project
Control Boards
• IT Quality Plan Deployment
Advantage Leadership, Inc.
AdvantageLeadership.com 18
18Advantage Leadership, Inc.
Beta assessed perception of testing and process effectiveness
among all IT managers and staff with roles in software projects
on 8 parameters.
Although Beta still had a need to improve further, scores on
all parameters increased significantly. Improvement came from
improved processes and standards, education/awareness, and
management focus on Cost of Quality.
Note: Beta IT and the corporation experienced several major
organizational changes during the deployment but the team’s
persistence paid off.
(Illustrated in next 3 slides)
AdvantageLeadership.com 19
BCBS MN Testing & QA
3.01
3.19
2.64
3.08
3.07
2.38
2.50
2.71
0
1
2
3
4
5
1. Continuous Improvement
2. Team Skills &
Responsibilities
3. Configuration Management
& Standards
4. Testing Process
Definition
5. Testing Execution
6. Training & Support
7. Risk & Defect
Management
8. Measurement Analysis
& Reporting
Baseline Assessment of Testing Effectiveness
19
Input solicited from all managers and staff associated with software projects
AdvantageLeadership.com 20
BCBS MN Testing & QA
0
1
2
3
4
5
Continuous Improvement
Team Skills &
Responsibilities
Configuration Management
& Standards
Testing Process
Definition
Testing Execution
Training & Support
Risk & Defect
Management
Measurement Analysis
& Reporting
Survey Dimension 2008 Response 2006 Response
Baseline = Blue vs. Year 3 Change = Purple
20Year 3 Response Year 1 Response
AdvantageLeadership.com 21
21
Change in Staff Perceptions
Of Quality Processes
Test Baseline SCORE
Change 2 years
Test Progress SCORE
Assessment Parameters PERCENT PERCENT
Continuous Improvement 60.2% 62.2%
Test Team Skills & Responsibilities
63.8% 67.8%
Requirements Management Team Skills, Responsibilities
N/A 66.2%
Configuration Management & Standards
52.8% 65.6%
Testing Process Definition
61.6% 68.4%
Testing Execution 61.4% 68.6%
Training & Support 47.6% 59.4%
Risk & Defect Management
50.0% 60.8%
Measurement Analysis & Reporting
54.2% 63.6%
AdvantageLeadership.com 22
22Advantage Leadership, Inc.
Perceptions of quality improvement were validated by dramatic
changes in Beta’s ability to find significantly more defects through
more robust testing and a dramatic decline in defects escaping into
production.
Over a 3-year period, Beta demonstrated over $6 MM in
cost savings and avoidance.
(Illustrated on next 3 slides)
AdvantageLeadership.com 23
Improved defect detection from improved testing processes
100%
90%
80%
70%
Year 1 Year 2 Year 3
78.3%
86.0%
90.5%
Defects found in Testing
AdvantageLeadership.com 24
Fewer defects enter production due to improve testing processes
20%
15%
10%
5%
Year 1 Year 2 Year 3
15.1%
9.5%
6.2%
Defects found in Production
AdvantageLeadership.com 25
Defect Costs Comparison: As focus on testing increased,
testing costs increased 14% & Production defect costs deceased 39%
$-
$2,000,000
$4,000,000
$6,000,000
$8,000,000
$10,000,000
$12,000,000
Require
ment
s
Analysis
/Desig
n
Constru
ction
Test
Config
urat
ion
Warra
nty
Produc
tion
Y1 Cost
Y2 Cost
Y3 Cost
Ind Cost
25
AdvantageLeadership.com 26
26
BETA began implementing Fagan Inspections
• Trained Inspectors & Moderators; BAs served as moderators, team members
• Established Defect Analysis and Reporting
• Compared similar projects with/without Inspections (Same Project Manager,
similar scope & application, 75% same team)
• 5 pilot Requirements Inspections
• Total defects found = 408
• Costs of defects found in Requirements ($50,592) + Inspection costs
($36,855) = Total cost of Inspection ($87,447)
• Cost avoided in Testing phase = $5,059,200
• Stopped Pilot because of success and began Inspection Roll Out
Fagan Inspections track record: Experienced teams find 85% of defects at
point of origin; cheaper to find & fix. (Illustrated on next 6 slides)
AdvantageLeadership.com 27
27
# Defects Cost of Defects
Stage Found
Project A (Inspections)
Project B(No Inspections)
Project A Project B
REQ 220 -0- $22,000 -0-
DES 2 1 $500 $250
CODE -0- -0- -0- -0-
TEST 12 83 $12,000 $83,000
Config -0- -0- -0- -0-
Warranty -0- -0- -0- -0-
PROD -0- 16 -0- $160,000
Inspection/ Train Cost
$11,346
TOTAL 234 100 $45,846 $243,250
Cost Avoidance
$518,404 -0-
Similar Projects With (A) & Without (B) Inspections
AdvantageLeadership.com 28
28
Year Req Des Test # Insp
Req Des Test Sev 1 Sev 2 Sev 3 - 5 Insp Defects
Year 1 5 0 0 5 406 0 0 104 114 188 406
Year 2 17 1 11 29 369 10 211 6 96 488 590
Year 3 0 0 11 11 0 0 162 9 33 120 162
Totals 22 1 22 45 775 10 373 119 243 796 1158
Severity 1 defects = total failure in production;
Severity 2 defects = Major problems in production;
Both must be fixed quickly, before release
Defects found in Inspections of Requirements, Design, and Test Plans – by Severity
AdvantageLeadership.com 29
29
5
0 0
5
17
1
11
29
0 0
11 11
0
5
10
15
20
25
30
Requirements Design Testing Total # of Inspections
2010 Data as of 5/31
Number of Inspections by High-level Work Product Type
Year 1
Year 2
Year 3
AdvantageLeadership.com 30
30
406
0 0
406
369
10
211
590
0 0
162 162
0
100
200
300
400
500
600
700
Requirements Design Testing Total Inspection Defects
2010 Data as of 5/31
Number of Inspection Defects by High-level Work Product Type
Year 1
Year 2
Year 3
AdvantageLeadership.com 31
31
104 11493
5936
406
6
96
202
250
36
590
933
80
391
162
0
100
200
300
400
500
600
700
Severity 1 Severity 2 Severity 3 Severity 4 Severity 5 Total Inspection Defects
2010 Data as of 5/31
Number of Inspection Defects by Severity Level
Year 1
Year 2
Year 3
AdvantageLeadership.com 32
32
Year 1 – Year 3 = 45 Inspections
• Requirements = 22
• Design = 1
• Test Plans = 22
Discovered 1158 Defects
• Severity 1 & 2 = 362
• Severity 3 – 5 = 796
If Severity1 & 2 Defects Undiscovered
• Cost in production: $3,826,600*
Inspection Costs & Savings/Avoidance
• Cost to perform 45 Inspections = $47,000
• Cost of Rework using Inspections = $432,480
• Cost if Defects found in Testing & Production = $2.3 MM
• Cost Avoidance with Inspections = $1.8 MM
*Conservative estimates because salaries had risen over the 3 years.
AdvantageLeadership.com 33
33Advantage Leadership, Inc.
Beta proceeded slowly with its improvement efforts but achieved
major savings.
Each process and change was practical with a proven track record.
Implementation required modest training with “train the trainer”
and coaching to build new processes into the company infrastructure.
Beta did not use significant root cause analysis to improve targeted
software–related processes.
AdvantageLeadership.com 34
34Advantage Leadership, Inc.
To Reduce the Cost of Quality
Increase early defect
detection with
Inspections
Improve testing
processes, training, tools penetration
Implement COQ
reduction plan as part
of IT strategic plan
AdvantageLeadership.com 35
Your Next Steps
35
Review Beta Case Study: Cost of Quality Analysis
Consider Cost of Quality Analysis to Discover Hidden Rework Costs
Consider Assessment of Mission Critical Processes
Consider Opportunities to Improve Results Using Practical, Proven Approaches
AdvantageLeadership.com 36
36Advantage Leadership, Inc.
Rebecca Staton-Reinstein, Ph.D., CSQA, PresidentADVANTAGE LEADERSHIP, INC.
320 S. Flamingo Road, Suite 291
Pembroke Pines, Florida 33027 USA
Phone: 305-606-9312 Fax: 305-397-2343
Irv Brownstein, PresidentTHE PRODUCTIVITY GROUP, INC. (TPG)
www.AdvantageLeadership.com
(Note: Logo photo by Whitney Cranshaw, Colorado State University, Bugwood.org;
Mantis religiosa (praying mantis) chows down on a red-legged grasshopper)