Presented by: Scott Aziz24 October 2013

Develop a Defect Prevention Strategy – Or Else!

2

Agenda

• Why is Defect Prevention Critical?• Cost & Quality / Time to Market Or Else!• Current Landscape & How Defect Prevention Can Help• Prevention Framework• Prevention Methods• Requirements Reviews

• Ambiguity• Pre-Test Defect Removal Activities• Summary

3

Premise

Why is Defect Prevention Critical?

• Testing schedules for low-quality, large software projects are 2-3X longer and more than 2X as costly as testing for high-quality projects.

• If defects remain undetected and unremoved until testing starts, it is too late to bring a software project back under control. It is much more cost-effective to prevent defects or to remove them prior to testing.

• Prevention and Appraisal activities remove many more defects per Engineer-Hour than Failure activities, but organizations often invest very little in the time-proven methods.

4

What can we hope to change with defect prevention?

5

The Concern

About 40-50% of the effort on current software

projects is spent on avoidable rework.1,2

$0.50 out of every $1.00 for new/maintenance.

Repair and rework is the major software cost

driver.

This is a restrictive business model.1. [Shull et al. 2002] Shull, F., V. Basili, B. Boehm, A.W. Brown, P. Costa, M. Lindvall, D. Port, I. Rus, R.

Tesoreiro, and M. Zelkowitz. 2002.What we have learned about fighting defects. Proceedings of the 8th

International Symposium on Software Metrics:

2. Jones, Capers and Bonsignour, Olivier; The Economics of Software Quality; Addison Wesley; 2011.

As an engineering discipline, we can do better. Much better.

6

Cost of Quality / Poor Quality Defined

• Prevention Costs─ Cost of activities to

prevent poor quality

• Appraisal Costs─ Cost of activities to

detect quality issues

• Internal Failure Costs─ Costs incurred to fix

quality issues before delivery to customer

• External Failure Costs─ Costs due to failures

after delivery tocustomer

Prevention Costs• Training of development team• Requirements Analysis

Appraisal Costs• Static Tests

─ Peer Review(also Prevention)

─ Code Walk-through• Dynamic Tests (Manual/

Automated)─ Functional Testing─ Performance Testing• Test Equipment

Internal Failure Costs• Churn – Characterize/Fix/Re-Test• Opportunity cost of delayed delivery

External Failure Costs• Bug fixing after delivery (All Types)• Customer support• Loss of business• Litigation• Fines

7

Why Should We Care about CoPQ?

It’s an accounting problem – right?Not my job…

Reduction of Cost is not the only factor• Quality, Time To Market, Customer Delight

Isn't this just a cost of doing business?

We have to take a leadership stance.• Manage IT as a business within a business.• Business growth enabled through lower costs.• Build the best product for lowest cost to best compete.

Poor quality costs are too large to ignore.

8

Align IT Goals / Business

1 1

2 2

3 3

4 4

5

6

7

8

9

10

5

6

7

8

9

10

13Not in top 20

Forrester Results From Survey Of Business And Technology Decision-Makers

KPI Business rank IT rank

IT cost per business service supported

Internal customer satisfaction survey score

IT spend by business objective (margin, revenue, growth, compliance, etc.)

Percentage of IT projects that meet or exceed expected benefits

Percentage of IT spend on run, grow, change the business

Yr/Yr IT budget growth vs. revenue growth

Yr/Yr unit cost of infrastructure, systems, apps maintenance

IT spend as a percentage of business revenue

Percentage of IT budget spend on R&D, emerging tech, pilots, innovations, etc.

External customer satisfaction (your organization's customers) survey score

Base: 16 CIOS, CMOs, and CFOs

Source: A commissioned study conducted by Forrester Consulting on behalf of The Technology Business Management Council, September 2013

9

External Failures

• Knight Trading Debacle in 2012 (Took 30 minutes to stop programmed trading; $440M loss)

• Nasdaq FaceBook IPO (Pre-Market Trading System)• $62M Compensation Fund• Race Condition – only could have been discovered under

heavy volume.2012.

• Investment firm AXA Rosenberg shelled out $217 million to cover investor losses from what it called a "significant error“in the computer code for one of its investment models.

• Issues are not limited to just financial companies…

2012.

10

Software Testing Profession Must Evolve

Why Are These Costly Failures Occurring?

• Minimal prevention activities– When more defects enter testing, the productivity of testing decreases due

to engineering churn; i.e., the costs of testing is higher and the time to complete testing increases.

• Casual test case design– multiple thorough studies show approximately 85% of defects in production

could have been detected by simply testing all possible pairs of values.1

• Testing by untrained/uncertified test personnel

What’s Next -> Regulation• The SEC is considering writing regulations that would require trading firms

and other market participants to disclose issues with their trading programs and test them before they are used on the open market. 2

1. Source: IEEE Computer: Combinatorial Software Testing Aug, 2009 by Richard Kuhn, Raghu Kacker, Jeff Lei, and Justin Hunter.2. Financial Times Newspaper, August 2012.

11

Visible / Hidden Internal & External Failure

Visible External Failures (Cost of Poor Quality)

Hidden Internal & External Failures (Cost of Poor Quality)

12

Internal & External Failures Example

Engineering Cost per Hour $96

Developer effort spent on QA, rework, process, etc. 20%

Management effort spent on QA, rework, process, etc. 10%

Average hours spent to correct a defect that resulted in a change 14

Average hours to close a defect with a no-change resolution (duplicate, etc.) 3

Support costs attributed to poor quality 30%

Group Change No Change

Group 1 5,675 2,481

Group 2 8,410 2,231

Group 3 912 326

Group 4 7,056 4,243

Group 5 2,258 1,330

Group 6 6,795 2,251

Group 7 3,205 1,564

TOTAL 39,938 14,426

Rework Support Total Staff Cost Total COPQ

TOTAL $53,076,827 $79,230,223 $444,129,231 $177,383,877

13

Minimal Early Activities = Technical Debt

Requirements Injection Phase

Design Injection Phase

Coding Injection Phase

Technical Debt

Prevention and Appraisal activities remove many more defects per Engineer-Hour than Failure activities.

Every defect that we can remove more economically leads to more money available to re-invest in better coverage methods and tools for testing.

Early Lifecycle Testing Activities are not well planned due to:

• Minimal Recognition• No Training• Few Tools• Project Inertia• Change Management

14

Testing is Too Late to Effect Enough Change

Take Control

15

What Do We Do?

The only way to be in control of Quality is to shift it from the uncontrollable Failure Costs to the

controllable Prevention/Appraisal Costs.

With each incremental increase in Appraisal activities, such as reviews, we can expect a corresponding

and larger reduction in our Failure activities.

Take Control -> Invest Upfront

Change Management is difficult

•Investments are needed to shift the majority of our Cost of Quality to the prevention and

appraisal side of the equation.

─ CoQ will not only be reduced significantly, but it will also be more predictable and more

manageable.

•Mind-set change

•Inspections are not the most enjoyable engineering task compared to designing and coding.

Inspections are labor intensive and low tech, but they do work.

•The importance of removing defects pre-test; organizational goals.

16

Framework for Prevention

Causal Analysis and Resolution: part of many process improvement models

(CMMI, ISO, SixSigma)• An organization-level team to coordinate

defect prevention activities exists.• A team to coordinate defect prevention

activities for the software project exists.• Adequate resources and funding are provided

for defect prevention activities at the project and organization levels.

• Members of the software engineering group and other software related groups receive required training to perform their defect prevention activities.

17

Prevention Activities

Pre-Code

Reusable requirements, architecture, design, and

code

Requirements / Design Reviews

Requirements Testing / Modelling

QFD, Kaizen for software

Using quality-strong methodologies such as RUP

and TSP

Agile: Test First, Then Code; Tests are the

Requirements.

Post-Code

Code Review

Static Analysis

Eliminate difficult work

Eliminate waste in process

Identify areas of

improvement

Experiment with solutions to problems

18

Measuring Prevention

Function Points

• Without prevention: Function Points ^1.2.

• With Prevention: Function Points ^1.05-1.15

Number of Defects = 250 @ 100 Function Points.

Quality Strong Methods = 199 = ^1.15

Requirements Testing = 150: ^1.05-1.10

19

Defect Prevention: Agile Practices

20

Requirement Error CategoriesFrom literature survey of software engineering, psychology and human cognitive fields

G.S. Walia, J.C. CarverA systematic literature review to identify and classify software requirement errorsInform. Softw. Technol., 51 (2009), pp. 1087–1109

If one person wrote it with one intent and another person reads it differently, it is ambiguous.

21

Requirement Ambiguities Across Other Fields

Requirements: What can go wrong?

• Ambiguity of reference

• Dangling Else

• Omissions─ Causes without effects─ Missing effects─ Effects without causes

• Ambiguous logical operators─ OR, AND, NOR, NAND─ Implicit connectors─ Compound operators

• Negation─ Scope of negation─ Unnecessary negation─ Double negation

• Ambiguous statements─ Verbs, adverbs, adjectives─ Variables, unnecessary aliases

Fact: If something is ambiguous in the specs it will nearly always result in a defect(s) in the code.

Examples:

It, This, The above, The previous, Them, These, They.

“Add field A to field B.

This number must be positive.”

Must be, will be, is one of, should be, could be.

“The value must be either A, B, or C.” Else? An error condition?

23

Code Reviews

• Keep it Small – Large teams are not productive and have diminishing

returns.

• Rigor and scheduling must be adhered to.

• Effectiveness of review depends on experience of reviewer.

• Can eliminate coding defects by 40-50% or more.

• Collaborative code review platforms like Gerrit, CodeFlow,

Collaborator, ReviewBoard or Crucible.

24

Static Analysis

• Is this the way we engineer software?• What is static analysis?

• Rules Engine: Code Patterns for Reliability, Performance, and Security.

• How can it help?• Leaks, crashes, deadlocks, security vulnerabilities, etc.—

problems that might otherwise take weeks to find.

25

Review Points

• Defect prevention and removal is a proven method to cost and schedule reduction. • The point at which most failing projects first show signs of serious trouble is when

testing starts. Many projects that are cancelled or have major delays for delivery showed no signs of distress until testing started.

• Although prevention and pre-test defect removal activities have been utilized since the software industry began, the literature on software quality is sparse: there is a 20 to 1 ratio between books on testing and books on prevention and pre-test removal activities.

• From an economic viewpoint prevention and pre-test defect removal is even more important than testing because it raises testing efficiency, lowers testing costs, shortens testing schedules, and generates a very solid return on investment.

• From both an economic and quality assurance standpoint, defect prevention, pre-test defect removal, and formal testing are all necessary to achieve a combination of low costs, short schedules, and low levels of defects present in the software when it is delivered to customers.