Software Engineering

Lecture 8: Quality Assurance

Today’s Topics Elements of Software Quality Quality Control & Assurance The Cost of Quality Defect Amplification & Removal The Formal Review Process Software Reliability & Safety

Software Quality Assurance

The goal of SE: “to produce high-quality software”

Question: “What is software quality?”

Poor definition:“Something you worry about after code has been generated”

SQA Principles Quality management approach Effective SE methods / tools Ongoing technical reviews Multi-tiered testing strategy Control of documentation, updates Standards compliance procedure Ongoing measurement, reporting

Variation Control

One goal is to minimize variation:• Among products, product versions • Process results (estimated vs. actual)• Behavior (defects noted over time)

Challenge: constantly evolve the product without degrading its reliability, functionality, etc.

What is Quality? Quality of design

• Requirements analysis• Specifications• Architectural design, detailed design

Evaluating the product• Does design follow requirements?• Does implementation follow design?• Does product meet performance goals?

Quality Control Inspections, reviews, tests:

• Each module meets requirements?• Feedback to development process• Automated, manual, hybrid tests

Key: All modules must have defined and measurable specifications for output evaluation

Quality Assurance

Auditing and reporting functions Goals:

• Provide management with information on product quality• Trigger identification & resolution of quality problems

Software Quality Assurance

Conformance to:• Explicitly stated functional and performance requirements• Explicitly documented development standards• Implicit characteristics of professional software

SQA [2]

Three important points:• requirements are the foundation for measuring quality• specified standards guide the development process• implicit requirements (e.g., maintainability)

The Cost of Quality Prevention costs

• quality planning• formal technical reviews• test equipment & training

Appraisal costs• “first time through”• single project, historical evolution• equipment calibration & maintenance• testing effort

The Cost of Quality [2] Failure costs include:

• Internal failure costs (before release)• rework & repair• failure mode analysis

• External failure costs (after release)• complaint resolution• product return and replacement• help line support• warranty work

Cost of Correcting Defects Relative costs to find and repair a defect increase

dramatically:• From prevention to detection• From internal vs. external

IBM example:• Prevention = ~$90/defect• Field fix = ~$25,000/defect

18x more expensive!

[Fro

m S

EPA

4/e]

SQA Activities Independent person or group

“...large classes of errors escape the originator more easily than … anyone else” [FRE90]

Prepare SQA Plan• Evaluations to be performed• Audits and reviews to be performed• Standards applicable to the product• Error reporting and tracking• Documents produced by SQA• Amount/type of feedback to developers

Types of Reviews Informal discussion of technical problems Formal presentation to customers / management /

staff Formal technical review (FTR)

“code walkthrough” Find errors before they become defects!

Reviews [2] Design activities introduce 50-65% of all errors Reviews can uncover up to 75% of all design

flaws Substantial cost savings in development and

maintenance

Defect Amplification Model

[from SEPA 4/e]

Defect Amplification: No Reviews

[From SEPA 4/e]

Defect Amplification: w/Reviews

[From SEPA 4/e]

Defect Removal

Percentage of Costly DefectsIs Higher (66% vs. 93%)

[From SEPA 4/e]

Cost Before testMuch Greater (34% vs. 7%)

Total Cost of DefectsMuch Greater

The Review Meeting 3-5 people Advance preparation

(< 2 hours per person)• Review software, read documentation

Meeting duration less than 2 hours Constraints limit focus of each meeting (e.g., single

component) Improve likelihood of uncovering errors

Review Meeting [2] Agenda:

• Introduction by developer• Walk-through with Q/A• Review “issues list” is created• Decide whether to:

• accept module without modifications• reject due to severe errors

(repeat review at a future time)• accept with revisions (no further review)

• Summary Report / Action Items

Review Guidelines “Review the product, not the producer” Set and maintain agenda Limit debate and rebuttal Enunciate, don’t solve problems Take written notes Insist on advance preparation

Review Guidelines [2]

Allocate resources for reviews Conduct meaningful training Review your early reviews!

Causes of Defects Incomplete / erroneous spec Misinterpret customer communication Intentional deviation from spec Programming standards not met Error in data representation Inconsistent module interface Error in design logic

Causes of Defects [2] Incomplete / erroneous testing Inaccurate / incomplete documentation Error in implementation of design Ambiguous / inconsistent UI

Statistical QA Collect info about defects Trace defects to causes, tabulate Prioritize (e.g. “80/20 rule”) Move to correct problems Create error index for each module, weighting

more heavily those errors that were discovered later

Software Reliability

“Probability of failure-free operation of a module in a specified environment for a specified time” [MUS87]

How to define “failure”? Annoying vs. catastrophic Varying effort to fix

Reliability Measures Mean time between failures (MTBF) Obscure defects occur rarely Some defects never detected Defects that are more frequent and costly are most

important

Software Safety

Identify hazards Prioritize (probability, severity) Design to minimize hazards Execution modeling:

• Fault tree analysis [VES81]• Real-time logic [JAN86]• Petri nets [LEV97

Poka-yoke devices:

Questions?