copyright 1995-2008, dennis j. frailey cse7315 – software project management cse7315 m34 - version...

Copyright 1995-2008, Dennis J. Frailey CSE7315 – Software Project ManagementCSE7315 M34 - Version 8.01

SMU CSE 7315Planning and Managing a Software Project

Module 34

Software Quality Assurance

Copyright 1995-2008, Dennis J. Frailey CSE7315- Software Project ManagementCSE7315 M34 - Version 8.01 2

Objectives of This Module

To examine the techniques of software quality assurance

To look at some techniques of software quality engineering

– Notably, cost of quality analysis


Software Quality Engineering and Assurance

References to text:

Managing Quality 16

Engineering Quality 9,10

Software Testing 11

Other Relevant Material 8


Evolution of Software Quality

QualityControl

QualityAssurance

QualityEngineeringThis Module Introduced

in This Module



Purposes:

– To provide management and developers with visibility into the process being followed and the products being built So they can manage the outcome

– To provide a more effective quality control mechanism



Typical Practices:

– Inspections

– Reviews

– Audits

– Communication

– Measurements

– Independent confirmation of compliance Standards, requirements, procedures


Quality Assurance

Goal: To assure that quality is achieved

-- Add quality during development

-- Improve processes, standards, and up-front evaluations of the product as it is being developed

QualityControl

QualityAssurance

QualityEngineering


Quality Assurance Looks at the Entire Process

Requirements

Development QC InspectionPass

Fail

Standards ofQuality

Process and Design Standards


Each Step of the Process Has an Effect on The Final Defect Rate

ProcessStep

I = Defects Input O = Defects Output

F = Defects Found and Fixed

C = Defects Created

O = I + C - F


Typical Responsibilities of QA Review all development plans for

completeness

Participate as inspection moderators in design and code inspections

Make sure the tests are appropriate and are run correctly

– Note – true quality assurance does not develop or run tests – it assures they are done correctly

Review a significant sample of all test results to determine adherence to standards

Periodically audit SCM performance to determine adherence to standards


At Each Step, QA must...

Inspect the product for

– Conformance with standards

– Satisfaction of requirements

Evaluate the process for

– Conformance with standards

– Opportunities to improve

– Process defects that may cause problems later on

Evaluate the support processes as well


At Each Step, QA must...

Monitor reviews, inspections, walkthroughs, etc. to see that they are accomplishing their objectives

Trace back to prior steps when defects are found

In addition to evaluating the development process, the QA procedures must also be tailored and revised for the needs of each

new project

In addition to evaluating the development process, the QA procedures must also be tailored and revised for the needs of each

new project


At the Requirements Analysis Step ...

Trace software requirements or specifications to original system or customer requirements

Inspect or evaluate software requirements against standards to see that they are

-- complete,

-- correct,

-- consistent,

-- testable, and

-- understandable

“Clean Room” technique takes this farther

PROGRAM SAMPLE (IN1,IN2)INTEGAR IN1,IN2,A,B,C;FOR I = 1 STEP 3 UNTIL 99 DO IF IN1 < A*B THEN IN2 := C + A*B ELSE IN2 := C + IN1 ENDIFENDDO** THE NEXT PART WILL SORTDO FOREVER Ii := I + 1 TEMP := ARRAY[I] - ARRAY[I+1] IF TEMP < 0, SWAP (ARRAY, i)ENDLOOPCALL CHECKORDER(ARRAY)** NEXT WE DO THE i/oCALL PRINT(ARRAY)CALL DEBUG (ARRAY, I)CASE I OF CALL SUB1 CALL SUB1


“Clean Room” Software Development Technique

Motive: don’t let the bugs get in

Method: filter them out from the beginning

-- Define the requirements using a formal notation

– Reduces ambiguity

– Some consistency/correctness issues can be checked automatically

-- Carry out rigid inspections

Benefits: fewer problems later in the development process (e.g., reduced testing).


At the Design Step ...

Trace the design to the requirements specification (plus “derived requirements”)

Evaluate the design against standards

-- complexity

-- cohesion

-- understandability

-- etc.

Evaluation of design effectiveness, correctness, consistency, completeness

Evaluate plans for testing and test cases


At the Coding Step ...

Trace code to design specifications

Evaluate code against coding standards (example: give code to a new maintainer and see if she or he can understand it)

Code walkthroughs or inspections to check on coding mistakes

Test coverage analysis

-- Do the tests address all requirements? (black box tests)

-- Do the tests adequately evaluate the design and its implementation? (glass box tests)


At the Module Testing Step ...

Test procedures and code analysis

-- Evaluate test code like any other code

-- Are the procedures documented?

-- Are the expected results documented?

Test results analysis

-- Were the tests run?

-- Were the results correct?


Two Philosophies of Testing

CodeTest

a LittleReview

ThoroughTest

Code ReviewThorough

Test


During Integration ...

Make sure that regression tests are performed

-- Retest all modules potentially affected by any change

Do independent tests

-- Someone other than the person who developed the software

Formal qualification tests (acceptance tests)


Planning for QA

Each development project should have a documented Software Quality Assurance Plan (SQAP)

– May be part of SW Development Plan

The SQAP documents tasks to be performed, standards of verification, procedures and organizational structure

There is an IEEE standard for SQAPs


Pitfalls in Implementing QA

Lack of sufficiently skilled staff in QA

– Software professionals are typically more interested in development jobs than QA

QA management cannot effectively negotiate with the development team

Senior management commitment may fade under schedule pressures

– The developers stop caring about QA & action revolves around insignificant issues


Pitfalls in Implementing QA(continued)

QA organization operates without suitably documented and approved development standards and procedures

Development team does not produce verifiable quality plans

– Eventually results in a debate over which bugs are more important to fix rather than whether the product has sufficient quality or not


Pitfalls in Implementing QA(continued)

Disputes over who has responsibility for product quality and for performing QA functions

– The software developers and managers are responsible for product quality

– Quality assurance is a method to help them achieve quality through independent evaluation of their work (like a coach)


Reporting Chain for QA Staff

Except in the highest maturity organizations, it is strongly recommended that the QA staff have a separate reporting chain from the project team

– This improves their objectivity

– And assures that disputes are resolved at a level in the organization that has responsibility for the possible consequences

Vice President

Quality Mgr

Project Mgr

Software Mgr

SoftwareEngineer

Quality Engineer


Benefits Achieved by Quality Assurance over Quality Control

Significant insight is gained into the process that can be used to gain long term benefits

– Development can focus on specific weak points in the quality

– Process improvements can be identified

– Steps that are difficult to perform can be automated


Problems with Quality Assurance

Can still be adversarial (although perhaps less than with quality control)

Motivation to fix the process is still weak

Can be more costly than quality control

It can be hard to show benefits until long after the product has been shipped


Quality Engineering

Goal: Build quality in as part of the software engineering process

A philosophical change that relies on the professional pride of the software

engineer

Engineering staff define and execute the quality assurance tasks

Team approach to quality, with rewards based on quality

Quality professionals are more like coaches and educators than evaluators

QualityControl

QualityAssurance

QualityEngineering


Quality Engineering Focuses on People As Well As Process

Finding errors is good -- it keeps them from leaking through to the customer

Problems result in process changes, not punishment of people

Everyone appreciates that a competitive process is the way to remain a competitor

Measurements are used so that decisions are based on fact (in addition to intuition)

Independent tests are welcome -- they tell you if you are as good as you think you are


Peer Reviews

Motivation: how to find problems in your work in a non-threatening way

Method: evaluation of one individual’s work by others at the same organizational level (i.e., team members, not supervisors)

Software Developers Software Managers


Peer Reviews - Cost

Cost: may require 10-20% of the total development time

-- this must be planned into the schedule

-- it must also be planned into progress measures and reward systems


Peer Reviews - Benefits

Benefits: reduced rework and faster development due to a more effective evaluation

-- the peers understand the software best

-- and they will not be as much of a threat when it comes to promotions and raises

-- reciprocity -- you help them and they help you


Big Problem with Peer Reviews

People don’t want to attend others’ peer reviews because they are behind in their own schedules

Solutions?

Class discussion


Benefits of the Quality Engineering Approach

Less adversarial

Motivation and information to improve

Flexibility to change the process in response to a problem

-- you understand the problem and its cause

-- you understand the consequences of a change in the process

Knowledge is the foundation of successful quality engineering


The Cost ofQuality Engineering

Your process may be more complex, costly and time consuming - at least when you start

But you gain knowledge

-- what and where to automate

-- how and where to improve the process

And you gain higher quality

And you can begin to optimize the process, just as you optimize software


Cost of Quality Analysis


Quality (Fewer Defects; Customer

Satisfaction)

Productivity Cycle Time

CustomerValue

Quality Improvements Cost Money

How can we justify them?


Managers and Technical Staff

must be convinced that

Quality problems are serious

Poor quality costs them money

Quality is worth fixing

Quality can be fixed by proper techniques

In Order to Justify Quality Improvements ...


The Fundamental Prejudice

Quality improvement is a non-value-added activity

– It costs overhead resources

– The benefits are not necessarily real

So we seek to avoid it

Cost of Quality (COQ) Analysis is used to address this prejudice


The First Question

This is what management and technical staff will typically ask when approached with a

proposal to improve quality

This is what management and technical staff will typically ask when approached with a

proposal to improve quality

“What does it costto improve quality?”


The Right Questions

“What’s the payoff?”

“What is the return on investment for quality

improvements?


Categorizing Quality-Related Costs

Quality Related Costs

Cost of Conformance

Cost of Non-Conformance

Prevention Costs

Appraisal Costs

Internal Failure Costs

External Failure Costs


Cost of Conformance

Things that improve quality

– Prevention costs - prevent poor quality

Predictive measurements, training, root cause analysis, process improvement

– Appraisal costs - detect poor quality

Tests, inspections, audits


Cost of Non-Conformance

The price you pay when you fail to achieve quality

– Internal failures

Costs incurred before product shipment

– External failures

Costs incurred after product shipment


Effects of Maturity on Costs

SEI Maturity Level

Cost

as

a P

erc

ent

of

Develo

pm

en

t C

ost

10

20

30

40

50

60

1 2 3 4 5PreventionAppraisal

InternalFailures

ExternalFailures

Total COQ

As reported by Knox (see references)


Module Summary

Quality Control focuses on the end product

Quality Assurance focuses on the process

Quality Engineering focuses on process and people, integrating QA with the software engineering process


References

Knox, 1993, Raytheon studies reported by Houston and Keats, Software Quality Matters, vol 5, no 1 (Spring, 1997), U. of Texas SW Quality Institute

Meyer, Bertrand, 1997, Object Oriented Software Construction

Musa, John D, 1992, “The Operational Profile,” in Software Reliability Engineering: An overview.

Weinberg, Gerald M., 1992, Quality Software Management, Volume 1, Systems Thinking, Dorset House, New York, ISBN 0-932633-22-6.


END OF

MODULE 34

copyright 1995-2008, dennis j. frailey cse7315 – software project management cse7315 m34 - version...

Documents

software project module

standards of quality

quality assurance goal

true quality assurance

software testing11

cost of quality analysis

module slide

procedures slide