january 20, 2000 cse 7315 - sw project management / chapter 12 – software quality engineering...

January 20, 2000

CSE 7315 - SW Project Management / Chapter 12 – Software Quality Engineering

& AssuranceCopyright © 1995-2000, Dennis J. Frailey,

All Rights Reserved

Slide # 1

CHAPTER 12

SOFTWARE QUALITY ENGINEERING &

ASSURANCENOTICE: This material is copyrighted and may be copied

or downloaded ONCE ONLY by students who are registered in this course at Southern Methodist University or National Technological University.

January 20, 2000



All Rights Reserved

Slide # 2

Software Quality Engineering and Assurance

References to text: • Managing Quality 16• Engineering Quality 9,10• Software Testing 11• Other Relevant Material 8

January 20, 2000



All Rights Reserved

Slide # 3

SW Quality AssuranceTypical Symptoms of a Problem

We have aprocedure to

prevent that! Why did it happen?

The procedurewas not followed. Nobody even knew

about it

January 20, 2000



All Rights Reserved

Slide # 4

Software Quality AssuranceSymptoms of Another Problem

They’ve violatedcompany policy and we are

facing a lawsuit!

Did they knowabout the policy?Who authorizedwhat they did?

January 20, 2000



All Rights Reserved

Slide # 5

Other Typical Symptoms of Quality Problems

• Management was surprised that the product did not work correctly after release–No one mentioned that it was not

really ready

• The customers complain that the software does not work as advertised–But we tested it!

January 20, 2000



All Rights Reserved

Slide # 6

Motivation of the Software Developer

Pride in Workmanship and Skill

Fear of Failure

QualitySoftware

Software Developer

January 20, 2000



All Rights Reserved

Slide # 7

Software Quality Assurance

Purposes:– To provide management and

developers with visibility into the process being followed and the products being built• So they can manage the outcome

– To provide a quality control mechanism

January 20, 2000



All Rights Reserved

Slide # 8

Software Quality Assurance

Typical Practices:– Reviews

– Audits

– Communication

– Measurements

– Independent confirmation of compliance• Standards, requirements, procedures

January 20, 2000



All Rights Reserved

Slide # 9

What Is Software Quality?

January 20, 2000



All Rights Reserved

Slide # 10

There Are Many Definitions• Crosby: Quality is Conformance to

Requirements• Juran: Quality is Fitness for Intended

Use• Webster: – (1) Quality is that which makes something

what it is– (2) Quality is the Degree of Excellence

• Weinberg: Quality is value to someone

January 20, 2000



All Rights Reserved

Slide # 11

The Essential Characteristics

• The product does what it is supposed to do

• The customer gets what they paid for

January 20, 2000



All Rights Reserved

Slide # 12

External vs. Internal Quality

External Quality:– Is determined by the factors whose

presence or absence may be observed by the users

– Is used as the ultimate criterion to judge the quality of a system

Internal Quality:– Factors invisible to the end users– Help achieve the external quality

January 20, 2000



All Rights Reserved

Slide # 13

External Quality Factors

• Correctness– Does the system conform to its

specs?

• Robustness– Does the software system react

appropriately to abnormal conditions?

• Extendibility– Ease of adapting the software product

to changes in its specifications

January 20, 2000



All Rights Reserved

Slide # 14

External Quality (continued)• Compatibility– The ease with which the system can be

combined with other systems• Efficiency– The ability of the software to put as little

demand on hardware as possible

• Portability– The ease of transferring software

products to different hardware/software platforms

January 20, 2000



All Rights Reserved

Slide # 15

External Quality (continued)

• Functionality– The extent of possibilities supported by

the software system

• Timeliness & Economy– The ability of the software to be cost

effectively developed and released in a timely manner

January 20, 2000



All Rights Reserved

Slide # 16

External Quality (continued)

• Ease of use– The ease with which people with

different backgrounds can learn to use the software and then effectively apply it to their benefit

• Verifiability– How easily and cost effectively can the

correctness of the system be judged?

January 20, 2000



All Rights Reserved

Slide # 17

Internal Quality Factors• Modularity– The degree to which elements of the

software are independent from each other

• Readability– How hard is it to read and understand

the source code?

• Reusability– The ability of the elements of the

software to be used in later systems

January 20, 2000



All Rights Reserved

Slide # 18

Internal Quality Factors

• Some of the external quality factors have an equally strong impact on the internal quality– Timeliness & Economy– Compatibility– Portability– Extendibility

January 20, 2000



All Rights Reserved

Slide # 19

Evolution of Software Quality

QualityControl

QualityAssurance

QualityEngineering

January 20, 2000



All Rights Reserved

Slide # 20

Quality ControlGoal: Keep Quality at an Acceptable Level by

Rejecting Unacceptable Products

Requirements

Development QC InspectionPass

Fail

Standards of

Quality

QualityControl

QualityAssurance

QualityEngineering

January 20, 2000



All Rights Reserved

Slide # 21

Problems with Quality Control

• Adversarial Relationship between QC and Developers

• No Motive to Improve• Little Help with Improvement

January 20, 2000



All Rights Reserved

Slide # 22

Software Quality Control

Criteria for QC Inspection– How many requirements are met?– How many tests have passed / failed?

Problems (maybe) Unique to Software– Are the tests adequate?– Do the fixes inject more errors? – Developers may intentionally fail to test

or avoid testing, to save time or because they are overconfident

January 20, 2000



All Rights Reserved

Slide # 23

Testing can detect the presence of errors, but not

their absence

Develop Test ?Errors

InjectedErrors

Reduced

How Many Errors are

Left?

January 20, 2000



All Rights Reserved

Slide # 24

Quality Assurance

Goal: To assure that quality is achieved -- Add quality during development -- Improve processes, standards, and

up-front evaluations of the product as it is being developed

QualityControl

QualityAssurance

QualityEngineering

January 20, 2000



All Rights Reserved

Slide # 25

Quality Assurance Looks at the Entire Process

Requirements

Development QC InspectionPass

Fail

Standards ofQuality

Process and Design Standards

January 20, 2000



All Rights Reserved

Slide # 26

Each Step of the Process Has an Effect on Final Defect Rate

ProcessStep

I = Defects Input

O = Defects Output

F = Defects Found and

Fixed

C = Defects Created

O = I + C - F

January 20, 2000



All Rights Reserved

Slide # 27

Responsibilities of QA

• Review all development plans for completeness

• Participate as inspection moderators in design and code inspections

• Review a significant sample of all test results to determine adherence to standards

• Periodically audit SCM performance to determine adherence to standards

January 20, 2000



All Rights Reserved

Slide # 28

At Each Step, QA must...• Inspect the product for – Conformance with standards– Satisfaction of requirements

• Evaluate the process for – Conformance with standards– Opportunities to improve– Process defects that may cause problems

later on

• Evaluate the support processes as well

January 20, 2000



All Rights Reserved

Slide # 29

At Each Step , QA must...

• Monitor reviews, inspections, walkthroughs, etc. to see that they are accomplishing their objectives

• Trace back to prior steps when defects are found

In addition to evaluating the development process, the QA procedures must also be tailored and revised for the needs of each

new project

January 20, 2000



All Rights Reserved

Slide # 30

At the Requirements Analysis Step ...

• Trace software requirements or specifications to original system or customer requirements

• Inspect or evaluate software requirements against standards to see that they are

-- complete, -- correct, -- consistent, -- testable, and -- understandable• “Clean Room” technique takes this farther

PROGRAM SAMPLE (IN1,IN2)INTEGAR IN1,IN2,A,B,C;FOR I = 1 STEP 3 UNTIL 99 DO IF IN1 < A*B THEN IN2 := C + A*B ELSE IN2 := C + IN1 ENDIFENDDO** THE NEXT PART WILL SORTDO FOREVER Ii := I + 1 TEMP := ARRAY[I] - ARRAY[I+1] IF TEMP < 0, SWAP (ARRAY, i)ENDLOOPCALL CHECKORDER(ARRAY)** NEXT WE DO THE i/oCALL PRINT(ARRAY)CALL DEBUG (ARRAY, I)CASE I OF CALL SUB1 CALL SUB1

January 20, 2000



All Rights Reserved

Slide # 31

“Clean Room” Software Development Technique

• Motive: Don’t let the Bugs Get In• Method: Filter Them Out from the Beginning -- Define the requirements using a formal

notation – reduces ambiguity– some consistency/correctness issues can be

checked automatically

-- Carry out rigid inspections• Benefits: fewer problems later in the development process (e.g., reduced testing).

January 20, 2000



All Rights Reserved

Slide # 32

At the Design Step ...• Trace the design to the requirements

specification (plus “derived requirements”)• Evaluate the design against standards -- complexity -- cohesion -- understandability -- etc.• Evaluation of design effectiveness,

correctness, consistency, completeness• Evaluate plans for testing and test cases

January 20, 2000



All Rights Reserved

Slide # 33

At the Coding Step ...• Trace code to design specifications• Evaluate code against coding standards

(example: give code to a new maintainer and see if she or he can understand it)

• Code walkthroughs or inspections to check on coding mistakes

• Test coverage analysis -- Do the tests address all requirements? (black box tests) -- Do the tests adequately evaluate the

design and its implementation? (white box tests)

January 20, 2000



All Rights Reserved

Slide # 34

At the Module Testing Step ...• Test procedures and code analysis -- Evaluate test code like any other code -- Are the procedures documented? -- Are the expected results documented?• Test results analysis -- Were the tests run? -- Were the results correct?

January 20, 2000



All Rights Reserved

Slide # 35

Two Philosophies of Testing

CodeTest

a LittleReview

ThoroughTest

Code ReviewThorough

Test

January 20, 2000



All Rights Reserved

Slide # 36

During Integration ...• Make sure that regression tests are

performed -- Retest all modules potentially

affected by any change• Do independent tests -- Someone other than the person

who developed the software• Formal qualification tests

(acceptance tests)

January 20, 2000



All Rights Reserved

Slide # 37

When do you Perform the Formal Qualification Test?

HardwareDevelopment

OperatingSystem

Development

SoftwareDevelopment

SystemIntegration

SystemFQT

Software FQT:

Here or Here

January 20, 2000



All Rights Reserved

Slide # 38

Planning for QA• Each development project should have

a documented Software Quality Assurance Plan (SQAP)– May be part of SW Development Plan

• The SQAP documents tasks to be performed, standards of verification, procedures and organizational structure

• There is an IEEE standard for SQAPs

January 20, 2000



All Rights Reserved

Slide # 39

Pitfalls in implementing QA

• Lack of sufficiently skilled staff in QA – Software professionals are typically more

interested in development jobs than QA

• QA management cannot effectively negotiate with the development team

• Senior Management commitment may fade under schedule pressures– The developers stop caring about QA &

action revolves around insignificant issues

January 20, 2000



All Rights Reserved

Slide # 40

Pitfalls in implementing QA(Continued)

• QA organization operates without suitably documented and approved development standards and procedures

• Development team does not produce verifiable quality plans– Eventually results in a debate over which

bugs are more important to fix rather than whether the product has sufficient quality or not

January 20, 2000



All Rights Reserved

Slide # 41

Pitfalls in implementing QA(continued)

• Disputes over who has responsibility for product quality and for performing QA functions– The software developers and managers

are responsible for product quality– Quality assurance is a method to help

them achieve quality through independent evaluation of their work (like a coach)

January 20, 2000



All Rights Reserved

Slide # 42

Reporting Chain for QA Staff

• Except in the highest maturity organizations, it is strongly recommended that the QA staff have a separate reporting chain from the project team– This improves their objectivity– And assures that disputes are

resolved at a level in the organization that has responsibility for the possible consequences

January 20, 2000



All Rights Reserved

Slide # 43

Benefits Achieved by Quality Assurance over Quality

Control• Significant insight is gained into the process

that can be used to gain long term benefits– Development can focus on specific weak points

in the quality– Process improvements can be identified– Steps that are difficult to perform can be

automated

January 20, 2000



All Rights Reserved

Slide # 44

Problems with Quality Assurance

• Can still be adversarial (although perhaps less than with quality control)

• Motivation to fix the process is still weak• Can be more costly than quality control• It can be hard to show benefits until long

after the product has been shipped

January 20, 2000



All Rights Reserved

Slide # 45

Quality EngineeringGoal: Build Quality In as Part of the Software

Engineering Process

A philosophical change that relies on the professional pride of the software engineer

• Engineering staff define and execute the quality assurance tasks

• Team approach to quality, with rewards based on quality

• Quality professionals are more like coaches and educators than evaluators

QualityControl

QualityAssurance

QualityEngineering

January 20, 2000



All Rights Reserved

Slide # 46

Quality Engineering Focuses on People as well

as Process• Finding errors is good -- it keeps them from

leaking through to the customer• Problems result in process changes, not

punishment of people• Everyone appreciates that a competitive

process is the way to remain a competitor• Measurements are used so that decisions

are based on fact (in addition to intuition)• Independent tests are welcome -- they tell

you if you are as good as you think you are

January 20, 2000



All Rights Reserved

Slide # 47

Peer ReviewsMotivation: how to find problems in your

work in a non-threatening wayMethod: evaluation of one individual’s work

by others at the same organizational level (i.e., team members, not supervisors)

Software Developers

Software Managers

January 20, 2000



All Rights Reserved

Slide # 48

Peer Reviews - CostCost: may require 10-20% of the total

development time -- this must be planned into the

schedule -- it must also be planned into progress

measures and reward systems

January 20, 2000



All Rights Reserved

Slide # 49

Peer Reviews - BenefitsBenefits: reduced rework and faster

development due to a more effective evaluation

-- the peers understand the software best

-- and they will not be as much of a threat when it comes to promotions and raises

-- reciprocity -- you help them and they help you

January 20, 2000



All Rights Reserved

Slide # 50

Big Problem with Peer Reviews

People don’t want to attend others’ peer reviews because they are behind in their own schedules

Solutions? (class discussion)

January 20, 2000



All Rights Reserved

Slide # 51

Benefits of the Quality Engineering Approach

• Less adversarial• Motivation and Information to improve• Flexibility to change the process in

response to a problem -- you understand the problem and its

cause -- you understand the consequences of

a change in the process• Knowledge is the foundation of

successful quality engineering

January 20, 2000



All Rights Reserved

Slide # 52

The Cost of Quality Engineering

• Your process may be more complex, costly and time consuming - at least when you start

• But you gain knowledge -- what and where to automate -- how and where to improve the

process• And you gain higher quality• And you can begin to optimize the

process, just as you optimize software

January 20, 2000



All Rights Reserved

Slide # 54

Quality (Fewer Defects; Customer

Satisfaction)

Productivity Cycle Time

CustomerValue

Quality Improvements Cost Money

• How can we justify them?

January 20, 2000



All Rights Reserved

Slide # 55

Managers and Technical Staffmust be convinced that

• Quality problems are serious

• Poor quality costs them money

• Quality is worth fixing

• Quality can be fixed by proper techniques

In Order to Justify Quality Improvements ...

January 20, 2000



All Rights Reserved

Slide # 56

The Fundamental Prejudice

• Quality improvement is a non-value-added activity– It costs overhead resources–The benefits are not necessarily real

• So we seek to avoid it

Cost of Quality (COQ) Analysis is used to address this prejudice

January 20, 2000



All Rights Reserved

Slide # 57

The First Question

“What does it cost to have quality?”

• This is what management and technical staff will typically ask when approached with a proposal to improve quality

January 20, 2000



All Rights Reserved

Slide # 59

Categorizing Quality-Related Costs

Quality Related Costs

Cost of Conformance

Cost of Non-Conformance

Prevention Costs

Appraisal Costs

Failure Costs

January 20, 2000



All Rights Reserved

Slide # 60

Cost of Conformance

• Things that improve quality–Prevention costs - prevent poor quality• Predictive metrics, training, root

cause analysis, process improvement

–Appraisal costs - detect poor quality• Tests, inspections, audits

January 20, 2000



All Rights Reserved

Slide # 61

Cost of Non-Conformance

• The price you pay when you fail to achieve quality–Internal failures• Costs incurred before product

shipment

–External Failures• Costs incurred after product

shipment

January 20, 2000



All Rights Reserved

Slide # 62

Effects of Maturity on Costs

SEI Maturity Level

Cost

as

a P

erc

ent

of

Develo

pm

en

t C

ost

10

20

30

40

50

60

1 2 3 4 5PreventionAppraisal

InternalFailures

ExternalFailures

Total COQ

As reported by Knox (see references)

January 20, 2000



All Rights Reserved

Slide # 63

Summary

• Quality Control focuses on the end product

• Quality Assurance focuses on the process

• Quality Engineering focuses on process and people, integrating QA with the software engineering process

January 20, 2000



All Rights Reserved

Slide # 64

References• Knox, 1993, Raytheon studies reported

by Houston and Keats, Software Quality Matters, vol 5, no 1 (Spring, 1997), U. of Texas SW Quality Institute

• Musa, John D, 1992, “The Operational Profile,” in Software Reliability Engineering: An overview.

• Weinberg, Gerald M., 1992, Quality Software Management, Volume 1, Systems Thinking, Dorset House, New York, ISBN 0-932633-22-6.

january 20, 2000 cse 7315 - sw project management / chapter 12 – software quality engineering...

Documents

systeminternal quality

is2 quality

software products

software developerpride

possibleportabilitythe

factors invisible

reusabilitythe ability

systemsefficiencythe