cse 8314 - sw metrics and quality engineering copyright 1995-2001, dennis j. frailey, all rights...

8/20/2001

Slide 1CSE 8314 - SW Metrics and Quality EngineeringCopyright © 1995-2001, Dennis J. Frailey, All Rights Reserved CSE8314M12

SMU CSE 8314 / NTU SE 762-N

Software Metrics and Quality Engineering

Module 12Software Reliability

Overview

8/20/2001


Contents

• Introduction• Measuring Reliability• Summary

8/20/2001


Introduction

8/20/2001


What can be Measured• It does what was specified• Failure rates are low• etc.

End-User’s Perspective

• It does what I want• It never fails• etc.

Not a Perfect Match

Reliability is the “Bottom Line” of Software Quality

• Reliability is the most conspicuous attribute of quality

• But what do we mean by reliability?

8/20/2001


• Assumption: failure results from physical changes– breakage, wearout, fatigue

• Or incorrect manufacturing processes

• Theory of reliability is founded on the statistical behavior of random molecules

Hardware Reliability Focuses on Materials and Production

8/20/2001


But Product Development can also be a Factor in Quality &

Reliability• Bulletproof Vest

– 2” thick steelvs.

– lightweight synthetic material

• Quality?• Reliability?• Suitability for Purpose?

8/20/2001


Poor Development Practices can Lead to Failure

• What if the design puts undue strain on a part?– Was the failure due to the part or the

design?• What if the product wasn't properly

tested?– Car overheats in the desert (never

tested that severely)

8/20/2001


Poor Software Development Practices can lead to Failure

• Software Failures are almost always attributable to product development practices

–Design - Testing–Coding - CM

8/20/2001


Software Reliability

• Failure: when it does not do what it is supposed to do

• Defect: the reason for the failure– Bad code/data/design/requirements– Bad configuration control– etc.

“The extent to which software correctly performs the functions

assigned to it”

8/20/2001


Improving Software ReliabilityOption 1

Design software to be fault tolerant– Redundancy– Multiple algorithms

• This approach has been shown to have very little effect on overall reliability

• It is a better fit to the hardware paradigm that involves fatigue of parts

8/20/2001


Improving Software ReliabilityOption 2

Develop software to be free of defects– Prevention activities– Detection activities

• This is where we will concentrate• Traditionally this has been viewed as

a lost cause• But using modern techniques of quality

improvement, there is hope for success

8/20/2001


Measuring Reliability

8/20/2001


DefinitionsFailure

– When the product does not do what it is expected to do for a given set of input or operating conditions.

Fault (depends on author) – A condition that causes failures.

Defect (depends on author):– A fault found before / after product release– Any cause of failure– Any error, regardless of whether it is caught

before release– Other terms: bug, mistake, malfunction, etc.

8/20/2001


What is Reliability?• Reliability is the probability that

software will operate for a given time interval (usually denoted by t) under given conditions (usually denoted by C or E) with no failures– t is a random, failure free time interval. – The question we are trying to answer is:

how long is t? – But since we cannot know this, we can only

estimate the probability for a given value of t.

8/20/2001


Reliability is not Correctness• Reliability means that it does what

you want it to do often enough to be satisfactory

whereas• Correctness is a binary, “yes or no”

condition• Software is almost never perfectly

correct• But it can be highly reliable

8/20/2001


This definition is usually expressed as a function:

R(t) = probability of operation without

failures in time t(i.e., in the interval 0-t)

• For example, with an exponential distribution, the classic reliability equation is: R(t) = e -t/

8/20/2001


Graph of a Reliability Function

Reliability Function for Exponential Distribution

0

0.2

0.4

0.6

0.8

1 2 3 4 5 6 7 8 9

t = time since product release

R(t) = e -t/

8/20/2001


Measures Reliability as a Constant

• For large values of , the probability of operation without failure remains high for a longer period of time

• For small values of , the probability of operation without failure deteriorates quickly

This works with an exponential distribution function. Otherwise, there may not be a simple

constant to measure reliability of the total product.

8/20/2001


Additional Notes about Reliability

• The desired value of t depends a lot on the application and the priorities

• Commercial application– t is large & the priority is to have

few defects over the life of the application in order to keep maintenance cost low

• Real time application – e.g. an aircraft application, t is relatively

short & failures in operation are critical

8/20/2001


Failure Function• Another popular approach is to look at

the probability of a failure:F(t) = 1 - R(t) = probability of failure in

time t• The latter is called a failure function.• It is the cumulative distribution

function of the time interval 0-t.• For the exponential distribution, the

failure function is:F(t) = 1 - e -t/

8/20/2001


Graph of a Failure Function

Failure Function for Exponential Distribution

00.20.40.60.8

11.2

1 2 3 4 5 6 7 8 9


F(t) = 1 - e -t/

8/20/2001


Probability Density Function• Attempts to put it in another form

that means something to a user: “(approximately) how many defects

are left in the software?”

For the exponential distribution, the formula is:

f(t) = dF(t)/dt

f(t) = -1e -t/

8/20/2001


Graph of Density Function

00.050.1

0.150.2

0.250.3

0.35

1 2 3 4 5 6 7 8 9


f(t) = -1e -t/

8/20/2001


Post Release DefectsPost- Release Defects

Product XYZ Release 2.3

020406080

0 1 2 3 4 5 6 7 8 9 10 11 12Months Since Release

Defe

cts

Defects Found Total Remaining

8/20/2001


All three functions( = 2)

00.20.40.60.8

11.2

1 2 3 4 5 6 7 8 9t = time since product release

R(t) F(t) f(t)

8/20/2001


Failure Rate ()(Hazard Function; Failure

Intensity)• This is an attempt to estimate the

probability of a failure at a given time, assuming no prior failures.

(t) = f(t)/R(t) = -dR(t)/dt

8/20/2001


How to Determine • For an exponential distribution, is a constant

• I.e., the higher the reliability, the lower the failure rate

= 1/

This works with an exponential distribution. Otherwise, may be a more complex function.

8/20/2001


Failure Ratevs Number of Defects

• Most models assume the failure rate is directly related to the number of defects remaining in the software.

Is this a reasonable assumption? (Discussion)

8/20/2001


Problems with the Assumptions for Classic

Definitions• Defects are random with respect to

the structure of the software– But some parts are harder to write than

others• Testing is uniform with respect to

software– But some parts are better tested than

others

8/20/2001


More Problems with Assumptions

• All defects are equally likely to occur– But it actually depends on the paths

taken most often• All defects produce equally serious

failures– Clearly not the case for software

• Testing correctly simulates normal, stressful and unusual conditions– Generally this is very hard to do

8/20/2001


Error ProbabilityHardware vs. Software

Probability of Error vs. Time

00.20.40.60.8

1

1 2 3 4 5 6 7 8 9 10 11 12

Hardware Software

8/20/2001


Another View

Def ect Rate af ter Product Release

05

10152025

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Hardware Software

Infant Mortality

Wearout

8/20/2001


Software May Not FitClassical Models of Reliability• There is very little solid data for any

of the models in use for software• Conditions that differ may produce

very different failure rates, even though defect count remains the same

8/20/2001


How to Measure Time

• The measure of time is a matter of considerable dispute

• This may dramatically affect how you determine reliability

8/20/2001


Three Ways to Measure Timefor a Software Product

• Real Time (Calendar Time)– Number of weeks or months since some

event• Use Time

– Number of hrs the software is in actual use• CPU Time

– Number of hours using the CPU• Each of these produces very different

results and fits very different models

8/20/2001


The Nature of The Application

• Different applications can have very different notions of reliability

8/20/2001


Different Applications - Different Reliability ImplicationsApplication: Financial TransactionsProblem: Floating Point Round off ErrorsNot a Problem: Excessive Time for Calculations

Application: Space Craft Flight Path Calculations (ground

based)See Financial Transactions

Application: Space Craft Local NavigationProblem: Excessive Time for CalculationsNot a Problem: Floating Point Round off Errors

8/20/2001


Summary• Reliability is important, but hard to

measure in a way that relates to customer expectations

• Software reliability is mainly determined by development practices rather than by manufacturing or materials

• Reliability, failure rate, and failure density are related measures

8/20/2001


References

• Lyu, Michael R., Handbook of Software Reliability Engineering, IEEE, 1996, Catalog # RS00030. ISBN 0-07-039400-8.

• Musa, John, Software Reliability Engineering: More Reliable Software, Faster Development and Testing, McGraw Hill. ISBN: 0-07-913271-5.

• Xie, M. Software Reliability Modeling, World Scientific, London, 1991. ISBN 981-02-0640-2.

8/20/2001


END OFMODULE 12

cse 8314 - sw metrics and quality engineering copyright 1995-2001, dennis j. frailey, all rights...

Documents