day 5 reliability

7/25/2019 Day 5 Reliability

1/77

Software Testing and ReliabilityReliability and Risk Assessment

Aditya P. Mathur

Purdue UniversityAugust 121!

" #uidant $or%orationMinnea%olis&St Paul' M(

Graduate Assistants: Ramkumar NatarajanBaskar Sridharan

Last update: August 16, 2002


2/77

Software Testing and 2

Reliability and risk assessment )earning ob*e+tives

1. ,hat is software reliability-

2. ow to estimate software reliability-

/. ,hat is risk assessment-

0. ow to estimate risk using a%%li+ation ar+hite+ture-


3/77


Referen+es1. Statistical Methods in Software Engineering: Reliability and

Risk' (oer . Sing%urwalla and Simon P. ,ilson' S%ringer'

1333.

2. Software Reliability, Measurement, Prediction,Application' 4ohn . Musa' Anthony 5annino' and

6auhira 7kumoto' M+#rawill 8ook $om%any' 139:.

/. A Methodology for Architecture Leel Reliability RiskAnalysis' S. M. ;a+oub and . . Ammar' 5


4/77


Software Reliability Softare re!ia"i!it# is the pro"a"i!it# of fai!ure free operation

of an app!i$ation in a spe$ified operating en%ironmentand

time period&

Reliability is onequality metric. Others include

performance, maintainability, portability, and

interoperability


5/77


7%erating


6/77


Un+ertainty Uncertaintyis a common phenomena in our daily lives.

In software engineering, uncertainty occurs in all phases of thesoftware life cycle.

Examples:

Will the schedule be met?

What is the number of faults remaining faults?

How many testers to deploy?

How many months will it take to complete the design?


7/77


Probability and statisti+s Uncertainty can be quantified and managed using probability

theory and statistical inference.

Probability theory assists with quantification and combination of

uncertainties.

Statistical inference assists with revision of uncertainties in

light of the available data.


8/77


Probability Theory In any software process there are known and unknown

quantities.

The knownquantities constitute history and is denoted by H.

The unknown quantities are referred to as random

quantities.

Each unknown quantity is denoted by a capital letter such

as TorX.


9/77


Random >ariables

When a random quantity can assume numerical values it is

known as a random variable.

Specific values of TandXare denoted by lower case letters t

andx and are known as realizationsof the corresponding

random quantities.

Example: IfXdenotes the outcome of a coin toss, thenXcan assume a value 0 (for head) and 1 (for tail).Xis a

random variable under the assumption that on each toss the

outcome is not known with certainty.


10/77


Probability

P(E|H)

or brevity we will su%%ressHand to denote the %robability

of Eas sim%ly

'(EP

The %robability of an event E+om%uted at time in light of

history H is given by


11/77


Random


12/77


8inary Random >ariables

A discrete random variableis one whose realizations are

countable.

When e1and e2are numerical values, such as 0 and 1, thenEis

known as a binary random variable.

Example: Number of failures encountered over four hoursof application

use.

A continuous random variableis one whose realizations are not

countable.

Example: Time to next failure.


13/77


Probability distribution fun+tion

'(xFX

5f E is the event that'()x then is known as

the distributionfun+tionof'and is denoted as

'( xXP

or a random variable'let Ebe the event that'Bx

0'( >= xXP 5f then C is said to have a %oint mass.

(ote that is nonde+reasing in D and ranges from = to

1.'(xF

X


14/77


Probability density fun+tion

'(xFX'(xFX

5f' is +ontinuous and takes all values in some interval I

and is differentiable with res%e+t to*for all*in I'

then is absolutely +ontinuous.

'(xFX The derivative of at D is denoted by and is

known as the %robability density fun+tion of'.

'(xfX

'(xfX dD is the a%%roDimate %robability that the random

variable'takes on a value in the interval*and*+d*.


15/77


0.

0

')( xf

x

xexXP

= ')(


16/77


8inomial istribution

Su%%ose that an a%%li+ation is eDe+uted ( times ea+h

with a distin+t in%ut. ,e want to know the numberof

in%uts''' on whi+h the a%%li+ation will fail.

(ote that the %ro%ortion of the +orre+t out%uts is a measure of

the reliabilityof the a%%li+ation.

'+an assume values x B=' 1' 2'E'(. ,e are interested in the

%robability that'Bx.


17/77


8inomial istribution G+ontd.H

Nxppx

NpxXP xNx ,&&&,0,'1('(')( ===

Under +ertain assum%tions' the following %robability

model' known as the 8inomial distribution' is used.

ere % is the %robability that Ci B 1 for iB1'E'(. 5n other words' % is the %robability of failure of any single run.

''*(*+(*'( xNxNx

N=


18/77


Poisson istribution

&&&&&2,1,0,*')( ===

xxexXP

x

,hen the a%%li+ation under test is almost error free and is

sub*e+ted to a large number of in%uts' then is large' I1pJ is

small' and -.!p/is moderate. The above assum%tion leads to a sim%lifi+ation of the 8inomial

distribution into the Poisson distribution given by the formula


19/77


Software Reliability@ Ty%es

Re!ia"i!it# on a sing!e ee$ution: P(X-1)H', mode!ed "#

Bernou!!i distri"ution&

Re!ia"i!it# o%er N ee$utions: P(X-x)H', for -0,1,2,.N,

gi%en "# Binomia! distri"ution or /oisson distri"ution for

!arge Nand sma!! parameter %a!ue p&

Re!ia"i!it# o%er an infinite num"er of ee$utions: P(X-x)H', for x-1,2,.N& Note that e are interested in the

num"er of inputs after hi$h the first fai!ure o$$urs& his

is gi%en "# geometri$ distri"ution&


20/77


Software Reliability@ Ty%es G+ontd.H

hen the inputs to softare o$$ur $ontinuous!# o%er

time, then e are interested in P(X-)H', i&e& the

pro"a"i!it# that the first fai!ure o$$urs after time units&his is gi%en "# the eponentia! distri"ution&

he time of o$$urren$e to the kth fai!ure $an "e gi%en "#

the Gamma distri"ution&

here are se%era! other mode!s of re!ia"i!it#, o%er one

hundred*


21/77


Software failures@ Sour+es of un+ertainty

Uncertainty about the presence and location of defects.

Uncertainty about the use of run types. Will a run for

a given input state cause a failure?


22/77


ailure Pro+ess

5n%uts arrive at an a%%li+ation at random times.

Some in%uts +ause failures and others do not.

T1' T2' Edenote I$PUJ times between a%%li+ation

failures.

Most reliability models are +entered around the interfailuretimes.


23/77


ailure 5ntensity and Reliability

ailure intensity is the number of failures eD%erien+ed

within a unit of time. or eDam%le' the failure intensity

of an a%%li+ation might be =./ failures&hr.

ailure intensity is an alternate way of eD%ressing

reliability' RIJ' whi+h is the %robability of no failures overtime duration .

or a +onstant failure intensity we have RIJBe. 5t is safe to assume that during testing and debugging' the

failure intensity de+reases with time and thus the reliability

in+reases.


24/77


4elinski and Moranda Model G13:2H

The a%%li+ation +ontains an unknown number of defe+ts.


25/77


4elinski and Moranda Model G+ontd.H

Thus' given =BS=KBS1KBE.KBSi' iB1' 2E and some +onstant c' we obtain the following

failure intensity' where S='S1'E'Si are su%%osed software failure times' failure rate rTiis

given by@

'1('( 1 += iNcStr iTi 1 iStfor

Note that the failure rate

drops by a constant amount.

ttimeS0=0 S1 S2 S3

'(tr
http://../Macintosh%20HD/Users/apm/Desktop/im8zb816.ppt


26/77


Musa7kumoto Model@ Terminology


27/77


Musa7kumoto Model@ Terminology G+ontd.H

(umber of inherent faults@ =BI I

(umber of sour+e instru+tions@ I

5nstru+tion eDe+ution rate@ r


28/77


Musa7kumoto@ 8asi+ Model

ailure intensity for basi+ eDe+ution time model

() = 0

[1

0 ]

() =0e00

R(' | ) = e{[0e

0

0

][1e0

0 '

]}


29/77


Musa7kumoto@ )ogarithmi+ Poisson Model

ailure intensity de+ay %arameter@

ailure intensity for logarithmi+ Poisson model@

() =0e

() = 0

0+1

R(' |) =[

0+1

0('+)+1]1/


30/77


ailure intensity +om%arison as a fun+tion

of average failures eD%erien+ed

0

0

Average number of failures experienced

F

ailureintensity

()

Logarithmic Poisson model

Basic model


31/77


ailure intensity +om%arison as fun+tion

of eDe+ution time

0

0

Execution time

Failureintensity

()

Basic model

Logarithmic Poisson model


32/77


,hi+h Model to use-

Uniform operational profile: Use the basic model

Non-uniform operational profile: Use the logarithmic Poisson

model


33/77


7ther issues

Counting failures

When is a defect repaired

Impact of imperfect repair


34/77


5nde%endent +he+k against +ode +overage

Reliability

estimate

Code coverage

CL CH

RL

RH

CL RL Unreliable estimate

CL RH Unreliable estimate

CH RL Reliable estimate

CH RH Reliable estimate


35/77


7%erational Profile

A quantitative characterization of how an application will be used. Thischaracterization requires a knowledge of input variables.

Input stateis a vector of values of all input variables.

Input variables:An interrupt is an input variable and so are all

environment variables and variables whose values are input by the user

via the keyboard or from a file in response to a prompt.

Internal variables, computed from one or more input

variables are not input variables.

Intermediate results and interrupts generated during the execution as a

result of the execution should not be considered as input variables.


36/77


7%erational Profile G+ontd.H

Runs of an application that begin with identical input states belong to the same runtype.

Example 1: Two withdrawals from the same person from the same

account and of the same dollar amount.

Example 2: Reservations made for two different people on the same

flight belong to different run types.

Function: Grouping of different run types. A function is conceived at

the time of requirements analysis.


37/77


7%erational Profile G+ontd.H

Function: A set of different run types. A function is conceived at the

time of requirements analysis. A function is analogous to a use-case.

Operation: A set of run types for the application that is built.


38/77


5n%ut S%a+e@ #ra%hi+al >iew

Input space

Inputstate

Inputstate

Inputstate

Function 1

Inputstate

Inputstate

Inputstate

Function 2

Function 3

Function 4Function k


39/77


un+tional Profile

un+tion Probability of o++urren+e

1 =.!

2 =./?

/ =.=?


40/77


7%erational Profile

un+tion 7%eration Probability of o++urren+e

1 711 =.0

712 =.1

71/ =.1

2 721 =.=?

722 =.1?

/ 7/1 =.1?

7// =.=?


41/77


Modes and 7%erational Profile

Mode un+tion 7%eration Probability of o++urren+e

(ormal 1 711 =.0

712 =.1

71/ =.1

2 721 =.=?

722 =.1?

/ 7/1 =.1?

7// =.=?


42/77


Modes and 7%erational Profile G+ontd.H

Mode un+tion 7%eration Probability of o++urren+e

Administrative A1 A711 =.0

A712 =.1

A2 A721 =.?


43/77


Reliability


44/77


Risk Assessment

Risk is a $om"ination of to fa$tors: Probability of malfunction

Consequence of malfunction

3#nami$ $omp!eit# and $oup!ing metri$s $an "e used to a$$ount for thepro"a"i!it# of a fau!t manifesting itse!f into a fai!ure&

Risk Assessment is usefu! in identif#ing: 5omp!e modu!es that need more attention

/otentia! trou"!e spots

7stimating test effort


45/77


Nuestion of interest

Gi%en the ar$hite$ture of an app!i$ation, ho does one 8uantif# the riskasso$iated ith the gi%en ar$hite$ture

Note that risk ana!#sis, as des$ri"ed here, $an "e

performed prior to the de%e!opment of an# $ode and

soon after the s#stem ar$hite$ture, in terms of its$omponents and $onne$tions, is a%ai!a"!e&


46/77


Risk Assessment Pro+edure

3e%e!op S#stem Ar$hite$ture

3etermine $omponent and $onne$tor $omp!eit#

3e%e!op operationa! s$enarios and their !ike!ihood

3e%e!op risk fa$tors

/erform se%erit# ana!#sis

/erform risk ana!#sis

3e%e!op 53G


47/77


$ardia+ Pa+emaker@ 8ehavior Modes

L1

What is paced?

A: Atrium

V: Ventricle

D: Dual; (both)

Behavior mode indicated by 3-letter acronym: L1L

2L

3

L2

Which chamber is

being monitored ?

A: Atrium

V: Ventricle

D: Dual; (both)

L3

What is the mode type?

I: Inhibited

T: Triggered

D: Dual pacing

Example: VVI: Ventricle is paced when Ventricular sense does

not occur, pace is Inhibited if a sense does occur


48/77


Pa+emaker@ $om%onents and $ommuni+ation

Reed

SwitchCommunication

Gnome

Coil

DriverAtrial

ModelVentricular

Model

enables

enables

heart

magnet

programming


49/77


$om%onent es+ri%tion

Reed Swit+h IRSJ@Magneti+ally a+tivated swit+hL must be

+losed before %rogramming +an begin.

$oil river I$J@Pulsed to send =Os and 1Os by the %rogrammer.

Atrial Model IARJ@$ontrols heart %a+ing.

$ommuni+ations #nome I$#J@Re+eives +ommands as bytes

from $ and send to AR and >T.

>entri+ular Model I>TJ@$ontrols sensing and the refra+tory

%eriod.


50/77


S+enarios

Programming@ Programmer sets the o%eration mode of the devi+e.

A>5@>T monitors the heart. ,hen a heart beat is not sensed the

AR %a+es the heart and a refra+tory %eriod is in effe+t.

AA5@ The AR +om%onent %a+es the heart when it does not sense any %ulse.

>>5@>T +om%onent %a+es the heart when it does not sense any

%ulse.

>>T@>T +om%onent +ontinuously %a+es the heart.

AAT@The AR +om%onent +ontinuously %a+es the heart.


51/77


Stati+ $om%leDity for 77 esigns

$ou%ling 8etween $lasses I$8$J@Total number of other

+lasses to whi+h a +lass is +ou%led.

$ou%ling@ Two +lasses are +onsidered +ou%led if methods fromone +lass use methods or instan+e variables from other +lass.


52/77


7%erational $om%leDity for State+harts

ynami+ +om%leDity fa+tor for ea+h +om%onent is based on +y+lomati++om%leDity of the state+hart s%e+ifi+ation for ea+h +om%onent.

#iven a %rogram gra%h # with e edges and n nodes'

the +y+lomati+ +om%leDity >I#JBenF2.

or ea+h eDe+ution s+enario Ska subset of the state+hart s%e+ifi+ation ofthe +om%onent is eDe+uted thereby eDer+ising state entries' state eDits'

and fired transitions.

The +y+lomati+ +om%leDity of the eDe+uted %ath for ea+h +om%onent3i is +alled the operational comple*ity denoted bycp*k -3i/.


53/77


ealing with $om%osite States

init t11

s11

Is1

init

s22

Is2

s21

t13

VGx (s11)

+VGa (t11)

+VGx (s1)

+VGa (t12)

+

VGe(s1) +VG

a(t13)+VGe(s22)

Cyclomatic complexity for the s11 to s22 transition:

VGp: p: x, a, e: Complexity of the exit, action, and entry code segments code segment

t12


54/77


ynami+ $om%leDity for State+harts

or ea+h eDe+ution s+enario these variables are u%dated with the

+om%leDity measure of the thread that is triggered for that %arti+ulars+enario.


55/77


$om%onent $om%leDity Seuen+e diagrams are develo%ed fo ea+h s+enario.


56/77



57/77


$onne+tor $om%leDity

Simulation is used to determine the dynami+ +ou%ling measurefor ea+h +onne+tor.

$ou%ling amongst +om%onents is re%resented in the form of amatriD.

$ou%ling values are normalied to the highest +ou%ling.


58/77


$om%onent $om%leDity >alues

RS $ $# AR >T

Programming I=.=1J 9./ !:.0 20./

A>5 I=.23J ?/.2 0!.9

AAT I=.1?J 1==

AA5 I=.2=J 1==

>>5 I=.1?J 1==

>>T I=.2=J 1==

of ar+hite+ture+om%leDity

.=9/ .!:0 .20/ ?=.209 09.?:2

(ormalied .=.==2 =.=1/ =.==? 1 =.3!/


59/77


$ou%ling MatriD

RS $ $# AR >T Prog. eart

RS .==10 .==10

$ .==/ .=11

$# .==2 .==10 .==10

AR .2? 1

>T .2: .9:/

Programmer .==10 .==!

eart .12/ ./=:


60/77


Severity Analysis

Risk fa+tors are asso+iated with ea+h +om%onent and +onne+tor

by %erforming severity analysis. 8asi+ failure modeIsJ of ea+h +om%onent&+onne+tor and its effe+t on the

overall system is studied using failure mode and effe+ts analysis IM


61/77


Severity Ranking

$atastro%hi+ I=.3?J@ailure may +ause death or total system loss.

$riti+al I=.:?J@ailure may +ause severe in*ury' %ro%ertydamage' system damage' or loss loss of %rodu+tion.

Marginal I=.?J@ailure may +ause minor in*ury' %ro%erty damage' system damage'delay or loss of %rodu+tion.

Minor I=.2?J@ ailure not serious enough to +ause in*ury' %ro%erty damage'or system damage but will result in uns+heduled maintenan+e or re%air.

omain eD%erts assign severity indi+es Isvrty iJ to the severity

+lasses.


62/77


euristi+ Risk a+tor

The highest severity indeD IsvrtyiJ +orres%onding to a severity level offailure of a given +om%onent i' is assigned as its severity value.

8y +om%aring the result of the simulation with the eD%e+tedo%eration' severity level for ea+h faulty +om%onent for a givens+enario is determined.

A euristi+ Risk a+tor IhrfiJ is then +om%uted for ea+h

+om%onent based on its +om%leDity and severity value.

iii svrtycpxhrf =


63/77


M


64/77


MT Send in+orre+t+ommand Ie.g.To7ffinstead ofTo5dleJ

5n+orre+tinter%retationof %rogrambytes

5n+orre+to%eration modeand %a+ing of theheart. evi+e stillmonitored by the

%hysi+ian'immediatemaintenan+ereuired.

Marginal


65/77


$om%onent Risk fa+tors@ Using ynami+ $om%leDity

RS $ $# AR >T

ynami+ +om%leDity .==2 .=1/ .==? 1 .3!/

Severity .2? .2? .? .3? .3?

Risk fa+tor .===? .==/2? .==2? .3? .3109?


66/77


$onne+tor Risk fa+tors@ Using ynami+ $om%leDity

RS $ $# AR >T Prog. eart

RS .===/? .===/?

$ .===:? .==2:?$# .===? .===: .===:

AR .2375 .95

>T .2565 .82935

Prog. .===/? .==1?

eart .11!9? .2916


67/77


$om%onent Risk fa+tors@ Using Stati+ $om%leDity

RS $ $# AR >T

$8$ =.0: =.9 1 =.! =.!

Severity =.2? =.2? =.? =.3? =.3?

Risk fa+tors based on$87

=.113 =.2 =.? =.?: =.?:


68/77


$om%onent Risk fa+tors@ $om%arison

Dynamic metrics better distinguish AR and VT components as highrisk when compared with RS, CD, and CG.

Using static metrics, CG is considered at the same risk level as AR andVT.

In pacemaker, AR and VT control the heart and hence are the highestrisk components which is confirmed when one computes the riskfactors using dynamic metrics.


69/77


$om%onent e%enden+y #ra%hs I$#sJ

A CDG is described by setsNandEwhereN is a set of of nodes and E is a setof edges. sand tare designated as the start and termination nodes and belong toN.

Each node n inN: , where Ci is the componentcorresponding ton,RCiis the reliability of Ci, andECi is the average

execution time of Ci .

Each edge einE: , where Tij is the transition from node

CitoCj,RTij is the reliability of this transition, and PTi jis the transition

probability.

In the methodology described here, risk factorsreplace the reliabilitiesof components and transitions.


70/77


#eneration of $#s

Estimate the execution probability of each scenario.

For each scenario estimate the execution time of each component and

then, using the probability of each scenario, compute the averageexecution time of each component.

Estimate the transition probability of each transition.

Estimate the complexity factor of each component.

Estimate the complexity factor of each connector..


71/77


$# for the Pa+emaker Iall transition labels not shownJ

t

s


72/77


Reliability Risk Analysis

Ar+hite+ture risk fa+tor is obtained by aggregating the risk fa+tors of individual+om%onents and +onne+tors.


73/77


Risk Analysis Algorithm7R %aths

8readth eD%ansions +orres%ond to Q7R %aths. The risk fa+tors

asso+iated with all nodes along the breadth eD%ansion aresummed u% weighted by the transition %robabilities.

e1:

e2:

n1:

n2:

RB1GI1=.?J=./FI1=.!J=.:H.

s

n1 n2

e1 e2

Traverse the $# starting at node sand sto% until either tisrea+hed or the average a%%li+ation eDe+ution time is +onsumed.


74/77


Risk Analysis AlgorithmA( %aths

The de%th of a %ath im%lies seuential eDe+ution. or eDam%le'su%%ose that node n.is rea+hed from node svia edge e.andthat node n4is rea+hed from n.via edgee4. Attributes of theedges and +om%onents are as follows@

e1:

e2:

n1:

n2:

RB1GI1=.?J=./ D I1=.!J=.:H. TimeBTimeF? F12

s

n1

n2

e1

e2

The QA( %aths take into +onsideration the +onne+tor risk fa+tors Ihrfi*J


75/77


Pa+emaker Risk

#iven the ar+hite+ture and the risk fa+tors asso+iated with+om%onents and +onne+tor' the risk fa+tor asso+iated with the%a+emaker is +om%uted to be a%%roD. =.3.

This value of risk is +onsidered high. 5t im%lies that the%a+emaker ar+hite+ture is +riti+al and failures are likely to be+atastro%hi+.

Risk analysis tells us that the >T and AR +om%onents are the highest risk +om%onents

Risk analysis also tells us that the +onne+tors between >T' AR and heart +om%onents are the highest risk+om%onents


76/77


Advantages of Risk Analysis

The $# is useful for the risk analysis of hierar+hi+al systems.Risks for subsystems +an be +om%uted. These +ould then beaggregated to +om%ute then risk of the entire system.

The $# is useful for %erforming sensitivity analysis. 7ne +ouldstudy the im%a+t of +hanging the risk fa+tor of a +om%onent onthe risk asso+iated with the entire system.

As the analysis is being done' most likely' %rior to +oding' one might+onsider revising the ar+hite+ture or use the same ar+hite+ture butallo+ate resour+es for +oding and testing based on individual riskfa+tors.


77/77

Summary

Reliability' modeling un+ertainty' failure intensity' o%erational%rofile' reliability growth models' %arameter estimation.

Risk assessment' ar+hite+ture' severity analysis' risk fa+tors'$#s.

day 5 reliability

Documents