2. Fault Tolerance

2

Fault - Error - Failure

Fault = physical defect or flow occurring in some component (hardware or software)

Error = incorrect behavior caused by a fault• – manifestation of fault

Failure = inability of the system to perform its specified service

Latent Fault = which has not yet produced error Latent Error = which has not yet produced failure

3

Fault - Error - Failure

Note: presents of fault does not ensure that error will occur, e.g. memory stuck-at-0

4

Origin of Defects in Objects (HW/SW)

Good object wearing out with age• – Hardware (software can age too)• – Incorrect maintenance/operation

Good object, unforeseen hostile environment• – Environmental fault

Marginal object: occasionally fails in target environment

• – Tight design/bad inputs Implementation mistakes Specification mistakes

Note: From Top to Down-> Increasing human responsibility

5

Bathtub Curve

Three phases of system lifetime• – Infant mortality• – Normal lifetime• – Wear-out period

6

Life-time of a Software system

7

Faults Characteristics

1- Cause– Specification errors

very dangerousgeneric fault

– Implementation errors• very hard to formally verify

– Random component faults• random, not manufacturing defects

– External disturbance• noise, EMP, vibration, radiation• much like random component

8

Faults Characteristics

2- Origin– software or hardware

• • Physical device level (HW)• • Logic level (HW)• • Chip level (HW)• • System level (HW/SW)

– interfacing, specifications, …

– don’t care, except: • hardware can be analog

• indeterminate voltage level

9

Faults Characteristics

3- Duration– Permanent fault

• occurs and doesn’t go away• easiest to diagnose

– Transient fault• occurs once and disappears• 10 times as expected as permanent fault

– Intermittent fault• occurs occasionally• may appear to be transient (if long period)• hard and expensive to detect

10

Faults Characteristics

4- Extent– Global

• A power supply fault

– Local• A memory fault

5- Value– Determinate

• memory stuck-at-0

– Indeterminate• A sensitive fault to data or time

11

What to do about Faults

Finding & identifying faults:• • Fault detection: is a fault there?• • Fault location: where?• • Fault diagnosis: which fault it is?

Automatic handling of faults• • Fault containment: blocking error flow

– Fault masking: fault has no effect• • Fault recovery: back to correct operation

12

System Response to faults

Error on output: may be acceptable in non-critical systems if happens only rarely

Fault masking: output correct even when fault from a specific class occurs

• – Critical applications: air/space/manufacturing

Fault-secure: output correct or error indication• – Retryable: banking, telephony

Fail safe: output correct or in safe state• – Flashing red traffic light, disabled ATM

13

What is Fault-Tolerance?

A fault-tolerant system is one that continues to perform at desired level of service according to their specification, in the presence of faults.

There are no failures in a fault-tolerant system.

Fault-tolerance is the ability of a system to provide a service complying with the specification in spite of faults.

A better title might have been Dependable or Reliable or Available computing

14

Fault Tolerance In the physical universe:

• - Fault detection• - Fault location• - Fault containment• - Fault recovery• - Continue servicing

In the informational universe:• - Error detection• - Error location• - Error containment• - Error recovery• - Continue servicing

15

Fault Recovery

How quickly is the fault detected? How soon can recovery begin?

• – Does is require human intervention• – How is the system admin notified?

How long does recovery take?• – Restore from backup?• – Purchase new HW?

16

Fault Coverage (C) Measure of system’s ability to perform:

• – fault detection• – fault location• – fault containment• – (and/or fault recovery)

C = P (fault detection | fault occurrence) C = P (fault recovery | fault occurrence) Note:

• – recovery implies that the system as a whole is operational• – this does not imply that a repair occurred• – e.g. duplex system with benign fault can recover to

continue operation on one non-faulty processor

17

Design Philosophies to Combat Faults

Fault avoidance (off-line)• Attempts to prevent fault in the:

• Design review• Component selecting• Quality control• Shielding• Testing

Fault masking (on-line)• Attempts to prevent a fault in a system from introducing

errors• Error correcting memory• Majority voting

Fault tolerance (on-line)• Attempts to provide a system to continue performing its

expected tasks after the occurrence of faults

18

Design Philosophies to Combat Faults

Fault avoidance Fault masking Fault tolerance

19

Fault Avoidance vs. Tolerance

Fault avoidance: eliminate problem sources• – Remove defects: Testing and debugging• – Robust design: reduce probability of defects• – Minimize environmental stress: Radiation shielding etc• – Impossible to avoid faults completely

Fault tolerance: add redundancy to mask effect• – Additional resources needed (more later)• – Examples:

• Error correction coding

• Backup storage

• Spare tire etc

20

Fault Forecasting vs. Tolerance Fault Tolerance

• Execution-time techniques that handle with the effects of faults

Fault Forecasting• Estimate current number, future incidence and likely

consequences• You can’t tolerate what you don’t expect• But if we expected it, we would avoid or eliminate the fault!• In general: We can itemize the classes of faults that can

occur• We can define what we want done if the fault occurs and if

the error is detected• Example: Automobile tire

• Lose air• Do not expect it to experience electrical overload

21

Fault Tolerant computing

Deterministic approaches• – Based on simplifying assumptions: “fault model”• – Obtain methods using the models: test generation• – Evaluation of effectiveness• – Used for Testing & combinatorial fault-tolerance

Probabilistic approaches• – We can’t predict exactly when a person will die, but

we can still get “life expectancy = 77.2”, if we have data• – Used for evaluating, achieving and optimizing

reliability• – Random testing

22

Fault Tolerant vs. Performance There are many Fault-tolerance approaches that

sacrifice performance to tolerate faults Ex. 1:

• – Periodically stop the system and checkpoint its state to disk.

• * If fault occurs, recover state from checkpoint and resume Ex. 2:

• – Log all changes made to system state in case recovery is needed

• * During recovery, undo the changes from the log Ex. 3:

• – Run two identical systems in parallel, compare their results before using them

Ex. 4:• – Run software with lots of error checking

23

Fault Tolerant vs. Cost

There are many Fault-tolerance approaches that sacrifice cost to tolerate faults

Ex. 1:• – Replicate the hardware 3 times and vote to determine

correct output

Ex. 2:• – Mirror the disks (RAID-1) to tolerate disk failures

Ex. 3:• – Use multiple independent versions of software to

tolerate bugs (Called N-version programming)

24

Fault Tolerant vs. Power There are many Fault-tolerance approaches

that sacrifice power to tolerate faults Ex. 1, 2 & 3 (same as previous slide)

• – Replicate the hardware 3 times and vote to determine correct output

• – Mirror the disks (RAID-1) to tolerate disk failures• – Use multiple independent versions of software to

tolerate bugs Ex. 4

• – Add continuously running checking hardware to system

Ex. 5• – Add extra code to check for software faults

25

Need for Fault Tolerance: Universal

Natural objects:• • Fat deposits in body: survival in starvation• • Duplication of eyes: graceful degradation

upon failure

Man-made objects• • Redundancy in ordinary text• • Asking for password twice during initial set-

up• • Duplicate tires in trucks

26

Forms of Redundancy

Hardware redundancy• – add extra hardware for detection or tolerating

faults Software redundancy

• – add extra software for detection and possibly tolerating faults

Information redundancy• – extra information, i.e. codes

Time redundancy• – extra time for performing tasks for fault

tolerance

27

Redundancy base

Time

Try Retry Retry

Space

Try

Try

Try