high availability survivable networks

Wayne D. Grover, Anthony Sack

9 October 2007

High Availability Survivable Networks:When is Reducing MTTR Better than Adding

Protection Capacity?

7-10 October 2007La Rochelle, France

Presented at:

High Availability Survivable Networks: When is Reducing MTTR Better than Adding Protection Capacity? DRCN 2007 2

Segue….back to DRCN 2005

Beautiful Ischia.

Question for Dr. Grover: (paraphrasing)

“In a network that is already designed for single failure restorability, to get yet higher availability of services, would you think it is better to add still more spare capacity to increase the dual-failure restorability or to invest at that point in MTRR reduction to enhance availability?”

GOOD QUESTION ! …and it lead to this study..

..the closing Panel Discussion.

High Availability Survivable Networks: When is Reducing MTTR Better than Adding Protection Capacity? DRCN 2007 3

Why this question is so insightful…

To minimize the impact of dual-failure events, we can:

Reduce physical MTTR values for network spans

physical repairs will happen faster

Time spent in an overlapping dual failure repair state will go down as MTTR-2

As MTTR ->0 there are no dual failures

Increase the network restorability to dual span failures By adding more spare

capacity

fewer dual-failure pairs will be outage causing

R2 = 1 means a triple failure will be needed to cause outage!

Which is best approach? Is there an optimal investment strategy?

High Availability Survivable Networks: When is Reducing MTTR Better than Adding Protection Capacity? DRCN 2007 4

In a survivable network, MTTR takes on new importance…

Key point is that in an R1=1 survivable network, outage requires two failures which interact in the network (restoration-wise) with repair processes that overlap in time

Otherwise the two failures are simply time-successive single failures.

This means that:

In a survivable network, unavailability drops as the square of the MTTR !

High Availability Survivable Networks: When is Reducing MTTR Better than Adding Protection Capacity? DRCN 2007 5

Illustrating the principle

TTR_Failure_1 TTR_Failure_2

No risk of outage(R1=1)

R2<1: risk of outage. Duration proportional to repair overlap time. Reduces as 2( )O MTTR

Decreasing time to repair

Increasing time to repair

High Availability Survivable Networks: When is Reducing MTTR Better than Adding Protection Capacity? DRCN 2007 6

Study mandate and design

Explore the trade-off between availability improvement through R2 enhancement and MTTR reduction.

What is the most cost-effective strategy for combined investment in capacity additions and repair time

improvements to maximize availability?

Framework: Total Availability Investment

100% to Dual-failure Restorability (R2)

100% to Physical MTTR Reduction

Interesting because the response of both variables to increasing investment is not linear.

High Availability Survivable Networks: When is Reducing MTTR Better than Adding Protection Capacity? DRCN 2007 7

“R1” is the level of single-failure restorability (range 0 to 1)

“R2” is the level of dual-failure restorability (range 0 to 1 as well)

Examples:

R1 = 1 indicates a network fully restorable to all single failures;

R2 = 0.60 means that 60% of failed working capacity units (or service paths) are restorable to dual failures, on average.

Some Terminology

High Availability Survivable Networks: When is Reducing MTTR Better than Adding Protection Capacity? DRCN 2007 8

Typical capacity cost profile of enhancing R2

Any network designed for100% R1 will always havea non-zero R2 level as well.

This is true even if the R1network design is optimal.

R2 vs. cost curve thenasymptotically approachesunity – always a diminishingreturn to further capacity investment.

This characteristic curve shape is well-known in the literature. Exact shapes for this curve have been found for different networks in, for example, the Ph.D. thesis by Clouquer

1.0

R2inherent

R2

Cost for restorabilityCR1 CR2

0.0

1.0

R2inherent

R2

Cost for restorabilityCR1 CR2

0.0

High Availability Survivable Networks: When is Reducing MTTR Better than Adding Protection Capacity? DRCN 2007 9

Properties of MTTR versus expenditure

Shape of the MTTR vs. costcurve is much less certain, but a plausible parametricmodel seems defensible.

For example, initialinvestments lead to largeMTTR reductions, withdiminishing returns thereafter.

Conceivably,however, this curve could alsobe convex – with initial investments leading to only small reductions, and larger investments required for larger changes.

Both scenarios will be tested in our experimental calculations.

0.0

MTTR

Cost for MTTR reduction0.0

0.0

MTTR

Cost for MTTR reduction0.0

High Availability Survivable Networks: When is Reducing MTTR Better than Adding Protection Capacity? DRCN 2007 10

Theoretical model development

Consider the availability of an individualreference path in an R1 survivable network:(for conceptual investigation, all spans taken as identical;S: number of spans in the network, N: number of spans on the path)

21 2 ( ) ( 1) (1 2)path spanA N S N N N U R

(One failureon the path)

(Both failures on the path)

Number of dual-failure scenarios that may cause

service path outage

Probability of any dual-failure event actually occurring

dual-span failure restorability of the

network

• This expression excludes contributions due to triple (or higher-order) independent failure scenarios.

High Availability Survivable Networks: When is Reducing MTTR Better than Adding Protection Capacity? DRCN 2007 11

Relation of unavailability to MTTR

We can now express the span unavailability in terms of MTTR:

/( )spanU MTTR MTTR MTTF Well-known expression:

spanU MTTR Approximation:(where λ is the failure rate)

2 22 ( 1) ( ) (1 2)pathU N S N MTTR R

Also: Switch to an unavailability orientation

Algebraically simplify “number of scenarios”

term from before

&

High Availability Survivable Networks: When is Reducing MTTR Better than Adding Protection Capacity? DRCN 2007 12

Unavailability as a function of MTTR and R2

We now have the operative expression that relates unavailability to both MTTR and R2:

2 22 ( 1) ( ) (1 2)pathU N S N MTTR R

Note that unavailability responds in a linear way to R2, but to the square of the MTTR or failure rate (λ).

Could availability improvements be most optimally gained through MTTR improvements or some type of

combined strategy with R2 enhancement?

High Availability Survivable Networks: When is Reducing MTTR Better than Adding Protection Capacity? DRCN 2007 13

MTTR and R2 as functions of cost

To study the economic tradeoff, we define MTTR and R2 as functions of cost:

2 ( )r rR f C( )m mMTTR f C

222 ( 1) ( ) 1 ( )path m m r rU N S N f C f C

If Cm + Cr is a constant total budget amount, then an optimum split of total investment must exist which minimizes Upath.

What is this optimum split for each of the two MTTR characteristic curve shapes postulated earlier?

High Availability Survivable Networks: When is Reducing MTTR Better than Adding Protection Capacity? DRCN 2007 14

Experimental calculations

For experimental calculations,we set a numerical value of 100“arbitrary cost units” to be dividedbetween MTTR and R2.(i.e. total investment remains constant,only the allocation changes)

Data points from the characteristic curves for both variables were used in the equation just presented to generate new curves for the unavailability of a typical reference path.

Other assumed parameters:N = 6 (length of the reference path)S = 20 (number of spans in the network)λ = 0.0005 (failure rate per span)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

100806040200

% of Total Budget Spent on R2 Enhancement

Av

era

ge

R2

(i,j)

Ac

hie

ve

d  

% of Total Budget Spent on R2 Enhancement

Ave

rag

e R

2(i,j

) A

chie

ved

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

100806040200

% of Total Budget Spent on R2 Enhancement

Av

era

ge

R2

(i,j)

Ac

hie

ve

d  

% of Total Budget Spent on R2 Enhancement

Ave

rag

e R

2(i,j

) A

chie

ved

N, S, λ

222 ( 1) ( ) 1 ( )path m m r rU N S N f C f C

0.0000

0.0005

0.0010

0.0015

0.0020

0.0025

0 20 40 60 80 100

% of Total Budget Spent on MTTR Reduction (Balance goes to R2 Enhancement)

Re

fere

nc

e P

ath

Un

av

aila

bili

ty  

% of Total Budget Spent on MTTR Reduction(Balance goes to R2 Enhancement)

Re

fere

nce

Pat

h U

na

vaila

bili

ty

0.0000

0.0005

0.0010

0.0015

0.0020

0.0025

0 20 40 60 80 100

% of Total Budget Spent on MTTR Reduction (Balance goes to R2 Enhancement)

Re

fere

nc

e P

ath

Un

av

aila

bili

ty  

% of Total Budget Spent on MTTR Reduction(Balance goes to R2 Enhancement)

Re

fere

nce

Pat

h U

na

vaila

bili

ty

0.0

4.0

8.0

12.0

16.0

20.0

24.0

28.0

32.0

36.0

40.0

0 20 40 60 80 100

% of Total Budget Spent on MTTR Reduction

MT

TR

(h

ou

rs)

% of Total Budget Spent on MTTR Reduction

MT

TR

(h

our

s)

0.0

4.0

8.0

12.0

16.0

20.0

24.0

28.0

32.0

36.0

40.0

0 20 40 60 80 100

% of Total Budget Spent on MTTR Reduction

MT

TR

(h

ou

rs)

% of Total Budget Spent on MTTR Reduction

MT

TR

(h

our

s)

Test one: concave MTTRTest two: convex MTTR

High Availability Survivable Networks: When is Reducing MTTR Better than Adding Protection Capacity? DRCN 2007 15

Experimental R2 curve (both tests)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

100806040200

% of Total Budget Spent on R2 Enhancement

Av

era

ge

R2

(i,j)

Ac

hie

ve

d  

% of Total Budget Spent on R2 Enhancement

Ave

rag

e R

2(i

,j) A

chie

ved

High Availability Survivable Networks: When is Reducing MTTR Better than Adding Protection Capacity? DRCN 2007 16

Experimental MTTR curve – concave (test one)

0.0

4.0

8.0

12.0

16.0

20.0

24.0

28.0

32.0

36.0

40.0

0 20 40 60 80 100

% of Total Budget Spent on MTTR Reduction

MT

TR

(h

ou

rs)

% of Total Budget Spent on MTTR Reduction

MT

TR

(h

ours

)

High Availability Survivable Networks: When is Reducing MTTR Better than Adding Protection Capacity? DRCN 2007 17

0.0000

0.0005

0.0010

0.0015

0.0020

0.0025

0 20 40 60 80 100

% of Total Budget Spent on MTTR Reduction (Balance goes to R2 Enhancement)

Re

fere

nc

e P

ath

Un

av

aila

bili

ty  

% of Total Budget Spent on MTTR Reduction(Balance goes to R2 Enhancement)

Re

fere

nce

Pat

h U

nav

aila

bilit

y

Test one result

Region 2 Region 3

Region 1

High Availability Survivable Networks: When is Reducing MTTR Better than Adding Protection Capacity? DRCN 2007 18

Test one: discussion

Three distinct regions evident.

In Region 1 Availability greatlybenefits from relatively easilyobtained initial reductions in MTTR.

In Region 3 MTTR reduction isa matter of diminishing returns. It would have been better to add more capacity with the same money, to enhance R2.

In Region 2, the overall availability is lowest and not very sensitive to exactly how the budget is spent on R2, or MTTR.

0.0000

0.0005

0.0010

0.0015

0.0020

0.0025

0 20 40 60 80 100

% of Total Budget Spent on MTTR Reduction (Balance goes to R2 Enhancement)

Re

fere

nc

e P

ath

Un

av

aila

bili

ty  

% of Total Budget Spent on MTTR Reduction(Balance goes to R2 Enhancement)

Ref

eren

ce P

ath

Una

vaila

bilit

y

0.0000

0.0005

0.0010

0.0015

0.0020

0.0025

0 20 40 60 80 100

% of Total Budget Spent on MTTR Reduction (Balance goes to R2 Enhancement)

Re

fere

nc

e P

ath

Un

av

aila

bili

ty  

% of Total Budget Spent on MTTR Reduction(Balance goes to R2 Enhancement)

Ref

eren

ce P

ath

Una

vaila

bilit

y

Region 2 Region 3

Region 1

Region 2Region 2 Region 3Region 3

Region 1Region 1

High Availability Survivable Networks: When is Reducing MTTR Better than Adding Protection Capacity? DRCN 2007 19

Experimental MTTR curve – convex (test two)

0.0

4.0

8.0

12.0

16.0

20.0

24.0

28.0

32.0

36.0

40.0

0 20 40 60 80 100

% of Total Budget Spent on MTTR Reduction

MT

TR

(h

ou

rs)

% of Total Budget Spent on MTTR Reduction

MT

TR

(h

ours

)

High Availability Survivable Networks: When is Reducing MTTR Better than Adding Protection Capacity? DRCN 2007 20

Test two result

0.0000

0.0005

0.0010

0.0015

0.0020

0.0025

0.0030

0.0035

0.0040

0 20 40 60 80 100

% of Total Budget Spent on MTTR Reduction (Balance goes to R2 Enhancement)

Re

fere

nc

e P

ath

Un

av

aila

bili

ty  

Re

fere

nce

Pat

h U

nav

aila

bilit

y

% of Total Budget Spent on MTTR Reduction(Balance goes to R2 Enhancement)

High Availability Survivable Networks: When is Reducing MTTR Better than Adding Protection Capacity? DRCN 2007 21

Test two discussion

This is the curve portraying whereit is very difficult (costly) to obtain any MTTR reductions.

We think less plausible a shape,but worthwhile as a “what if” to show the range of strategythis analysis can inform an operator on.

In this case, the preferred strategy is strongly on capacity addition to enhance R2,

0.0000

0.0005

0.0010

0.0015

0.0020

0.0025

0.0030

0.0035

0.0040

0 20 40 60 80 100

% of Total Budget Spent on MTTR Reduction (Balance goes to R2 Enhancement)

Re

fere

nc

e P

ath

Un

av

aila

bili

ty  

Ref

eren

ce P

ath

Una

vaila

bili

ty

% of Total Budget Spent on MTTR Reduction(Balance goes to R2 Enhancement)

0.0000

0.0005

0.0010

0.0015

0.0020

0.0025

0.0030

0.0035

0.0040

0 20 40 60 80 100

% of Total Budget Spent on MTTR Reduction (Balance goes to R2 Enhancement)

Re

fere

nc

e P

ath

Un

av

aila

bili

ty  

Ref

eren

ce P

ath

Una

vaila

bili

ty

% of Total Budget Spent on MTTR Reduction(Balance goes to R2 Enhancement)

High Availability Survivable Networks: When is Reducing MTTR Better than Adding Protection Capacity? DRCN 2007 22

Concluding comments

Suffices to show that at least conceptually an optimum combined strategy in MTTR and R2 investment exists.

A unique / interesting phenomenon arising specifically in the context of networks that are already “R1=1” survivable by design. (Note MTTR has no special role for R1=1 design)

Once a network is R1=1, however, MTTR takes on new importance because thereafter U ~ O(MTTR-2)

Other factors to consider: MTTR improvements are probably annual expenses, manpower R2 improvement is, however, probably a capital investment. Added capacity never hurts in a network (throughput, flexibility, grown) But fast repairs will be directly appreciated by users too

High Availability Survivable Networks: When is Reducing MTTR Better than Adding Protection Capacity? DRCN 2007 23

thank you

Thank You

(And thanks again to the great question from the DRCN 2005 Panel Discussion !)

www.telus.com www.trlabs.ca www.ualberta.ca