reliability and reslience

SystemReliability and

Resilienceand stuff

Some things need to be cleared up first

http://en.wikipedia.org/wiki/Vedette_(cabaret)

tuple

//Initialize customer and invoiceInitialize(customer, invoice);

public void Initialize(Customer customer, Invoice

invoice){

customer.Name = “asdf”;invoice.Date = DateTime.Now;

}

Initialize(customer, invoice);//did something happen to customer// and/or invoice?

customer.Name =

InitNameFrom(customer, invoice);invoice.Date =

InitDateFrom(customer, invoice);

customer.Name =

GetNameFrom(customer, invoice);invoice.Date =

GetDateFrom(customer, invoice);

var results = Initialize(customer,

invoice);

customer.Name = results.Item1;invoice.Date = results.Item2;

public tuple<string, DateTime>Initialize(customer,

invoice){

return new Tuple<string, DateTime>

(“asdf”, DateTime.Now);}

public static bool TryParse(string s, out DateTime result)

or

public static tuple<bool, DateTime?>

TryParse(string s)

tuple• Avoid side effects• Avoid out parameters•multiple values without a specific type

null object

private ILogger _logger;public MyClass(ILogger logger) {

_logger = logger;}

…

if (_logger != null) {_logger.Debug(

“it worked on my machine!”);}

null checks for everyone!

forget one and…

public class NullLogger : ILogger {

public void Debug(string text) {

//do sweet nothing}

}

private ILogger _logger = new NullLogger();

public MyClass(ILogger logger) {_logger = logger;

}

…

_logger.Debug(“it worked on my machine!”);

null object• Can eliminate null checks• Simple to implement

Circuit Breaker

Retry

You

r A

pp

licati

on O

ut o

f Pro

cess

Dep

en

dency

N times

Ou

t of P

roce

ss D

ep

en

dency

N times*

Y clients

= Denial of Service Attack

Limit the # of retries

N * Ybecomes

5 * Y

Y isstill a

problem

Circuit Breaker

State Machine

On :: Off

On Offwhen not healthy

Off Onmanually

Get to softwarebefore we ask you to dance

Healthyor

Unhealthy

Ou

t of P

roce

ss D

ep

en

dency

State is independent of requestor

Ou

t of P

roce

ss D

ep

en

dency

You

r A

pp

licati

on

Has many independent external dependencies

You

r A

pp

licati

on

Can throttle itself

You

r A

pp

licati

on

Has a wait threshold

Your Application

External Dependency

Circuit Breaker

Threshold = 2Pause = 10msTimeout = 30sState = ClosedRequest

Request

Failure (i.e. HTTP 500)Failure Count = 1Pause 10ms

Request

Failure (i.e. HTTP 500)Failure Count = 2State = Open

OperationFailedException

Threshold = 2Pause = 10msTimeout = 30sState = OpenRequest

30s has not passed

CircuitBreakerOpenException

Request

30s has not passed

CircuitBreakerOpenException

System can try to

become healthyfor 30s

Your Application

External Dependency

Circuit Breaker

Threshold = 2Pause = 10msTimeout = 30sState = ½ OpenRequest

Request

Failure (i.e. HTTP 500)

Failure Count = 2State = Open

OperationFailedException

30s has passed

Your Application

External Dependency

Circuit Breaker

Threshold = 2Pause = 10msTimeout = 30sState = ½ OpenRequest

Request

Failure Count = 0State = Closed

Response

30s has passed

Response

Your Application

External Dependency

Circuit Breaker

ClosedOpen

½ Open

½ Open is like a

manual reset

PauseTimeout

Pausebetween calls

in the loop

Timeoutbefore you

can call again

Exceptions

OperationFailed:

AggregateException

CircuitBreakerOpen:

ApplicationException

Don’t Loose Exception Info

Always use InnerException(s)

Threshold = 3State = ClosedRequest

Request

Failure (i.e. HTTP 500)Request

Failure (i.e. HTTP 500)Failure Count = 2

Failure Count = 0State = Closed

Response

Response

Request?

Your Application

External Dependency

Circuit Breaker

Failure Count = 1

SegregateDependencies

circuitBreaker(“database”)

circuitBreaker(“weatherservice”)

Dependency type, endpoint svc,

endpoint

Where?

You

r A

pp

licati

on O

ut o

f Pro

cess

Dep

en

dency

Cir

cuit

Bre

ake

r

Pro

xy

Watch forInception

You

r A

pp

licati

on W

eb

Serv

iceC

ircu

it B

reake

r

Cir

cuit

Bre

ake

r

Pro

xy

Data

base

Reposi

tory

circuit breaker• retry looping• slow down attempts• good neighbour

¡Muchas gracias!

gracias

Donald Belcham@dbelcham

donald.belcham@igloocoder.com