dependable cloud architecture - swocc edition
DESCRIPTION
TRANSCRIPT
“Failure is alwaysan option.”
Image: Discovery Channel, Fair Use
Protection From:
What are we looking for?
Check out: http://bit.ly/wazbizcontImages: Office ClipArt & Godzilla Releasing Corp (Fair Use)
Hardware Failure Data Corruption Network Failure Loss of Facilities
Image: FOX, Fair Use
Human Error
What we’re trying to achieve
1. Monitoring2. Resilient Solutions
Image: Cohdra
Image: Office ClipArt
Cost vs Risk
99.999% $1, … ,000.00
To get more 9’s here add more 0’s here.
Image: NASA
Monitoring
Functional Transparency
Image: Office ClipArt
Logging Messages
Hardware Health
Dependent Services Health
Telemetry
Image: NASA
Analyze your Data
ResilienceImage: Office ClipArt
Remember: Failure is always an option.
Common Points of Failure• Machine\application crashes• Throttling (exceeding capacity)• Connectivity\Network• External service dependencies
Focus less on the uptime of hardware and more about how the solution handles it WHEN
something fails!
Try/catch != Resilient
private void createFile() {
string fileName = @"c:\workingDirectory\someFileName.txt";
try {
File.Create(fileName);}catch (DirectoryNotFoundException ex)
{Trace.WriteLine(String.Format("Unable to create {0}. {1}",
fileName, ex));
throw; } } }
Image: Michael Wood
Decompose your system…
Capacity BufferingContent Delivery Networks (CDN’s)
Distributed Application Cache
Local Content Cache
Enables recovery during outages or
spikes in load
Image: jepler
Always carry a spare75% Capacity, half of our load 75% Capacity, half of our load
50% more capacity then needed• Can absorb of temporary spikes• Time to react if need to add capacity
100% of load, 150% Capacity0% Capacity, redirect all load
Over allocated, but still functioning• Degrade, but don’t fail
SYSTEM FAILURE!!!
Image: Kevin Rosseel
Request Buffering
Image: Joe Shlabotnik
QueuesRetry PoliciesAsync Workloads
Dept. of Redundancy Dept.
Have a backup, somewhere elseMore than one? Cost to benefit Ratio?
Ready StateHot = full capacityWarm = scaled down, but ready to growCold = mothballed, starts from zero
Image: Mr. White
Redundancy - Its about probability
95% uptime 95% uptime 95% uptime 95% uptime
1 box : 5% downtime or 438hrs per year
2 boxes : 5/100 * 5/100 = 25/10,000 = 0.25% downtime or 22hrs per year
4 boxes : 5/100 * 5/100 * 5/100 * 5/100 = 625/100,000,0000.000625% downtime or 3.285 MINUTES per year
(that’s 18 ½ days!)
Total Outage duration =
Time to Detect+ Time to Diagnose+ Time to Decide+ Time to ActImage: Office ClipArt
Dynamic Addressing & Configuration
What about your data?
Image: barrymieny
Availability via Degradation
Image: Michael Wood
Images: Gizmodo
Virtualization and Automation
Images: Orion Pictures owns Terminator Franchise
The “HI” Point
Check out: http://bit.ly/wazinternalsImages: Office Clip Art
Image: NASA
“Don't be too proud of this technological terror you've constructed…”
ADMIT:• Your Solution WILL fail at some point• You can learn from others just as
well as yourself
DO:• Root cause analysis• Read other root cause analysis• Plan for failure
DON’T:• Get cocky• Stick your head in the sand
Images: LucasFilm, Fair Use
