twilio signal 2016 chaos patterns

a

CHAOS PATTERNS

BRUCE M. WONG | @BRUCE_M_WONG

LESSONS ABOUT FAILING WELL AND FAILING OFTEN

FAILURE HAPPENS


“EVERYTHING FAILS ALL THE TIME”-WERNER VOGELS, CTO, AMAZON WEB SERVICES

HTTP://THENEXTWEB.COM/2008/04/04/WERNER-VOGELS-EVERYTHING-FAILS-ALL-THE-TIME/


THE ORIGINAL CHAOS MONKEY

CREATED BY NETFLIX CLOUD ARCHITECT, GREG ORZELL - @CHAOSSIMIA 2010


HTTPS://WWW.LINKEDIN.COM/IN/GORZELL

a

A STATE OF XENAWS EC2 REBOOT, 2014


HTTP://XENBITS.XEN.ORG/XSA/ADVISORY-108.HTML

HTTP://TECHBLOG.NETFLIX.COM/2014/10/A-STATE-OF-XEN-CHAOS-MONKEY-CASSANDRA.HTML

HTTP://AWS.AMAZON.COM/BLOGS/AWS/EC2-MAINTENANCE-UPDATE/

22 COMPLETE NODE FAILURE

2700+ C* NODES, 218 REBOOTS

0 DOWNTIME


http://xenbits.xen.org/xsa/advisory-108.html

LESSON #1 : TRUST YOUR RESILIENCE


SLOW IS HARD


UNBOUND QUEUES - ELASTIC ISN’T INFINITE


SLOW IS HARD


LATENCY MONKEY


SLOW IS HARD


LATENCY TESTING 2.0 - FIT

HTTP://TECHBLOG.NETFLIX.COM/2014/10/FIT-FAILURE-INJECTION-TESTING.HTML


http://techblog.netflix.com/2014/10/fit-failure-injection-testing.html

SLOW IS HARD


SLOW IS HARD

START SLOW

•ACCOUNT LEVEL •+10MS BEFORE +100MS •+1% ERRORS BEFORE +80% ERRORS

DIAL IT UP •A -> D NOT * -> D


LESSON # 2 : FIXING ONE FAILURE MODE EXPOSES NEW ONES


WHATS SO SPECIAL ABOUT CHAOS


CHAOS IS A CHOICE

WHATS SO SPECIAL ABOUT CHAOS


OUTAGES VS CHAOS


OUTAGES VS CHAOSUncontrolled Controlled

Unpredictable Scheduled

Time to Detect: Minutes 0 Time to Detect

Time to Resolve: ???? Time to Resolve: seconds*

Analysis Time: ???? Root Cause Analysis: Intentional

MYTH OF RESILIENCE

NATION’S BUSINESS, 1977


LATENCY MONKEY


LESSON # 3 : THE CULTURE ASPECTS OF CHAOS ARE HARD



MOST ENTERPRISES HIRE PEOPLE TO FIX THINGS. NETFLIX HIRES PEOPLE TO BREAK THINGS….

…WE SHOULD EMBRACE NETFLIX'S CULTURE OF "CHAOS ENGINEERING" THROUGHOUT ORGANIZATIONS OF ALL SHAPES AND SIZES.

SEEK PROGRESS OVER PERFECTIONTWILIO LEADERSHIP PRINCIPLE


GAME DAYS - BENEFITS

•Training New Engineers

•Discover Instrumentation gaps

•New Product Launches

•Incident Management Practices


GAME DAYS - THE SETUP

•Two “on-call” teams

•Separate rooms, separate slack channels

•Master of Disaster

•Incident Commander


LEVERAGE EXISTING TESTBOTS

•Functionally test fallback code

•Early warning!

•Existing Integrations with Telemetry, PagerDuty, Slack

•Incorporate into Canary processFUTURE


RECAP

Lesson # 1 : Trust your resilience

Lesson # 2 : Fixing one failure mode exposes new ones

Lesson # 3 : The culture aspects of Chaos are HARD

Get started today!

Game Days are your friend - do them early and often

Testbots + focus on developer productivity


WHEN YOU WISH UPON A BLUE MOON


twilio signal 2016 chaos patterns

Software