![Page 1: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/1.jpg)
Keynote ✨ ✨ ✨ ✨
![Page 2: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/2.jpg)
The Paper Edition!
! " # $ %♥ ' ! " # $ %♥ ' ! " # $ %♥ ' ! "
! " # $ %♥ ' ! " # $ %♥ ' ! " # $ %♥ ' ! "
#$ %♥ ' ! "# $ %♥ ' ! "# $ %♥ ' ! " $ %
♥ ' ! "# $ %♥ ' ! "# $ % # $ %♥♥ ' ! " $
! " # $ %♥ ' ! ! "' ! "% # $ %♥ '♥ $ %♥
Agile, Lean, Rugged
![Page 3: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/3.jpg)
First
.Introductions
![Page 4: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/4.jpg)
@Randommood
Ines Sombra
![Page 5: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/5.jpg)
@adriancolyerAdrian Colyer
![Page 6: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/6.jpg)
The Rules Only 5 minutes per paperFoundation
!Frontier
A challenge!
No Cheating!
![Page 7: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/7.jpg)
A paper tour of
Agile
![Page 8: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/8.jpg)
Foundation!
![Page 9: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/9.jpg)
We disdain old software
![Page 10: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/10.jpg)
“The only systems that don’t get changed are those that are so bad nobody wants to use
them”
![Page 11: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/11.jpg)
When software gets older
![Page 12: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/12.jpg)
Design for change Embrace modularity & information hiding Stress clarity & documentation Amputate disease-ridden parts Plan for eventual replacement
Preventative medicine
![Page 13: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/13.jpg)
Frontier
![Page 14: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/14.jpg)
What do we want? We want agile Development Testing and verification Delivery
and we want agility of
operations too!
![Page 15: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/15.jpg)
Facebook Scuba
"Data lives in server’s heap
![Page 16: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/16.jpg)
The problem with state
Restarting a database clears its memory Reading 120GB of data from disk takes about 3 hours per server (8 per machine) Even with orchestrated restarts & partial queries total of ~12 hours to restart a fleet
Operationally expensive & slow!
#
![Page 17: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/17.jpg)
“When we shutdown a server for a planned upgrade, we
know that the memory state is good… so we decided to
decouple the memory’s lifetime from the process’s
lifetime“
![Page 18: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/18.jpg)
2-3 minutes per server
Fleet restarts < 1 hour now!
$
![Page 19: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/19.jpg)
A paper tour of
Lean
![Page 20: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/20.jpg)
Foundation!
![Page 21: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/21.jpg)
Which system is better?
![Page 22: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/22.jpg)
Which system is better?
![Page 23: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/23.jpg)
Single-minded pursuit of scalability
is the wrong goal
![Page 24: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/24.jpg)
Common wisdom Effective scaling is evidence of solid system building
Why does this happen?
McSherry et al. Any system can scale arbitrarily well with a sufficient lack of care in its implementation
![Page 25: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/25.jpg)
! " # $ %♥ ' ! " # $ %♥ ' ! " # $ %♥ ' ! "
#$ %♥ ' ! "# $ %♥ ' ! "# $ %♥ ' ! " $ %
♥ ' ! "# $ %♥ ' ! "# $ % # $ %♥♥ ' ! " $
! " # $ %♥ ' ! ! "' ! "% # $ %♥ '♥ $ %♥
#$ %♥ ' ! "# $ %♥ ' ! "# $ %♥ ' ! " $ %♥ ' ! "# $ %♥ ' ! "# $ % # $ %♥♥ ' ! " $
! " # $ %♥ ' ! " # $ %♥ ' ! " # $ %♥ ' ! "
#$ %♥ ' ! "# $ %♥ ' ! "# $ %♥ ' ! " $ %
♥ ' ! "# $ %♥ ' ! "# $ % # $ %♥♥ ' ! " $
! " # $ %♥ ' ! ! "' ! "% # $ %♥ '♥ $ %♥#$ %♥ ' ! "# $ %♥ ' ! "# $ %♥ ' ! " $ %
♥ ' ! "# $ %♥ ' ! "# $ % # $ %♥♥ ' ! " $
COST
Configuration that outperforms a single thread COST of a system is the hardware platform (number of cores) required before the platform outperforms a competent single threaded implementation
![Page 26: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/26.jpg)
![Page 27: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/27.jpg)
“If you’re building a system, make sure it’s better than
your laptop. If you’re using a system, make sure it’s better
than your laptop”McSherry
![Page 28: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/28.jpg)
Frontier
![Page 29: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/29.jpg)
![Page 30: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/30.jpg)
![Page 31: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/31.jpg)
Sampling works!
![Page 32: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/32.jpg)
Error bounds & confidence
![Page 33: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/33.jpg)
Don’t ask wasteful questions
![Page 34: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/34.jpg)
A paper tour of
Rugged
![Page 35: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/35.jpg)
Foundation!
![Page 36: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/36.jpg)
Strategies to enhance ruggedness in the presence of failures Better way to think about system availability
Ruggedness as availability
![Page 37: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/37.jpg)
Harvest: fraction of the complete result
Yield: fraction of answered queries
![Page 38: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/38.jpg)
Yield as response ruggedness
Close to uptime (% requests answered successfully) but more useful because it directly maps to user experience Failure during high & low traffic generates different yields. Uptime misses this
Focus on yield rather than uptime
![Page 39: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/39.jpg)
Harvest as quality of response
From Coda Hale’s “You can’t sacrifice partition tolerance”
Server A Server B Server C
Baby AnimalsCute
![Page 40: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/40.jpg)
Harvest as quality of response
From Coda Hale’s “You can’t sacrifice partition tolerance”
Server A Server B Server C
Baby AnimalsCute X66% harvest
![Page 41: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/41.jpg)
#1: Probabilistic Availability
Graceful harvest degradation under faults
Randomness to make the worst-case & average-case the same
Replication of high-priority data for greater harvest control
Degrading results based on client capability
![Page 42: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/42.jpg)
#2 Decomposition & Orthogonality
Decomposing into subsystems independently intolerant to harvest degradation (fail by reducing yield). But app can continue if they fail
Only provide strong consistency for the subsystems that need it
Orthogonal mechanisms (state vs functionality)
%
![Page 43: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/43.jpg)
Frontier
![Page 44: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/44.jpg)
Ruggedness via verification
Formal Methods Testing
TOP-DOWN
FAULT INJECTORS, INPUT GENERATORS
BOTTOM-UP
LINEAGE DRIVEN FAULT INJECTORS
WHITE / BLACK BOX
WE KNOW (OR NOT) ABOUT THE SYSTEM
HUMAN ASSISTED PROOFS
SAFETY CRITICAL (TLA+, COQ, ISABELLE)
MODEL CHECKING
PROPERTIES + TRANSITIONS (SPIN, TLA+)
LIGHTWEIGHT FM
BEST OF BOTH WORLDS (ALLOY, SAT)
&
![Page 45: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/45.jpg)
! " # $ %♥ ' ! " # $ %♥ ' ! " # $ %♥ ' ! "
#$ %♥ ' ! "# $ %♥ ' ! "# $ %♥ ' ! " $ %
♥ ' ! "# $ %♥ ' ! "# $ % # $ %♥♥ ' ! " $
! " # $ %♥ ' ! ! "' ! "% # $ %♥ '♥ $ %♥
#$ %♥ ' ! "# $ %♥ ' ! "# $ %♥ ' ! " $ %♥ ' ! "# $ %♥ ' ! "# $ % # $ %♥♥ ' ! " $
! " # $ %♥ ' ! " # $ %♥ ' ! " # $ %♥ ' ! "
#$ %♥ ' ! "# $ %♥ ' ! "# $ %♥ ' ! " $ %
♥ ' ! "# $ %♥ ' ! "# $ % # $ %♥♥ ' ! " $
! " # $ %♥ ' ! ! "' ! "% # $ %♥ '♥ $ %♥#$ %♥ ' ! "# $ %♥ ' ! "# $ %♥ ' ! " $ %
♥ ' ! "# $ %♥ ' ! "# $ % # $ %♥♥ ' ! " $
MOLLY: Lineage Driven Fault Injection
Reasons backwards from correct system outcomes & determines if a failure could have prevented it MOLLY only injects the failures it can prove might affect an outcome
![Page 46: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/46.jpg)
Ruggedness with MOLLY
“Without explicitly forcing a system to fail, you have no confidence that it will operate correctly in failure modes”
Caitie McCaffrey’s pearls of wisdom
' (
VerifierProgrammer
![Page 47: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/47.jpg)
MOLLY helps us undestand failure
![Page 48: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/48.jpg)
“Presents a middle ground between pragmatism and formalism, dictated by the
importance of verifying fault tolerance in spite of the
complexity of the space of faults”
![Page 49: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/49.jpg)
Now let’s
.Wrap things
![Page 50: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/50.jpg)
Agile Lean Ruggedtl;dr - foundations
A scalable system may not be a lean system Pursuing scalability out of context can be COSTly
Designing for change is designing for success
Think about availability in terms of yield and harvest
Graceful degradation is a design outcome
! !
![Page 51: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/51.jpg)
Agile Lean Ruggedtl;dr - Frontiers
Asking the wrong question is wasteful
Think about what is truly needed
Use approximations
State can be challenging
Saving state in shared memory allows us to restart DB processes faster
Reasoning backwards from correct system output helps us determine the execution failures that prevent it from happening
![Page 52: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/52.jpg)
Join your local PWL and read The Morning Paper!
github.com/Randommood/GotoLondon2015Papers are a lot of fun!)
![Page 53: Agile, Lean, Rugged: The Paper Edition - Ines Sombra's keynote at GOTO London](https://reader034.vdocuments.mx/reader034/viewer/2022051709/58718d671a28ab2c198b6bb9/html5/thumbnails/53.jpg)
✨ ✨*+ ,- *+ ,- *+ ,- *+ ,- *+ ,- *+ ,
*+ ,-*+ ,-*+ ,-*+ ,-*+ ,-*+ ,
DRANKS!