testing & integration (the remix)
TRANSCRIPT
Testing & Integration
THE REMIX!
Globally distributed & highly available
DS TestingContinuous IntegrationBring Home
"
!
$
Our AgendaDS =
Distributed Systems
Visual Guide ^
Distributed Systems Testing
" $!
Why do we test?We test to gain confidence that our system is doing the right thing (now & later)
" $!
MEDIUM
Unit Integration System Acceptance
SMALL
Unit Integration (maybe)
COMPLEX SYSTEMFault injection Stress / performance Canary Regression
Unit Integration System Acceptance Compatibility
Many Types of Tests " $!
Nondeterminism
TimingFailures
Unbounded inputs
Ordering
Many states
Concurrency
No centralized view
Challenges of DS Testing " $!
Behavior is aggregate
Components tested in isolation also need to be tested together
?
!
A
!
B
" $!Challenges of DS Testing
Reliability of a linear system is the product of the reliability of each of the
system’s components
" $!Challenges of DS Testing
.95 * .95 * .95 = .85 Your system is only
85% reliable
Component A
95% reliable
!
Component B
95% reliable
!
Component C
95% reliable
!
" $!Challenges of DS Testing
Hierarchy of Errors*BYZANTINE FAILURES
(fail by doing whatever I want)
OMISSION FAILURES
(fail by dropping messages)
CRASH FAILURES
(fail by stopping)
* Stolen from Henry Robinson’s PWLSF talk
Deadlocks
Livelock / starvation
Under specification
Over specification
" $!
Distributed Systems TestingDifficult to approach & many factors in play
Behavior is aggregate
Goal is to gain confidence of proper system behavior now & later
" $!
Continuous Integration
CI for short
" $!
Waiting till the end of a project to integrate leads to 🔥🔥 hell 🔥🔥 (a special kind)
" $!
What is CI?Development practice
Integrate code to master branch daily
Each integration is automatically verified
" $!
Why CI Today?Often overlooked/ Taken for granted
“Eat your veggies”
Impacts your risk & quality
" $!
Commit frequently
100% test must pass
Fix broken tests
Automation is 👑
CI Practices" $!
CI gives usRapid feedback
Rapid deployment
Repeatable automated testing
Reduce risk
" $!
Revisiting Tests
Differentiated by required setup
which affects running time
Unit
Integration
System
Functional (also Smoke /Acceptance)
" $!
CI ChallengesComplex systems / dependencies
Multiple architectures
Type of applications & types of tests
" $!
CI Challenges " $!
Overfitted/ brittle / slow tests
Writing good tests is hard
Maintenance
Consider Regulations (PCI)
“Manage the CI
System or be
controlled by manual processes”
Is it worth it?
" $!
You integrate to master very often (daily)
All of your tests pass & you fix broken builds!
Success indicators: # commits / build status
Am I doing CI right? " $!
CI In the Wild
" $!
Tools & Services
How to pick?
Security, Cost
Language support
Services & integrations
Scalability
" $!
" $!
CI In the WildBranches & pull requests. Merges & deploys
Consolidated view of build statuses across all codebases
" $!
Master PRs
Kick off a suite
StacksOS + Our
Image
ScenarioLive setup + assertions
SuiteCollection of
scenarios
My Assertions: Installing, backing
up/restoring, setting up, replicating,
promoting, etc
Any prod deploy
Kicks off a suite
" $!Dredd TestsSystems,
boundaries, & integration
CI In the Wild
CI In the Wild " $!
* Stolen from Google’s talk on their Continuous Build System
" $!CI In the Wild
* Slide stolen from Seth Falcon’s ChefConf 2015 talk
CI In the Wild: CD tools* " $!
CI Matters Predictor of professional maturity at the organizational & individual level
Adds visibility & allows us to manage risk and quality
" $!
🏆
How are we doing?
Bringing it home
" $!
" $!Getting Started
Test the full distributed system. This means testing the client, system, and provisioning code!
HAVE DECENT TESTS
ADD VISIBILITY
VERIFY & ENHANCE
CI & Code QualityAutomated inspection as part of the build
Customizable by Org
Test coverage
Code duplication
" $!
CI & Quality " $!
Languages help you: go cov, race detector,go lint
Infrastructure testing matters
But beware of misguided processes
😩
CI & QualityCyclomatic Complexity Number (CNN)
Afferent/Efferent Coupling
Software instability as “quantifiable”
" $!
And more! " $!
Use it for docs generation
Architectural/design diagrams too !
Accessibility testing
Used to test mobile architectures too
📄
Pro-Tips " $!
Add CI early
Different machine for integrating & put everything you need to built in git
Parallelize builds to scale up or down
Rapid feedback is critical. Slow is 😭💀
Staggered builds: run fast tests first, then second phase kicks in
The more likely a test is to fail the sooner you should execute it
" $!Pro-Tips
Tl;DRDS TESTING
Different types of tests Can be tricky Behavior is aggregate
CI PRACTICES
Everyone commits to baseline daily Automate build & make it self-testing Keep builds fast Everyone can see builds & status
ON QUALITY
CI can be used to drive good coding standards Some metrics available to gauge stability / quality Keep track of them
Rantifesto: Build a Culture of Quality
✨The technology is just a tool but your culture is really the key ✨
Lower the risk of change with tools & culture
" $!
“Quality means doing it right when
no one is looking”
Henry Ford
Thank You!
Special thanks to: Caitie McCaffrey, Jordan West, Jon Hyman, Bruce Spang, Devon O’Dell, Kelsey Gilmore-Innis, Aysylu Greenberg, Alan Kasindorf, Paul Reed, André Arko, Mike O’Neill, Thomas Mahoney, Eric Kustarz, Ian Fung, Neha Narula, Karl Smith, and Greg Bako.
[email protected] | @Randommood | github.com/Randommood/CraftConf2015
