three years of microservices at soundcloud - distributed matters berlin 2015
TRANSCRIPT
No free lunch, indeed:Three years
Phil CalçadoSoundCloud
of microservicesat SoundCloud
No free lunch, indeed:Three years
Phil CalçadoSoundCloud
of microservicesat SoundCloud
> 11 hours of audio uploaded every minute
~ 300 million people every month
heaps have been written about microservices in
the past year-ish
tl;dr
• Rapid provisioning• Basic Monitoring• Rapid application deployment
These make sense
• Rapid provisioning• Basic Monitoring• Rapid application deployment
this makes sense
why does it make me so nervous?
the SoundCloud story you might know
we moved to microservices
because $reasons
http://bit.ly/how_we_ended_up_with_microservices
the pre-history
SoundCloud, circa 2011
let’s prepare for the “microservices
explosion"
#1 provisioning
what was cool in 2010-11
much better than anything else at the
time
a problem
no resource limits (i.e. cgroups)+
naïve scheduling=
loud neighbour in your own datacentre
a problemmade for
most of our services migrated to
the problem
time
start workon v1
v1 100% deployed
start work on v2
attempt #1 - before we go sophisticated, let’s simplify what we have
warmed up pool
machineintake
just a bit of glue code
just a bit of glue codea LOT
machineintake
#2 telemetry
state of telemetry tools circa 2011-12 wasn’t
great
StatsD
let’s build our own!
but that’s not what broke…
obvious with a monolith
monolith
not so much now
?
?
?
standardise dashboards
standardise operations
https://twitter.github.io/twitter-server/Features.html#http-admin-interface
visualise
#3 deployment
> git
SquashFS
> make
unit tests integration tests
acceptance tests perf tests
> make
/dev/null
we ended up with 7 different deployment
scripts
> make > gitunit tests integration tests
unit tests integration tests
acceptance tests perf tests
containers let you spawn your mini-
SoundCloud
but why was I so nervous?
because wemessed up
there are simple and incremental ways to address these
• Rapid provisioning• Basic Monitoring• Rapid application deployment
“wtf? do you think Netflix got it right the first time?”
— @adrianco
some of the good things
Q&A