Making SoundCloud’s µ-services safe(r) to deploy
Move Fast and Consumer-Driven-Contract-Test Things
XP Days Ukraine 2017
2.5 years @ SoundCloud Monetization Team Backend Engineer
@alonpeer
About SoundCloud
a cloud full of sounds
135M tracks, 175M listeners, 12 hours/minute
300+ services / 50+ teams
Agenda
● Testing, monolith style
● My first steps @ SoundCloud
● Introducing contract tests
● Pactifying our services
● Caveats
● QA Q&A
Testing, monolith style
● One service, one client
● API changes are easy
● One client team to sync with for migrations
● API deprecation is easy
Testing, monolith styleThe good ol’ days
● Different requirements per client
=> Code complexity increases
● Harder to deploy without breaking at least one client
Testing, monolith styleMo clients, mo problems
● More manual QA
expensive, slow, prone to human error,
doesn’t scale
● More end-to-end tests
Maintenance nightmare, flaky, slow,
creates bottlenecks
Testing, monolith styleMo clients, mo problems
My first steps @ SoundCloud
My first steps @ SoundCloud
Whoa, micro-services
My first steps @ SoundCloudTesting strategy
E2E
Unit
Integration
My first steps @ SoundCloudDigging into post-mortems
© ofsmallthings
My first steps @ SoundCloud
“A lack of trust in the acceptance tests caused us to largely ignore the warnings they generated.”
“We couldn’t figure out the impact of the broken acceptance tests, and assumed the only problem was the tests themselves, rather than the underlying code.”
“The commit went through as there weren't any tests for the serialisation from the client to the backend.”
Digging into post-mortems
© ofsmallthings
Introducing contract tests
Introducing contract testsMy daily routine
© LockeSteerpike
Unit tests are not enoughIntroducing contract tests
“#cake” >> { result = call(“cake”) assert(result == “lie”)}
GET “/cake”: return “lie”
Unit tests are not enoughIntroducing contract tests
GET “/cake”: return “truth”
“#cake” >> { result = call(“cake”) assert(result == “lie”)}
Unit tests are not enoughIntroducing contract tests
import JsonLibFoo
GET “/cake”: return toJson(current_state)
// { “current_state” : “lie” }
import JsonLibFoo
“#cake” >> { result = fromJson(call(“cake”)) assert(result == “lie”)}
Unit tests are not enoughIntroducing contract tests
import JsonLibBar
GET “/cake”: return toJson(current_state)
// { “currentState” : “lie” }
import JsonLibBar
“#cake” >> { result = fromJson(call(“cake”)) assert(result == “lie”)}
Introducing contract testsHow this works?
Consumer Provider
End-to-End Test
Introducing contract testsHow this works?
Consumer Provider
Unit Test Unit Test
Introducing contract testsHow this works?
Consumer Provider
Unit Test Unit Test
Introducing contract testsHow this works?
➢ Requesting “/cake”.
➢ Expecting a JSON response
with a key “current_state”.
➢ Asserting deserialization
of the response succeeds.
import JsonLibFoo
GET “/cake”: return toJson(current_state)
// { “current_state” : “lie” }
Consumer
Provider
What’s Pact?Pactifying our services
● A family of frameworks designed for consumer driven
contract testing.
● Supports JSON over HTTP.
● Impl. for Java, Scala, Go, Ruby, .Net, Swift, JS etc.
● Tests run locally, no external dependencies.
What’s Pact?Pactifying our services
Step 2: Verify expectations on provider
© pact.io
Consumer: client code (LikesClient)
Consumer: client code (LikesClient)
Consumer: client code (LikesClient)
Consumer: client code (LikesClient)
Consumer: client code (LikesClient)
Consumer: client code (LikesClient)
Consumer: client unit test
Consumer: client unit test
Consumer: client unit test
Consumer: client unit test
Consumer: client contract unit test
Consumer: client contract unit test
Consumer: client contract unit test
Consumer: client contract unit test
Consumer: client contract unit test
Consumer: client contract unit test
Consumer: client contract unit test
Consumer: client contract unit test
Consumer: my-consumer-likes.json
Consumer: my-consumer-likes.json
Consumer: my-consumer-likes.json
Consumer: my-consumer-likes.json
Consumer: my-consumer-likes.json
Consumer: my-consumer-likes.json
Consumer: my-consumer-likes.json
Pactifying our servicesProvider side verification
● Collect pact files from consumers and verify them all.
● Keep your provider environment isolated.
○ Dockerized databases.
○ Test doubles for upstream services.
Pactifying our servicesProvider states
Pactifying our servicesProvider states
LikesApp
LikesDB
Pactifying our servicesProvider states
Set state:“User 1000 has liked 2 tracks”
LikesApp
LikesDB
Pactifying our servicesProvider states
INSERTlikes
Set state:“User 1000 has liked 2 tracks”
LikesApp
LikesDB
Pactifying our servicesProvider states
INSERTlikes
Set state:“User 1000 has liked 2 tracks”
HTTP Request: GET /likes/1000
LikesApp
LikesDB
Pactifying our servicesProvider states
HTTP Request: GET /likes/1000
HTTP Response: 200 OK
INSERTlikes
Set state:“User 1000 has liked 2 tracks”
LikesApp
LikesDB
Pactifying our servicesProvider states
LikesApp
LikesDB
HTTP Response: 200 OK
INSERTlikes
Set state:“User 1000 has liked 2 tracks”
HTTP Request: GET /likes/2000
HTTP Response: 404 Not Found
Set state:“User 2000 has liked no tracks”
HTTP Request: GET /likes/1000
DELETElikes
Pactifying our servicesPact broker
● Share pact files between projects.
● API for pact frameworks to fetch all pacts by provider.
● Support for versioning and tagging.
● Useful UI and visualization of the network.
Pactifying our servicesOur CI pipeline
Push new change
Generate consumer contracts
DeployUpload
pacts & tag with ‘prod’
Consumer pipeline
Pactifying our servicesOur CI pipeline
Push new change Deploy
Upload pacts & tag with ‘prod’
Consumer pipeline
Push new change
Verify all ‘prod’-tagged
pactsDeploy
Provider pipeline
Generate consumer contracts
● Communication is key.
Caveats
● Communication is key.
● New moving parts.
Caveats
● Communication is key.
● New moving parts.
● Learning curve per pact framework.
Caveats
● Communication is key.
● New moving parts.
● Learning curve per pact framework.
● Frameworks are WIP.
Caveats
● Communication is key.
● New moving parts.
● Learning curve per pact framework.
● Frameworks are WIP.
● Provider side setup is time consuming.
Caveats
● Communication is key.
● New moving parts.
● Learning curve per pact framework.
● Frameworks are WIP.
● Provider side setup is time consuming.
● Automating consumer-triggered provider verification.
Caveats
● Communication is key.
● New moving parts.
● Learning curve per pact framework.
● Frameworks are WIP.
● Provider side setup is time consuming.
● Automating consumer-triggered provider verification.
● Adoption is essential.
Caveats
Recap
Recap
Write contract tests
Cons
umer
s
Recap
Write contract tests Generate pact files
Cons
umer
s
Recap
Write contract tests Generate pact files
Publish to broker
Cons
umer
s
Recap
Write contract tests Generate pact files
Publish to brokerVerify pacts
Cons
umer
s
Prov
iders
Recap
Write contract tests Generate pact files
Publish to brokerVerify pacts
Cons
umer
s
Prov
iders
QA Q&A
@alonpeer
