acceptance testing for continuous delivery · • what we really want is to deploy once, and run...
TRANSCRIPT
![Page 1: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/1.jpg)
Dave Farley http://www.davefarley.net @davefarley77
http://www.continuous-delivery.co.uk
Acceptance Testing for Continuous Delivery
http://www.continuous-delivery.co.uk
![Page 2: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/2.jpg)
The Role of Acceptance Testing
Local Dev. Env.
Source Repository
![Page 3: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/3.jpg)
The Role of Acceptance Testing
Artifact Repository
Local Dev. Env.
Deployment PipelineCommit
Production Env.
Deployment App.
CommitAcceptance
Manual
Perf1Perf2
Staged
Production
Source Repository
Acceptance
Component Performance
System Performance
Staging Env.
Deployment App.
Manual Test Env.
Deployment App.
![Page 4: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/4.jpg)
The Role of Acceptance Testing
Artifact Repository
Local Dev. Env.
Deployment PipelineCommit
Production Env.
Deployment App.
CommitAcceptance
Manual
Perf1Perf2
Staged
Production
Source Repository
Acceptance
Component Performance
System Performance
Staging Env.
Deployment App.
Manual Test Env.
Deployment App.
Staging Env.
Deployment App.
Manual Test Env.
Deployment App.
Component Performance
System Performance
Acceptance
![Page 5: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/5.jpg)
The Role of Acceptance Testing
Artifact Repository
Local Dev. Env.
Deployment PipelineCommit
Production Env.
Deployment App.
CommitAcceptance
Manual
Perf1Perf2
Staged
Production
Source Repository
Acceptance
Component Performance
System Performance
Staging Env.
Deployment App.
Manual Test Env.
Deployment App.
Staging Env.
Deployment App.
Manual Test Env.
Deployment App.
Component Performance
System Performance
Acceptance
![Page 6: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/6.jpg)
What is Acceptance Testing?• Asserts that the code does what the users want.
• An automated “definition of done”
• Asserts that the code works in a “production-like” test environment.
• A test of the deployment and configuration of a whole system.
• Provides timely feedback on stories - closes a feedback loop.
• Acceptance Testing, ATDD, BDD, Specification by Example, Executable Specifications.
![Page 7: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/7.jpg)
What is Acceptance Testing?
A Good Acceptance Test is:
An Executable Specification of the Behaviour of the System
![Page 8: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/8.jpg)
What is Acceptance Testing?
Unit Test CodeIdea Executable spec. Build Release
![Page 9: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/9.jpg)
What is Acceptance Testing?
Unit Test CodeIdea Executable spec. Build Release
![Page 10: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/10.jpg)
What is Acceptance Testing?
Unit Test CodeIdea Executable spec. Build Release
![Page 11: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/11.jpg)
So What’s So Hard?• Tests break when the SUT changes (Particularly UI)
• Tests are complex to develop
• This is a problem of design, the tests are too tightly-coupled to the SUT!
• The history is littered with poor implementations:
• UI Record-and-playback Systems
• Record-and-playback of production data
• Dumps of production data to test systems
• Nasty automated testing products.
![Page 12: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/12.jpg)
So What’s So Hard?• Tests break when the SUT changes (Particularly UI)
• Tests are complex to develop
• This is a problem of design, the tests are too tightly-coupled to the SUT!
• The history is littered with poor implementations:
• UI Record-and-playback Systems
• Record-and-playback of production data
• Dumps of production data to test systems
• Nasty automated testing products.
Anti-Pattern!Anti-Pattern!Anti-Pattern!Anti-Pattern!
![Page 13: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/13.jpg)
Who Owns the Tests?
• Anyone can write a test
• Developers are the people that will break tests
• Therefore Developers own the responsibility to keep them working
• Separate Testing/QA team owning automated tests
![Page 14: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/14.jpg)
Who Owns the Tests?
• Anyone can write a test
• Developers are the people that will break tests
• Therefore Developers own the responsibility to keep them working
• Separate Testing/QA team owning automated tests Anti-Pattern!
![Page 15: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/15.jpg)
Who Owns the Tests?
Developers Own Acceptance Tests!
![Page 16: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/16.jpg)
Properties of Good Acceptance Tests• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient
![Page 17: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/17.jpg)
Properties of Good Acceptance Tests• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient
![Page 18: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/18.jpg)
Public API FIX APITrade
Reporting Gateway
…
“What” not “How”
API Traders Clearing Destination
Other external
end-pointsMarket Makers
UI Traders
![Page 19: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/19.jpg)
Public API FIX APITrade
Reporting Gateway
…
“What” not “How”
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case Test
Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
![Page 20: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/20.jpg)
Public API FIX APITrade
Reporting Gateway
…FIX API
“What” not “How”
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case Test
Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case Test
Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
![Page 21: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/21.jpg)
Public API FIX APITrade
Reporting Gateway
…
“What” not “How”
API Traders Clearing Destination
Other external
end-pointsMarket Makers
UI Traders
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case Test
Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
![Page 22: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/22.jpg)
Public API FIX APITrade
Reporting Gateway
…
“What” not “How”
FIX API
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case Test
Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
API External StubsFIX-APIUI FIX-APIFIX-API
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case Test
Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
![Page 23: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/23.jpg)
Public API FIX APITrade
Reporting Gateway
…
“What” not “How”
FIX API
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case Test
Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
API External StubsFIX-APIUI FIX-API
![Page 24: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/24.jpg)
Public API FIX APITrade
Reporting Gateway
…
“What” not “How”Test
CaseTest Case
Test Case
Test Case
Test Case
Test Case
Test Case Test
Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
Test Case
API External StubsFIX-APIUI FIX-API
![Page 25: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/25.jpg)
Public API FIX APITrade
Reporting Gateway
…
“What” not “How”
API External StubsFIX-APIUI FIX-API
Test infrastructure common to all acceptance tests
![Page 26: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/26.jpg)
“What” not “How” - Separate Deployment from Testing
• Every Test should control its start conditions, and so should start and init the app.
• Acceptance Test deployment should be a rehearsal for Production Release
• This separation of concerns provides an opportunity for optimisation
• Parallel tests in a shared environment
• Lower test start-up overhead
![Page 27: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/27.jpg)
“What” not “How” - Separate Deployment from Testing
• Every Test should control its start conditions, and so should start and init the app.
• Acceptance Test deployment should be a rehearsal for Production Release
• This separation of concerns provides an opportunity for optimisation
• Parallel tests in a shared environment
• Lower test start-up overhead
Anti-Pattern!
![Page 28: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/28.jpg)
Properties of Good Acceptance Tests• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient
![Page 29: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/29.jpg)
Properties of Good Acceptance Tests• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient
![Page 30: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/30.jpg)
Test Isolation• Any form of testing is about evaluating
something in controlled circumstances
• Isolation works on multiple levels
• Isolating the System under test
• Isolating test cases from each other
• Isolating test cases from themselves (temporal isolation)
• Isolation is a vital part of your Test Strategy
![Page 31: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/31.jpg)
Test Isolation - Isolating the System Under Test
![Page 32: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/32.jpg)
Test Isolation - Isolating the System Under Test
External System ‘A’
External System ‘C’
System Under Test ‘B’
![Page 33: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/33.jpg)
Test Isolation - Isolating the System Under Test
External System ‘A’
External System ‘C’
System Under Test ‘B’
![Page 34: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/34.jpg)
Test Isolation - Isolating the System Under Test
External System ‘A’
External System ‘C’
System Under Test ‘B’
![Page 35: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/35.jpg)
Test Isolation - Isolating the System Under Test
External System ‘A’
External System ‘C’
System Under Test ‘B’?
![Page 36: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/36.jpg)
Test Isolation - Isolating the System Under Test
External System ‘A’
External System ‘C’
System Under Test ‘B’Anti-Pattern!
![Page 37: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/37.jpg)
Test Isolation - Isolating the System Under Test
System Under Test ‘B’Test Cases Verifiable
Output
![Page 38: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/38.jpg)
Test Isolation - Validating The Interfaces
![Page 39: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/39.jpg)
Test Isolation - Validating The Interfaces
External System ‘A’
External System ‘C’
System Under Test ‘B’
![Page 40: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/40.jpg)
Test Isolation - Validating The Interfaces
External System ‘A’
External System ‘C’
System Under Test ‘B’
![Page 41: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/41.jpg)
Test Isolation - Validating The Interfaces
External System ‘A’
External System ‘C’
Test Cases Verifiable Output
System Under Test ‘B’
Test Cases Verifiable Output
Test Cases Verifiable Output
![Page 42: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/42.jpg)
Test Isolation - Validating The Interfaces
External System ‘A’
External System ‘C’
Test Cases Verifiable Output
System Under Test ‘B’
Test Cases Verifiable Output
Test Cases Verifiable Output
![Page 43: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/43.jpg)
Test Isolation - Isolating Test Cases• Assuming multi-user systems…
• Tests should be efficient - We want to run LOTS!
• What we really want is to deploy once, and run LOTS of tests
• So we must avoid ANY dependencies between tests…
• Use natural functional isolation e.g.
• If testing Amazon, create a new account and a new book/product for every test-case
• If testing eBay create a new account and a new auction for every test-case
• If testing GitHub, create a new account and a new repository for every test-case
• …
![Page 44: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/44.jpg)
• We want repeatable results
• If I run my test-case twice it should work both times
Test Isolation - Temporal Isolation
![Page 45: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/45.jpg)
• We want repeatable results
• If I run my test-case twice it should work both times
Test Isolation - Temporal Isolation
def test_should_place_an_order(self): self.store.createBook(“Continuous Delivery”);
order = self.store.placeOrder(book=“Continuous Delivery")
self.store.assertOrderPlaced(order)
![Page 46: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/46.jpg)
• We want repeatable results
• If I run my test-case twice it should work both times
Test Isolation - Temporal Isolation
def test_should_place_an_order(self): self.store.createBook(“Continuous Delivery”);
order = self.store.placeOrder(book=“Continuous Delivery")
self.store.assertOrderPlaced(order)
![Page 47: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/47.jpg)
• We want repeatable results
• If I run my test-case twice it should work both times
Test Isolation - Temporal Isolation
def test_should_place_an_order(self): self.store.createBook(“Continuous Delivery”);
order = self.store.placeOrder(book=“Continuous Delivery")
self.store.assertOrderPlaced(order)
![Page 48: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/48.jpg)
• We want repeatable results
• If I run my test-case twice it should work both times
Test Isolation - Temporal Isolation
def test_should_place_an_order(self): self.store.createBook(“Continuous Delivery”);
order = self.store.placeOrder(book=“Continuous Delivery")
self.store.assertOrderPlaced(order)
Continuous Delivery
![Page 49: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/49.jpg)
• We want repeatable results
• If I run my test-case twice it should work both times
Test Isolation - Temporal Isolation
def test_should_place_an_order(self): self.store.createBook(“Continuous Delivery”);
order = self.store.placeOrder(book=“Continuous Delivery")
self.store.assertOrderPlaced(order)
Continuous Delivery
![Page 50: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/50.jpg)
• We want repeatable results
• If I run my test-case twice it should work both times
Test Isolation - Temporal Isolation
def test_should_place_an_order(self): self.store.createBook(“Continuous Delivery”);
order = self.store.placeOrder(book=“Continuous Delivery")
self.store.assertOrderPlaced(order)
Continuous Delivery1234
![Page 51: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/51.jpg)
• We want repeatable results
• If I run my test-case twice it should work both times
Test Isolation - Temporal Isolation
def test_should_place_an_order(self): self.store.createBook(“Continuous Delivery”);
order = self.store.placeOrder(book=“Continuous Delivery")
self.store.assertOrderPlaced(order)
Continuous Delivery1234Continuous Delivery6789
![Page 52: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/52.jpg)
• We want repeatable results
• If I run my test-case twice it should work both times
Test Isolation - Temporal Isolation
def test_should_place_an_order(self): self.store.createBook(“Continuous Delivery”);
order = self.store.placeOrder(book=“Continuous Delivery")
self.store.assertOrderPlaced(order)
Continuous Delivery1234Continuous Delivery6789
• Alias your functional isolation entities
• In your test case create account ‘Dave’ in reality, in the test infrastructure, ask the application to create account ‘Dave2938472398472’ and alias it to ‘Dave’ in your test infrastructure.
![Page 53: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/53.jpg)
Properties of Good Acceptance Tests• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient
![Page 54: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/54.jpg)
Properties of Good Acceptance Tests• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient
![Page 55: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/55.jpg)
Repeatability - Test Doubles
External System
![Page 56: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/56.jpg)
Repeatability - Test Doubles
External System
Local Interface to External System
![Page 57: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/57.jpg)
Repeatability - Test Doubles
External System
Local Interface to External System
Communications to External System
![Page 58: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/58.jpg)
Repeatability - Test Doubles
External System
Local Interface to External System
Communications to External System
TestStub Simulating External
System
Local Interface to External System
![Page 59: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/59.jpg)
Repeatability - Test Doubles
External System
Local Interface to External System
Communications to External System
TestStub Simulating External
System
Local Interface to External System
Production
Test Enviro
nment
kjhaskjhdkjhkjh askjhl lkjasl dkjas lkajl ajsd lkjalskjlakjsdlkajsld j lkajsdlkajsldkj
lkjlakjsldkjlka laskj ljl akjl kajsldijupoqwiuepoq dlkjl iu lkajsodiuqpwouoi la ]laksjdiuqoiwuoijds
oijasodiaosidjuoiasud
kjhaskjhdkjhkjh askjhl lkjasl dkjas lkajl ajsd lkjalskjlakjsdlkajsld j lkajsdlkajsldkj
lkjlakjsldkjlka laskj ljl akjl kajsldijupoqwiuepoq dlkjl iu lkajsodiuqpwouoi la ]laksjdiuqoiwuoijds
oijasodiaosidjuoiasud
Configuratio
n
![Page 60: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/60.jpg)
Test Doubles As Part of Test Infrastructure
TestStub Simulating External
System
Local Interface to External System
![Page 61: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/61.jpg)
Test Doubles As Part of Test Infrastructure
TestStub Simulating External
System
Local Interface to External System
![Page 62: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/62.jpg)
Test Doubles As Part of Test Infrastructure
TestStub Simulating External
System
Local Interface to External System
Public Interface
![Page 63: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/63.jpg)
Test Doubles As Part of Test Infrastructure
TestStub Simulating External
System
Local Interface to External System
Public Interface
![Page 64: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/64.jpg)
Test Doubles As Part of Test Infrastructure
TestStub Simulating External
System
Local Interface to External System
Test Infrastructure
Test Case
Test Case
Test Case
Test Case
Test Infrastructure Back-Channel
Public InterfaceSystem Under Test
![Page 65: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/65.jpg)
Properties of Good Acceptance Tests• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient
![Page 66: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/66.jpg)
Properties of Good Acceptance Tests• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient
![Page 67: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/67.jpg)
Language of the Problem Domain - DSL
• A Simple ‘DSL’ Solves many of our problems • Ease of TestCase creation • Readability • Ease of Maintenance • Separation of “What” from “How” • Test Isolation • The Chance to abstract complex set-up and scenarios • …
![Page 68: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/68.jpg)
Language of the Problem Domain - DSL @Test public void shouldSupportPlacingValidBuyAndSellLimitOrders() { trading.selectDealTicket("instrument"); trading.dealTicket.placeOrder("type: limit", ”bid: 4@10”); trading.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to buy 4.00 contracts at 10.0"); trading.dealTicket.dismissFeedbackMessage();
trading.dealTicket.placeOrder("type: limit", ”ask: 4@9”); trading.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to sell 4.00 contracts at 9.0"); }
![Page 69: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/69.jpg)
Language of the Problem Domain - DSL @Test public void shouldSupportPlacingValidBuyAndSellLimitOrders() { trading.selectDealTicket("instrument"); trading.dealTicket.placeOrder("type: limit", ”bid: 4@10”); trading.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to buy 4.00 contracts at 10.0"); trading.dealTicket.dismissFeedbackMessage();
trading.dealTicket.placeOrder("type: limit", ”ask: 4@9”); trading.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to sell 4.00 contracts at 9.0"); }
@Test public void shouldSuccessfullyPlaceAnImmediateOrCancelBuyMarketOrder() { fixAPIMarketMaker.placeMassOrder("instrument", "ask: 11@52", "ask: 10@51", "ask: 10@50", "bid: 10@49");
fixAPI.placeOrder("instrument", "side: buy", "quantity: 4", "goodUntil: Immediate", "allowUnmatched: true"); fixAPI.waitForExecutionReport("executionType: Fill", "orderStatus: Filled", "side: buy", "quantity: 4", "matched: 4", "remaining: 0", "executionPrice: 50", "executionQuantity: 4"); }
![Page 70: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/70.jpg)
Language of the Problem Domain - DSL @Test public void shouldSupportPlacingValidBuyAndSellLimitOrders() { trading.selectDealTicket("instrument"); trading.dealTicket.placeOrder("type: limit", ”bid: 4@10”); trading.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to buy 4.00 contracts at 10.0"); trading.dealTicket.dismissFeedbackMessage();
trading.dealTicket.placeOrder("type: limit", ”ask: 4@9”); trading.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to sell 4.00 contracts at 9.0"); }
@Test public void shouldSuccessfullyPlaceAnImmediateOrCancelBuyMarketOrder() { fixAPIMarketMaker.placeMassOrder("instrument", "ask: 11@52", "ask: 10@51", "ask: 10@50", "bid: 10@49");
fixAPI.placeOrder("instrument", "side: buy", "quantity: 4", "goodUntil: Immediate", "allowUnmatched: true"); fixAPI.waitForExecutionReport("executionType: Fill", "orderStatus: Filled", "side: buy", "quantity: 4", "matched: 4", "remaining: 0", "executionPrice: 50", "executionQuantity: 4"); }
@Before public void beforeEveryTest() { adminAPI.createInstrument("name: instrument"); registrationAPI.createUser("user"); registrationAPI.createUser("marketMaker", "accountType: MARKET_MAKER"); tradingUI.loginAsLive("user"); }
![Page 71: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/71.jpg)
Language of the Problem Domain - DSL public void placeOrder(final String... args) { final DslParams params = new DslParams(args, new OptionalParam("type").setDefault("Limit").setAllowedValues("limit", "market", "StopMarket"), new OptionalParam("side").setDefault("Buy").setAllowedValues("buy", "sell"), new OptionalParam("price"), new OptionalParam("triggerPrice"), new OptionalParam("quantity"), new OptionalParam("stopProfitOffset"), new OptionalParam("stopLossOffset"), new OptionalParam("confirmFeedback").setDefault("true"));
getDealTicketPageDriver().placeOrder(params.value("type"), params.value("side"), params.value("price"), params.value("triggerPrice"), params.value("quantity"), params.value("stopProfitOffset"), params.value("stopLossOffset"));
if (params.valueAsBoolean("confirmFeedback")) { getDealTicketPageDriver().clickOrderFeedbackConfirmationButton(); }
LOGGER.debug("placeOrder(" + Arrays.deepToString(args) + ")"); }
![Page 72: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/72.jpg)
Language of the Problem Domain - DSL @Test public void shouldSupportPlacingValidBuyAndSellLimitOrders() { tradingUI.showDealTicket("instrument"); tradingUI.dealTicket.placeOrder("type: limit", ”bid: 4@10”); tradingUI.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to buy 4.00 contracts at 10.0"); tradingUI.dealTicket.dismissFeedbackMessage();
tradingUI.dealTicket.placeOrder("type: limit", ”ask: 4@9”); tradingUI.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to sell 4.00 contracts at 9.0"); }
@Test public void shouldSuccessfullyPlaceAnImmediateOrCancelBuyMarketOrder() { fixAPIMarketMaker.placeMassOrder("instrument", "ask: 11@52", "ask: 10@51", "ask: 10@50", "bid: 10@49");
fixAPI.placeOrder("instrument", "side: buy", "quantity: 4", "goodUntil: Immediate", "allowUnmatched: true"); fixAPI.waitForExecutionReport("executionType: Fill", "orderStatus: Filled", "side: buy", "quantity: 4", "matched: 4", "remaining: 0", "executionPrice: 50", "executionQuantity: 4"); }
![Page 73: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/73.jpg)
Language of the Problem Domain - DSL @Test public void shouldSupportPlacingValidBuyAndSellLimitOrders() { tradingUI.showDealTicket("instrument"); tradingUI.dealTicket.placeOrder("type: limit", ”bid: 4@10”); tradingUI.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to buy 4.00 contracts at 10.0"); tradingUI.dealTicket.dismissFeedbackMessage();
tradingUI.dealTicket.placeOrder("type: limit", ”ask: 4@9”); tradingUI.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to sell 4.00 contracts at 9.0"); }
@Test public void shouldSuccessfullyPlaceAnImmediateOrCancelBuyMarketOrder() { fixAPIMarketMaker.placeMassOrder("instrument", "ask: 11@52", "ask: 10@51", "ask: 10@50", "bid: 10@49");
fixAPI.placeOrder("instrument", "side: buy", "quantity: 4", "goodUntil: Immediate", "allowUnmatched: true"); fixAPI.waitForExecutionReport("executionType: Fill", "orderStatus: Filled", "side: buy", "quantity: 4", "matched: 4", "remaining: 0", "executionPrice: 50", "executionQuantity: 4"); }
![Page 74: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/74.jpg)
Language of the Problem Domain - DSL
@Channel(fixApi, dealTicket, publicApi) @Test public void shouldSuccessfullyPlaceAnImmediateOrCancelBuyMarketOrder() { trading.placeOrder("instrument", "side: buy", “price: 123.45”, "quantity: 4", "goodUntil: Immediate”);
trading.waitForExecutionReport("executionType: Fill", "orderStatus: Filled", "side: buy", "quantity: 4", "matched: 4", "remaining: 0", "executionPrice: 123.45", "executionQuantity: 4"); }
![Page 75: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/75.jpg)
Language of the Problem Domain - DSL
@Channel(fixApi, dealTicket, publicApi) @Test public void shouldSuccessfullyPlaceAnImmediateOrCancelBuyMarketOrder() { trading.placeOrder("instrument", "side: buy", “price: 123.45”, "quantity: 4", "goodUntil: Immediate”);
trading.waitForExecutionReport("executionType: Fill", "orderStatus: Filled", "side: buy", "quantity: 4", "matched: 4", "remaining: 0", "executionPrice: 123.45", "executionQuantity: 4"); }
![Page 76: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/76.jpg)
Properties of Good Acceptance Tests• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient
![Page 77: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/77.jpg)
Properties of Good Acceptance Tests• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient
![Page 78: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/78.jpg)
Testing with Time
• Test Cases should be deterministic
• Time is a problem for determinism - There are two options:
• Ignore time
• Control time
![Page 79: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/79.jpg)
Testing With Time - Ignore TimeMechanism
Filter out time-based values in your test infrastructure so that they are ignored
Pros:• Simple!
Cons:• Can miss errors • Prevents any hope of testing complex time-based
scenarios
![Page 80: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/80.jpg)
Mechanism
Treat Time as an external dependency, like any external system - and Fake it!
Pros:• Very Flexible! • Can simulate any time-based scenario, with time under the
control of the test case.
Cons:• Slightly more complex infrastructure
Testing With Time - Controlling Time
![Page 81: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/81.jpg)
Testing With Time - Controlling Time
@Test public void shouldBeOverdueAfterOneMonth() {
book = library.borrowBook(“Continuous Delivery”); assertFalse(book.isOverdue());
time.travel(“+1 week”); assertFalse(book.isOverdue());
time.travel(“+4 weeks”); assertTrue(book.isOverdue());
}
![Page 82: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/82.jpg)
Testing With Time - Controlling Time
@Test public void shouldBeOverdueAfterOneMonth() {
book = library.borrowBook(“Continuous Delivery”); assertFalse(book.isOverdue());
time.travel(“+1 week”); assertFalse(book.isOverdue());
time.travel(“+4 weeks”); assertTrue(book.isOverdue());
}
![Page 83: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/83.jpg)
Testing With Time - Controlling Time
![Page 84: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/84.jpg)
Testing With Time - Controlling Time
Test Infrastructure
Test Case
Test Case
Test Case
Test Case
System Under Test
public void someTimeDependentMethod() {
time = System.getTime(); }
System Under Test
![Page 85: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/85.jpg)
Testing With Time - Controlling Time
Test Infrastructure
Test Case
Test Case
Test Case
Test Case
System Under Test
include Clock;
public void someTimeDependentMethod() {
time = Clock.getTime(); }
System Under Test
![Page 86: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/86.jpg)
Testing With Time - Controlling Time
Test Infrastructure
Test Case
Test Case
Test Case
Test Case
System Under Test include Clock;
public void someTimeDependentMethod() {
time = Clock.getTime(); }
public class Clock { public static clock = new SystemClock();
public static void setTime(long newTime) { clock.setTime(newTime); }
public static long getTime() { return clock.getTime(); } System Under Test
![Page 87: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/87.jpg)
Testing With Time - Controlling Time
Test Infrastructure
Test Case
Test Case
Test Case
Test Case
System Under Test include Clock;
public void someTimeDependentMethod() {
time = Clock.getTime(); }
public void onInit() { // Remote Call - back-channel systemUnderTest.setClock(new TestClock()); } public void time-travel(String time) { long newTime = parseTime(time); // Remote Call - back-channel systemUnderTest.setTime(newTime); }
Test Infrastructure Back-Channel
public class Clock { public static clock = new SystemClock();
public static void setTime(long newTime) { clock.setTime(newTime); }
public static long getTime() { return clock.getTime(); } System Under Test
![Page 88: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/88.jpg)
Test Environment Types• Some Tests need special treatment.
• Tag Tests with properties and allocate them dynamically:
![Page 89: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/89.jpg)
Test Environment Types• Some Tests need special treatment.
• Tag Tests with properties and allocate them dynamically:
@TimeTravel @Test public void shouldDoSomethingThatNeedsFakeTime() …
@Destructive @Test public void shouldDoSomethingThatKillsPartOfTheSystem() …
@FPGA(version=1.3) @Test public void shouldDoSomethingThatRequiresSpecificHardware() …
![Page 90: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/90.jpg)
Test Environment Types• Some Tests need special treatment.
• Tag Tests with properties and allocate them dynamically:
@TimeTravel @Test public void shouldDoSomethingThatNeedsFakeTime() …
@Destructive @Test public void shouldDoSomethingThatKillsPartOfTheSystem() …
@FPGA(version=1.3) @Test public void shouldDoSomethingThatRequiresSpecificHardware() …
![Page 91: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/91.jpg)
Test Environment Types
![Page 92: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/92.jpg)
Test Environment Types
![Page 93: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/93.jpg)
Properties of Good Acceptance Tests• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient
![Page 94: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/94.jpg)
Properties of Good Acceptance Tests• “What” not “How”
• Isolated from other tests
• Repeatable
• Uses the language of the problem domain
• Tests ANY change
• Efficient
![Page 95: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/95.jpg)
Production-like Test Environments
![Page 96: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/96.jpg)
Production-like Test Environments
![Page 97: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/97.jpg)
Production-like Test Environments
![Page 98: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/98.jpg)
Production-like Test Environments
![Page 99: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/99.jpg)
Production-like Test Environments
![Page 100: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/100.jpg)
Production-like Test Environments
![Page 101: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/101.jpg)
Production-like Test Environments
![Page 102: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/102.jpg)
Make Test Cases Internally Synchronous
![Page 103: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/103.jpg)
Make Test Cases Internally Synchronous
• Look for a “Concluding Event” listen for that in your DSL to report an async call as complete
![Page 104: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/104.jpg)
Make Test Cases Internally Synchronous
Example DSL level Implementation…
public String placeOrder(String params…) {
orderSent = sendAsyncPlaceOrderMessage(parseOrderParams(params)); return waitForOrderConfirmedOrFailOnTimeOut(orderSent); }
• Look for a “Concluding Event” listen for that in your DSL to report an async call as complete
![Page 105: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/105.jpg)
Make Test Cases Internally Synchronous
Example DSL level Implementation…
public String placeOrder(String params…) {
orderSent = sendAsyncPlaceOrderMessage(parseOrderParams(params)); return waitForOrderConfirmedOrFailOnTimeOut(orderSent); }
• Look for a “Concluding Event” listen for that in your DSL to report an async call as complete
![Page 106: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/106.jpg)
Make Test Cases Internally Synchronous
• Look for a “Concluding Event” listen for that in your DSL to report an async call as complete
• If you really have to, implement a “poll-and-timeout” mechanism in your test-infrastructure
• Never, Never, Never, put a “wait(xx)” and expect your tests to be (a) Reliable or (b) Efficient!
• Look for a “Concluding Event” listen for that in your DSL to report an async call as complete
![Page 107: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/107.jpg)
Make Test Cases Internally Synchronous
• Look for a “Concluding Event” listen for that in your DSL to report an async call as complete
• If you really have to, implement a “poll-and-timeout” mechanism in your test-infrastructure
• Never, Never, Never, put a “wait(xx)” and expect your tests to be (a) Reliable or (b) Efficient!
• Look for a “Concluding Event” listen for that in your DSL to report an async call as complete
Anti-Pattern!
![Page 108: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/108.jpg)
Scaling-Up
Artifact Repository
Deployment Pipeline
Acceptance
Commit
Component Performance
System Performance
Staging Env.
Deployment App.
Production Env.
Deployment App.
Source Repository
Manual Test Env.
Deployment App.
![Page 109: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/109.jpg)
Scaling-Up
Artifact Repository
Deployment Pipeline
Acceptance
Commit
Component Performance
System Performance
Staging Env.
Deployment App.
Production Env.
Deployment App.
Source Repository
Manual Test Env.
Deployment App.
Deployment PipelineCommit
Manual Test Env.
Deployment App.
Artifact Repository
Acceptance Acceptance Test Environment
![Page 110: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/110.jpg)
Scaling-Up
Artifact Repository
Deployment Pipeline
Acceptance
Commit
Component Performance
System Performance
Staging Env.
Deployment App.
Production Env.
Deployment App.
Source Repository
Manual Test Env.
Deployment App.
Deployment PipelineCommit
Manual Test Env.
Deployment App.
Artifact Repository
Acceptance Acceptance Test Environment
AA
![Page 111: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/111.jpg)
Scaling-Up
Artifact Repository
Deployment Pipeline
Acceptance
Commit
Component Performance
System Performance
Staging Env.
Deployment App.
Production Env.
Deployment App.
Source Repository
Manual Test Env.
Deployment App.
Deployment PipelineCommit
Manual Test Env.
Deployment App.
Artifact Repository
Acceptance Acceptance Test EnvironmentA
A
![Page 112: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/112.jpg)
Scaling-Up
Artifact Repository
Deployment Pipeline
Acceptance
Commit
Component Performance
System Performance
Staging Env.
Deployment App.
Production Env.
Deployment App.
Source Repository
Manual Test Env.
Deployment App.
Deployment PipelineCommit
Manual Test Env.
Deployment App.
Artifact Repository
Acceptance Acceptance Test Environment
Test HostTest Host
Test Host
Test Host
Test Host
A
A
![Page 113: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/113.jpg)
Scaling-Up
Artifact Repository
Deployment Pipeline
Acceptance
Commit
Component Performance
System Performance
Staging Env.
Deployment App.
Production Env.
Deployment App.
Source Repository
Manual Test Env.
Deployment App.
Deployment PipelineCommit
Manual Test Env.
Deployment App.
Artifact Repository
Acceptance
Acceptance
Acceptance Test Environment
Test HostTest Host
Test Host
Test Host
Test Host
AA
![Page 114: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/114.jpg)
Anti-Patterns in Acceptance Testing
![Page 115: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/115.jpg)
Anti-Patterns in Acceptance Testing• Don’t use UI Record-and-playback Systems
![Page 116: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/116.jpg)
Anti-Patterns in Acceptance Testing• Don’t use UI Record-and-playback Systems
• Don’t Record-and-playback production data. This has a role, but it is NOT Acceptance Testing
![Page 117: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/117.jpg)
Anti-Patterns in Acceptance Testing• Don’t use UI Record-and-playback Systems
• Don’t Record-and-playback production data. This has a role, but it is NOT Acceptance Testing
• Don’t dump production data to your test systems, instead define the absolute minimum data that you need
![Page 118: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/118.jpg)
Anti-Patterns in Acceptance Testing• Don’t use UI Record-and-playback Systems
• Don’t Record-and-playback production data. This has a role, but it is NOT Acceptance Testing
• Don’t dump production data to your test systems, instead define the absolute minimum data that you need
• Don’t assume Nasty Automated Testing Products(tm) will do what you need. Be very sceptical about them. Start with YOUR strategy and evaluate tools against that.
![Page 119: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/119.jpg)
Anti-Patterns in Acceptance Testing• Don’t use UI Record-and-playback Systems
• Don’t Record-and-playback production data. This has a role, but it is NOT Acceptance Testing
• Don’t dump production data to your test systems, instead define the absolute minimum data that you need
• Don’t assume Nasty Automated Testing Products(tm) will do what you need. Be very sceptical about them. Start with YOUR strategy and evaluate tools against that.
• Don’t have a separate Testing/QA team! Quality is down to everyone - Developers own Acceptance Tests!!!
![Page 120: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/120.jpg)
Anti-Patterns in Acceptance Testing• Don’t use UI Record-and-playback Systems
• Don’t Record-and-playback production data. This has a role, but it is NOT Acceptance Testing
• Don’t dump production data to your test systems, instead define the absolute minimum data that you need
• Don’t assume Nasty Automated Testing Products(tm) will do what you need. Be very sceptical about them. Start with YOUR strategy and evaluate tools against that.
• Don’t have a separate Testing/QA team! Quality is down to everyone - Developers own Acceptance Tests!!!
• Don’t let every Test start and init the app. Optimise for Cycle-Time, be efficient in your use of test environments.
![Page 121: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/121.jpg)
Anti-Patterns in Acceptance Testing• Don’t use UI Record-and-playback Systems
• Don’t Record-and-playback production data. This has a role, but it is NOT Acceptance Testing
• Don’t dump production data to your test systems, instead define the absolute minimum data that you need
• Don’t assume Nasty Automated Testing Products(tm) will do what you need. Be very sceptical about them. Start with YOUR strategy and evaluate tools against that.
• Don’t have a separate Testing/QA team! Quality is down to everyone - Developers own Acceptance Tests!!!
• Don’t let every Test start and init the app. Optimise for Cycle-Time, be efficient in your use of test environments.
• Don’t include Systems outside of your control in your Acceptance Test Scope
![Page 122: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/122.jpg)
Anti-Patterns in Acceptance Testing• Don’t use UI Record-and-playback Systems
• Don’t Record-and-playback production data. This has a role, but it is NOT Acceptance Testing
• Don’t dump production data to your test systems, instead define the absolute minimum data that you need
• Don’t assume Nasty Automated Testing Products(tm) will do what you need. Be very sceptical about them. Start with YOUR strategy and evaluate tools against that.
• Don’t have a separate Testing/QA team! Quality is down to everyone - Developers own Acceptance Tests!!!
• Don’t let every Test start and init the app. Optimise for Cycle-Time, be efficient in your use of test environments.
• Don’t include Systems outside of your control in your Acceptance Test Scope
• Don’t Put ‘wait()’ instructions in your tests hoping it will solve intermittency
![Page 123: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/123.jpg)
Tricks for Success
![Page 124: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/124.jpg)
Tricks for Success• Do Ensure That Developers Own the Tests
![Page 125: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/125.jpg)
Tricks for Success• Do Ensure That Developers Own the Tests
• Do Focus Your Tests on “What” not “How”
![Page 126: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/126.jpg)
Tricks for Success• Do Ensure That Developers Own the Tests
• Do Focus Your Tests on “What” not “How”
• Do Think of Your Tests as “Executable Specifications”
![Page 127: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/127.jpg)
Tricks for Success• Do Ensure That Developers Own the Tests
• Do Focus Your Tests on “What” not “How”
• Do Think of Your Tests as “Executable Specifications”
• Do Make Acceptance Testing Part of your “Definition of Done”
![Page 128: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/128.jpg)
Tricks for Success• Do Ensure That Developers Own the Tests
• Do Focus Your Tests on “What” not “How”
• Do Think of Your Tests as “Executable Specifications”
• Do Make Acceptance Testing Part of your “Definition of Done”
• Do Keep Tests Isolated from one-another
![Page 129: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/129.jpg)
Tricks for Success• Do Ensure That Developers Own the Tests
• Do Focus Your Tests on “What” not “How”
• Do Think of Your Tests as “Executable Specifications”
• Do Make Acceptance Testing Part of your “Definition of Done”
• Do Keep Tests Isolated from one-another
• Do Keep Your Tests Repeatable
![Page 130: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/130.jpg)
Tricks for Success• Do Ensure That Developers Own the Tests
• Do Focus Your Tests on “What” not “How”
• Do Think of Your Tests as “Executable Specifications”
• Do Make Acceptance Testing Part of your “Definition of Done”
• Do Keep Tests Isolated from one-another
• Do Keep Your Tests Repeatable
• Do Use the Language of the Problem Domain - Do try the DSL approach, whatever your tech.
![Page 131: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/131.jpg)
Tricks for Success• Do Ensure That Developers Own the Tests
• Do Focus Your Tests on “What” not “How”
• Do Think of Your Tests as “Executable Specifications”
• Do Make Acceptance Testing Part of your “Definition of Done”
• Do Keep Tests Isolated from one-another
• Do Keep Your Tests Repeatable
• Do Use the Language of the Problem Domain - Do try the DSL approach, whatever your tech.
• Do Stub External Systems
![Page 132: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/132.jpg)
Tricks for Success• Do Ensure That Developers Own the Tests
• Do Focus Your Tests on “What” not “How”
• Do Think of Your Tests as “Executable Specifications”
• Do Make Acceptance Testing Part of your “Definition of Done”
• Do Keep Tests Isolated from one-another
• Do Keep Your Tests Repeatable
• Do Use the Language of the Problem Domain - Do try the DSL approach, whatever your tech.
• Do Stub External Systems
• Do Test in “Production-Like” Environments
![Page 133: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/133.jpg)
Tricks for Success• Do Ensure That Developers Own the Tests
• Do Focus Your Tests on “What” not “How”
• Do Think of Your Tests as “Executable Specifications”
• Do Make Acceptance Testing Part of your “Definition of Done”
• Do Keep Tests Isolated from one-another
• Do Keep Your Tests Repeatable
• Do Use the Language of the Problem Domain - Do try the DSL approach, whatever your tech.
• Do Stub External Systems
• Do Test in “Production-Like” Environments
• Do Make Instructions Appear Synchronous at the Level of the Test Case
![Page 134: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/134.jpg)
Tricks for Success• Do Ensure That Developers Own the Tests
• Do Focus Your Tests on “What” not “How”
• Do Think of Your Tests as “Executable Specifications”
• Do Make Acceptance Testing Part of your “Definition of Done”
• Do Keep Tests Isolated from one-another
• Do Keep Your Tests Repeatable
• Do Use the Language of the Problem Domain - Do try the DSL approach, whatever your tech.
• Do Stub External Systems
• Do Test in “Production-Like” Environments
• Do Make Instructions Appear Synchronous at the Level of the Test Case
• Do Test for ANY change
![Page 135: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/135.jpg)
Tricks for Success• Do Ensure That Developers Own the Tests
• Do Focus Your Tests on “What” not “How”
• Do Think of Your Tests as “Executable Specifications”
• Do Make Acceptance Testing Part of your “Definition of Done”
• Do Keep Tests Isolated from one-another
• Do Keep Your Tests Repeatable
• Do Use the Language of the Problem Domain - Do try the DSL approach, whatever your tech.
• Do Stub External Systems
• Do Test in “Production-Like” Environments
• Do Make Instructions Appear Synchronous at the Level of the Test Case
• Do Test for ANY change
• Do Keep your Tests Efficient
![Page 136: Acceptance Testing for Continuous Delivery · • What we really want is to deploy once, and run LOTS of tests ... order = self.store.placeOrder(book=“Continuous Delivery") self.store.assertOrderPlaced(order)](https://reader034.vdocuments.mx/reader034/viewer/2022050715/5f20320c39746a282f1ab1fb/html5/thumbnails/136.jpg)
Q&A
http://www.continuous-delivery.co.uk
Dave Farley
http://www.davefarley.net
@davefarley77