national software testing conference monday
DESCRIPTION
Testing within an agile environment in the National Software Testing Conference MondayTRANSCRIPT
![Page 1: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/1.jpg)
Testing within an agile environmentChris Gollop & Beyza Sakir
![Page 2: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/2.jpg)
Overview
History Changing Testing Concerns Adaptations Future Conclusion Further Reading
![Page 3: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/3.jpg)
History – what we do
Release every 2 weeks Trade $billions/day Average 0.4ms execution time for MTF members 10,000 orders second 1 tester per 5-6 developers Very low rate of production issues
![Page 4: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/4.jpg)
History - company
Conceived as an Agile company Agile has matured but we’re still adapting Agile testing:
Acceptance Tests Integration Tests Exploratory Testing
![Page 5: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/5.jpg)
History - company
• This has allowed us to experiment and adapt:• Time freed up• Quick feedback on changes• Historic data available to analyse
![Page 6: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/6.jpg)
History – test team
4 test specialists with different strengths complement each other as a 'team‘ Rotate through all teams• No Test Manager• Part of development team though test interests have a separate iteration
meeting• Testing is responsibility of everyone
![Page 7: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/7.jpg)
History – team structure
l
Team One Team ThreeTeam Two
Test Specialists
Business Specialists
Development Specialists
![Page 8: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/8.jpg)
History - innovate
Keep changing, even if something appears to work there may be something better
“If I had asked people what they wanted, they would have said faster horses. I gave them a car” (Henry Ford)
![Page 9: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/9.jpg)
History – measure and feedback
Measure To know if it is effective Many monitoring and measuring tools
Feedback Tying feedback to ideas Quicker this is the faster you can confidently adapt Can be numerical or social
![Page 10: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/10.jpg)
Changing Testing Concerns Over Time
Traditional Testing Maturity Models start at “ad-hoc” and move through structured testing before finishing at “Continuous Improvement”
Some of our test concerns (over time) Decrease Manual Testing & increase automated Acceptance Tests Performance DSL Speed of Feedback Increasing number of Edge Cases as system becomes more complex
![Page 11: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/11.jpg)
Changing Testing Concerns Over Time
9,000 Behavioural Tests 8,000 Acceptance Tests (UI, API) 1,000 Integration Tests (subsets of services using internal interfaces)
Acceptance Test would test a rejection Integration Tests would test the different rejection types
![Page 12: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/12.jpg)
Changing Testing Concerns Over Time
Was right at the time as needed Acceptance Tests to get good coverage quickly, trade-off is they are slower to run repeatedly.
Focus is now quicker feedback.
![Page 13: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/13.jpg)
Adaptations – Fitness Landscape
![Page 14: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/14.jpg)
Adaptations – Fitness Landscape
Each square is similar to it's neighbour Higher it is the better the solution – valleys are bad, mountaintops are good
You don't fully know where you are on the landscape You feel your way by trying out new ideas and recognising failure You may be half-way up the mountain rather than at the top And the mountains move!
Some peaks move faster than others: Google v McDonalds
Evolution isn't just biological it's everywhere
![Page 15: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/15.jpg)
Adaptations – Palchinsky Principles
Peter Palchinsky was a Russian engineer who advised on (against) Lenin Dam and Magnitogorsk – large projects which essentially failed due to being too big and unable to adapt.
Palchinsky Principles are shaped to encourage stronger innovation, better leadership and more effective policies:
Variation - seek out new ideas and try new ideas Survivability - when trying something new do it on a scale where failure is
survivable Selection - seek out feedback and learn from mistakes as you go along, avoid
an instinctive reaction of denial
![Page 16: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/16.jpg)
Adaptations – Recognise failure quickly
Being able to recognize a failure means you can re-cast it into something more likely to succeed.
Accept something is failing, don’t deny or ignore it.
"Hard to see what changes would improve current process when they’re currently sweeping all problems under the ‘our context’ rug." (Anthony Green)
The essential first step is being willing to fail.
![Page 17: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/17.jpg)
Adaptations - Google v Apple
Both considered as innovative companies.
Apple innovates in things that are core to its business Google innovates on things that indirectly contribute to its business
Apple releases products that are polished and well-scoped Google releases many different products (Buzz, Wave, Docs, AppEngine) early in
their development and lets the community develop and adapt them.
Apple create new mountains on the fitness landscape Google is surfing along on, or near, the moving fitness landscape mountain-top by
adapting its strategy and offerings.
![Page 18: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/18.jpg)
Adaptations – a brief history of what we have tried
![Page 19: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/19.jpg)
Adaptations – what worked
Production Data Testing in Live Big Feedback Intermittent Tests Tester Pairing
![Page 20: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/20.jpg)
Adaptations – Sanitised Production Data
Raw data (database) Patterns of use Data profile patterns
Migration (automated) Real world data Larger than we can create Many edge cases Try and reincorporate into our Acceptance Tests Performance Staging
Progression in what and how we use the data
![Page 21: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/21.jpg)
Adaptations – Testing in Live
Uses same hardware and code as it is just an isolated venue Runs a subset of the Acceptance Tests
When we expanded our Exchange by adding a multi-tenanted concept known as a venue we took advantage for testing
Verifies that the various components of the exchange are hooked up and configured correctly and that the deployment process worked correctly
Runs automatically throughout the day as one of our monitoring systems
![Page 22: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/22.jpg)
Adaptations – Big Feedback
![Page 23: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/23.jpg)
Adaptations – Big Feedback
Important CI pipeline builds at the top Red (failed) items rise to the top Continuously and easily expanded Useful features survive
Failure ownership/failures at a glance - AutoTrish Intermittent tests - Magic Eight Ball Auto Embargo Performance metrics
Build sheriff to keep check-in embargo
![Page 24: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/24.jpg)
Adaptations – Intermittent Tests
Tests that can fail even if feature is working Have 1 or 2 intermittent tests per run (out of 8,200) Each about 1 test run in 20
Hides genuine failures Have previously missed a release as couldn't tell if genuine
Can result from Asynchronous messaging Tests changing global settings/lack of isolation End of day transitions Date/time
![Page 25: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/25.jpg)
Adaptations – Big Feedback
anon
anon
![Page 26: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/26.jpg)
Adaptations – Intermittent Tests
All Failures run twice Fail twice = failure Fail once, pass once = intermittent Accepted as something we need to deal with
Work to remove DSL Fix a genuine issue (race condition) End of iteration is given over to removing intermittency Build Sheriff also works on this
![Page 27: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/27.jpg)
Adaptations – Intermittent Tests
![Page 28: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/28.jpg)
Adaptations – Tester Pairing
Each tester has their own strengths
Noticed that at rotation handover the new tester generally found a bug Issue with the testers (denial) Something we can harness (acceptance)
Already do development pairing (Dev & Dev and Dev & Tester)
When do we pair? At the right time for the story
Initial brainstorming when creating a mindmap When we feel we need to (pull) When someone has time (push)
![Page 29: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/29.jpg)
Adaptations – mindmap
![Page 30: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/30.jpg)
Adaptations – what didn't work
Not many absolute failures
Although many reasons it can fail
Even if something fails you learn something
Natural selection will continue
You can't stay still, unless you are improving you are losing ground as others will innovate or adapt
![Page 31: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/31.jpg)
Adaptations – facts
Of the top 25 industrial corporations in the United States in 1900, only two remained on that list at the start of the 1960s. And of the top 25 companies on the Fortune 500 in 1961, only six remain there today (IBM)
Average life expectancy of a Fortune 500 company has declined from around 75 years half a century ago to less than 15 years today, and heading towards 5 years (Deloitte)
![Page 32: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/32.jpg)
Future
Expanding the way we test
Speed up feedback Increasing integration test coverage Use PhantomJS for UI Acceptance Tests
Improving feedback/communication Test team standups
Documentation What is just enough How to get people up to speed in a complex system
![Page 33: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/33.jpg)
Future
Is about the people
Team make-up is varied and we recruit to keep it this way
“’Testing is a strictly formal process that by its nature should not vary’ - this is a typical belief by people who know nothing about testing.” (James Bach).
![Page 34: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/34.jpg)
Conclusion
Need to experiment and adapt to stay ahead
Even if something works it does not mean there isn't something better
Make changes on a scale where failure can be survived
Do not be afraid of failure but know how to recognise it quickly
![Page 35: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/35.jpg)
Further Reading
• Tim Harford• Richard Wiseman• Malcolm Gladwell• Daniel Kahneman• Dan Ariely• Richard Thaler
• LMAX Staff blogs - http://blogs.lmax.com/staff-blogs/
![Page 36: National Software Testing Conference Monday](https://reader034.vdocuments.mx/reader034/viewer/2022051501/577cc99b1a28aba711a43009/html5/thumbnails/36.jpg)