experiences of test automation at spotify€¦ · •hard-to-test sut (experiences of test...
TRANSCRIPT
![Page 1: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/1.jpg)
Kristian Karl, Spotify
Experiences Of Test Automation at Spotify
www.eurostarconferences.com
@esconfs#esconfs
Insert speaker picture here, no more than 150x150 pixels
![Page 2: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/2.jpg)
October 16, 2013
Experiences Of Test Automation
@ SpotifyKristian Karl
![Page 3: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/3.jpg)
![Page 4: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/4.jpg)
Scaling Agile @ Spotify with Tribes, Squads, Chapters & Guilds, by Henrik Kniberg & Anders Ivarsson
![Page 5: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/5.jpg)
![Page 6: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/6.jpg)
Why automate?
• We want to be faster
• Shorter release cycles
• Facilitate the testing in
agile dev cycle
From: ”Intelligent Test Automation” by Harry Robinson
![Page 7: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/7.jpg)
What to automate?
Graphical user interface testingUsability testingSoftware performance testingSystem testingFunctional testingLoad testingVolume testingStress testingSecurity testingScalability testingSanity testing
Unit testingSmoke testingComponent testingAPI testingRegression testingInstallation testingMaintenance testingRecovery and failover testing.Accessibility testingMonkey testingIntegration testing
Graphical user interface testing
![Page 8: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/8.jpg)
Our challenges• Hard-to-test SUT
(Experiences of test automation: case study 1, An Agile Team’s Test
Automation Journey: The First Year), Dorothy Graham & Mark
Fewster)
• Maintenance of automation
• Peoples expectations
• Flaky SUTs
• Flaky tests
• Re-prioritizations of non-critical but test hindering bugs
• Test data, test environments
• Supporting services
![Page 9: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/9.jpg)
Our goals• Create automated end-user regression tests on 4 major
platforms1) Desktop – Windows and OSX
2) iOS – iPhone and iPad
3) Android
4) Webplayer
• To facilitate the workload for testers
• To deliver automated regression tests for a feature as a part of
definition of done
• To deliver short feedback loops to teams using Dashboards
![Page 10: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/10.jpg)
![Page 11: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/11.jpg)
Model-based testing
• The models are the abstraction layer
• The testers designs the automation
using models
• The developers implements the code
of the automation
![Page 12: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/12.jpg)
Login view displayed
Main view displayed
Valid credentials
App not running
Start app
Log out
Close
Exit app
Invalid credentialsToggle
’Remember Me’
Start app
![Page 13: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/13.jpg)
![Page 14: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/14.jpg)
public interface SimpleLogin {
public void e_Close();
public void e_Exit();
public void e_Init();
public void e_InvalidCredentials();
public void e_Logout();
public void e_StartClient();
public void e_ToggleRememberMe();
public void e_ValidPremiumCredentials();
public void v_ClientNotRunning();
public void v_LoginPrompted();
public void v_WhatsNew();
}
![Page 15: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/15.jpg)
Dem
o
![Page 16: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/16.jpg)
Developers and developers
• Why not use developers for TA?
• Why use developers for TA?
• Using Test API’s
• Defined by TA
• Implemented by developers
![Page 17: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/17.jpg)
![Page 18: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/18.jpg)
Before
android.view.View seekBarView =
solo.getView(com.spotify.mobile.android.ui.view.CancellableSeekBar.class, 0);
int[] xy = new int[2];
seekBarView.getLocationOnScreen(xy);
solo.clickOnScreen(xy[0] + 9 + (seekBarView.getWidth() - 18) * position, xy[1] +
seekBarView.getHeight() / 2.0f);
After
page().seekTrack(position)
![Page 19: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/19.jpg)
Supporting services
• Continuous testing
• QA Lab
• TDS – Test Data Service
• TRS – Test Result Service
![Page 20: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/20.jpg)
![Page 21: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/21.jpg)
![Page 22: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/22.jpg)
![Page 23: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/23.jpg)
Open source tools that we use
• yEd – editing FSM models [www.yworks.com]
• GraphWalker – generate test sequences from models [graphwalker.org github.com/spotify/python-graphwalker]
• Sikuli – image recognition tool [www.sikuli.org]
• NuRemote - communicating with iOS client [github.com/nevyn/NuRemoting]
• TestNG – group tests into suites and and run them [testng.org]
• Java – code implementation
• Jenkins – [test] job scheduler 24/7 [jenkins-ci.org]
![Page 24: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/24.jpg)
Demo
![Page 25: Experiences Of Test Automation at Spotify€¦ · •Hard-to-test SUT (Experiences of test automation: case study 1, An Agile Team’s Test Automation Journey: The First Year), Dorothy](https://reader034.vdocuments.mx/reader034/viewer/2022052102/603c4a140e09850a485ef445/html5/thumbnails/25.jpg)
Q & A