heavenly hell – automated tests at scale wojciech seliga
DESCRIPTION
Last Summit, we walked you through a migration from SVN to Git. So now you’re on Git – but what do you do with those branches? Branching in Git is great, but can be intimidating for those coming over from other SCMs. In this talk, you will learn all about branches – how to use them, organize them, and even revert an accidental merge. Embrace those branches and make them work for your team.TRANSCRIPT
![Page 1: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/1.jpg)
#atlassian
![Page 2: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/2.jpg)
WOJCIECH SELIGA • SENIOR DEV MANAGER • ATLASSIAN • @WSELIGA
Heavenly HellAutomated Tests at Scale
![Page 3: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/3.jpg)
• Coding since 6 yo• Agile Practices (inc. TDD) since 2003• Dev Nerd, Tech Leader, Agile Coach,
Speaker, PHB• 7 years with Atlassian
(JIRA Senior Dev Manager)• Spartez Co-founder & CEO
About me
![Page 4: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/4.jpg)
XP Promise
Cos
t of
Cha
nge
Time
WaterfallXP
![Page 5: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/5.jpg)
XP Promise
Cos
t of
Cha
nge
Time
WaterfallXP
![Page 6: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/6.jpg)
The Story
![Page 7: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/7.jpg)
About 2.5 years ago
![Page 8: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/8.jpg)
Almost 10 years of accumulating
legacy automatic tests
![Page 9: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/9.jpg)
About 20 000 tests on all levels of abstraction
*just in core JIRA
![Page 10: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/10.jpg)
Very slow (even hours)and fragile feedback loop
![Page 11: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/11.jpg)
Serious performance and reliability issues
![Page 12: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/12.jpg)
Dispirited devs accepting RED as a norm
![Page 13: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/13.jpg)
FeedbackSpeed
`Test
Quality
![Page 14: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/14.jpg)
Test Code is Not Trash
Design
MaintainRefactor
Share
Review
Prune
Respect
Discuss
Restructure
Rewrite
![Page 15: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/15.jpg)
Test Pyramid
Unit Tests (including QUnit)
REST / HTML Tests
Selenium
Fastest, lowest overall confidence
Slowest, highest overall confidence
![Page 16: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/16.jpg)
Test Pyramid
Unit Tests (including QUnit)
REST / HTML Tests
Selenium
90%
9%
1%
![Page 17: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/17.jpg)
Optimum Balance
![Page 18: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/18.jpg)
Optimum Balance
Isolation
![Page 19: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/19.jpg)
Optimum Balance
Isolation Speed
![Page 20: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/20.jpg)
Optimum Balance
Isolation Speed Coverage
![Page 21: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/21.jpg)
Optimum Balance
Isolation Speed Coverage Level
![Page 22: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/22.jpg)
Optimum Balance
Isolation Speed Coverage Level Access
![Page 23: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/23.jpg)
Optimum Balance
Isolation Speed Coverage Level Access Effort
![Page 24: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/24.jpg)
Dangerous to temper with
![Page 25: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/25.jpg)
Dangerous to temper with
Quality / Determinism
![Page 26: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/26.jpg)
Dangerous to temper with
MaintainabilityQuality / Determinism
![Page 27: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/27.jpg)
Almost two years later…
![Page 28: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/28.jpg)
People
![Page 29: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/29.jpg)
People - Motivation Making GREEN the norm
![Page 30: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/30.jpg)
Shades of Red
![Page 31: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/31.jpg)
Build Tiers and Policy
Tier A1 - green soon after all commits
Tier A2 - green at the end of the day
Tier A3 - green at the end of the iteration
unit tests and functional* tests
WebDriver and bundled plugins tests
supported platforms tests, compatibility tests
![Page 32: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/32.jpg)
Wallboards: Constant
Awareness
![Page 33: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/33.jpg)
Training
• Favouring assertThat over assertTrue/False and assertEquals
• Avoiding races - Atlassian Selenium with its TimedElement
• Favouring unit tests over functional tests (including QUnit over WebDriver)
• Promoting Page Objects
• Brownbags, blog posts, code reviews
![Page 34: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/34.jpg)
Quality
![Page 35: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/35.jpg)
Automatic Flakiness Detection Quarantine
Re-run failed tests and see if they pass
![Page 36: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/36.jpg)
Quarantine - Healing
![Page 37: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/37.jpg)
SlowMo - expose races
![Page 38: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/38.jpg)
Selenium 1
![Page 39: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/39.jpg)
Selenium 1
![Page 40: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/40.jpg)
Selenium ditching Sky did not fall in
![Page 41: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/41.jpg)
Ditching - benefits
• Freed build agents - better system throughput
• Boosted morale
• Gazillion of developer hours saved
• Money saved on infrastructure
![Page 42: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/42.jpg)
Ditching - due diligence
• conducting the audit - analysis of the coverage we lost
• determining which tests needs to rewritten (e.g. security related)
• rewriting the tests (good job for new hires + a senior mentor)
![Page 43: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/43.jpg)
Flaky Browser-based Tests
Playing with "loading" CSS class does not really help
Races between test code and asynchronous page logic
![Page 44: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/44.jpg)
Races Removal with Tracing
// in the browser:!function mySearchClickHandler() {! doSomeXhr().always(function() {! // This executes when the XHR has completed (either success or failure)! JIRA.trace("search.completed");" });!}!// In production code JIRA.trace is a no-op
// in my page object:!@Inject!TraceContext traceContext;! !public SearchResults doASearch() {! Tracer snapshot = traceContext.checkpoint();! getSearchButton().click(); // causes mySearchClickHandler to be invoked! // This waits until the "search.completed" // event has been emitted, *after* previous snapshot ! traceContext.waitFor(snapshot, "search.completed"); ! return pageBinder.bind(SearchResults.class);!}!
![Page 45: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/45.jpg)
Speed
![Page 46: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/46.jpg)
Can we halve our build times?
![Page 47: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/47.jpg)
Parallel Execution - Theory
End of Build
Batches
Start of Build
![Page 48: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/48.jpg)
Parallel Execution
End of Build
Batches
Start of Build
![Page 49: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/49.jpg)
Parallel Execution - Reality Bites
End of Build
Batches
Start of Build
Agent availability
![Page 50: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/50.jpg)
Dynamic Test Execution Dispatch - Hallelujah
![Page 51: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/51.jpg)
Dynamic Test Execution Dispatch - Hallelujah
![Page 52: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/52.jpg)
"You can't manage what you can't measure."
not by W. Edwards Deming
![Page 53: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/53.jpg)
"You can't manage what you can't measure."
not by W. Edwards Deming
If you believe just in it
you are doomed.
![Page 54: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/54.jpg)
You can't improve the systemif you can't measure it
![Page 55: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/55.jpg)
You can't improve the systemif you can't measure itProfiler, Build statistics, Logs, statsd → Graphite
![Page 56: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/56.jpg)
Anatomy of Build*
CompilationPackaging
Executing Tests
![Page 57: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/57.jpg)
Anatomy of Build*
CompilationPackaging
Executing Tests
Fetching Dependencies
![Page 58: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/58.jpg)
Anatomy of Build*
CompilationPackaging
Executing Tests
Fetching Dependencies
*Any resemblance to maven build is entirely accidental
![Page 59: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/59.jpg)
Anatomy of Build*
CompilationPackaging
Executing Tests
Fetching Dependencies
*Any resemblance to maven build is entirely accidental
SCM Update
![Page 60: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/60.jpg)
Anatomy of Build*
CompilationPackaging
Executing Tests
Fetching Dependencies
*Any resemblance to maven build is entirely accidental
SCM Update
Agent Availability/Setup
![Page 61: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/61.jpg)
Anatomy of Build*
CompilationPackaging
Executing Tests
Fetching Dependencies
*Any resemblance to maven build is entirely accidental
SCM Update
Agent Availability/Setup
Publishing Results
![Page 62: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/62.jpg)
JIRA Unit Tests Build
Compilation (7min)
![Page 63: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/63.jpg)
JIRA Unit Tests Build
Compilation (7min)
Packaging (0min)
![Page 64: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/64.jpg)
JIRA Unit Tests Build
Compilation (7min)
Packaging (0min)
Executing Tests (7min)
![Page 65: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/65.jpg)
JIRA Unit Tests Build
Compilation (7min)
Packaging (0min)
Executing Tests (7min)
Publishing Results (1min)
![Page 66: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/66.jpg)
JIRA Unit Tests Build
Compilation (7min)
Packaging (0min)
Executing Tests (7min)Fetching Dependencies (1.5min)
Publishing Results (1min)
![Page 67: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/67.jpg)
JIRA Unit Tests Build
Compilation (7min)
Packaging (0min)
Executing Tests (7min)Fetching Dependencies (1.5min)
SCM Update (2min)
Publishing Results (1min)
![Page 68: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/68.jpg)
JIRA Unit Tests Build
Compilation (7min)
Packaging (0min)
Executing Tests (7min)Fetching Dependencies (1.5min)
SCM Update (2min)
Agent Availability/Setup (mean 10min)
Publishing Results (1min)
![Page 69: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/69.jpg)
Decreasing test execution time to
ZERO alone would not let us achieve our goal!
![Page 70: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/70.jpg)
Agent Availability/Setup
• starved builds due to busy agents building very long builds
• time synchronization issue - NTPD problem
![Page 71: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/71.jpg)
• Proximity of SCM repo
• shallow git clones are not so fast and lightweight + generating extra git server CPU load
• git clone per agent/plan + git pull + git clone per build (hard links!)
• Much less load on Stash server (no need to queue up)
SCM Update - Checkout time
![Page 72: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/72.jpg)
• Proximity of SCM repo
• shallow git clones are not so fast and lightweight + generating extra git server CPU load
• git clone per agent/plan + git pull + git clone per build (hard links!)
• Much less load on Stash server (no need to queue up)
SCM Update - Checkout time
2 min → 5 seconds
![Page 73: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/73.jpg)
![Page 74: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/74.jpg)
• Fix Predator
• Sandboxing/isolation agent trade-off: rm -rf $HOME/.m2/repository/com/atlassian/*
intofind $HOME/.m2/repository/com/atlassian/ -name “*SNAPSHOT*” | xargs rm
• Network hardware failure found (dropping packets)
Fetching Dependencies
![Page 75: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/75.jpg)
• Fix Predator
• Sandboxing/isolation agent trade-off: rm -rf $HOME/.m2/repository/com/atlassian/*
intofind $HOME/.m2/repository/com/atlassian/ -name “*SNAPSHOT*” | xargs rm
• Network hardware failure found (dropping packets)
Fetching Dependencies
1.5 min → 10 seconds
![Page 76: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/76.jpg)
Compilation
• Restructuring multi-pom maven project and dependencies
• Maven 3 parallel compilation FTW!
-T 1.5C *optimal factor thanks to scientific trial and error research
![Page 77: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/77.jpg)
Compilation
• Restructuring multi-pom maven project and dependencies
• Maven 3 parallel compilation FTW!
-T 1.5C *optimal factor thanks to scientific trial and error research
7 min → 1 min
![Page 78: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/78.jpg)
Unit Test Execution
• Splitting unit tests into 2 buckets: good and legacy (much longer)
• Maven 3 parallel test execution (-T 1.5C)
3000 poor tests (5min)
11000 good tests (1.5min)
Rewritten entirely over next year
![Page 79: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/79.jpg)
Unit Test Execution
• Splitting unit tests into 2 buckets: good and legacy (much longer)
• Maven 3 parallel test execution (-T 1.5C)
7 min → 5 min
3000 poor tests (5min)
11000 good tests (1.5min)
Rewritten entirely over next year
![Page 80: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/80.jpg)
Functional Tests
• Selenium 1 removal did help
• Faster reset/restore (avoid unnecessary stuff, intercepting SQL operations for debug purposes - building stacktraces is costly)
• Restoring via Backdoor REST API (JIRA TestKit)
• Using REST API for common setup/teardown operations
![Page 81: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/81.jpg)
Functional Tests
![Page 82: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/82.jpg)
Publishing Results
• Server log allocation per test → using now Backdoor REST API (was Selenium)
• Bamboo DB performance degradation for rich build history
![Page 83: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/83.jpg)
Publishing Results
• Server log allocation per test → using now Backdoor REST API (was Selenium)
• Bamboo DB performance degradation for rich build history
1 min → 40 s
![Page 84: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/84.jpg)
Unexpected Problem
• Stability Issues with our CI server (hardware)
• The bottleneck changed from I/O to CPU
• Too many agents per physical machine
![Page 85: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/85.jpg)
JIRA Unit Tests Build Improved
Compilation (1min)
![Page 86: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/86.jpg)
JIRA Unit Tests Build Improved
Compilation (1min)
Packaging (0min)
![Page 87: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/87.jpg)
JIRA Unit Tests Build Improved
Compilation (1min)
Packaging (0min)
Executing Tests (5min)
![Page 88: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/88.jpg)
JIRA Unit Tests Build Improved
Compilation (1min)
Packaging (0min)
Executing Tests (5min)
Publishing Results (40sec)
![Page 89: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/89.jpg)
JIRA Unit Tests Build Improved
Compilation (1min)
Packaging (0min)
Executing Tests (5min)
Fetching Dependencies (10sec)
Publishing Results (40sec)
![Page 90: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/90.jpg)
JIRA Unit Tests Build Improved
Compilation (1min)
Packaging (0min)
Executing Tests (5min)
Fetching Dependencies (10sec)
SCM Update (5sec)
Publishing Results (40sec)
![Page 91: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/91.jpg)
JIRA Unit Tests Build Improved
Compilation (1min)
Packaging (0min)
Executing Tests (5min)
Fetching Dependencies (10sec)
SCM Update (5sec)
Agent Availability/Setup (3min)*
Publishing Results (40sec)
![Page 92: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/92.jpg)
Improvements Summary
Tests Before After Improvement %
Unit tests 29 min 17 min 41%
Functional tests 56 min 34 min 39%
WebDriver tests 39 min 21 min 46%
Overall 124 min 72 min 42%
* Additional ca. 5% improvement expected once new git clone strategy is consistently rolled-out everywhere
![Page 93: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/93.jpg)
Better speed increases responsibility
Fewer commits (authors) per single build
vs.
![Page 94: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/94.jpg)
The Quality Follows
![Page 95: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/95.jpg)
The Quality Follows
![Page 96: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/96.jpg)
The Quality Follows
![Page 97: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/97.jpg)
But that's still bad
We want CI feedback loop in a few minutes maximum
![Page 98: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/98.jpg)
Splitting The Codebase
![Page 99: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/99.jpg)
Inevitable Split - Fears
• Organizational concerns - understanding, managing, integrating, releasing, coordinating
• Mindset change - if something worked for 10+ years why to change it?
• Trust - does this library still work?
• We damned ourselves with big buckets for all tests - where do they belong to?
![Page 100: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/100.jpg)
Splitting code base
• Step 0 - JIRA Importers Plugin (3.5 years ago)
• Step 1- New Issue View and Navigator
• Step 2 - now everything else follows (e.g. Workflow Designer)JIRA 6.0
![Page 101: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/101.jpg)
Getting back from hell to heaven is difficult. Hell sucks in your soul.
![Page 102: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/102.jpg)
Key takeaways:
• Visibility and problem awareness help• Maintaining huge testbed is difficult and costly• Measure the problem - to baseline• No prejudice - no sacred cows• Automated tests are not one-off investment, it's a continuous journey• Performance is a damn important feature
#atlassian
![Page 103: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/103.jpg)
Test performance is a damn
important feature!
![Page 104: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/104.jpg)
XP vs Sad Reality
Cos
t of
Cha
nge
Time
WaterfallXP - idealSad Reality
![Page 105: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/105.jpg)
• Green Traffic Light - by flrnt, CC-BY-SA-2.0
• Turtle - by Jonathan Zander, CC-BY-SA-3.0
• Loading - by MatthewJ13, CC-SA-3.0
• Merlin Tool - by L. Mahin, CC-BY-SA-3.0
• Flashing Red Light - by Chris Phan, CC BY 2.0
• In Heaven - by Daniel Pascoal, CC BY-NC-ND 2.0
Images - Credits
![Page 106: Heavenly hell – automated tests at scale wojciech seliga](https://reader031.vdocuments.mx/reader031/viewer/2022012916/54576ebbaf7959234b8b5631/html5/thumbnails/106.jpg)
Thank you!
WOJCIECH SELIGA • SENIOR DEV MANAGER • ATLASSIAN • @WSELIGA