reshaping the testing pyramid
TRANSCRIPT
2
About Me…
• Quality Analyst
• Automation enthusiast
• Worked with
3
Product Background
• Desktop application• Offline, Online and Network modes of operation
• Has multiple versions in production
• Multiple teams committing to the same repository
4
As a team
5
As a team
6
As a team
7
Observations• Road blocks
• 2 different UI test automation approaches (White and TestComplete)
• Loads of UI automation tests
• Some of the UI automation tests were not part of the build pipelines
• Flaky builds
• 2 weeks of manual regression testing before each release
8
Road Blocks• In Detail
• Number of UI automation tests: ~450
• Running on 2 different modes: ~900 UI automation tests
• Parallel execution using 8 VMS
• Average time taken to execute a test suite: ~35-40 mins
• More than ~400 TestComplete UI automation tests
9
Observations
10
Goal
11
The Headache
12
Experiment• Analysis/Flow
• Analyse existing UI tests
• Check the integration tests coverage
• Identify integration tests candidate and approach
• Share the outcome with the team and finalize the approach
13
Guinea Pig for the Experiment• One of the most flaky UI automation test suite:
Reports
• Number of UI tests: 93
• Time taken to complete execution: ~65 mins
• Going to add 11 new features in Reports
14
Result of the Experiment• Added 5 integration tests in BDD format
15
Outcome of the Experiment• Added 145 integration tests in BDD format
• Execution time: ~ 3 mins
• Deleted 50 UI automation tests
• Execution time for UI automation time: ~25 mins
16
Going Forward…• More collaborative effort
• Review unit tests coverage
• Add BDD format integration tests
• Add UI automation tests only if needed
17
Going Forward…• New Feature: Reports
• Reviewed unit tests coverage
• Added BDD format integration tests: 72 (~2 mins)
• Added UI automation tests: 11 (~4 mins)
• No manual regression testing needed
18
Key Learning• How to conduct an experiment
• Placing the right tests at right level
• Team Buy-In
• Balancing the existing work with rewrite of tests/improvement approach
19
Questions