CONTINUOUS DELIVERY AND THE CHALLENGES FOR THE TEST AUTOMATIONDISA KAKKO (MAY 2016)

DISA.KAKKO@BASWARE.COMHTTPS://WWW.LINKEDIN.COM/IN/DISAKAKKO

CONTINUOUS DELIVERY- IDEAL SITUATION

• Continuous integration against continuous test automation.• Continuous automatic builds and release packages.• Automatic deployment to the QA and production environments (includes

automatic verification of the deployment successfulness).• QA personnel is focused on testing the new functionality with exploratory testing.

Test automation covers smoke and regression testing of the existing functionality on all test automation levels.• Monitoring data from the production environment is gathered and the information

is utilized to develop each phase preceding the production deployment.

REALITY?

• Some of the teams in the same SW program/project are doing test automation for all levels of sw (unit/API/system), but not all teams. In some cases some test automation level is skipped completely (e.g. unit tests).

• There is no pressure to do test automation since the QA personnel will manually test all new features and the existing functionality for every code change.

• DoD does not exist or it is not followed.• Release packaging and deployment are done manually. Deployment verification is done

by performing manual tests. • Monitoring of the production environment does not exist or the monitoring data is not

utilized by the R&D.

WHAT KIND OF TEST AUTOMATION COVERAGE CONTINUOUS DELIVERY REQUIRES

CHALLENGES IN THE SOFTWARE DEVELOPMENT- CONSEQUENCES TO THE TEST AUTOMATION?

WHAT CAUSES THE AVALANCHE?

• ”We’re too busy” is often the excuse for dismissing the needed internal QA actions for R&D (refactoring, test automation, static analysis, etc). It is more important to get features released, than to do them properly the first time (which would mean test automation and non-functional aspects taken into account for the new functionality).• QA actions for the R&D (such as test automation or refactoring) is put to the same

backlog with the new features. This way the responsibility of doing quality is outsourced to the product management. • If there is a pressure from the sales promises for the new features, the internal QA

actions for the R&D (such as test automation work) is moved down in the backlog. Eventually test automation does not get done. -> test automation is not seen as part of the new feature development.

WHAT CAUSES THE AVALANCHE?

• There is no advocate for the test automation work within a team (e.g. the lead developer).• A viewpoint that the QA personnel will take care of the quality (and

therefore all leaked bugs are seen as the QA personnel’s fault, not as the team’s fault). • Test automation framework that would suite the team’s needs is

missing. • No-one is following the DoD.

WHAT CAN BE DONE WHEN THE TEST AUTOMATION COVERAGE IS POOR?

HOW TO CHANGE THE ORGANIZATION CULTURE TO BE ”PRO TEST AUTOMATION” • Velocity allocation within each team for team internal QA development work -

> Effort estimates for the new features should include test automation work.• A test automation advocate for each team.• TDD/BDD• Test automation creation based on the leaked bugs.• Top-down approach when starting to create the missing test automation

coverage (unit test level is a ”lost case” when there hasn’t been any test automation done).• Production monitoring data utilization (e.g. estimates how data amounts will

grow)

HOW TO CHANGE THE ORGANIZATION CULTURE TO BE ”PRO TEST AUTOMATION” • Test automation framework taken into use that addresses the needs of the team.• Automatic tests as part of the CI system and the radiator. • Team’s commitment to the test automation work:

• Responsibility agreements (developers focusing on the unit/integration test automation and QA personnel to the UI/system test automation).

• Test follow-up responsibilities within the team (test status from the CI and updating the tests when needed).

• DoD dicipline (minimum set that the team will commit to).• Test automation work done as part of the feature branch work (or behind the

same feature flag)

CASE: BASWARE – ACTIONS TAKEN

• Started to automate the different phases of the deployment and the checking of the deployment successfulness.• Setting up a system for generating the test environments automatically.• Automatic deployment of builds into the automatically generated environments.• System level smoke and regression test automation. Also all APIs covered with test

automation.• Test data tool development based on the production monitoring data (possibility to

generate average data amounts or amounts greater than the known maximum amounts).• Rotating test automation upgrade responsibility within each team (to avoid test

automation to be commented out instead of fixing the outdated tests).• Automatic checks to JIRA so that it is not possible to move a feature item from one state

to another if the DoD requirements are not fulfilled.

THANK YOU! • Thoughts that came to Your mind based on this speech?• Questions?