test automation

Building an effective test automation suite

As we grow an automated acceptance test suite which we can rely upon will become a valuable resource.


The process of building an effective and sustainable test automation suite involves following a simple five step process.

Identify

Describe

Implement

Run

Integrate


Identify test scenarios to include in our suite.

Identify

Describe

Implement

Run

Integrate


Describe the test scenarios in a way that is easily understood by all product stakeholders.

Identify

Describe

Implement

Run

Integrate


Implement the test steps using our chosen testing frameworks.

Identify

Describe

Implement

Run

Integrate


Run our tests on all supported product development environments.

Identify

Describe

Implement

Run

Integrate


Integrate our completed tests into our Continuous Integration and Testing system.

Identify

Describe

Implement

Run

Integrate


Test Identification should take place as an ongoing process throughout the product development lifecycle.

Identify


Test Identification should take place as an ongoing process throughout the product development lifecycle.

Identify Identify tests during Story creation. Stories should be described in Gherkin/BDD

style to develop an ubiquitous language and easily testable deliverables. Identify tests during Acceptance Criteria meetings. After reviewing the initial

requirements it may become clear that edge cases can also be defined. Identify tests during development. As the deliverables begin to become fleshed

out, additional test cases may become apparent, and should be defined. Identify tests during testing. When the testing process begins, more test

scenarios may be discovered, and should be defined and documented. Identify tests when errors appear. Errors/Bugs in any environment, including our

live environment, should result in a bug report and tests to prove when its fixed. Identify tests at any other time. Whenever a valuable test case is identified then

it should be documented and added to the test automation backlog.


We describe our product requirements using the Gherkin syntax – Given, When, Then.

Describe Identify tests during Story creation. Stories should be described in Gherkin/BDD









Describe Identify tests during Story creation. Stories should be described in Gherkin/BDD







Our BDD scenarios should be kept as short as possible – with a focus on making them easy to read and understand in as few sentences as possible.

The focus should be to explain the intended behavior of a particular piece of functionality – not to describe the steps required to make it happen.

GIVEN steps are used to describe the expected system state before our desired behavior occurs.

WHEN steps are used to describe the action that we would like to carry out on the system.

THEN steps describe the expected reaction or the state of the system after we have carried out the action in the When step.



A Good BDD Example Identify tests during Story creation. Stories should be described in Gherkin/BDD







Story: Account Holder withdraws cash

As an Account HolderI want to withdraw cash from an ATMSo that I can get money when the bank is closed

Scenario: Card has been disabledGiven the card is disabledWhen the Account Holder requests $20Then the ATM should retain the cardAnd the ATM should say the card has been retained

http://dannorth.net/whats-in-a-story/



A Good BDD Example Identify tests during Story creation. Stories should be described in Gherkin/BDD







Story: Account Holder withdraws cash

As an Account HolderI want to withdraw cash from an ATMSo that I can get money when the bank is closed

Scenario: Card has been disabledGiven the card is disabled <= Description of state before the actionWhen the Account Holder requests $20 <= The action to be carried outThen the ATM should retain the card <= How the system should react to the actionAnd the ATM should say the card has been retained <= The systems state afterward

http://dannorth.net/whats-in-a-story/


Once our scenarios have been Identified and Described then they need to be implemented as automated tests in our chosen test automation tools - currently Cucumber & Capybara.

Implement Identify tests during Story creation. Stories should be described in Gherkin/BDD








Once our scenarios have been Identified and Described then they need to be implemented as automated tests in our chosen test automation tools - currently Cucumber & Capybara.

Implement Identify tests during Story creation. Stories should be described in Gherkin/BDD







The Gherkin scenarios are included as feature files in our test automation project (PACT).

All defined tests are initially tagged as @not_started so they are not included in test runs.

When a test is being implemented then the tag changes from @not_started to @wip. Only one test should ever have a @wip tag.

When the test is successfully implemented then the @wip tag changes to a @complete tag so it is included in test runs.

Tests may also have other tags to describe what area they belong to. For example @messenger or @page_manager. This allows selective test runs.

Tests should be implemented to run on all our testable environments or should be tagged to indicate the environments on which they cannot be run. e.g. @not_staging or @not_production.


Completed tests should be run as often as possible, by as many people as possible, on as many environments, operating systems and browsers as possible.

Run


Completed tests should be run as often as possible, by as many people as possible, on as many environments, operating systems and browsers as possible.

Run Identify tests during Story creation. Stories should be described in Gherkin/BDD







During development of tests, they should be run on the local PC of the test implementer using their local OS and Browser configurations.

After completion of the tests then then should be run on a variety of supported OS and Browser combinations using our own machines or virtualized systems such as Saucelabs on a regular basis.

During our sprint testing cycles the test suites should be run by QA engineers to prove existing functionality and catch potential regressions.

As a regular ongoing process the test suites should be run against the latest code which is considered complete by the developers.


In order to maximize the value of our test automation and to ensure any potential problems are caught as early as possible – they should be integrated in a CI environment.

Integrate


In order to maximize the value of our test automation and to ensure any potential problems are caught as early as possible – they should be integrated in a CI environment.

Integrate Identify tests during Story creation. Stories should be described in Gherkin/BDD







Our tests are designed to be run regularly on a variety of environments without any manual setup or configuration steps. They are also designed to be quick and independent of each other.

To ensure that the tests are run as often as possible and are constantly proving the correctness of our systems, they will be integrated into a continuous testing environment, and run on a regular basis against all our testable environments.

We will use Jenkins, the continuous integration tool, to automate test runs. These will occur on both a scheduled basis and as new versions of our code are deployed onto the testable environments.

Some environments will receive more comprehensive testing on a more regular basis. For example, all tests would be run on the Test Automation environment whenever it is deployed (multiple times every day) but only a subset of tests would be run on Production (once every 2 weeks).


Everyone on the team has a part to play when it comes to ensuring that we have a comprehensive test suite which describes the intended behavior of Wildfire’s software.

What does this mean to me?

PO’s Dev’s QA’s TA’s



Product Owner Ensure that all the sprints stories have enough Gherkin/BDD scenarios to get the

discussion started during planning sessions. The scenarios should be short but concise and written from the perspective of a

user of the system. Ideally these scenarios would be written in advance of any planning sessions and

circulated to the rest of the team for review. Follow up with the rest of the team both before, during and after work on the

story has been undertaken to ensure that your vision has been understood by everyone involved in the development process.

Ask to see the automated tests running when the story is marked as Done.



Developers Review the Gherkin/BDD scenarios of a story prior to the planning and AC

meetings to have a clear understanding of what is being asked. Give feedback and ask questions before, during and after the planning sessions. Work with the PM and QA’s to identify any other core scenarios and edge cases

which need to be added to the feature to ensure it is complete. Work with the QA’s to ensure that the feature is testable in both a manual and

automated fashion on all of our testable environments. When code complete – run the automated acceptance test suite on your

environment to ensure that nothing else got broken. Follow up with the test automators to ensure that all environments are tested.



Quality Assurance Engineers

Review the Gherkin/BDD scenarios of a story prior to the planning and AC meetings to have a clear understanding of what is being asked.

Give feedback and ask questions before, during and after the planning sessions. Work with the PM and Dev’s to identify any other core scenarios and edge cases

which need to be added to the feature to ensure it is complete. Work with the test automators to create a prioritized backlog of tests which

should be added to the automated test suite. Checkout the automated test suite and run it on a regular basis to help catch

regressions and identify environmental issues. Work with the automators to ensure your project has good test coverage.



Test Automators Review the Gherkin/BDD scenarios of a story prior to the planning and AC

meetings to have a clear understanding of what is being asked. Give feedback and ask questions before, during and after the planning sessions. Work with the PM and Dev’s to identify any other core scenarios and edge cases

which need to be added to the feature to ensure it is complete. Work with the QA’s to create a prioritized backlog of tests which should be

added to the automated test suite. Work with the Developers to ensure that automated tests can be run in any

environment and to ensure the test automation environment is suitable for test. Keep the test suite in a well maintained state so that it can be run at any time.


Finally we have some goals which we should aim for with our Test Automation.


Finally we have some goals which we should aim for with our Test Automation.

Our test automation suite should deliver wide ranging yet concise coverage of the existing functionality provided by Wildfire’s applications. This is to help identify and fix regressions at the earliest possible opportunity.

As new stories are implemented on a sprint by sprint basis we should identify every potentially automatable story and add it to the automation backlog.

The test suite should be deployable by anyone on the team and should require no manual intervention or manual setup tasks to run.

In collaboration with the developers and system administrators we must work towards having a completely isolated test automation environment which is free of dependencies on third party systems.

Our ultimate goal should be to have a self testing system which automatically deploys and tests itself on various environments and provides feedback on potential problems and tells us when it is ready to be deployed to our production environment – all without any manual intervention.


One last thing…


One last thing…

An automated test suite is not, and never will be, a replacement for manual testing activities.

The goal of an automated test suite is to provide fast and regular feedback on the known and predictable elements within our system.

We will always require manual testing in order to identify and understand the unknown or unpredictable elements within our system.

Our goal should be to find the right balance between manual and automated testing which enables us to deliver software in a fast, efficient and confident manner.


This is still a work in progress. Please give your feedback to make this a more accurate representation of the process we want to achieve.Thanks..

test automation

Technology

test scenarios

test steps

test automation backlog

additional test cases

valuable test casesystem

completed tests

testing process

product requirements