scalable test automation sqa 2014 01 14 v3.pdfproject for a new teller system large, state of the...

© 2014 LogiGear Corporation. All Rights Reserved

Scalable Test Automation SQuAD, Denver, January 14th, 2014

Hans Buwalda LogiGear Corporation

hans @ logigear.com

www.logigear.com www.testarchitect.com www.happytester.com

© 2010 LogiGear Corporation. All Rights Reserved © 2010 LogiGear Corporation. All rights reserved. © 2014 LogiGear Corporation. All rights reserved.

The 5% Rules of Test Automation

No more than 5% of all test cases should be executed manually

No more than 5% of all efforts around testing should involve automating the tests


Vision on Automation

Have a good method integrated method for testing and automation testers in the driver seat maximum flexibility, best technology minimal technical efforts maximum maintainability

Have a good tool that supports that method specifically designed to provide a framework for testing and

automation does not stand in the way of the engineers encourages testers to follow the method supports maximum manageability of

• products • process

Succeed in the three "success factors for automation" test design automation solution organization


Record and Playback

select window "Logon"

enter text "username", "administrator"

enter text "password", "testonly"

push button "Ok"

select window "Main"

push button "New Customer"

expect window "Customer Information"

select field "First Name"

type "Paul"

select field "Last Name"

type "Jones"

select field "Address"

type "54321 Space Drive"

.

.

.


Scripting

Test Case

Design

Test Case

Automation

Test Case

Execution

TEST DESIGNER

AUTOMATION ENGINEER

PLAYBACK


Example scripting

/// <summary> /// AddItems - Use 'AddItemsParams' to pass parameters into this method. /// </summary> public void AddItems() { #region Variable Declarations WinControl uICalculatorDialog = this.UICalculatorWindow.UICalculatorDialog; WinEdit uIItemEdit = this.UICalculatorWindow.UIItemWindow.UIItemEdit; #endregion // Type '{NumPad7}' in 'Calculator' Dialog Keyboard.SendKeys(uICalculatorDialog, this.AddItemsParams.UICalculatorDialogSendKeys, ModifierKeys.None); // Type '{Add}{NumPad2}{Enter}' in 'Unknown Name' text box Keyboard.SendKeys(uIItemEdit, this.AddItemsParams.UIItemEditSendKeys, ModifierKeys.None); }

State of the art, but stuff for coders . . .


Action Based Testing

Modularized keyword-driven method for automated testing

Particularly suited for large and complex projects

High degree of agility, even when used in traditional projects

High level of automation, 95% or more

Puts the focus on test design, less on technology


number name quantity

new product P-9009 Sledge Hammer 5

number quantity

add quantity P-9009 20



number quantity

check quantity P-9009 34

Organize the test cases into "test modules", each with a clear and differentiated scope

Test modules contain test objectives and test cases

Test cases are written as a sequence of "actions", each with an action keyword and arguments

The automation engineers focus on automating actions, not test cases


actions, each with a

keyword and

arguments

"34" is the

expected value

here

read from top

to bottom

fragment from a test module with actions



Test Development Plan

Test Module 1

Test cases

Test Objectives

Test Module 2 Test Module N

Actions

ACTION AUTOMATION

Break down

Automate

Test cases

Test Objectives

Test cases

Test Objectives


Test Design

Effective test breakdown (into test modules) make sure every test module has a clear focus keep different kinds and levels of tests separate

Right level of actions as “high level” if possible, hiding as many details

as much as possible but not if the details are relevant for the test

It is my believe that successful automation is not a technical

challenge. It is most of all a test design challenge.


The 3 "Holy Grails" of Test Design

Effective breakdown of tests into test modules each with a focused and differentiated scope

Right approach for each test module which test design technique who to involve, etc

Proper level of detail in the test lines what actions what checks


What's the trick...


What's the trick...

Have or acquire facilities to store and organize you content

Edit your stuff

Decide where to put what assign and label the shelves

Put it there

If the organization is not sufficient anymore, add to it or change it


Breakdown Criteria

Straightforward Criteria Functionality (customers, finances, management information, ...)

-> follow the CRUD for items, with tests for UI, interaction, and business rules

Architecture of the system under test (client, server, protocol, sub

systems, components, modules, ...)

Kind of test (navigation flow, negative tests, response time, ...)

Ambition level (smoke test, regression, aggressive, …)

Additional Criteria Stakeholders (like "Accounting", "Compliance", "HR", ...)

Complexity of the test (put complex tests in separate modules)

Technical aspects of execution (special hardware, multi-station, ...)

Overall project planning (availability of information, timelines, sprints, ...)

Risks involved (extra test modules for high risk areas)


Example of business level test module

Consists of an (1) initial part, (2) test cases and (3) a final part

Focus is on business functionality, with a clear business scope

Navigation details are avoided

TEST MODULE Car Rental Payments

user

start system john

TEST CASE TC 01 Rent some cars

first name last name car

rent car John Doe Ford Escape

rent car John Doe Chevvy Volt

last name amount

check payment Doe 140.4

FINAL

close application


Example of an interaction level test module

Interaction details that are the target of the test are not hidden

Focus is not on business ("is the payment amount correct"), but on interaction ("can I retrieve the payment amount")

TEST MODULE Screen Flow

user

start system john

TEST CASE TC 01 Order button

window button

click main create order

window

check window exists new order

FINAL

close application


Case: American Bank

Project for a new teller system

Large, state of the art

Many system releases, many adjustments

Need for very high level of automation

Over 1 million test lines, in over 650 test modules

Initially little attention paid to "holy grails" UI and functional tests in the same modules virtually un-maintainable, came close to killing the project test design forced upon the team by a powerful stakeholder

who did not care much for methods...

Emergency re-organization of the test modules after system changes the tests would run again within a day


Example of using actions

In this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function

• the "#" means an expression, in this case a variable

• the ">>" assign to a variable for use later on in the test

key

key navigate F7

key navigate 3

page tab

locate page tab Scan Criteria

w indow

wait for controls loaded search

text

check breadcrumb general functions > search

w indow control value

select search scan direction Backward

w indow control value

enter value search business date match # bus date

source control

click search go

w indow

wait for controls loaded search results

w indow control variable

get search results sequence number >> seq num


In this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function

• the "#" means an expression, in this case a variable

• the ">>" assign to a variable for use later on in the test

variable

get sequence number >> seq num

Example of using actions

© 2010 LogiGear Corporation. All Rights Reserved © 2014 LogiGear Corporation. All rights reserved.

TestArchitect


TestArchitect for Visual Studio®


Focus on Test Development

Traditional testing tools are designed around test management and automation

Test Management

Automation

Test Management

Automation Test Development

TestArchitect™ is intended as a tool for test development but has a lot of test management and automation too


Automation Framework

Actions

Test Modules

Action

Definitions

Low Level

Actions

Interface

Definitions

(UI, SQL, OLE, ...)

Test Architect

Built-in Additional

Scripting (C#, Java, Python, ...)

90% +

• split up in lower level actions

• provided defaults values for

arguments

• shield (volatile) technical

details of an interface

• provide "logical" names for

controls and their classes

• test objectives

• test cases Test Modules


Example application


A Test Module for the application

We click a tree node, and then do a check

The actions here are built-in in the framework


Making a new "action"

This action definition uses existing actions to create a new action called "check bitrate"

Argument names can be used in cell expressions, that start with "#", and support the usual string and numeric operators

create a node path

from the first two

arguments the expected value is

given by the 3rd argument

the arguments of

the new action

name of the new action


Using the action in a test

These test lines don't care about the navigation in the UI of the application, the focus is functional: verify data

Such functional tests are easier to read with high level actions, and the reduced dependency on navigation makes them (much) easier to maintain in the long term


Low-level, high-level, mid-level actions

Low-level: detailed interaction with the UI (or API) generic, do not show any functional or business logic

examples: "click", "expand tree node", "select menu"

High-level: represent a business function specific to the

scope of the test hide the interaction

examples: "enter customer", "rent car", "check balance"

Mid-level: auxiliary actions that represent common

sequences of low level actions usually to wrap a form or dialog

greatly enhance maintainability

example: "enter address fields"

enter customer

enter address fields

enter select set . . . . . .


Example Script for an Action

# get table object, column number and column count

windowName = LIBRARY.NamedArgument("window")

tableName = LIBRARY.NamedArgument("table")

columnName = LIBRARY.NamedArgument("column")

table = ABT.OpenElement(windowName, tableName)

column = table.GetColumnIndex(columnName)

rowCount = table.GetRowCount()

# check the sort order, row by row

previous = table.GetCellText(0, column)

for i in range(1, rowCount):

current = table.GetCellText(i, column)

if current < previous :

LIBRARY.AdministerCheck("order", "sorted", "fails " + str(i+1), 0)

return

previous = current

LIBRARY.AdministerCheck("order", "sorted", "all rows in order", 1)

The following action script will verify whether the rows in a table are sorted:

find the table in the UI

if a value is smaller than before, fail the test

if all rows are ascending, pass the test

get arguments from the test line

def action_checkSortOrder():


Mapping the interface

An interface mapping will map windows and controls to names

When the interface of an application changes, you only have to

update this in one place

The interface mapping is a key step in your automation success,

allocate time to design it well


Some Tips to Get Stable Automation

Test your automation develop separate tests for the actions themselves

Let developers supply values for hidden identifying properties of UI element, to ease automated identification like “accessible name” in .Net and Java, or “id” in Web controls a lot of time is spent on identifying and re-identifying UI elements

Use "active" timing wait based on conditions in the system under test, avoid fixed “sleeps” if needed request developers to provide hooks

Use automation to identify differences between versions of the system under test test in a loop if all control can still be found

Keep an eye on the test design unsuccessful automation virtually always can be traced back to poor test

design


Hidden Identifying Property


Variables and expressions with keywords

This test does not need an absolute number for the available cars, just wants to see if a stock is updated

As a convention we denote an assignment with ">>"

The "#" indicates an expression

TEST CASE TC 02 Rent some more cars

car available

get quantity Chevvy Volt >> volts




car expected

check quantity Chevvy Volt # volts - 2


Data driven testing with keywords

The test lines will be repeated for each row in the data set

The values represented by "car", "first" and "last" come from the selected row of the data set

DATA SET cars

car first last value

Chevvy Volt John Doe 40000

Ford Escape Mary Kane 22500

Chrysler 300 Jane Collins 29000

Buick Verano Tom Anderson 23000

BMW 750 Henry Smyth 87000

Toyota Corolla Vivian Major 16000

TEST CASE TC 03 Check stocks

data set

use data set /cars

car available

get quantity # car >> quantity


rent car # first # last # car

car expected

check quantity # car # quantity - 1

repeat for data set


Multiple System Access

System

Under Test

Action Automation

API

access

protocol

access

UI

access

database

access

Test Modules


Device Testing

Software Under Test

Agent ABT

Automation

Interface

Info

Testing Host Device

Android


Life Cycles

Product life cycles, rather than task life cycles

The project planning and execution largely determines when the

products are created

system

development

test

development

test

automation


Typical Time Allocation

TEST DEVELOPMENT

AUTOMATION

time

effort

s


ABT in Agile

Test Module

Definition

(optional)

Test Module Development

Interface Definition

Action Automation

Test Execution

Sprint Products Product Backlog

Test re-use

Automation re-use

product owner

team prod owner

& team

User stories

Documentation

Domain understanding

Acceptance Criteria

PO Questions

Situations

Relations

Agile life cycle

Test development

Main Level Test Modules

Interaction Test Modules

Cross over Test Modules


Using ABT in Sprints (1)

Aim for "sprint + zero", meaning: try to get test development and automation "done" in the same sprint, not the next one next one means work clutters up, part of team is not

working on the same sprint, work is done double (manually and automated), ...

Agree on the approach: questions like does "done" include tests developed and

automated? do we see testing and automation as distinguishable tasks

and skillsets? is testability a requirement for the software?



Just like for development, use discussions with the team and product owners deepen understanding, for the whole team help identify items like negative, alternate and unexpected

situations

Start with the main test modules, that address the user stories and acceptance criteria try to keep the main test modules at a similar level as those

stories and criteria test modules can double as modeling device for the sprint

Plan for additional test modules: low-level testing of the interaction with the system under

test (like UI's) crossing over to other parts of the system under test



Create good starting conditions for a sprint: automation technology available (like hooks, calling

functions, etc) how to deal with data and environments understanding of subject matter, testing, automation, etc

Do interface mapping by hand, using developer provided identifications saves time by not having to use the viewer or other spy

tools recording of actions (not tests) will go better

Make testing and automation part of the evaluations and hardening sprints


Summary

Keywords is one of the techniques for automated testing, in addition to record & playback and scripting

In itself keywords are not a silver bullet. It needs a good approach, careful planning and good organization to be successful, maintainable and scalable

Thanks for your time

scalable test automation sqa 2014 01 14 v3.pdfproject for a new teller system large, state of the...

Documents