Effective Test Automation a la Carte

Martin Gijsen, Consultant,

The Netherlands

Europe’s Premier Software Testing Event

Stockholmsmässan, Sweden

WWW.EUROSTARCONFERENCES.COM

“Testing For Real, Testing For Now”

Effective Test Automationà la Carte

Martin Gijsen

Test automation architectIndependent consultant

Martin@DeAnalist.nl© 2009

Overview

Effective test automation

Domain Specific Test Languages

Three example solutions (free tools only)

Where to get the automator

Conclusion

Q & A

Test automation scenarios

time

total cost / effort

1. manual testing2. effective automated testing3. less effective automated testing

3

About 2/3 of all test automation projects fail sooner or later– Brian LeSuer (testing experience magazine,December 2008 issue, page 47)

Effective test automation

Record & Playback only works in case of (very) little manual maintenance to the testware

An automated test is a program

Test automation requires software engineering

For continuity of benefits, focus on:

3 R's: Tests must be easy to write, review and revise

Maintainability and maintenance sensitivity

Domain Specific Test Languages

Used by testers to express automated tests

Instructions defined by (or with) testers

Only the essence of a test case should remain:

Natural, high abstraction level

No irrelevant interfacing details

No irrelevant tooling details

The result is low maintenance

No programming

Clear test report

A good DSTL based test caseaccount first name last name address

open account 1234567890 John Doe 11 Test street, Testburg

open account 1234567891 Jane Doe 27 Testability lane, Testville

account amount

deposit John 12.345,00

amount from account to account

transfer 1234 John Jane

account amount

check balance John 11.111,00

check balance Jane 1.234,00

Example instruction documentation

Name and aliases 'begin test case' and 'begin testcase'

Description Indicates the beginning of a (new) test case.

Parameters

1The identification of the test case, preferably unique within the test. Optional.

2 The test case description. Optional.

Pre-condition None.

Post-conditionIf the previous input line was part of a test case, that test case is closed. A new test case is opened and assigned the next sequence number for the report(s), starting at one.

Error condition If in a procedure definition.

Example ...

Developer domain: Scripting Source code

Focus is on HOW to test itRequires software engineering skills

Tester domain: Instructions

Focus is on WHAT to testRequires analysis skills

How does it work?

Testautomation

solution

TestTest

report

SystemUnderTest

Instructiondefinitions

Three examples

Real systems and test solutions:

1. Web services based billing system

2. Web application for batch payments

3. Generic solution for payment systems

Different systems, interfaces & tooling

Web services based billing system

Interface: web services (SOAP / XML)

Interface type: synchronous messaging

Tooling: ETA Framework, Axis, Java

Multiple implementations of instructionsfor phased test execution

A sample test caseurl

select end point http://12.34.56.78:1234/billing

name

select wholesale customer Joy inc.

cust nr first name last name ...

create end customer 12345 John Doe ...

cust nr service ... ...

add subscription 12345 X ... ...

item nr cust nr ... ...

check billing item X12345 12345 ... ...

The design

Test engine

TestTest

report

Billingsystem

Instruction logic

Web serv. client

WSDL Axis

Web app for payment batches

Interface: web application (J2EE)

Interface type: Graphical user interface

Tooling: ETA Framework, WebDriver, Java

A sample test case

name

set screen item name order letter account accountNr

set screen item name order letter amount amount

set screen item name order letter date desiredDate

user name

log on JohnDoe

account amount date ...

enter order letter 123456789 12345,67 today ...

The design

Test engine

TestTest

report

Webapplication

Instruction logic

WebDriver

Payment systems

Interfaces:

Iso15022 / SWIFT MT messages

Iso20022 / SWIFT MX / UNIFI messages

Interface type: asynchronous messaging

Tooling: ETA Framework, Java

Low abstraction level is raised using procedures (not in example)

A sample (low level) test casename

define structure msgOut

name value

set msgOut.field1 value1

set msgOut.field2 value2

connection type name

send message toSut mt103 msgOut

connection name period

receive message fromSut msgIn within 10 seconds

value 1 value 2

compare ?msgIn.field1 value1

compare ?msgIn.field2 value2

The design

TestTest

report

Payment system

Test engine

Instruction logic

UNIFI protocol SWIFT protocol Incoming msgs

Connection logic

Test engine

WS logic Web app logic Payment logic

WS client WebDriver

Bringing it all together

System Under Test

Protocol X Protocol Y Incoming msgs

Connection logic

The test automation menu

1. DSTL

– Define your own

2. Test engines

– ETA Framework

– Fit / FitNesse

– STAF

– xUnit

– ...

3. Interfacing components

– Web services

– Web app

– Messaging

– Java Swing

– ...

4. Vendor lock-in (optional)

Where to get the automator

Main criteria:

Skills (programming ≠ software engineering)

Availability

Cost

Train someone from the test team

Borrow from development

Hire or reserve developer for test team

Hire external consultant

Conclusions

Test automation is software engineering- apply the right skills to succeed

A DSTL enables testers and reduces maintenance

Automated testing with free software is possible for many systems and interfaces (even combined)

While every system and DSTL is different,the basic design of test solutions can be the same

Non-programming testers and non-testing developers can be effective at automated testing

Q & A

?

See also

“Exploring no man's land with keyword driven testing” - presentation at Free Test 2009

“Effective automated testing with a DSTL”-white paper

www.DeAnalist.nl for the ETA Framework and its documentation