the difference that makes the difference

The difference that makes the difference

Make your test automation project a success story

Magnus NilssonLemontree

Agenda

The Test automation concept– Success factors– Components– Test cases– Framework

Walkthrough of a test automation project– Analysis phase and the business case– Maintenance of an automated solution

Test automation - success factors

Develop functional components

Decouple in- and output data from the script

Central data storage

Development project – strategy and – center of

excellence

Combine components to test cases by actions as input data

Functionalcomponents

Test automation

In-/outdata

Central data

storage

Project

Test Automation

Concept

Test automation - building up the components

Functionality available in test – Development of the functionality

as it comes available

Requirement

Design and development

Test

Phase 1 Phase 2

Function 1

Function 2

Function 1

Function 2

Function 1

Func. 2

Each functionality is part of a component (or a component itself)

Func. 3

Func. 4

Test Automation

Concept

6


CRM System

Function 1: Create customer using CRM

CRMCustomer

7


http://www.webshop.se

Function 1: Create customer using Webshop

WebCustomer

Function 1: Create customer using mobile application

8


AppCustomer

Test automation – test cases

Login

Customer Account[new]

Inventory

Login Credit check

[create]

Subscription Order

[new]

Account [new]

Customer [open]

Billing

Test case 1:

Test case 2:

Test Automation

Concept

Test automation – test cases

Login

Customer

Inventory

Login Credit check

[create]

Subscription Order

[new]

Account [new]

Customer [open]

Billing

Test case 1:

Test case 2:

Account [new]

ver 2.0

ver 1.0

Account[new]

ver 1.0

Account [new]

ver 2.0

Account[new]

ver 2.0

Test Automation

Concept

1st generation script– Using automation tool for capture and replay only– Hardcoded data

2nd generation script– Code refactoring for maintainable scripts– Uses generic functions for iterations and error handling

3rd generation script– Data driven test (DDT) where dynamic data is taken from data table rows– Keyword driven test (KDT) where action words controls the actions– Use of driver scripts for executing multiple test cases– SME for test designs and requirement– More complex scripting by developers

Test automation - development of a framework

Framework

Development of a


HP QuickTest Professional/Functional tester– Framework itself with

• test management connection ALM, • keyword driven action scripts with • data table,• function libraries,• Repositories, • BPT,• recovery scenario

– Generates code Library definitions, template of standard functions Structured data handling, input and output data

Repository

objects

Application GUI layer

List object

Text object

Table object

Object recognition

engine

Data table

M

O Business ID will be validated if provided

Out

N/A

CustomerID_Out

BusinessID

FamilyName

FirstName

In

In

In

M

N/A The created CRM customer ID

Family name

First name of customer

MOfficialName

In Private or Business

OutScriptstate_Out

Status of execution

LinkEntity_In

In O Entity linked from previous component

Function library A

Function library B

ALM

Test resources

Test plan

Test lab

Dashboard

Recovery scenario

Test action script

Framework

Development of a


HP QuickTest Professional/Functional tester

Library definitions, template of standard functions– Application specific libraries– Standard actions in applications should be in application library– Component libraries enabling Configuration Management

• Branched Release Management for ST, AT and dev• Multiple checkout and merge• Only driver scripts

– Framework libraries handling framework and best practices Structured data handling, input and output data

Framework

Development of a

HP QuickTest Professional/Functional tester Library definitions, template of standard functions

Structured data handling, input and output data– Central data source– Decouple structure and data– Central structured data source


Data table

Component A sheet

M


Out

N/A

CustomerID_Out

BusinessID

FamilyName

FirstName

In

In

In

M

N/A

The created CRM customer ID

Family name


MOfficialName


OutScriptstate_Out

Status of execution

LinkEntity_In


Component B sheet

M


Out

N/A

CustomerID_Out

BusinessID

FamilyName

FirstName

In

In

In

M

N/A


Family name


MOfficialName


OutScriptstate_Out

Status of execution

LinkEntity_In


CreateCustomer sheet

M


Out

N/A

CustomerID_Out

BusinessID

FamilyName

FirstName

In

In

In

M

N/A


Family name


MOfficialName


OutScriptstate_Out

Status of execution

LinkEntity_In


ALM

Test resources

Test plan

Test lab

DashboardData &Structure

Framework

Development of a

Function library A

Master DB

Data

Data &Structure

Test action script

The test automation projectPhase 1

Development

System test

Acceptance test

Week

Analysis

Detailed design

Requirement

Plan next phase

Documentation & Training

Phase 2

Detailed design

Development

…

Walkthrough

Project

Phase 0 The Business Case

Test automation project - business case

Why do we need a business case?– Test automation cost time and money– The automated solution must at least ensure the time spent in development, maintenance

and execution is less than a manual solution would cost to accomplish the same goal.

What is a test automation business case?– List of all costs and savings associated with test that possibly can be covered by test

automation.– Not all drivers are possible to calculate directly as quality. But quality can be expressed as an

estimation of the number of testing hours to ensure the desired level of quality.– Additional test automation drivers as repeated executions is calculated indirectly as increased

quality.

How can a business case be used?– Business case can be used to predict effectiveness as automation deliveries should follow the

calculated schedule.– ROI should be calculated at regular audits based on the business case.

Walkthrough

Project

Test automation project - business caseCost for manual regression tests

Cost for error handling

External and business costs for errors

ProductionAT

Number of errors found per year

Handling time per error (report, reproduce, administer, test data, close defect)

Number of FTE's for error handling per year

Error handling cost per hour

Annual error handling costs

1000

2

2

100

200000

Error source

500

4

2

100

200000

Total test case executions per cycle

Number of test cycles

Execution time per test, h

Execution cost per hour

10004

0.5

100

Annual cost for execution of regression test

200000

Grand total

Annual error correction costs for errors

Business costs for productions errors1)

0 100000

2

400000

Cost multiplier for Production Errors

1)Goodwill, compensation, customer service, SLA, penalties, handling costs etc. Multiplier based on 20% of the lowest industry rule of thumb factor: cost for finding errors in production is 10-100 times the cost for finding them in test

1100000

Test automation project - Business case

Savings - Manual Test Labour

Savings - Error Reduction

Implementation Cost (Year 1)

Operational costs (running)

Description Reduction ratio

Annual reduction/cost

Acc reduction

100000

70000

70000

Phase 1

Phase 2

Phase 3

Total savings

270000

90000

180000

Year 1

Year 2

Year 3

10%

20%

30% 540000

90000

270000

Year 1 120000

160000

140000Year 2

Year 3

60%

70%

80%

120000

420000

260000

Year 1 proj support, SME

Year 2 TAC OP

Year 3 TAC OP

50000

60000

60000 Acc savings

Year 1

Year 2

Year 3

-80 000

260000

370000

180000

550000

-80 000

Walkthrough

Project

Test automation project - Business case

Walkthrough

Project

Technology2

1

1

Mixed

Yes

5%

1.5

2

3

Create

5

1

2

Maintenance

Reusability

Simple TC

Complex TC

Framework

Test data

Integrations

Selection Value

45

61

New

Implementation cost calculations– Initial estimates– Design estimates

CRM Customer

20

10

10

10

15

10 20

Create

Terminate

WS Customer

Create

App Customer

Create

Analyis&Design Development

20

Component/Functions

Business case – follow up short term

Analysis

Detailed design

Development

System test

Acceptance test

Plan next phase

Phase 1

Phase 2

Detailed design

Development

Week

DeliveredPlannedTest cases

Week

Walkthrough

Project

Business case – follow up long term

”Test automation strategy”

Description of the standard approach on how to use the tools Company specific set of rules on how automation should be done Generic solution that is not dependent on individuals Maintenance focused Encapsulated by test automation centre Enforced by framework Ensured by the automation project

But we need to know what we are testing!

We should not test manually what we

execute automatically!

Walkthrough

Project

Analysis Analysis

– Analysis based on existing documentation, test cases, use cases and requirements (static test review)– Provides the prioritization to an analysis structure template

Disposition– Decision making: if we go live what does and does not work?– What business processes are working and not working or not tested?

• Often details on number of test cases passed and failed on integration and system level is not vital• Test and defect status itself often only aids prioritizing of resources/focus

Structure– Common and clear reporting on a level that enables correct business decisions– Structured test design and coverage analysis– Will control the test automation deliveries– Used in test management to create common priorities for both manual and automated tests

Implementation– After the analysis and approved structure design, the structure may be implemented using

available tools in HP QC/ALM

Walkthrough

Project

Analysis structure - disposition

what is the system used for?– how is it done?• which are the requirements on process level?

»which data conditionsare covered by use cases?

Use case

Process

Requirement

Data condition 1

Data condition 2

Walkthrough

Project

Example

Customer

Web Self Care

New

Private

Business

Analysis structure - disposition

Requirements

Folder area Process Description Prio

Use cases/product

Customer Account Subscription

Create Terminate New New

Sales Web self care 1

Modify Cancel

3 1 10Web shop

Sales Customer operations

1 3 1 1 3 1 15CRM

Sales Device 2 3App

Sales Device 2 1 1 1Online

1

Product Inventory 1Product configurations

3

1

11

Walkthrough

Project

Requirement and component coverage

Requirements

Folder area

Requirement Process

Data conditionTest case Prio

Components

Customer Account Customer

Create Terminate New Create

SalesWeb self care R1.1

1

Modify Cancel

Create Customer

Sales Customer op R2.3.4

1

Sales 2Terminate customer

Sales 2Create bank account

1

CRM

Create Customer

Web Shop

X

Customer op R2.4.1

X

Customer op R4.1.1

X

X

SalesCustomer op R4.1.1

Create invoice account

X

Test case

TC23

TC12

TC1

TC45

TC6

Walkthrough

Project

X

Coverage analysis and status report; BC follow up

Customer

Customer operations

New

Terminate

New

2 1

0

3

2 1

1

No runNot

completedFailed PassedStatus Not covered

1

3 2 71

3Passed

1

1

1

3

Not covered

Not completed

Web self care

Mobile device

New Failed 0

1 3

2

2 3

2

Coverage analysis and status report, HP ALM

Coverage in ALM Test either

– manual,– automated or– business

componentWalkthrough

Project

Maintenance

Test cases created Test data prepared Result reporting arranged Press the button

Walkthrough

Project

Then what? X

Automation Engineer

Functional tester

Automation stakeholder

Subject Matter Expert

Test Automation Center

Project organisation - Use roles to enforce competence

Automation Engineer

Automation Analyst

Automation Solution Architect

Automation Project Lead

Walkthrough

Project

Project A



Project B



Project C



Project organisation - Test automation center

Development

Execution

Maintenence

Walkthrough

Project

CommunicatingReportingPlanned activities

Test automation center

TAC Service Management

Management

Test management

Release management

Project planning

Change request

TAC Execution

TAC Maintenance

TAC Developmen

t

Request

Result

Operations

ST SIT AT

Project AProject B

Project C

Maintenance

Supplier

Defect and Issue Management

Resolution

Short about Lemontree

Swedish company established 1999

Awarded Gasell and Superföretag 2008-2012

Turnover +98 MSEK 2011

AAA-rated

About 100 employees, growing intensive

Owners are actively working in the company, no external capital

Offices in Stockholm and Oslo

QM trainee program since 2012

Focus areas

• Quality Management

• eGovernment

• Enterprise Solutions

Quality Management AQUA solutions

Thanks!Magnus Nilsson

Lemontree Solution [email protected]

the difference that makes the difference

Documents

test development

test designs

multiple test casessme

test management connection

dynamic data

function libraries

components crm systemfunction

keyword driven action