an automation framework that really works

An Automation Framework That Really Works

ByBasivi Reddy Junna

QA Practice Manager and Sr. Automation Architect

Thursday, May 3,2012

An Automation Framework That Really Works - Basivi Reddy Junna

2

Almost anything that you would

ever want to do in

programming can be done

with Functions


3

Traditional approach to

automation

Why automation

frameworks at all?

Disadvantages of

automation frameworks

Available frameworks of

automation

Data Driven Framework

Keyword Driven framework

Modular Framework

Functional Decomposition Framework• Conceptual overview• Components• Workflow• Driver

Hybrid Framework

Q&A

Introduction

4

Create Tests

• Record the test or Program it manually

Enhance Tests

• Synchronization• Parameterization• Checkpoints

Execute Tests

• One by one• Batch mode

Analyze and Report

• Analyze Results• Report and Manage Defects

Traditional Approach to Automation


Create

Enhanc

e

Execute

Analyze

and Repor

t

5

Traditional Approach to Automation Continued…


Pros

• Best approach for automating few tests• Quick and easy to implement

Cons• In the absence of any common pattern or structure, reusability of automation code is very limited

• Standardization is a challenge• Difficult to maintain as tests are standalone and each test is a

separate point of maintenance• As the number of tests increase, coverage analysis becomes

difficult and test suite may quickly go out of control• Difficult to consolidate and analyze results

6

Standard wish list expected to be fulfilled by automation


Test Development

Promote reusability of assets

Provide rapid test building capabilities

Even manual testers should be able to create automated tests

Test Data Management

Test data management should be a breeze.

Data coverage analysis should be real simple

Use the data generated by one test in others

Handle non-reusable sets of test data without manual intervention

Test Execution

Execution setup should be centralized.

Even non technical people should be able to setup and run automated tests.

Run tests in multiple environments concurrently

Run a batch of tests with just click of a button

Schedule tests to run over night

Recover from any errors and continue with other tests in the batch

Re-run tests that did not run completely in previous run. Re-run them from the point where they left off.

Re-run tests that have one or more failed checkpoints or assertions in previous run

Test Results and Reporting

Notify designated people when tests fail to execute

Create summarized and detailed reports with results of execution.

Keep history of results.

Compare results of two or more test runs

Maintenance

Fewer points of maintenance

Quickly identify maintenance points when application changes

Maintenance should be super simple

7

Why Use Automation Frameworks At All?


RobustDo not break easily. Recover from errors for un-interrupted executionCentralized Execution Controland unattended execution.

ScalableCan easily be up-scaled in terms of number and size of tests and yet retain control

FlexibleWith little modifications assets can be leveraged to create new tests and cover more scenarios

MaintainableEasy to maintain - Few maintenance points due code re-use

ReusableAssets are reusable and provide rapid

test building capabilities

8

Difficult to implement

Availability of skilled resources is an issue

Require more up front investment than traditional approach

Over engineering trap

Wrong selection of framework

Get obsessed with framework

Disadvantages of Automation Frameworks


9


Key word Driven Framework

Modular Framework

Functional Decomposition Framework

Hybrid Framework

Available Frameworks of Automation


10

Data Driven Tests:

• Parameterized tests are commonly referred to as data driven tests• Parameterization: Fixed input values for test steps are replaced with variables and variables are supplied

with values during runtime• Same steps and logic are repeated across multiple test data sets

Data Driven Framework:

• Will have all features of data driven tests• Logic is controlled by test data. In other words script’s control flow and other behavior is dictated by test data• Suitable for large data intensive applications with limited variations in functional flows and simple to moderately complex business rules• Usually made up of few but very large test scripts whose behavior is controlled by data populated in test data repository• Test maintenance is mainly taken care by updating test data



11

Data Driven Framework - Example


Test

Id TestProd_Id TestAddress_Id TestPmt_Id

1 2 1 2

2 3 2 1

Id Product_Id Action

1 10 Search

2 3 Add

2 4 Remove

Tests

Test_ProductsId Product_Name

1 Kindle Fire

2 iPod Charger

3 Google Nexus

4 Blackberry

ProductsId Address_Id Action

1 3 Add

2 1 Edit

Test_Addresses

Id Address City

1 … Chicago

2 … Chicago

3 70 York… Lombard

Addresses

InputTest Id

12

Pros

• Limited numbers of tests to maintain• Script maintenance is largely handled through test data updates

Cons

• Complex to implement: Test data repository may become as complex as the application data model itself• Not suitable for applications with heavy variations in functionalities and flows• Tests scripts are too big to fail



13

Testing activities can be divided into basic actions called keywords – Click, Select, Enter etc

Keyword(s) => Function(s)

Test scripts are written using key words

Keywords (functions) are invoked by providing as inputs - data and application objects

Automated scripts are automatically generated behind the scene, while you are creating manual scripts using keywords

Keyword Driven Framework


14

Pros:

• You create manual tests and have automated scripts generated behind the scene automatically• Even manual testers can create automated tests without any knowledge or experience of automation.• Two birds with one shot - Produces highly structured manual tests so you can test manually in case if

automation is NOT working for any reason

Cons:

• Debugging is very tedious• Heavy upfront investment• Suitable for small and medium applications only• Difficult to handle dynamic objects• Grouping steps and controlling iterations is extremely difficult• Complex test flows cannot be specified using keywords

Keyword Driven Framework


15

Divide the applications into logically identifiable sections called modules • Re-usable• Data Driven – Modules behave according to data• Perform multiple functions and validations• Handle errors• Report results

Develop automation scripts for the modules

Modules characteristics:

Actual tests are created by calling the reusable modules in a required sequence

Can be treated as an additional layer of abstraction on the top of functional decompositions framework

Modular Framework


16

Pros:

• Large number of tests are created by relatively less number of modules. Hence, fewer points of maintenance

• Provides much higher level of abstraction than Keyword driven or functional decomposition frameworks making it easy for users to understand the test flows.

Cons:

• Best described as half data driven and half functional decomposition framework. It has disadvantages of both the frameworks but not the advantages of Functional Decomposition Framework

• Restricted maneuverability - Not suitable for application with wide variety of functionalities, twists and turns(flows). Integration of modules into actual tests and still conforming to navigational rules of the application is difficult to achieve

• Each module performs multiple functions. Selective execution of functions is difficult to achieve• Combination of higher level of abstraction and underlying data driven framework make it difficult to understand the test data being passed to and consumed by tests

Modular Framework


17

Divide the application into logically identifiable functions

Create functions using scripting language of the tool

Make the functions data driven using parameters

Create actual tests using the functions

Suitable for medium to highly complex applications

Can handle wide variety of functionalities

Capable of being data driven



18

Functional Decomposition Framework: Example Test 1


Launch Application

Login

Search Products

Add To Cart

Edit Cart

Checkout

Shipping Address

Payment Method

Place Order

Logout

Test 1Launch Application

Login

Search Products

Add To Cart

Edit Cart

Checkout

Shipping Address

Payment Method

Place Order

Logout

Launch Application

Login

Search Products

Add To Cart

Edit Cart

Checkout

Select Shipping Address

Select Payment Method

Place Order

Logout

URL

UID and PWD

Criteria & Product Info

19

Functional Decomposition Framework: Example Test 2


Launch Application

Login

Search Products

Add To Cart

Edit Cart

Checkout

Shipping Address

Payment Method

Place Order

Logout

Test 2Launch Application

Login

Search Products

Add To Cart

Edit Cart

Checkout

Shipping Address

Payment Method

Place Order

Logout

Launch Application

Login

Search Products

Add To CartEdit Cart

Checkout

Select Shipping Address

Select Payment Method

Place Order

Logout

URL

UID and PWD

Criteria & Product Info

Quantity

Capture Order Number

20

Test Data Repository

Business Functions

Common Functions

Generic Functions

Framework Functions

Functions for

Web Applications

FunctionsJAVA

Applications

Test1Function 1Function 2

…Function n

Test2Function 1Function 5

…Function n

Test3Function 6

Function 10…

Function n

DriverTest1Test2Test3

…Test n

Functional Decomposition Framework – Conceptual Overview

Functions for

Mainframe Applications

Etc…

21

Business Functions: These are re-usable functions that perform specific tasks by reading data from test data repository. All functions are data driven. These functions are specific to the application under test. Using common functions (explained later) will significantly accelerate the development of business functions• Validate the inputs• Verify execution status of previous functions in the test• Verify start status• Implement synchronization and perform validations• Recover from errors• Report results

Characteristics:

Components of the Functional Decomposition Framework


22

Common Functions: These functions are independent of any business domain/application and are commonly used. There are three sub categories of common functions: 1. Generic functions (string functions, date functions etc) 2. Framework functions (Reporting, configuration, error handling functions etc) 3. Environment specific functions such web functions, Power builder functions etc. These functions are mostly called by business functions

Automated Tests: Automated Tests are created by simply calling a series of re-usable business functions in required sequence using parameters to read appropriate data from test data repository.

Test Data Repository: A place to store , organize and manipulate test data. It is usually made up of any one or combination of flat files, excel workbooks, relational databases. This is also where results of the automated test execution will be stored. The idea is to have a centralized location for controlling test data and for reporting.

Driver: It is a batch test which calls other automated tests. It will have error handling/recovery capability where if one test in the batch fails to run due to any error, that test will be ignored and execution control will move to next test in the batch.

Components of the Functional Decomposition Framework


23

Appendix A - Structure of Business Functions


<Function Name>(Arg1,Arg2,...)‘Verify if the test failed in previous functionIf STATUS <>"PASS" Then

Exit FunctionEnd If'Validate parameters passed to the function

'for example you verify if they are null. or in some cases verify if they are numbers or not etc'If IsNull(Arg1) or IsNull(Arg2)... Then

'Perform appropriate action based on whether any argument is mandatory for the function to continue'Report about missing or incorrect argument and Exit Action

'End IfValidate Start Status

'Describe in what state the application should be (screen name and short description) when this function starts'Create Expected Result (ER) strings for all conditions this function seeks to verify

'Expected Result1=“…"'Expected Result2=“…“

write code to verify each Expected Results‘Wherever necessary implement synchronization, parameterization, Error handling, Insert output values and

checkpoints'Capture Actual Result for each Expected Result ; Compare Expected and Actual Result'Report Pass if they match or Fail if they don't along with expected and actual results‘While performing above steps recover from errors if any and also call Common functions as needed

<End Function>

24

Example Setup


Setup file - Setup IDs sheet

Setup file - Setup Details sheet

25

Example Setup


Setup file – Run Setup IDs sheet Setup file – Tests Sheet

Setup file – Run Tests Sheet

26

Sample functions for a fictitious E-Commerce Application


27

Functional Decomposition Framework Sample Test and Results


28

Workflow of the Functional Decomposition Framework


29

Functional Decomposition Framework - Driver

30

Functional Decomposition Framework – Run Test

31

Workflow of the Functional Decomposition Framework – Test Data Repository Considerations


Systems/ Applications

With UI

No UI (Headless)

32



UI Based Applications

Free form text type

Text box

Combo box

Text area

Single Select Type

Single select list view

Single select tree view

Radio button group

Dropdown list

Multi select type

Multi select list view

Multi select tree view

Checkbox

33



Non UI Based /Faceless Applications

Single values

Set of values

34

Workflow of the Functional Decomposition Framework – Test Data Repository Design Recommendations


Data Tables Static Data Tables

Contain static data . Examples: list of states, product categories like furniture, electronics etc

1. For each set of static data create a separate data table and assign unique Id for each row

2. Use these IDs delimited by ; inside dynamic tables if the function needs to handle multi select type fields or a set of

values as input3. Use the static data values directly inside the dynamic tables

if the need is to handle single select type fields or a single value as input

Dynamic Data TablesMade up of:

1. Free form text, numbers and dates 2. Different permutations and combinations of data from

static tables

35

Test Data Repository Design - Sample


Table to store values captured from Application

36

Pros:

• More flexible than other frameworks• Better maneuverability means test flows involving complex twists and turns can be created easily• Better control over test data consumed in your tests• The building blocks (functions) are not too granular, not too high level and present fewer points maintenance• One of the best frameworks suitable for automation in agile environments

Cons:

• Becomes complex when the test gets big - as the test size increases, so will the number of function calls and failure points, debug points

• When a test becomes really big, test flow is not as clearly evident as in the of modular framework


An Automation Framework That Really Works Basivi Reddy Junna

37

Hybrid Framework(s)


Keyword + Functional Decomposition + Modular Framework

Keyword + Functional Decomposition Framework

Functional Decomposition + Modular Framework

Keyword + Modular Framework

Do not copy or distribute. Proprietary and Confidential to Scalar USA Inc.

38

Q&A

???


Do not copy or distribute. Proprietary and Confidential to Scalar USA Inc.

39

Thank you!!!

[email protected]


an automation framework that really works

Documents

batch of tests

automation architect

number of tests

tests quick

batch rerun tests

use automation frameworks

previous run test results

reusability of automation