test automation frameworks final

Advanced Test AutomationAdvanced Test AutomationPractical Application of Test Automation Practical Application of Test Automation

FrameworksFrameworks

Lee BarnesFounder & Chief Technology Officer

Utopia Solutions

Copyright 2007 Utopia Solutions

AgendaAgenda

• Assumptions & Definitions• Introduction to Automation Frameworks• Overview of Common Frameworks• Selecting the Right Framework• Case Study• Q & A


AssumptionsAssumptions

• Audience has:– Exposure to automation best practices– Exposure to contemporary test automation tool

(e.g. WinRunner, QuickTest Pro, SilkTest, QA Run, Robot)

• Addressing technical issues such as custom interface objects not in scope

• Presentation is tool agnostic


DefinitionsDefinitions

• Data-driven• Business Process• Keyword• Test case• Test scenario (script)• Test engine

Introduction to Automation Introduction to Automation FrameworksFrameworks


What is an “Automation What is an “Automation Framework” ?Framework” ?

• Structured approach to automation utilizing data-driven techniques, reusable test code assets and standards/guidelines

Automation Standards & Guidelines

Test Scenario

Input

Test Scenario Results

Test EngineReusableScripts/Modules

UtilityFunctions


Why?Why?

• Promote the three key characteristics automated test suites must possess to be successful:– Reliable– Maintainable– Scalable


How?How?

• A properly designed framework will possess these characteristics if they incorporate three key attributes:– Modular data-driven design– Separation of test definition and test

execution– Error detection and recovery


Modular & Data-Driven Modular & Data-Driven DesignDesign

• Promotes maintainability through reuse and reduced test suite size

• Promotes scalability by abstracting test data from test code


Separation of Test Definition Separation of Test Definition and Executionand Execution

• Promotes scalability by allowing test creation to be performed by non-technical testers and SME’s without test code modification

• Allows testers/business analysts and automation engineers to focus on their strengths – separately

• Enables execution of single test scenario across multiple platforms


Separation of Test Definition Separation of Test Definition and Test Executionand Test Execution

Test Engine

Test Scenario

InputTest

Scenario Results

Software Testing and Functional Subject Matter Experts

Automation Experts

ReusableScripts/Modules

UtilityFunctions

Test DefinitionInterface


Error Detection and Error Detection and RecoveryRecovery

• Promotes reliability by enabling unattended execution

• Best Practices– Use tool capabilities for the simple stuff

• Trapping unexpected pop-ups• Handling/logging unexpected return values from

built-in tool functions (e.g. object recognition errors)

– Keep it simple and consistent– Carefully plan execution status into framework


Handling Execution Handling Execution StatusStatus

• Need to track progress and handle recovery after restart• Assess possible statuses - PASS, FAIL, INCOMPLETE, etc.• Build into test suite modules & input files…

Business Process Module

Initialize status to INCOMPLETE…Perform business process…If validation condition = TRUE

Set status = PASSElse

Set status = FAILLog execution status

Test Scenario Input File

Common Automation Common Automation FrameworksFrameworks

Business Process FrameworkKeyword Framework

Integrated Framework

Business Process (BP) Business Process (BP) FrameworkFramework


Business Process Business Process FrameworkFramework

• Main components of test engine parallel AUT functionality at a business process level

• Sample Components:

• Login• PickOrder• AdjustInventory• VerifyInventory• …

BPFunctions

• LogResults• LogError• OpenTestInput• ReadTestInput• …

SupportingFunctions

• Design/Coding Standards

• Test Definition Guide

• Maintenance & • Execution Guide

Standards &Guidelines


Test Engine


Test Scenario

Files

Login, test_user_01, password_01

Create_Order, <ord1>,SKU10045,100,…


Ship_Order,<ord1>,…

Ship_Order,<ord2>,…

Verify_Inventory,SKU10045,…


Business Process Scripts

Utility Functions



• Relatively simple to implement

• Straight forward test definition interface for test creators

• Efficient execution

Advantages Disadvantages

• Requires automation expertise if business process steps change

• Requires automation expertise to support new business processes

• Not suitable for other types of testing



• Ideal for testing that involves the consistent execution of a finite number of end-to-end business processes

Keyword (KW) FrameworkKeyword (KW) Framework


Keyword FrameworkKeyword Framework

• Main components of test engine parallel object/action level functionality required to perform test actions

• Sample Components:

• FieldSetText• FieldGetText• ListSelectItem• ButtonClick• …

Object/ActionFunctions

• LogResults• LogError• OpenTestInput• ReadTestInput• …

SupportingFunctions

• Design/Coding Standards

• Test Definition Guide

• Maintenance & • Execution Guide

Standards &Guidelines


Test Engine

Keyword-Driven ModelKeyword-Driven Model

Object/ Action

Functions

Utility Functions

VerifyState, login_page, EXISTS

SetText, user_id, testuser01

SetTextSecure, password, j4ghjs39

ClickButton, submit

VerifyState, main_menu, EXISTS

…


Test Scenario

Files



• Suitable for many types of testing

• Automation expertise not required to maintain or add business processes


• More complex test definition interface

• Can result in large test scenario input files

• Can result in inefficient execution



• Ideal framework for:– Verbatim conversion of manual test scripts– Detailed field-level testing– Teams with technical test creators– Serving as base for integrated frameworks (discussed next)

Integrated Business Process / Integrated Business Process / Keyword FrameworkKeyword Framework


Integrated FrameworkIntegrated Framework

• Business Process framework wrapped around a keyword framework

• Contains all Keyword framework components plus business process templates to allow test scenarios to be defined at the BP level


Integrated BP/KW Integrated BP/KW FrameworkFramework

Business Process

Templates

Test Scenario

File


SetText, user_id, <user_id>

SetTextSecure, password, <pw>

ClickButton, submit


Test Engine

Object/ Action

Functions

Utility Functions




…



s

• Incorporates the benefits of both BP and KW frameworks


• Can be complex to implement and maintain

• Can muddy the functional/technical boundary



• Ideal for testing that requires both business process and keyword frameworks

Selecting the Right FrameworkSelecting the Right Framework


Aligning Automation Aligning Automation Framework with Test Framework with Test

ObjectivesObjectives

• Testing objectives come in different sizes and shapes

• Choose the framework that best aligns with your needs:– Test coverage– Type of test cases (GUI, BP, data validation, etc.)– Quantity of test cases

• “Distill” your test cases (discussed later)


Understanding Test Understanding Test CoverageCoverage

Test Script

• Login as user test01

• Select Create PO link and verify user has access

• Verify Create PO page is defaulted with the following data …

• Verify SKU field rejects non-numeric data

• Create PO using the following data …

• Verify …

Requirements

• Functional• Order Mgmt.• Inventory Mgmt.• Customer Mgmt.• …

• Security• User access• …

• User Interface• Screen

navigation• Input validation• …


What’s MissingWhat’s Missing

Test Script






• Verify …

Requirements





?


Distilling Test CasesDistilling Test Cases

Test Cases

• Create PO• Create PO under

credit limit• Create PO over

credit limit• …

• Receive Order• Full receipt• Partial receipt• …

• Input Field Validation

• Valid values• Max length• …

Test Script






• Verify …

Requirements






Why Distill?Why Distill?






• Verify Customer credit limit...

Manual Testing

User Interface

Input Validation

Business Process

User Interface

User InterfaceUser Interface

User Interface

User Interface

Input Validation

Input Validation

Input Validation

Input Validation

Input Validation

Business ProcessBusiness Process

Business ProcessBusiness Process

Business Process


Distilled Test Cases MappedDistilled Test Cases MappedTo FrameworksTo FrameworksAutomated Testing

Automated Test Suite

Business ProcessTesting Framework

User InterfaceTesting Framework

Input ValidationTesting Framework

Business Process

User Interface

Input Validation

Case StudyCase Study

Multi-Platform Warehouse Management System


SituationSituation

• Warehouse Management System– Web-based client used by office staff– Subset of functionality available to warehouse staff via

hand-held RF device– One-to-one mapping of screens & fields between platforms

• Existing manual regression test suite– Focused on BP-level testing– 35% of test scenarios executed on both platforms

• Objectives– Create automated test suite to include existing test

coverage– Single input source for multi-platform test scenarios


Multi-Platform FrameworkMulti-Platform FrameworkAutomation OptionsAutomation Options

1. BP framework with parallel modules for each business process requiring execution on both platforms

2. KW framework with parallel functions for each object/action requiring support on both platforms

3. KW framework with “intelligent” functions that “sense” the execution platform and execute appropriately

4. Integrated (BP wrapped around KW) frameworkWhat criteria would you use to decide?


Multi-Platform FrameworkMulti-Platform FrameworkConceptual DesignConceptual Design

Test Scenario

Input

Tes

t D

efin

itio

nIn

terf

ace

Test Engine


Desktop ExecutionFramework

Hand-held ExecutionFramework

FunctionalSME’s

AutomationEngineers


Best Framework Choice?Best Framework Choice?

• Integrated framework to incorporate the advantages of both BP and KW frameworks

• Low-level “intelligent” object/action functions best support multi-platform execution


Integrated BP/KW Integrated BP/KW FrameworkFramework

Business Process

Templates

Test Scenario

File


SetText, user_id, <user_id>

SetTextSecure, password, <pw>

ClickButton, submit


Test Engine

Object/ Action

Functions

Utility Functions




…


Translate Manual Scenario Translate Manual Scenario to Input Fileto Input File

• Log into warehouse A3 as testuser01

• Set inventory levels for SKU1001/Bin 47 and SKU1010/Bin 49 to 150

• Pick order ORD0034

• Pick order ORD0046

• Verify inventory level for SKU1001/Bin 47 is 140

• Verify inventory level for SKU1010/Bin 49 is 135

Test Scenario Test Scenario Input File

Login, test01, test01, warehouseA3

Adjust_Inventory, SKU1001, BIN047, 150

Adjust_Inventory, SKU1010, BIN049, 150

Pick_Order, ORD0034

Pick_Order, ORD0046

Verify_Inventory, SKU1001, BIN047, 140

Verify_Inventory, SKU1010, BIN049, 135


Translate BP’s to Keyword Translate BP’s to Keyword StepsSteps

Test Scenario Input File

Business Process Template


Test Driver Test Driver

Get BP

Instruction

Login, test01, test01, warehouse3

Load BP

Template

Map BP

Data to BP

Steps

Execute BP

Steps

VerifyState, Login Page, EXISTS

SetText, User ID, <user_id>

SetText, Password, <pw>

SetText, Warehouse, <warehouse>

ClickButton, Submit

VerifyState, Login Page, EXISTS

SetText, User ID, test01

SetText, Password, test01

SetText, Warehouse, warehouse3

ClickButton, Submit


Required Keyword Functions Required Keyword Functions ??

• All object/action pairs required to execute multi-platform test scenarios

• Additional hybrid keywords for common step sequences (e.g. login, navigation, etc.)

• Review existing manual test scenarios and get input from your customers – test suite users


Multi-PlatformMulti-PlatformKeyword FunctionsKeyword Functions

Function Set_Text, Parameters Object, Value

Get execution_platform

Select Case execution_platform

Case BROWSER

status = Set web Object = Value

Case RF

status = Set RF Object = Value

Get debug level

If debug level = TRACE

log execution detail

Return status

Multi-Platform Code

Mange multi-platform differences at the tool level if possible


Supporting FunctionsSupporting Functions

• Reading input files• Results logging, progress tracking, etc.• Test suite metrics (pass/fail, component

usage, etc.)• Error handling• Custom object interaction• Custom synchronization


Supporting FunctionsSupporting FunctionsResults LoggingResults Logging

• Start with the end in mind– Data– Level of detail– Format– Medium


Sample Summary Result Sample Summary Result FileFile

ACCUMULATED TOTALS PERCENTAGE------------------------------------------STEPS EXECUTED: 14TEST CASES EXECUTED: 5TEST CASES PASSED: 3 60.0TEST CASES FAILED: 1 20.0TEST CASES INCOMPLETE: 1 20.0SCRIPT/APPLICATION ERRORS: 3

TEST CASE RESULT

---------------------------------------------

TC0001.........................PASS

TC0002.........................PASS

TC0003.........................PASS

TC0004.........................INCOMPLETE

TC0005.........................FAIL

Summarized Test Case Results

Accumulated Execution Metrics


Sample Detail Log FileSample Detail Log File

BEGIN STEP: BP0001

START TIME: 14:46:48

S0001 Verify_State.............OK

S0002 Set_Text.................OK

S0003 Set_Text.................OK

S0004 Set_Text.................OK

S0005 Click_Button.............OK

BP0001.........................OK

END STEP: BP0001

END TIME: 14:48:04

BEGIN STEP: BP0002

START TIME: 14:48:06

S0001 TSL error handler invoked

S0001 Err: -10011, function: set_window

S0001 Verify_State............FAIL

One or more script errors were detected

BP0002........................FAIL

END STEP: BP0002

END TIME: 14:50:21Page 1 Page 2


Results Logging/Metrics Results Logging/Metrics FunctionsFunctions

Test

Driver

Keyword

Functions

CreateLogFiles

StepBegin

StepEnd

Writes step start time to log files

Writes step status, end time to log files, input file, metric totals & other mediums

Creates log files

LogMetrics Write accumulated test execution metrics to log files

Writes detailed step execution status to log files

StepDetail

Logs component usageLogUsage


…and Answers

Please direct any future questions to: [email protected]

Questions…Questions…

test automation frameworks final

Documents