Test Automation Framework Designs

Designing standardized frameworks instead of simple scripting

By Martin Lienhard

The Great Divide

• There has been a divide separating QA and Dev due in large part by 3rd party tool vendors and their proprietary test tools

• Open source tools have helped significantly to bridge that divide, so that we can work together to create better software

• Most QAs have the bigger picture of what they want to test, and how to do it, but have trouble building robust automation frameworks, because they lack the technical skills.

• Most Developers focus on testing code in isolation, but lack the testing experience from a system-wide or end-to-end perspective, so they have trouble building a test automation framework that suits the needs of QA.

1st Phase of Test Progression

• Most QA start out creating test scripts by using record & playback tools

• Lack conditional logic and looping• Contain static data, UI locators, URLs, etc.• Cannot be executed iteratively• Cannot be reusable modules• Should not be chained together as dependencies• Lack configuration and global properties• Cannot abstract away from the test tool or test runner

Record & Playback Test Scripts

2nd Phase of Test Progression

• QA usually improves their tests by adding data-driven techniques

• Create variables and arrays to store data• Create parameterized functions or methods to store data• Add looping to recorded scripts

3rd Phase of Test Progression

• QA usually improves their tests by creating a business function library

• Modular, reusable test libraries of functions• Create file readers that pull in parameters from text files, CSV files,

etc.• Abstract out the data from the tests by externalizing data• Create parameterized or iterative tests that are data-driven• Abstract out the UI locators from the tests by externalizing UI maps• Create global properties and configuration files• A vast array of disparate data pools and UI maps are created,

which usually become unmanageable after a couple of years

4th Phase of Test Progression

• QA usually improves their tests by creating a keyword-driven library:

• Create ODBC functions that read data from spreadsheets and databases

• Create spreadsheets of keywords that represent business domain language– Low-level keywords describe test operations– High-level keywords describe business language– Intermediate-level keywords map business functions to test

operations– Tests become readable by business users and non-technical

QA

…continued

• Create spreadsheets of data that drive the tests• Query data from the application database as input and

verification points for tests• A hierarchy of abstracted keyword spreadsheets are created• Now non-technical BAs and QAs are creating automated tests• A vast array of disparate keyword spreadsheets are created,

which usually become unmanageable after a couple of years• Nearly all 3rd party vendor test tools live in this area today– Most of these tools use legacy scripting languages, which

are not compatible with today’s technology

5th Phase of Test Progression

• QAs become more technical and improve their tests by using object oriented testing techniques:

• Objects are designed and created to further modularize and reuse code with very low maintenance

• Test code is stored in objects– The page object design is used– Component object designs are used

• Test data are stored in objects• Configuration and properties are stored in objects• Calls are made to the application and developer written APIs

to make tests “smarter”

…continued

• Continuous integration environments are utilized to run tests• QA moves into white box testing to drive quality deeper into

the application– UI tests are ran “headless” to make them execute faster

• QA expands the test framework to include multiple test tools and test runners– Databases, web services, and APIs are created to reuse

code and data across multiple test tools– Test tools and 3rd party utilities can be upgraded to newer

versions with enhanced functionality with minor impact to the test framework

…continued

• Developers can now run the tests, and contribute to the overall test suites

• More 3rd party vendors are rewriting their test tools to include object oriented languages to accommodate these technical QAs– Due to the cost of the tools and their proprietary nature, it

is difficult to place these tools into a continuous integration environment to share with developers

What is a test automation framework?

• A test automation framework is an application that allows you to write a series of tests without worrying about the constraints or limitations of the underlying test tools

Benefits of building a Test Framework

• Tool agnostic– Abstracts away low level commands– Utilize many test tools• Test tools such as Selenium, WebDriver, etc.• Test runner such as unit test frameworks• Other common utilities

– Perform multiple levels and types of testing• Functional, regression, load, performance, unit,

integration, etc.

…continued

• Generic test API– Common codebase and database– Exception and error handling– Modularized, reusable, and maintainable code and data– Standardized test idioms

• Test configuration– Configurable test suites– Global test properties

…continued

• Automatic test versioning– No need to get a specific version from the source control

repository • Always get the latest version

– Eliminates deployment to multiple machines for testing different versions of an application• Deploy to a single server machine

…continued

• Multi-threading– Run tests in parallel– Runs on test machines in the grid or cloud– Test across multiple environments and application

versions simultaneously• Continuous integration– Run in a continuous integration environment– Share tests and collaborate with other teams

Test Framework Architecture

Test Suite

• A collection of tests to run• Have test parameters– i.e.: application, environment, version, etc.

• Have test groups– i.e.: regression, build, etc.

Test Suite example

Test Configuration

• Any configurable test item that is separate from the test suite– i.e.: web site URLs, database URLs, usernames, passwords, lab

machine configurations, etc.

• Separated from the rest of the framework for information security and regulatory compliance

Test Configuration example

Test Properties

• Are thread specific• Have global scope and can be accessed by any class method

Test Properties example

Test Data Repository

• Abstracts away changes made only to test data• Data can be stored in databases and files such as XML, JSON,

XLS, TXT, CSV, etc.• Data can be retrieved from data sources such as databases,

web services, APIs, EDI, etc.• Metadata can be used to describe additional information

about the test data such as versions, application names, required fields, invalid formats, etc.

• Share data across multiple screens, applications, and tools

Test Data example

Parameterized Test example

JUnit4 TestNG

Test Parameters

• Uses a hash map of key-value pairs– Very useful for both functional and load testing

• Hash maps eliminate the need for– Creating fixed method parameters that change frequently– Changing data access objects that change frequently

• Hash maps can be easily populated from any data source• Retrieve the test data and process the format for input to

various test tools• Handles version differences between test data

Test Parameters example

User Interface (UI) Mapping

• Abstracts away changes made only to UI object locators• UI object identifiers can be stored in databases and files such

as XML, JSON, XLS, TXT, CSV, etc.• Metadata can be used to describe additional information

about the UI objects such as versions, application names, form fields, etc.

• Share UI identifiers across multiple screens, applications, and tools

UI Map example

Test Controls

• Retrieve the UI locators and process the format for input to various test tools

• Handles version differences between UI locators• Develop custom controls to mimic UI objects• Self test verification

Test Control Interface example

Test Control Class example

Test Components

• A class object that represents a piece of a form, page, or screen

• Groups of common objects that can be found across more than one page

• Standardized component verification tests

Test Components example

Test Forms

• A class object that represents a piece a form• Groups of components and controls that mimic the UI form• Standardized form verification tests

Test Form Interface example

Test Form Class example

…continued

Test Page or Screen

• A class object that represents a web page or screen – commonly referred to as “page objects”

• Groups of forms, components, and controls that mimic the UI page or screen

• Self test verification

Test Page Class example

Instances when NOT to return a new Page Object for each Page

• Submitting a page can return one of many different pages, or versions of pages, which would be represented by many different classes– Page A could return B, C, or D– Page A could return B v1.0, B v2.0, or Cv1.0, C v2.0, or D

v1.0, D v2.0, etc.– This means that you would need to do the following:• Create multiple methods with similar names that

return different page objects– Page A.submitB() returns B– Page A.submitC() returns C

…continued

• Return an page object of an inherited type, and then cast the type of the page in the script– Page A.submit() returns page X– Pages B and C inherit from page X– Script checks the type of X and casts to B or C

– This can get very ugly in a script…• Submitting a page may initiate an AJAX call, which can rewrite

the same page– Page A could display an AJAX (modal) dialog– Page A could display some other AJAX widget– Page A could be rewritten to look like page B

…continued• If the page flow is not what is being tested, then an unexpected

page or AJAX call could fail and kill the test– Test case:

• We are trying to test the functionality of page Z• The primary page flow is as follows:

– A to B, B to C, C to D, and D to Z• The alternative page flows could also correctly occur, but

are unpredictable base on many unknown variables:– A to H, H to L, L to M, and M to Z– A to R, R to S, and S to Z– A to X, X to Y, and Y to Z

…continued

– What if page A unexpectedly returned page H, R, or X instead of page B?

– If page H, R, or X verified itself and asserted a failure, then how would the test ever reach page Z?

JavaScript & HTML DOM

• There are many instances when we need the test framework to mimic a human reading the content of a page and then making a decision on what action to perform next– Verifying the list of items on a search results page– Selecting items to purchase from a list– Selecting elements to edit, update, or delete

• Scenarios like these usually occur as a result of some previous action, and display content generated dynamically to the user

• The test framework may have to query an application database or web service to provide the input to trigger the dynamic page content

…continued

• The test framework will then need to interrogate the page to determine if the expected elements, attributes, or content are present, and then make a decision on whether to perform an action on some element

• JavaScript can be used to inspect the HTML DOM to find these elements and their attributes and content

• JavaScript expressions can be evaluated by the test tool, and then return some result that the test framework can use to make a decision

• Checking if an element is present or getting a value will not work on a dynamically generated element

JS DOM example

HTML JavaScript

Test Application Container

• A class object that represents the application, and contains the major page and component objects by version

• It provides the test services available to the test scripts• Is thread specific

Container example

Interface Class

Parallel Environment Tracks with Different Build Versions

• Parallel development tracks competing to deploy to production

• Run automated build verification tests• Run automated functional and regression tests• Run automated user acceptance tests• Run automated tests for production patches and fixes• Run automated sanity tests against production post-

deployment• Run automated tests across versions for production rollback

strategy

Parallel Environment Tracks with Different Build Versions

Test Scripts

• Test scripts can be written using standard unit test frameworks

• Classes and methods are written using domain language• Script details and complexity are abstracted to the framework

layers• Scripts are shorter and easier to read

Test Script Class

Test Reports and Logs

• Reports can be generated from– The unit test framework– External reporting tools– Custom report utilities can be written

• Test execution activity can be written to standard logging utilities

• Test failures, errors, and exceptions can be written to the

reports and logs

Q&A

Thank you for your time!

Martin Lienhardlienhardtestautomation@gmail.com