building an automation framework around open source technologies charles thangamuthu 20 th november...

Building an Automation Framework around Open Source Technologies

Charles Thangamuthu20th November 2010

Disclaimer – This is a personal case study and does not represent PTC’s view on the topic

2

Thanks SiliconIndia!

3

Before we step in….

What this presentation is not

What this presentation is

Does not say open source technology is better than commercial testing tools

Does not say the components talked about are the best in their class

Does not talk about a PTC proprietary or certified framework

Does not claim all your automation woes are solved

An example of how open source tools can be used to form an automation framework

To say there are solutions readily available for many of your problems – don’t have to reinvent the wheel

More of a case study

4

Topics

Automation Framework– Definition– Requirements– Functions

Generic Architecture

Framework components– Automation Tool– Version Control System– Continuous Integration– Test Runner– Data Manager

Advantages– Flexibility– Scalability

Benefits

5

Automation Framework is not a single tool or process but a collection of tools and processes working together to support automated testing of an application

Is an integrated environment that sets the rules, assumptions, concepts and tools to aid automated testing

It integrates function libraries, test data, object details and various reusable modules

These components act as building blocks which can be assembled to represent a business process

Test Automation Framework

Definition - What is it?

6

Data driven capability

Customizable result reporting

Ease of debugging and logging

Version control of tests

Execute unattended runs

Repeat tests as needed

Extensible and Customizable

Status notification

Distributed test execution


Requirements – What do we need?

Automation Framework

Platform and OS independence

7


Functions – What does it do?

8


Architecture – What does it consist of?

Application under test (AUT)

Data Manager

Test Runner

Integration (CI)

Version Controller

Automation Tool

Test artifacts

Test ScriptsLibrary Functions

Config FilesData FilesLoggers

9

Automation Tool

Automation Tool Selection - Considerations

• Operating system support• Resource requirements

Technical Information

Features & Usability

Event Capturing

Scripting Support

In-built Functions

• GUI abstraction features • GUI spying features• Capturing comfort

• Installation• Test Execution• Result reporting

• Supported languages• Object orientation• Compiler / Interpreter

• Browser handling• XML handling• Interaction with File System• Wait /Sleep functions

10

Automation Tool

Automation Tool Selection – Considerations continued…

• Editor features• Debugger• Version management

Integrated Development Environment

Data Driven Testing

Result Reporting

Documentation

Community Support

• Output formats• Flexibility for customization

• Supported file formats

• Ease of use• Users manual• Online help

• FAQs• WIKIs• Forums• Technical support

11

Automation Tool

Focus on Selenium

Selenium

12

Automation Tool

Why Selenium?

13

Automation Tool









Status notification


Fulfills the following requirements

Win Runner

HP Quick Test Professional

IBM Rational Functional Tester

Parasoft SOATest

Rational Robot

Silk Test

QFTest

Test Partner

Other popular Automation Tools


14

Version Control System

Git

Purpose

Is a way of ensuring the evolution of the tests corresponds to the evolution of the application features

Allows concurrent updates to the same test

Maintains the history and baselines of the automated test

Is a free, fast and open source Distributed Revision Control System

Can work offline

Everyone has a local sandbox

Branching and Merging are fast and easy

15

Fulfills the following requirement

Version Control System

Clearcase

Subversion

Fossil

Aegis

Bonsai

Concurrent Version System (CVS)

Revision Control System (RCS)

Other Version Control Systems









Status notification



16

Is a Free, Open Source CI server written in Java

Monitors executions of repeated jobs like– Building a software project– Executing automated tests– Sending e-mail notifications

Can execute – Apache Ant and Apache Maven based projects– External jobs on a remote machine

Supports SCM tools including CVS, Subversion, Git and Clearcase

Easy to install and configure

Supports plug-ins with RSS/E-mail/IM Integration

Continuous Integration System

Purpose HUDSON

Gives early warning of broken or incompatible code and conflicting changes

Does immediate unit testing of all changes in code

Initiates and monitors repeated jobs like test automation runs at predetermined schedule or event

17

Fulfills the following requirements

Continuous Integration System

CruiseControl

CruiseControl.NET

CruiseControl.rb

Apache Gump

Apache Continuum

BuildBot

CABIE

Other CI Systems









Status notification



18

Is an automation framework component that provides a simple workflow for execution of tests, result collection, re-running failures, and comprehensive reporting

Is an application built in-house on top of JUnit

Some of its features are – Compile APIs/Tests as needed– Generate Test Suite as per requirement– Selective execution of Tests in unattended mode– Re-runs the failed tests for the number of times specified– Creates comprehensive report with total number of tests passed and failed, time taken by each

test, reasons for failure and logs– Localization Support– Even Video capture!

Test Runner

Features of RADAR

19

An essential feature for repeating tests and testing in parallel

Is a set of APIs (developed in-house) allowing users to save and restore data in the database and LDAP

Creates multiple identical database schema clones from a Master and assigns them to multiple AUT installations

Data Manager

Features of ROSLIN

Virtualization could be used as well

20

Fulfill the following requirements

Test Runner & Data Manager









Status notification



21


Example

AUT

ROSLIN

Test artifacts

JUnit testsScriptsXMLs

Java APIsAnnotations

Log4j, Dom4j

Version Mgmt

GITAutomation

Tool

SELENIUM RC

Data Manager

Test Runner

RADAR

Cont. Integrator

HUDSON

22

Advantages - Flexibility

Framework design considerations

What is a Solvent layer?

Reduce maintenance efforts to a minimum

Provide a means to help people with lesser programming expertise to create automated tests

A common infrastructure for the Development and QA teams to build and execute automated tests

Solvents are a set of API that help create Selenium tests, without executing Selenium commands directly

Solvents package the logic and Selenium commands required to perform various user actions into simple, reusable classes

If you are wondering what’s in the name - Solvents reduce the amount of Selenium required to create a test (sometimes to none) thereby diluting Selenium, hence the name ‘Solvent’

23


Layout with Solvent Layer

Additional Layer

24

It aids reusability of code

They can be ‘wired’ together to form test scenarios

Changes need to be made at a single place and not in multiple tests

Makes the test / script more readable


Advantages of the additional layer

POINT TO NOTE – The automation framework is flexible enough to accommodate an additional layer as needed

25

The framework could be integrated with tools that help determine code coverage like Corbertura, EMMA and Jtest

– Assess if the testing done is sufficient– Analyze if the testing focus is as planned

Advantages - Scalability

Other possibilities

Mechanism for code review like Gerrit could be integrated with the framework

– Enforce standards, conventions and best practices for creating tests– Inspect the overall quality of the tests

26

POINT TO NOTE – The automation framework can be enhanced by adding additional capabilities

Advantages - Scalability

Enhanced

AUT

ROSLIN

Test artifacts

XMLsJava APIs

AnnotationsLog4j, Dom4j

EclipseAnt

JUnit Tests

Version Mgmt

GITAutomation

Tool

SELENIUM RC

Data Manager

Test Runner

RADAR

Cont. Integrator

HUDSON

Gerrit

EMMA

Newly Added

Newly Added

27

A consistent, flexible and scalable framework will help automation play a role at every stage in the software development process

Benefits

Software Development Ecosystem

Not just test automation, some aspect of process automation is also in-built

28

Maintainability – Significantly reduces the test maintenance effort

Reusability – Due to modularity of test cases and library functions

Manageability – Effective test design, execution, and traceability

Accessibility – To design, develop & modify tests whilst executing

Availability – Scheduled execution can run unattended on a 24/7 basis

Reliability – Due to advanced error handling and scenario recovery

Flexibility – Framework independent of AUT or environment

Measurability – Customizable reporting of test results ensure quality

Benefits

Key Advantages

building an automation framework around open source technologies charles thangamuthu 20 th november...

Documents