c:\documents and settings\selvam.mc\my documents\automation testing process

25
Large - Scale Test Automation Production Date :09 th July’2010 Place :Chennai Olympia Tech Park, India Anbhu Selvam.MC…MS(CS).MBA. (Senior Software Quality Engineer, EDS. India)

Upload: selvam-mc

Post on 22-Nov-2014

945 views

Category:

Technology


2 download

DESCRIPTION

Automation Testing framework document for QTP, LoadRunner

TRANSCRIPT

Page 1: C:\documents and settings\selvam.mc\my documents\automation testing process

Large - Scale Test Automation Production

Date :09th July’2010Place :Chennai Olympia Tech Park, India

Anbhu Selvam.MC…MS(CS).MBA.(Senior Software Quality Engineer, EDS. India)

Page 2: C:\documents and settings\selvam.mc\my documents\automation testing process

Com

mon

Cha

lleng

es

Automation Technology–Multiplatform (cross-platform) support

•Ability to interface with multiple

platforms•Ease of extensibility to new platforms

•Reusability of test suites/scripts across

multiple platforms–Output/Object validation support

•High rate-of-change–Interface changes–Functionality changes

Page 3: C:\documents and settings\selvam.mc\my documents\automation testing process

Lar

ge-s

cale

Aut

omat

ions

M

etho

d-C

entr

ic

Automation is Method-Centric—it is more

than about the technology; it is about the

effective application of technology

–Maintainability—handling high rate-of-

change–Reusability—ability to reuse common test

components without having to

program/code–Scalability—ability to automate large

volume of tests through reusability (and

team-based cost efficiency)

–Visibility•Tests auditable by management and non-

programmer staff•Productivity is structurally measurable

Page 4: C:\documents and settings\selvam.mc\my documents\automation testing process

Eff

icie

ncy

is K

ey!

Efficiency Key Contributors

•Test designDefinition: A test (case) creation activity with intent to

serve one or both of the following

objectives: 1.Exposing bugs/errors (or show otherwise,

it works as intended), and

2.Optimizing maintainability and

scalability (especially for automation-

readiness).•Test automation (large-scale)

–Method-centric, not tool-centric

–Team-based, not individual coders

Page 5: C:\documents and settings\selvam.mc\my documents\automation testing process

Aut

omat

ion

Stra

tegy

M

ust I

nclu

de

Efficient Practices–Methodology•High maintainability (low maintenance cost)

•High reusability–Technology•High extensibility•Object validation capability•People and process–High scalability (large-volume and team-based

methodology)–Just-in-time automation—test engineers are

skilled at test design and automation can be

done before code-complete and/or

concurrent to exploratory testing

–High manageability (high visibility)

Page 6: C:\documents and settings\selvam.mc\my documents\automation testing process

"Agi

le"

Syst

em

Dev

elop

men

t

Establish objectives (global)•Develop tests•Develop the system

–in iterations, one subsystem at the time

–test-driven•ReleaseAgile ―Home Ground*‖

•Low mission-critical•Senior developers•Requirements change often

•Small number of developers•Culture that thrives on chaos

Page 7: C:\documents and settings\selvam.mc\my documents\automation testing process

ReservedRole of AutomationEffective automation can allow tests to support agile system

development

•Automation should not dominate. Don't make it into an "agile automation" project

•A keyword driven method is, in my view, essential for successful test development w/ automation in agile development environment

–Automation is separate from test development

–Usually doesn’t need user interaction (only some with testers)

–Automation focuses on action-keywords, not on tests

Page 8: C:\documents and settings\selvam.mc\my documents\automation testing process

Agile Test D

evelopment

ProcessDEVELOPMENT

Testers Stakeholders

AUTOMATION

System

Development

AUTOMATION

Level –II Development AUTOMATION

Page 9: C:\documents and settings\selvam.mc\my documents\automation testing process

Reco

mm

endati

ons

Have a global test-design scheme

–Keep it short and simple: focused on breakdown of the

tests

–This global test design can (and usually should) be

revisited throughout the project cycle

•Separate (1) test objectives and (2) test cases

–Test objectives are easier to create/describe than test

cases•Use your stakeholders (specialists, power-users, etc.)

wisely–Focus the efforts on the relevant tests

–Don’t bother them more than needed

•Avoid involving them in low-and medium-level tests

–Observe various types of input:

•How does this work?•What do we need to test?

•What is interesting to test/How can we break the

system?•Keep automation separate

–Test developers are not consumed by automation

–Use keyword-driven approach (e.g., Action Based

Testing)

Page 10: C:\documents and settings\selvam.mc\my documents\automation testing process

EAS: Extended Automation Support

Page 11: C:\documents and settings\selvam.mc\my documents\automation testing process

Scalability Illustration3000

3000

6000

200

250

20 22

TESTS Application (High-level)Actions

Built-in/System(Low-level)Actions

Page 12: C:\documents and settings\selvam.mc\my documents\automation testing process

Division of Work

ApproachWorkload Allocation

Automation Engineering

Test Case Production

Test Data Construction / Selection

Record & Playback

N/A Test Engineer N/A

Scripting Automation Engineer

Automation Engineer

N/A

Data-driven Automation Engineer

Automation Engineer

Test Engineer

Action-driven Automation Engineer

Test Engineer Test Engineer

Page 13: C:\documents and settings\selvam.mc\my documents\automation testing process

Key

Tak

eaw

ays

1. Fully understand automation cost-of-ownership

2.Don’t underestimate the challenge of keeping maintenance

costs low3.You need to get efficient—optimize your volume of test to

exceed 50% coverage4.Efficiency is key, and it will come from excellent test

design and automation methodology (e.g., action-

driven), and a well architected framework technology

5.Minimize programming tests

6.High scalability comes from high reusability of common

―actions‖ and team-based staffing model

7.High maintainability comes from keeping maintenance

activities at the lowest level

8.High visibility in your automation program to give you

control and measurability, which ultimately leads to

manageability9.Practice just-in-time automation

10.Have a global test-design scheme that separate test

objectives from test cases; and automation from test

development

Page 14: C:\documents and settings\selvam.mc\my documents\automation testing process

Automation Cost of Ownership&

The Need for Large-Volume Automation

Page 15: C:\documents and settings\selvam.mc\my documents\automation testing process

Autom

ation Cost of

Ow

nership1.Technology = Tool Licensing + Development/Customization Cost

2.Production Cost = *One-time Cost + **Recurring Cost

*NEW–One-time Cost of Design, Implementation & Execution

–Test Case or Test Script Creation

»Test line creation

»Sequencing test logic or order of test lines (test steps)

»Testing and debugging test script

–Function or Keyword Creation

–Interface Capturing or Mapping

–Test Dataset Creation

**EXISTING—Recurring Cost Test Execution & Maintenance Cost

–Monitoring test suit execution

–Investigating and troubleshooting false negatives

–Maintaining Test Case or Test Script

»Test line modification

»Updating test logic or re-sequencing order of test lines (test steps)

»Testing and debugging updated test script

»Function or Keyword Modification

»Interface Recapturing or Remapping

»Test Dataset Modification

Page 16: C:\documents and settings\selvam.mc\my documents\automation testing process

The C

ost of

Autom

ation

The Cost of Owning an

Automated Test

Cost of an Automated Test*=

**Cost of Ownership /

Volume (#) of Tests

* All test cases are not equal—a clear and structured definition of a test case is required.

Furthermore, this cost will be a moving target over time

** Cost of Ownership = Technology Cost + Production Cost

Page 17: C:\documents and settings\selvam.mc\my documents\automation testing process

The N

eed for Large

Volum

e

Cost of an Automated Test*

=**Cost of Ownership/Volume

(#) of Tests

* All test cases are not equal—a clear and structured definition of a test case is required.

Furthermore, this cost will be a moving target over time

** Cost of Ownership = Technology Cost + Production Cost

You need to get efficient by optimizing the Volume

of Tests!

Page 18: C:\documents and settings\selvam.mc\my documents\automation testing process

Rules of A

utomation

Built-to-L

ast1.Tests are treated as product asset, along with the source

code.

2.Tests, good or bad are dependent on the design.

3.Tests, manual or automated must be optimized for

visibility, reusability, scalability and maintainability.

4.Tests must be automation-ready.

5.Tests, if they are worth automating, should follow the

5% rule:

No more than 5% of all tests should be executed manually

No more than 5% of all efforts around testing should

involve automating the tests

No more than 5% of coded test scripts against non-coded

test scripts.

Page 19: C:\documents and settings\selvam.mc\my documents\automation testing process

Action Based Testing Example

Page 20: C:\documents and settings\selvam.mc\my documents\automation testing process

Exa

mpl

e w

ith T

est

Arc

hite

ct™

(TA

)

AT Computer

AT Agent

TCP/IP

Wireless

AT Agent

AT Agent

AT Agent

Mobile API

Page 21: C:\documents and settings\selvam.mc\my documents\automation testing process

Res

erve

d Te

st

Exa

mpl

e

Page 22: C:\documents and settings\selvam.mc\my documents\automation testing process

Act

ion

Def

initi

on f

or

―se

nd S

MS

mes

sage

and

ch

eck‖

act

ion

Page 23: C:\documents and settings\selvam.mc\my documents\automation testing process

TA Connects to Window Mobile via TCP/IPTCP/IP

Wireless

Page 24: C:\documents and settings\selvam.mc\my documents\automation testing process

Inte

rfac

e D

efin

ition

for

mes

sage

edi

tor‖

Ent

ity

Page 25: C:\documents and settings\selvam.mc\my documents\automation testing process

Questions ?