c:\documents and settings\selvam.mc\my documents\automation testing process
DESCRIPTION
Automation Testing framework document for QTP, LoadRunnerTRANSCRIPT
Large - Scale Test Automation Production
Date :09th July’2010Place :Chennai Olympia Tech Park, India
Anbhu Selvam.MC…MS(CS).MBA.(Senior Software Quality Engineer, EDS. India)
Com
mon
Cha
lleng
es
Automation Technology–Multiplatform (cross-platform) support
•Ability to interface with multiple
platforms•Ease of extensibility to new platforms
•Reusability of test suites/scripts across
multiple platforms–Output/Object validation support
•High rate-of-change–Interface changes–Functionality changes
Lar
ge-s
cale
Aut
omat
ions
M
etho
d-C
entr
ic
Automation is Method-Centric—it is more
than about the technology; it is about the
effective application of technology
–Maintainability—handling high rate-of-
change–Reusability—ability to reuse common test
components without having to
program/code–Scalability—ability to automate large
volume of tests through reusability (and
team-based cost efficiency)
–Visibility•Tests auditable by management and non-
programmer staff•Productivity is structurally measurable
Eff
icie
ncy
is K
ey!
Efficiency Key Contributors
•Test designDefinition: A test (case) creation activity with intent to
serve one or both of the following
objectives: 1.Exposing bugs/errors (or show otherwise,
it works as intended), and
2.Optimizing maintainability and
scalability (especially for automation-
readiness).•Test automation (large-scale)
–Method-centric, not tool-centric
–Team-based, not individual coders
Aut
omat
ion
Stra
tegy
M
ust I
nclu
de
Efficient Practices–Methodology•High maintainability (low maintenance cost)
•High reusability–Technology•High extensibility•Object validation capability•People and process–High scalability (large-volume and team-based
methodology)–Just-in-time automation—test engineers are
skilled at test design and automation can be
done before code-complete and/or
concurrent to exploratory testing
–High manageability (high visibility)
"Agi
le"
Syst
em
Dev
elop
men
t
Establish objectives (global)•Develop tests•Develop the system
–in iterations, one subsystem at the time
–test-driven•ReleaseAgile ―Home Ground*‖
•Low mission-critical•Senior developers•Requirements change often
•Small number of developers•Culture that thrives on chaos
ReservedRole of AutomationEffective automation can allow tests to support agile system
development
•Automation should not dominate. Don't make it into an "agile automation" project
•A keyword driven method is, in my view, essential for successful test development w/ automation in agile development environment
–Automation is separate from test development
–Usually doesn’t need user interaction (only some with testers)
–Automation focuses on action-keywords, not on tests
Agile Test D
evelopment
ProcessDEVELOPMENT
Testers Stakeholders
AUTOMATION
System
Development
AUTOMATION
Level –II Development AUTOMATION
Reco
mm
endati
ons
Have a global test-design scheme
–Keep it short and simple: focused on breakdown of the
tests
–This global test design can (and usually should) be
revisited throughout the project cycle
•Separate (1) test objectives and (2) test cases
–Test objectives are easier to create/describe than test
cases•Use your stakeholders (specialists, power-users, etc.)
wisely–Focus the efforts on the relevant tests
–Don’t bother them more than needed
•Avoid involving them in low-and medium-level tests
–Observe various types of input:
•How does this work?•What do we need to test?
•What is interesting to test/How can we break the
system?•Keep automation separate
–Test developers are not consumed by automation
–Use keyword-driven approach (e.g., Action Based
Testing)
EAS: Extended Automation Support
Scalability Illustration3000
3000
6000
200
250
20 22
TESTS Application (High-level)Actions
Built-in/System(Low-level)Actions
Division of Work
ApproachWorkload Allocation
Automation Engineering
Test Case Production
Test Data Construction / Selection
Record & Playback
N/A Test Engineer N/A
Scripting Automation Engineer
Automation Engineer
N/A
Data-driven Automation Engineer
Automation Engineer
Test Engineer
Action-driven Automation Engineer
Test Engineer Test Engineer
Key
Tak
eaw
ays
1. Fully understand automation cost-of-ownership
2.Don’t underestimate the challenge of keeping maintenance
costs low3.You need to get efficient—optimize your volume of test to
exceed 50% coverage4.Efficiency is key, and it will come from excellent test
design and automation methodology (e.g., action-
driven), and a well architected framework technology
5.Minimize programming tests
6.High scalability comes from high reusability of common
―actions‖ and team-based staffing model
7.High maintainability comes from keeping maintenance
activities at the lowest level
8.High visibility in your automation program to give you
control and measurability, which ultimately leads to
manageability9.Practice just-in-time automation
10.Have a global test-design scheme that separate test
objectives from test cases; and automation from test
development
Automation Cost of Ownership&
The Need for Large-Volume Automation
Autom
ation Cost of
Ow
nership1.Technology = Tool Licensing + Development/Customization Cost
2.Production Cost = *One-time Cost + **Recurring Cost
*NEW–One-time Cost of Design, Implementation & Execution
–Test Case or Test Script Creation
»Test line creation
»Sequencing test logic or order of test lines (test steps)
»Testing and debugging test script
–Function or Keyword Creation
–Interface Capturing or Mapping
–Test Dataset Creation
**EXISTING—Recurring Cost Test Execution & Maintenance Cost
–Monitoring test suit execution
–Investigating and troubleshooting false negatives
–Maintaining Test Case or Test Script
»Test line modification
»Updating test logic or re-sequencing order of test lines (test steps)
»Testing and debugging updated test script
»Function or Keyword Modification
»Interface Recapturing or Remapping
»Test Dataset Modification
The C
ost of
Autom
ation
The Cost of Owning an
Automated Test
Cost of an Automated Test*=
**Cost of Ownership /
Volume (#) of Tests
* All test cases are not equal—a clear and structured definition of a test case is required.
Furthermore, this cost will be a moving target over time
** Cost of Ownership = Technology Cost + Production Cost
The N
eed for Large
Volum
e
Cost of an Automated Test*
=**Cost of Ownership/Volume
(#) of Tests
* All test cases are not equal—a clear and structured definition of a test case is required.
Furthermore, this cost will be a moving target over time
** Cost of Ownership = Technology Cost + Production Cost
You need to get efficient by optimizing the Volume
of Tests!
Rules of A
utomation
Built-to-L
ast1.Tests are treated as product asset, along with the source
code.
2.Tests, good or bad are dependent on the design.
3.Tests, manual or automated must be optimized for
visibility, reusability, scalability and maintainability.
4.Tests must be automation-ready.
5.Tests, if they are worth automating, should follow the
5% rule:
No more than 5% of all tests should be executed manually
No more than 5% of all efforts around testing should
involve automating the tests
No more than 5% of coded test scripts against non-coded
test scripts.
Action Based Testing Example
Exa
mpl
e w
ith T
est
Arc
hite
ct™
(TA
)
AT Computer
AT Agent
TCP/IP
Wireless
AT Agent
AT Agent
AT Agent
Mobile API
Res
erve
d Te
st
Exa
mpl
e
Act
ion
Def
initi
on f
or
―se
nd S
MS
mes
sage
and
ch
eck‖
act
ion
TA Connects to Window Mobile via TCP/IPTCP/IP
Wireless
Inte
rfac
e D
efin
ition
for
―
mes
sage
edi
tor‖
Ent
ity
Questions ?