automation testing strategy, approach & planning

IBM Global Services – Testing Competency

IBM Confidential © Copyright IBM Corporation 200501/23/2005

Strategy, Approach & Planning

2

IBM Global ServicesIBM Global Services – Testing Competency

© Copyright IBM Corporation 2005IBM Confidential2

Agenda

Introduction

Effective Automation

Fundamental Approach

Real Facts & Think Factor

Considerations for Automation

Automation Start-up Strategies to Avoid

Guidelines & Approach

Automation Methodologies

Overview on Automation Tools

References

3



Introduction

GUI automation is always viewed as the SILVER BULLET, which will save money, time and resources

Test automation raises our hopes, yet often frustrates & disappoints us

Promises to deliver us from a tough situation, yet can create as many problems as it solves

"Embarking on a successful automation test project is similar to embarking on a successful software development project“

4



Effective Automation

Reduced utilization of manual resources

More efficient use of scarce hardware resources

Reduced test cycle time

Reduced cost of testing, after initial cost of implementation

Improved reliability of the testing

Repeatable tests, thus overcoming boredom-induced missed bugs

Improved test coverage, since testing may be done more frequently

Development of programming skills

May allow the use of less skilled staff to do the actual testing

5



Fundamental Approach

Understand the need / requirement for automation

Point out pitfalls to avoid

Promote an evolving implementation

Begin simple and increase in complexity as the team's skills increase

"Learning from others is the most effective way to grow your automation expertise"

6



Real Facts & Think factors

Real Fact

Automation is initially expensive and it may not do a better job than manual testing in some situations

Think Factors

1. How much will automating and running a test cost, versus running it manually?

2. Automated test has a limited lifetime in which it must recoup the additional cost. Will this test be reusable long enough? What factors will prevent it from being reusable?

3. The automated test will find bugs it was designed to test for and in addition it will find bugs that are not directly related to the test. What is the likelihood that this test will find additional bugs?

7



Real Facts & Think factors

Real Fact

Automation is initially expensive and it may not do a better job than manual testing in some situations

Think Factors

1. How much will automating and running a test cost, versus running it manually?

2. Automated test has a limited lifetime in which it must recoup the additional cost. Will this test be reusable long enough? What factors will prevent it from being reusable?

3. The automated test will find bugs it was designed to test for and in addition it will find bugs that are not directly related to the test. What is the likelihood that this test will find additional bugs?

8



Considerations for Automation

Test cases those are long running, repetitive, and non-subjective

Projects / Products that have a working, relatively stable GUI

Projects / Products that will span multiple releases.

9



Guidelines & Approach

Record / Playback initially using the Testing tool’s wizard or toolbar will be easier but hard to maintain because they represent not only WHAT needs to be tested, but HOW it is to be tested

Guideline: 1

Once familiarity in scripting is achieved, continue using the scripting language of an automation tool to provide your own implementation logic (framework) that is separate from your test case scripts

Requires more investment upfront, but the code will be less expensive to maintain over the life span of multiple product releases

Example:

Imagine having recorded test case scripts for a banking application that initially allowed the user to simply click "OK" to commit a sensitive transaction, then the application developers changed the navigational logic requiring the user to type "yes" to confirm these transactions. Every recorded test case that has hard coded "OK" clicks relating to the sensitive transaction commits would require maintenance.

10



Guidelines & Approach (Contd..)

Automation code without any Discipline - Tendency to jump in and start developing new test scripts without collaborating with other team's members and without establishing a plan, structure, and/or standards is natural. However, in the long term this undisciplined approach is not productive. Any test case scripts and frameworks that are produced in this manner tend to lack such characteristics as: maintainability, reusability, readability, flexibility, reliability, effectiveness, and efficiency

Guideline: 2

Establish complete Automation plan, Structure, and Standards

“Quality automation effort should follow a software development process”

11




Guideline: 3

“Separating the What from the How”

An approach to separate what needs to be tested from how it is to be tested. To accomplish this separation, guideline would suggest to have the test script writers write the test scripts to identify what needs to be tested and automation developers write a framework to handle the implementation (i.e., the navigation logic, window processing logic, and verification logic) of your tests.

Framework - "a supporting or enclosing structure”

12



Guideline: 4

Use the 6 phases for implementing the best course of Automated Testing action

- Evaluation

- Planning

- Design

- Implementation

- Deployment

- Review

Iteration Cycle – Plan for future changes


13




Guideline: 5

A. Define Automation Goals & Prioritize

At every point in the specification process, you can check the "testability" of the requirement or specification by ensuring it is S.M.A.R.T.

S - Specific

M - Measurable

A - Attainable or Achievable

R - Realistic

T - Timely

B. Define Automation Strategy

- GUI - Roundtrip / Segment, Define objects, Synchronization

- Types of Testing

- Strategy Considerations

1. Partial automation

2. Time cost

3. Testing delay

4. Inertia

5. Risk of missed bugs

14




Guideline: 6

Define Users and Skill Level Required / Desired

- Test Script Writers

- Automation Developers

Guideline: 7

Resource Planning / Source Code Management

- Hardware

- Software

- People

15




Guideline: 8

Define Design Goals & Prioritize

- Maintainability

- Structured

- Well Documented

- Reviewed

- Framework Specific Goals

- Modularity

- Reusability

- Usability

- Scalability

- Reusability

- Portability

- Robustness

- Reliability

- Efficiency

- Flexibility

16




Guideline: 9

Determine the Automation Interface Architecture

- Data Driven Interface

- Programming Interface

Guideline: 10

Determine the Automation Framework Architecture

- Error Recovery

- Support of Custom Controls

- Product Testability Issues

- NLS – Localization

Guideline: 11

Define Standards – Naming & Code Conventions

17




Components of Automated Test System

Test Script Writers

Automation Developers

Test Scripts

“WHAT”

Interface

Framework

“HOW”

Documents

Framework Usage Notes

Guidelines

Naming Conventions

18



Automation Methodologies

Functional Decomposition Method

This methodology is to reduce all the test cases to their most fundamental tasks, and write User-Defined Functions, Business Function Scripts, and "Sub-routine" or "Utility" Scripts which perform these tasks independently of one another

Key-Word Driven or Test Plan Driven Method

This method uses the actual Test Case document developed by the tester using a spreadsheet containing special "Key-Words". This method

preserves most of the advantages of the "Functional Decomposition" method, while eliminating most of the disadvantages. In this method, the entire process is data-driven, including functionality.

The Keyword controls the entire processing

19



Overview on Automation Tools

Most of the Automated Tools in the market remain as an integrated suite for Test Documentations, Scripting, Executions & Defect Tracking

Automation tool does exist for specific testing needs such as

Performance Testing

Code Coverage

Memory Leak Detection

Automation Tools available in the market which are more on focus today includes

Rational Tools – Robot, Test Manager, ClearQuest

Mercury Interactive – WinRunner, LoadRunner, QTP, Astra Load Test

Segue – Silk Performer

20



References

Web Sites for Additional Resources

1. http://www.io.com/~wazmo/qa/ Software Testing Hotlist, maintained by Bret Pettichord, has links to many of the articles and papers mentioned in this document

2. http://www.softwareqatest.com/index.html Software QA and Testing Resource Center, maintained by Rick Hower, links

3. http://qaforums.com/ QA Forums, maintained by individual moderators, links to forums on Quality Assurance, Tools, Consulting & Employment

Bibliography

1. Dustin, Elfriede, Rashka, Jeff, and Paul, John (1999) Automated Software Testing: Introduction, Management and Performance. Addison-Wesley Longman, Inc. Reading, Massachusetts

2. Fewster, Mark and Graham, Dorothy (1999) Software Test Automation: Effective use of test execution tools. Addison-Wesley: ACM Press, New York

3. Kaner, Cem, Falk, Jack, and Nguyen, Hung Quoc (1999) Testing Computer Software. Wiley Computer Publishing, New York

automation testing strategy, approach & planning

Education

resources test automation

test cost

automation test cases

automation code

new test scripts

automation expertise

recorded test case scripts

factors real fact automation