automation concepts

Key to successful Automation

© Nishant Worah 2009 2

Murphy’s Rules for Testers

Never assume anything.

There is no such thing as stupid question.

Record everything thoroughly.

If you don’t understand, it’s not your fault.


What is Testing ?

Testing is a process of executing the program with intent to finding the bugs

Testing is the set of processes which ensure that the functionality and performance planned through design has been delivered through the system.

Testing is required to improve the quality of the product before handling over the product to the customer.


Type of testing

Manual Testing

Automated testing


Manual Testing

Traditional form of testing Slow , tedious , boring Costly , more time & effort Difficult to Manage Reliable ? Flexible (A lot have been said about the Slow

and tediousness of the manual testing but it’s the flexibility that it provides that overshadows all other points. With the frequent change in the software its not viable to change the test script regularly. So when changes are brought into the software regularly, the manual testing is the best option).


Automation and its Need

Automation is done to reduce repeated and redundant testing. Automation maintains a single standard of quality for your application over time — across releases, platforms, and networks.

It is used to replace or supplement manual testing with a suite of test programs. Benefits include increased software quality, improved time and resource efforts , repeatable test procedures, and reduced testing costs.


Myths about Automated testing

Find more bugs:- In most places the test cases are written by test engineers who are familiar with the application they are testing. From test cases to test scripts, automation does not add anything in the process to find more bugs. The test scripts will work only as good as the test cases when comes to finding bugs.

Eliminate or reduce manual testers:-For the stable application we can say this, but for an application that required any change this is totally a myth. As said earlier that the test scripts are as good as the test cases itself so for the new test cases manual tester will definitely be required.

Automated Testing doesn't mean Automatic Testing …. It means Computer aided testing


Advantages of Automated testing

Ease the testing process Quicken the testing, by reducing the time and effort involved in manual

testing . A typical automated test suite will run in less than 24 hours, without any human intervention required. For a sophisticated product, manual testing may require dozens of staff months to perform the same testing

Maintain consistency – After a new drop is taken, the same script would be run against the same data, thereby eliminating the chances of any inconsistency. With a complex testing process manual testing often yields inconsistent coverage and results depending on the staff and schedule employed. An automated test suite ensures the same scope and process is used repeatable each time testing is performed.

Accelerate release schedule by testing at any point in the development cycle

IMPROVED TESTING PRODUCTIVITY. With its much shorter execution time an automated test suite can be run multiple times over the course of a product development cycle. By testing earlier and more often bugs are detected and corrected earlier and at much reduced expense.

IMPROVED PRODUCT QUALITY. The sum of improved test procedures and testing productivity is a substantial improvement in product quality. Automated testing detects functional and performance issues more efficiently, allowing test staff to focus on quality in areas such as documentation, installation, hardware compatibility, etc.

Reuses test across platforms Encourage more thorough testing without straining manual resources


Advantages of Automated testing

Cost effective - Automated testing has an upfront cost to develop, but over the lifetime of a product it will offer substantial net savings. An average automated test suite development is 3-5 times the cost of a complete manual test cycle. Over multiple product releases with multiple cycles per release, this cost is quickly recouped.

Decrease the product’s time to market and prevent QA from being the bottleneck in product delivery

Improve coverage of regression testing Improve the quality of testing by reducing the human

error Make test cycle more maintainable, consistent and

thorough Improve the reusability of tests Provide the organized, detailed test log and audit trail of

executed tests Simplifies the debugging by allowing the exact test case

to be replicated Allows cross-referencing of automated test cases against

bug logged in the bug tracking system.


Starting with Automation… The tester works begin once the developer is ready with the software. On an average effort

require to automate 1 test case is equivalent to effort required for executing the test case 8 times manually.

There are several factors to consider before anyone go for the automated testing.

Cost effectiveness of the automation.

Resources availability.

Reusability of the automated test script.

Tool selection.


Typical Automated testing process

• Create a test plan

• Record a test script ( Use agreed norms )

• Customise Code , Use Reusable components and functions

• Add Check points and parameterize

• Use standard programming conventions

• Save and run Test Script

• Create a Test Suite


Automation Tools

WinRunner (From Mercury Interactive www.merc-int.com) .

Quick Test Pro (From Mercury Interactive www.merc-int.com) .

Silk Test (From Segue www.segue.com) .

Visual Test (From Rational www.rational.com) .

Rational Robot (From Rational www.rational.com) .

QA Run and Test Partner ( www.compuware .com)


Maintainability

Reliability

Most important criteria for Automation


Current Trend ….

Programmers quickly create application using GUI tools which has lead to increase in their productivity

Pressure on Testers Test more and more code in less time Improve their productivity Move towards automation for GUI , CLI and API based

application Test Tools – a misnomer Lack of Programming skills Record and play back scripts , very sensitive to change

How to effectively Design , Develop and maintain automated suites?


Who should Automate Tests Good Testing and development skills Understand Testing Requirements and situations

Testers face. Testers who want to be programmers ? Automate 100 % testing ? A test automator needs to know how to develop

software. He needs to be particularly aware of issues such as maintenance and reliability. Making the system easy to update with changes to the product under test should be the priority.

Independent contractors . Who will maintain tests after they have left ?

Rejects from Development or Testing as Automators ?


What and When to Automate

Is 100 % automation possible ? Testing is the art of pragmatic – Good software testing

requires good judgment The area where one spends a lot of time testing manually Regression Testing Smoke Testing Don’t focus on areas that might otherwise go untested

( Difficult to maintain ) Rigorous Manual testing of the functionality before Very Strong Manual Testing Process and Procedures Automating everything to make job exciting ? Problems

associated Test automation takes 8 times the time to test manually continued ….


What and When to Automate Desire to be a programmer Maintainability should be the top priority – Aim to run tests

unattended Usage of Reusable components and distribution of work First run on test and then build test suite – Focus not on how

much code has been written but how many test set have been automated and how reliable and maintainable they are .

All areas that are run frequently should be automated . Don’t automate everything .. Look for parts that are big

paybacks. Introduction of Automation in Development cycle – Timing is

important.


Building Maintainable and Reliable Test Suites

Biggest challenge – Product Interface / components changes

Accurate automated Test suites

No False Positives Abortions and False

negatives are still o.k. Usability and

comprehension Standard naming

conventions and programming practices

Using Error Recovery Systems - Exception handling

Test Case Independence – base state restoration



Apply Software Development Process

Improve the Testing Process

Define Requirements Prove the concept Champion Product

Testability Design for Sustainability Plan for Deployment Face the Challenges of

Success



Apply Software Development Process

Avoid Pattern Zero status (no distinction between user and developer )

Dedicated resources to Test Automation and treating like development activity

Plan , Design Effective coding practices Configuration management , version

control Bug Tracking and testing Usage of comments , functions etc Usage of Re-usable components Comprehensive documentation

including Best practices guide



Improve the Testing Process Robust Manual Testing and Defect

Management process and procedures. Objective is to streamline Test process ,

allowing things to move more quickly without delays.

Effective Regression and documentation of the same

Document Testing Approach Names , data for tests , preconditions etc Expected results Test Designs

Creation of Logs Identify product improvements that would help

Manual Testing ( eg simplify installation process )

More computers for Testing



Define Requirements What needs to be tested – Test Design Automation Requirements – Goals for automation

Speed up testing to accelerate releases Allow testing to happen more frequently Reduce costs of testing by reducing manual labor Improve test coverage Ensure consistency Improve the reliability of testing Allow testing to be done by staff with less skill Define the testing process and reduce

dependence on the few who know it Make testing more interesting Develop programming skills

Agree on all party expectations



Prove the concept Proof of concept – A meaningful test suites that

demonstrates suitability for Regression Testing Configuration Testing GUI / NON GUI Testing

Demonstrate advantages of automation to higher management

Secure support and necessary approvals Tool Evaluation and Feasibility study

Need to investigate if the tool and approach will work for your product and staff . Is it even possible to automate tests for your product ?

Vendor issues, support , license etc Budget and other considerations



Champion Product Testability CLI / API / GUI Testing GUI is toughest to automate

Heavy Customization of automated scripts ( programming)

Technical challenge of getting the tool to work with the product

Add inns Non standard components

Keeping up with frequent design change to GUI .

API Automation – Unit Testing CLI Testing simpler – For e.g. .Install shield silent

installation



Design for Sustainability Long term focus on the need for maintenance and

reliability so that the product remains functional and relevant with new releases.

Integrity of the tests are paramount Trust automation results , All checkpoints covered No False positive or False Alarms Tests have PASS, FAILED and NOTRUN Status

Performance Attention on testing automation , automated code etc. Proper Code Reviews , remove redundant code etc

Ease of Analysis Analyze failure due to product defects or improper

automation False Alarm Improve Error Reporting Remove unreliable tests that are redundant or

obsolete


Key to successful Automation Reviewability

Ease of understanding the objectives and working of each test

Good documentation and code coverage Maintainability

Keep in mind product changes by using Centralized object repository , libraries etc

Independence Must for successful execution of a test suite as one

failure could cascade others. Ideally two tests should be independent unlike Manual

Testing Each test should create its own test environment Though redundancy increases there is more reliability Object is to run tests unattended .



Repeatability Tests should execute in the same way each and

every time they are run Action Libraries

Reusable components Effective documentation Differentiate between errors in Code or called

function Data Driven Tests

Also called Table Driven or third generation automation

Tests are written in simplified table format Parser is written to interpret and execute test

statements. Heuristic Verification

Correct Result verification


Key to successful Automation Plan for Deployment

Document setup/installation, how to run tests and analyze failures

Package tests for other people to use Helpful error messages Treat Test Suites as a product – Least dependencies on

external libraries /services. Test suites readily available to concerned parties

Face the challenges of success Staff should know how to diagnose failures . Minimize calling automators for every issue Defects should be reproduced manually Repair Old tests / Add new / Maintenance etc Formal Review of Test suites after each release. – Avoid

Old Oak syndrome


Key to successful Automation Reviewability

Ease of understanding the objectives and working of each test Good documentation and code coverage

Maintainability Keep in mind product changes by using Centralized object

repository , libraries etc Independence

Must for successful execution of a test suite as one failure could cascade others.

Ideally two tests should be independent unlike Manual Testing Each test should create its own test environment Though redundancy increases there is more reliability Object is to run tests unattended .

automation concepts

Technology

time testing

quality of testing

bugs testing

thorough testing

automatic testing

redundant testing

automated testing doesnt

supplement manual testing