Ananya Das

Commercial Test Automation Tools and their real benefits

An Automation Fable Once upon a time in the peaceful land of Testing, the wizard proposes , “ Let there be automation ..” and the King agrees “Yes.. that is the way to be”

Enter Chandan Nilekani “delegate” Reviews commercial tools, selects , buys , costs …. Assigns to Narad Murthy “overworked “ Narad has several other responsibilities , tool is complicated , tries customer care Finally gets an expert – works and works – several days pass … but no automation

Enter Nazim Premji “short timer” Fresh graduate, new ideas, trainings Works full time … library, gets some parts working After 6 months … gets a great offer …..

Enter Shankar Nadar “hardluck” Tries to execute scripts, no documentation Incomplete scripts, poor error messages, gets some bugs … king happy Next version of product.. .Changes …. Changes scripts … long hours … gets working Product shipped Customer very unhappy … lots of bugs … why ?? Automation Programming error had dismissed these bugs …

King – “Off with their Heads !!!!!!!!!“

Some Commercial test Automation tools

•QTP , RFT, •TestPartner•Silktest•IBM Rational Robot

Functional Testing

•Load Runner•QA LoadPerformance

Testing

•Quality Center•Test Director

Test Management

Evaluating a commercial tool How easy is it to use? Learning Curve? Continuous Integration (CI) friendly? Call & Log without manual intervention? Work with

existing systems ? How does it interact? IE (client side)? HTTP layer? What types of tests does it support?

Web? WinForms? Performance? Framework for Integration Test (FIT)? SQL?

Does it support recording? Does it support scripting?

Languages? Easy creation of scripts ? Object Oriented Scripting Support? Interactive Debugging Tools? Easy integration with External libraries?

Ability to identify objects using multiple parameters? Ability to run multiple scripts consistently in a Batch mode? Ability to store test application's objects so that scripts are well maintainable? Reusability of Scripts and Tests? Reporting mechanism? Object parametrization? Ajax? Popup handling? Keyword driven? Automatic Exception

Handling? Need to support IE7? How easy is it to transfer tests to/from platform?

Evaluating a commercial tool

How well is the tool supported? Commercial? Community?

Support for mocking + stubbing? Test case organization?

Organizing tests similar to structured manual test plans? Support cross frame script calls? Web Objects? Cost/Licensing? Integrated with other test components? VPN / Remote Desktop friendly? Active Community of Users?

Forums? Blogs?

Advantages of Commercial Tools

•Many of the open source tools come and go with little to no support •Most commercial tools are constantly being updated as technologies

change

Customer support

•Most commercial tools usually have more functionality (QTP can test various GUI applications: Web, .Net, Java, VB, C/C++ etc)

•Commercial tools are integrated with Test Management tools which makes reporting and execution much simpler.

Functionality

•Commercial tools usually have a large community of users, which translates into better availability of qualified resources

•Commercial tools require less advanced programming

Resources

•More test automation frameworks are available for commercial tools

Framework

Truth or Myth ?Commercial test tools are expensive ?

Myth Under the influence of this myth some companies, especially the small

ones:o Try to develop their own test automation toolso Use scripting lang. like Perl and Ruby,use shareware test toolso Do not consider test automation at all

Reality It has been calculated that if implemented properly, the cost of this

tool is just 3% of the person who uses it, but productivity gain can be very significant

Truth or Myth ?Recorders make test automation easier

Myth

The Truth:They make recording easy …But they also make it easy to create automation that is difficult

and expensive to maintainA recorder can serve as a training device to help you learn a

scripting language

Truth or Myth ?You can’t develop test automation for an application

until it is stable

Myth

The Truth:You should develop test plans as soon as the Requirements are

documented, You can also develop automated tests from wire-frames, mock-ups and prototypes

In the case of wire-frames, you can even test some of your methodsIf tests are structured properly, only a small portion of your work

will have to be maintained when GUI changes are made to an application that has not yet stabilized.

Truth or Myth ?Test automation is most effective for regression testing

Part Truth:

Test automation is very effective for regression testingIt is also effective for first pass testing when a data driven

approach is used Automated tests should be designed to find bugs not just verify

that regression tests don’t break

Truth or Myth ?It is difficult to automate a dynamic application where data

changes constantly

Myth

The Truth:Data can be separated from test code so that only the data needs to be

changedBetter yet, in some cases input data can be captured at runtime to meet

the criteria required for a testOr a database query can be executed to get data at runtimeVerification data can also be captured at runtime so that tests remain

flexible in a dynamic environment

Truth or Myth ?Test automation is a software development task

Truth:Automated tests should be designed, developed and testedYou need to have some kind of a programming background to

implement test automation. Test Automation is not as complex as C++/C#/Java development.

Automated test components are assets that should be treated like application source code

Test automation standards should be developedTest automation should be subject to peer reviews

Truth or Myth ?Productivity gains will be realized from test tools right

away

Myth:

Test automation is an investmentQuality gains will be realized right awayProductivity gains will be realized over time as skills develop and

manual testing tasks are offset by completed automation

Truth or Myth ?Automated tests are difficult to maintain

Myth

The Truth:Separate business rules, test methods and data to promote

maintainabilityIf properly structured, automated tests can be maintained across

many releases of a target applicationIt’s important to run automated tests on every build and make

incremental changes as needed

Why is Automation great Speed up testing to accelerate releasesAllow testing to happen more frequentlyReduce costs of testing by reducing manual laborImprove test coverageEnsure consistencyImprove the reliability of testingAllow testing to be done by staff with less skillDefine the testing process and reduce dependence on the few who

know itMake testing more interestingDevelop programming skillsBUT ………….

Then … why failure ?About 63 % of the Automation projects fail …. Why ?

MisconceptionsTesting tools don’t work and are difficult to use This application is too complex and the schedules are too tight Senior Management will never support itAutomation will eliminate the need for all manual testingManual testing must have taken place before automating a feature

Critical Success Factors

Clearly define Automation Objectives

Get management support for the project

Clearly demonstrate the short-term & well as long-term financial benefits to be accrued by investing quality resources and time in the project.

Ensure that all stakeholders are in agreement on the end objective of the project. Top Management, the IT team, the developers, the tester, and the automation team--may have different

requirements from the project. However, success will be elusive unless they all agree to a common goal.

Treat Automation as Software development

Study the requirements Planning and tracking Set realistic goals , assess existing skill sets and resources Evaluate the testing tools; perform automation feasibility; Identify reusable functions

Critical Success FactorsDesign and develop a framework

Design and document extensible automation architecture, which clearly outlines the foundation and rules for the project.

Incorporate flexibility and scalability into the architecture to be able to cope with growth in the requirements

Also, provide the testing team with a proof-of-concept test suite and make test case maintenance a top priority

Evaluate testing tools Validate the tools and approach at the earliest and evaluate its compatibility with

the organization’s requirement and the AUT

Check the availability of trained/experienced resources

Critical Success factors` Decide what needs to be automated

Include short and simple transactions and tests that are executed regularly. Execution with multiple datasets Do not automate where it is difficult to predict results or are not going to be repeated. Avoid automating long or complex transactions, cross multiple applications

Spend time on Automation Strategy and Plan

List of activities Organization test objective Reusable and Modular test script identification and planning

Run Automation as a full time project

Look at it as a long term investment

Co-operation with functional testers

GM Regression Automation

Test Logs and bugs

Coverage Matrix

Alpha Test Cases

Components and procedures

Automation Framework

ROI

ROI = BENEFIT/COST

Automation Cost = Price of Tool + Dev Cost + Maint. Cost + Execution Cost

Manual Testing Cost = Development Cost + Maintenance Cost + Execution Cost

ROI = (Manual Testing Cost - Automation Cost)/Automation Cost

Looks right, Doesn’t it? NO …..

You can’t compare Automated Testing and Manual Testing. They are not the same and they provide different information about the AUT.

You can’t compare cost of multiple execution of automated tests vs. manual tests. You would never dream of executing that many test cases manually.

Automation ROI

•People available to work on other projects•Higher market share•Higher margins

Reduce Time to Market

•Defects found early•More testing gets done fasterTest

Efficiency

•Effective regression tests•AccuracyTest

Effectiveness

Conclusion … Long Live the King Commercial Automation tools are useful if implemented and

managed correctly

ROI should be managed

CSFs should be considered seriously

And then ... the King said – “ Let there be automation ….” And there were profits, productivity, growth and success all thru the Kingdom

THE END