optimize manual software testing - ibm · 2010-01-15 · optimize manual software testing taking...

Optimize manual software testingTaking the manual out of testing with Rational Manual Tester

Skill Level: Introductory

Sandra WilkeyIBM

24 Jan 2005

Manual testing is still the most popular method of software application testing. If yourmanual testing has room for improvement, learn how you can use Rational ManualTester to improve manual testing documentation, test execution, and test resultscollection. This tutorial will guide you through the process of evaluating what IBMRational Manual Tester can provide to your manual testing projects.

Section 1. Introduction

About this tutorial

Few organizations have fully adopted test automation. For some, manual testing isprimarily done by analysts and subject matter experts who are not test automationexperts. In other cases, the application under test does not have the necessaryinterfaces to use test automation tools. As a result, manual testing remains animportant part of the testing process.

Manual testing is deceptively simple at first glance but quite challenging whenimproperly implemented. Project teams quickly learn the difficulty of successfullymanaging multiple spreadsheets and other test and test-result storage mediums.They note the high volume of rework that results from the lack of reuse andmodularity across the many test scripts involved. And they struggle to minimizehuman error and inconsistency. The very objective of manual testing - evaluating

Optimize manual software testing© Copyright IBM Corporation 1994, 2008. All rights reserved. of 23

http://www.ibm.com/legal/copytrade.shtml

application quality and exposing defects - is jeopardized by these inefficiencies.

In this tutorial you'll learn how IBM® Rational® Manual Tester can help you to:

• Document clear and concise manual tests that are easy for the tester tofollow.

• Leverage the ability to share test content among multiple tests.

• Assist testers with common error prone activities - data entry and datavalidation.

• Accurately execute and record the results of manual tests.

Who should take this tutorial?

IBM Rational Manual Tester addresses the needs of teams performing at least aportion of their testing manually. It overcomes the ad hoc nature of hands-onverification techniques to ensure optimal testing and defect discovery in what canotherwise potentially be a high-investment, low-return activity. This tutorial is foranyone who needs to improve the accuracy and speed of building, executing, andmaintaining manual tests to test software applications.

Prerequisites

To complete the steps and exercises in this tutorial, you need to install IBM RationalManual Tester. Download a trial copy.

After you have downloaded this application, install it using the defaults. RationalManual Tester includes a sample application that will be used with this tutorial.

You will also need to download RationalMT_Tutorial.zip, which is a set of manualtests that have been created for use with this tutorial. Download the attached zip fileand unzip the content to any folder on your local hard drive. Be sure to preserve thefolder names.

Section 2. Getting started

Overview

developerWorks® ibm.com/developerWorks

Optimize manual software testing of 23 © Copyright IBM Corporation 1994, 2008. All rights reserved.

http://www.ibm.com/developerworks/downloads/r/rmt/?S_TACT=105AGX15&S_CMP=TUT

RationalMT_Tutorial.zip


IBM Rational Manual Tester is a tool designed to support anyone who performs atleast some portion of their software application testing manually. Its ease of usemakes it particularly valuable to business analysts and non-technical testers.Because IBM Rational Manual Tester is independent of the developmentenvironment used to build the application under test, it can be used to test any typeof software application: Java® application, Web, legacy, thick-client, ERP/CRMapplications, and so forth. It can also be used to document and execute tests forhardware components, peripherals, or other non-software systems.

Rational Manual Tester provides a solution for documenting detailed manual tests,streamlining test maintenance, ensuring accurate test execution, and collecting andstoring test results.

Starting Rational Manual Tester

1. Select Start > Programs > IBM Rational > IBM Rational Manual Tester6.1 > IBM Rational Manual Tester. The following Welcome Screenopens the first time you start the tool. If you have previously openedRational Manual Tester, skip to step 3 below.

2. Click the curved arrow in the upper right corner to go to the Manual Testwindow.

ibm.com/developerWorks developerWorks®



3. The main window of Rational Manual Tester opens.

4. In Manual Tester, select Help > IBM Rational Product Updater. Clickthe Find Updates button to find available updates, and then click theInstall Updates button to install them. Once you have completed installingthe available updates, you are ready to continue.

Understanding the user interface

There are five main tabs in the Rational Manual Tester window.

• In the middle of the window is the Editor. The editor is used to documenta manual test. A default, empty test titled "untitled.rmt" is opened in theeditor.




• The tab on the left side is Outline. It displays a short summary of eachstep documented in the Editor. The Outline tab is also used to link (share)content among tests.

• Next to the Outline tab is Recent Files. This tab displays the files youhave been working with to make it easy to open a recently used test ortest result.

• The tab at the top right is Reuse. Test content that will be reused amongmultiple tests can be placed on the Reuse tab for quick access.

• The tab at the bottom right is Properties. It contains additionalinformation about the test currently displayed in the editor.

Section 3. Author a manual test

Types of test statements

Manual tests can contain four types of statements. Each statement type is denotedby an icon at the beginning of each statement in the test editor.

• Step -- A step is a statement that instructs the tester to perform some

action. "Start the application" or "Click on the New button" are examplesof common test step instructions.

• Verification Point -- A verification point is a statement that instructs

the user to verify a particular behavior of the application under test. "Verifythat the Login button appears in the top right corner" or "Does theapplication return the date in the format mm/dd/yy?" are examples ofcommon verification point statements. When the tester executes a testthat contains a verification point, the tester must enter a result for eachverification point.

• Reporting Point -- A reporting point is a high-level verification point. It

is often used to ask a high-level or summary question about the successor failure of the entire test.

• Group -- A group indicates a block of related statements. For

example, you might have a group called Login. Under this group would bethe specific steps to log in to an application.




Jump-starting the test authoring process

There are a couple of routes to jump-start your use of Rational Manual Tester. If youalready have manual tests documented in Microsoft Word or Excel, you can use theFile > Import option to import your tests into Rational Manual Tester.

If you do not have existing test documentation that you want to import, you cansimply begin documenting your tests in the Editor. Rational Manual Tester providesa rich text editor to help you document manual tests that are detailed and easy forthe tester to follow.

Setting the reuse file location

Before continuing, set the Reuse file to the location and start the sample applicationused in this tutorial.

1. Select Window > Preferences.

2. Use the Browse button to set the Reuse File location to the locationwhere you unzipped the tests. The folder name you need to point to isRationalMT_Tutorial.

3. Click Apply.

4. Click OK.

5. To start the sample application select Start > Programs > IBM Rational> IBM Rational Manual Tester 6.1 > ClassicsJavaA. This starts asample application called ClassicsCD for which you will create a test toverify the process of ordering a CD.

Rich text editing

One of the easiest ways to understand the features of rich text editing is to look at amanual test that has taken advantage of these features.

Rich text editing facilitates creating robust, detailed, easy-to-follow tests. Clear andconcise tests will help to ensure that tests execute accurately.

1. Click Recent Files in the Rational Manual Tester window anddouble-click the Place_Order.rmt test to open it. The test should displayin the editor as shown below.




In this test, you see that the author has used fonts and colors to callattention to important keywords or selections that the tester must makewhen executing the test. The author has also used embedded images,which give the tester context for the steps they are performing. Imagesare easily added to the test using a built-in image capture feature.

2. To insert an image, place your cursor at the end of the first statement andpress <CTRL><ENTER>.

3. Select the menu item Edit > Insert Image.

4. Select Region, then click OK.




5. Draw a box around ClassicsCD application window. The image willappear in the body of the test.

6. Resize the image to a smaller size. The test should now look similar tothe picture below.

7. In the second statement, highlight the word Haydn and click the Boldbutton on the Rational Manual Tester toolbar.

8. Click the Font color button on the Rational Manual Tester toolbar and

select a color other than black.




Fonts can be used to identify text that the tester should type when executing thetest, buttons, or list choices to select in the application under test, or for any otherneed to call special attention to an action.

Using verification points

Verification points are an important element of a manual test. They instruct the testerto verify a specific behavior of the application under test. During test execution, thetester will need to respond to a verification point with a result (pass, fail,inconclusive, and so forth), an optional comment, and a file attachment if needed.Verification point results are stored in the test log upon completion of the test.

Verification points are marked with the icon . You can have zero, one, or multiple

verification points in a test depending on the scope of the test. Verification pointstatements generally begin with text such as "check that..." or "verify that...", or theymight pose a question to the tester.

1. In the test editor, locate the statement: Verify that the Place an Orderdialog appears.

2. Select this statement and click the button on the toolbar. It changes

the icon next to the statement to mark it as a verification point.

3. Locate the following three statements and set them to Verification pointsas well:Highlight the item Symphonies No. 99 & 101 as shown below, press<CTRL> C to copy to the clipboard, then press the toolbar buttonlabeled "Compare copied text to expected text".

Verify that the window shows the order amount plus shipping andhandling as shown below.

Does the application display an order number similar to the onebelow? (Note that the order number will be different each time yourun the test.)

Using reporting points

Reporting points are also an important element of a manual test. Like verificationpoints, they instruct the tester to answer a question or report a result. However,reporting points typically pose a higher level question about the overall success orfailure of the test. For example, in a test, you might have multiple verification points




asking about discrete behaviors of the application under test. At the end of the test,you will typically have a reporting point that asks whether the overall test case orbusiness use case was successful or not. During test results analysis, you canchoose to report on verification points for detailed results analysis, reporting pointsfor a higher level test status, or both.

Reporting point statements are marked with the icon . You can have zero, one, or

multiple reporting points in a test.

1. Scroll to the end of the test.

2. Place your cursor at the end of the last statement and press Enter.

3. Type the statement: Did the order complete successfully?

4. Leave your cursor on this line and click the button on the toolbar.

The icon next to this statement changes to the reporting point icon.

5. Save your changes using File > Save.

You now have a test that contains statements that provide instructions to the tester,embedded images to add clarity, four verification points to verify the behavior of theapplication, and a reporting point to determine the overall success of this test.

Section 4. Reduce manual testing errors

Automated data entry

One of the most common problems with manual testing is human error. Applicationsthat require a large amount of data entry are particularly prone to human error.Rational Manual Tester can help reduce data entry errors by providing assisted dataentry during test execution. When a test is authored, the data that must be enteredto perform a test can be stored with the test instructions. During test authoring, awizard extracts the data to be entered directly from the application under test.Alternatively, the data can be entered directly into the test step properties. Duringtest execution, rather than manually entering the data, the tester can paste the datafrom the manual test execution window into the application under test. The benefit ofusing automated data entry is greater accuracy and speed during test execution.




Perform the following steps to learn how to use the automated data entry feature:

1. In the Composers list of the ClassicsCD application, expand the Haydnfolder.

2. Select Symphonies Nos. 99 & 101 then click Place Order.

3. Click OK in the Login window.

4. Place your cursor in the Credit Card field and enter 1234 5678 12345678.

5. Switch back to the Manual Test window.Now you want to capture the data in the application and place it in themanual test step that instructs the tester to enter the credit card number.

6. In the test editor, place your cursor on the test step that reads: Place thecursor in the credit card number line and use <CTRL> V to paste inthe test card number.

7. From the Manual Tester menu, select Capture > Data for Insert.

8. A window with instructions opens.

9. Press <ATL> <TAB> and switch to the Place an Order window in theClassicsCD application.

10. Highlight the credit card number that you entered earlier and press<CTRL> C to copy the text to the clipboard.

11. Press <ALT> <TAB> to return to the Manual Tester instructions windowand click OK. The text is inserted into the Paste Data property of the test.When this test is executed, the tester will be able to place their cursor inthe credit card field and press <CTRL> V to have the credit card numberautomatically inserted. If you are not familiar with common testautomation techniques, using assisted data entry with manual testing willintroduce you the concept of tool-based data insertion -- which iscommonly used with automation tools. If you are planning to leverageautomated testing in the future, familiarity with this concept will make thetransition to test automation easier.

Automated data comparisons




Verifying the accuracy of data that is output from the application under test isanother error prone task when manually testing applications. Rational Manual Testerprovides a built-in data comparator that compares the expected output data to theactual data output by the application under test. As with the automated data entryfeature, the expected output data is stored in the manual test. Using this feature canspeed the process of test execution and ensure accuracy when validating data. Thesteps below show you how to capture text to be used for an automated datacomparison when the test is executed. This time, instead of using the wizard tocapture data, paste the data directly into the Verification Point property. Eithermethod for capturing the data can be used.

1. Switch back to the Place an Order window in the ClassicsCD applicationand highlight the text Symphonies No. 99 & 101 as shown below.

2. Press <CTRL> C to copy the data to the clipboard.

3. Switch to Rational Manual Tester and place your cursor on theVerification Point statement in the manual test that reads: Highlight theitem Symphonies No. 99 & 101 as shown below and press <CTRL> Cto verify that the correct selection is displaying.

4. In the lower right corner of the window is the Properties tab with aproperty called Compare Data. Place your cursor in the Value field nextto this property and press <CTRL> V to paste in the expected data.

5. Let's make a slight change to this expected data so that you can see howRational Manual Tester will identify differences when the test is executed.Edit the data that you copied so that it is slightly different from the text thatthe application under test will display. Change one of the Symphonynumbers, such as changing the number 99 to 94.That's it! You have now stored the expected content of this field alongwith the manual test.

When this text is executed, the tester will highlight the text from theapplication they are testing. Press <CTRL> C, then Rational Manual




Tester automatically compares the two text strings.

6. Save the changes.

Section 5. Reuse test content

Principles of reuse and modularity

In testing, the concepts of test modularity and test reuse are important concepts tomaster to get the fullest benefit from test automation. Automated testing tools havelong promoted the creation of small modular tests that cover a single use case orfunction. The reason for this approach is two-fold. First, it makes test maintenanceeasier. For example, when the application under test changes, and therefore the testdocumentation or test script must change, having modular tests will help to reducethe number of tests that must be updated. Automated testing tools have alsopromoted the concept of test reuse by providing mechanisms to call one test fromanother. Building modular tests that document commonly performed tasks, thenreusing those tests, will help to ensure that consistency and accuracy is maintainedwhen test maintenance is required.

There have been varying degrees of success when applying these same principlesto manual testing. In most cases the problem stems from the fact that the toolstraditionally used to document manual tests, such as Microsoft Word and Excel, donot provide a framework for implementing test modularity or reuse. Thus, manualtest documentation often results in long, difficult to maintain tests, many of whichcontained similar content for commonly performed actions in an application. Whentests needed to be updated, the test team had to search through all the testdocumentation, cutting and pasting updates.

Rational Manual Tester facilitates the development of modular tests and the reuse oftests by providing a simple framework for test content sharing. Test content can beshared among multiple tests and multiple testers. By reusing test content, testingteams are encouraged to create smaller, modular tests, without the worry of havingnumerous test scripts to update when the application under test changes. Commonlyused test content can be placed on a Reuse pallet and made available to multipletests created by multiple team members. So whether you are a single tester, part ofa small team, or part of an enterprise testing team, this capability will help you tobuild easily maintainable tests.




Reusing test content

1. Click the Recent Files tab. The two tests that have already been createdis displayed.

Each of these tests covers testing a specific use case of the application.As previously discussed, it is a best practice to keep tests short, modular,and focused.

However, because of the design of this application, there are some stepsthat are repeated in multiple tests. For example, each time you place anorder, you are prompted to enter your password. Because of this design,it would be convenient if the tester has the login steps embedded in anytest where an order is placed. However, duplicating these steps inmultiple tests can lead to a maintenance nightmare if the login steps wereto change.

A better solution is to make commonly used test instructions reusable.Follow the steps below to see how Rational Manual Tester can improvetest maintenance and consistency through reusable test content:

2. Double-click on the Login test to open it.




3. The content, which is organized into a group, can be placed on the Reusepallet to make it available for reuse in other tests. You can reuseindividual statements or groups of statements.

4. Click the Outline tab and to see an outline of the test steps.

5. Click the Login group in the Outline tab and drag it to the ReusableStatements icon in the Reuse tab on the right side of the window.Your Reuse tab should look as shown below.

Content on the Reuse tab can be linked to any other test. Linked contentis shared, not duplicated. Changes made to linked content will bereflected in every test in which that content is reused. You no longer needto search through multiple documents to find repeated instances ofcommonly used test instructions.

6. Now click the Place_Order tab in the Editor.Reuse the Login steps in the Place_Order test.

7. Click the Outline tab to see the outline of the Place_Order test.

8. Drag the Login folder from the Reusable Statements palette to theOutline tab. Drag it to directly beneath the statement Click the PlaceOrder button. The Outline tab has the content shown below.




The login steps are now reused in the Place_Order test. If you look at thetest in the editor, the Login steps have a light gray background and theicon to the left of each reused step has a small arrow in it. This icondenotes linked content. If the steps to log in to the application change, alltests that reuse the login steps are automatically updated when the Logintest is updated. If you later wish to make the statements local (break thelinking), right-click the linked statement or folder and select Edit > MakeStatement Local.

Section 6. Execute manual tests

Running a test

When you are ready to test your application, Rational Manual Tester displays anexecution window that prompts the tester with each step of the test. As each step isperformed, the tester can enter comments, attach files (such as screen shots), andenter a result for each Verification or Reporting Point. When the test is completed,the results are stored in a permanent log. Continue on to learn how to execute atest.

The execution window




1. Select File > Save to save all your edits to the Place_Order test.

2. Close the ClassicsCD application.

3. From the Rational Manual Tester menu, select Run > Run script. Thefollowing window opens.

The left side of the window contains the statements in the test. Theexecution window has a translucency feature that allows the tester to seethe application under test through the execution window. This makes iteasier to see the application under test and the test instructions at thesame time. You can turn the translucency feature off or set it to anydesired degree of translucency. You can also resize the window to makeit easier to view the full list of steps. To the right of each statement is aResult group. Use the Apply button to indicate that a step was completedor to enter a result for a Verification Point or Reporting Point. On the rightside of the execution window is the Properties pane. Use the Propertiespane to add information such as a comment or a file attachment to furtherdetail the behavior of the application during the test.

Executing the test

1. Start the ClassicsJavaA application, then click Apply.

2. Position the ClassicsCD application window at the top of the screen andthe Manual Test execution window at the bottom of the screen so that youcan easily see both applications at the same time.

3. Expand the Haydn folder as instructed in step 2. When you have donethat, again, click Apply.

4. Select Symphonies Nos 99 & 101, then press the Place Order button.




Click the Apply button next to both of these steps.You are now ready to log in to the application.

5. Follow the steps in the Login group, and click Apply as you completeeach of the four steps. Your test execution window should now looksimilar to the one below.

6. The next statement is a Verification Point asking if the Place an Orderwindow displayed. Assuming it does, select pass from the Resultdrop-down list then click Apply.




7. In the next Verification Point, use the automated data comparison to verifythat the correct text displays in the Item heading field in the Place anOrder window. Follow the instructions in the test to highlight the data inthe application, copy it to the clipboard, then press the toolbar button

to automatically compare the expected text to the actual text in theapplication.The comparator shown below displays and highlights the differences. Thedifference you see will depend on the edits you made to the text aftercopying it to the Properties tab.

8. Click OK to dismiss the comparator window.

9. Select Fail from the Result drop-down list then click Apply.

10. In the next step, use the automated data insert feature to enter the creditcard number. Follow the instructions in the test to insert the credit cardnumber automatically, then click Apply.

11. Enter the credit card expiration date as instructed, then click Apply.

12. In the next Verification Point check that the right total appears, selectpass from the Results drop-down list, then click Apply.

13. Follow the remaining steps and VPs, and click Apply after completingeach one.

14. The last statement is a Reporting Point that asks if the order completedsuccessfully. This step prompts you to enter a summary result for theentire test. Reporting Points results are often rolled up for managementreports for a summary test status report. (For the purposes of this tutorial,




you can select either Pass or Fail.)

15. When prompted to save the results, respond Yes, use the default nameprovided, select the folder RationalMT_Tutorial, then press Save.

16. After running the test for the first time, you might want to run this testagain using only the instructions in the Manual Test execution window(and not this guide) for guidance. If you choose to do that, shut down theClassicsCD application before running the test. You can overwrite theresults of the first test with the results of your second execution.

Section 7. Analyze test results

Storing test results

After a test is executed, its results are stored in an execution file. The default nameis testscriptname.execution You can place result files in any folder location that youchoose. The file is accessible through the file system or from the Recent Files tab.Each execution file stores results for an individual test. If you want to collate theresults of multiple tests, export the results to a .csv file then import them intoMicrosoft® Excel or any other analysis tool.

Understanding the result file

Follow the steps below to open and explore the results file for the test you justexecuted.

1. Click the Recent files tab.




2. Double-click the file Place_Order.execution to open it.The execution file contains an Overview and an Events tab. In theOverview tab is a Verdict, which is the result of the last Verification orReporting Point. By placing a summary question at the end of the test youwill always have an accurate high-level "verdict" for the results of the test.The start and stop times for the test are also in the Overview tab.

3. Click the Events tab at the bottom of the window.

4. Expand the Group Root as shown below:

Each statement in the test is listed in the Group. If you click a statement,you will see the full text in the description field, the time that statementwas executed, and a verdict for any Verification or Reporting Points. Thiswindow can be used for a quick evaluation of a single test.

Collating and analyzing test results

After executing a series of tests, you may want to combine all of the results andanalyze them collectively to make an overall assessment of your test progress andthe application's readiness for delivery. To do that, you can export the contents ofmultiple .execution files and place them in a .csv file using File > Export. Theexported results can then be imported into your central project reporting tool togenerate reports. In addition, Rational Manual Tester can integrate with IBMRational TestManager, which will allow you to use the results analysis and reporting




features already available in this test management tool. A license for RationalTestManager is included with Rational Manual Tester.

Section 8. Conclusion

Summary

Rational Manual Tester provides a solution to create, execute, and control manualtesting. By dedicating focus on manual testing productivity, Rational Manual Testerhelps teams overcome the risks to the success of a project. The result is a greaterfocus of time and effort on completing high value, informative manual tests which willlead to more thorough testing and a more accurate assessment of the quality of theapplication.

Other ideas you might want to explore include:

• Customizing Rational Manual Tester result choices to ensure test resultsare collected using your team's standards

• Customizing the test properties to collect additional data during testdevelopment or test execution

• Modifying shared content to see how tests with shared content areautomatically updated

• Using IBM Rational ClearCase to version control test scripts. (RationalClearCase LT is included with each purchase of Rational Manual Tester.)




Resources

• Participate in the discussion forum for this content.

• Rational Manual Tester Features and Benefits

• Rational Manual Tester System Requirements

• IBM Rational Manual Tester Data Sheet

• developerWorks Rational Manual Tester resources

• Rational demos

About the author

Sandra WilkeySandra Wilkey is a technical marketing specialist in the IBM Rational SoftwareGroup. Her expertise is in the software quality tools that are part of the IBM SoftwareDevelopment Platform. Sandra has been with the IBM Rational organization for over12 years and has spoken at numerous user conferences and industry trade shows.Sandra most recently spoke at the 2004 Rational Software Developer UserConference in Dallas, Texas, where she gave a well-received talk on tips and trickswith Rational TestManager.



http://www.ibm.com/developerworks/community/

http://www.ibm.com/software/awdtools/tester/manual/features/index.html

http://www.ibm.com/software/awdtools/tester/manual/sysreq/index.html

http://www.ibm.com/developerworks/rational/products/manualtester/

http://www.ibm.com/software/rational/technical/DEMO/


optimize manual software testing - ibm · 2010-01-15 · optimize manual software testing taking...

Documents