intermediate_and_adv

Session # AT22

Rational User Conference 2002Rational User Conference 2002

©199

8, 1

999,

200

0, 2

001,

200

2 Ra

tiona

l Sof

twar

e - A

ll rig

hts

rese

rved

Intermediate and Advanced Tips and Tricks for Performance Testing with Rational Suite

TestStudio

Intermediate and Advanced Tips and Tricks for Performance Testing with Rational Suite

TestStudio

Scott SilverstoneJeff Robbins

Rational Software

Scott SilverstoneJeff Robbins

Rational Software

Your PresentersYour Presenters Scott Silverstone, Rational Marketing Engineer

Jeff Robbins, Rational Product Manager

Special thanks to presentation reviewers

Kent Siefkes

Dawn Haynes

Scott Silverstone, Rational Marketing Engineer

Jeff Robbins, Rational Product Manager

Special thanks to presentation reviewers

Kent Siefkes

Dawn Haynes

What Does A Performance Tester Do?What Does A Performance Tester Do?

Execute/Monitor Test

Evaluate Results

Manage Test Data

Create Test Suite

Plan Test

Design Test

Tester

(from Actors)

Create Test Scripts

Getting StartedGetting Started


Evaluate Results

Manage Test Data

Create Test Suite

Plan Test

Design Test

Tester

(from Actors)

Create Test Scripts

Performance Test Planning & DesignPerformance Test Planning & Design Several other presentations cover Planning & Design

AT03 Basic Tips and Tricks for Performance Testing AT32 Modeling the Real World for Load Testing Web Sites RTL09 Introduction to Functional and Performance Testing

See www.rational.com/ruc for other presentations

Due to coverage in these presentations, this presentation will not cover planning & design

Several other presentations cover Planning & Design AT03 Basic Tips and Tricks for Performance Testing AT32 Modeling the Real World for Load Testing Web Sites RTL09 Introduction to Functional and Performance Testing

See www.rational.com/ruc for other presentations

Due to coverage in these presentations, this presentation will not cover planning & design

Digging InDigging In


Evaluate Results

Manage Test Data

Create Test Suite

Plan Test

Design Test

Tester

(from Actors)

Create Test Scripts

Creating Test Scripts: Hint OutlineCreating Test Scripts: Hint Outline

Use of “non-traditional” performance test scripts

Use of INFO SERVER for monitoring

VU language trick to conserve hardware




Use Of “Non-Traditional” Test ScriptsUse Of “Non-Traditional” Test Scripts You are not limited to using VU language scripts

Other scripting languages are supported Java Command line (compiled executables,shell scripts, perl scripts, etc) Visual Basic

Presentations on Java & Visual Basic scripting: AT28 - Designing Scalable Performance Test Scripts for Java Environments AT30 - Custom Load Generation with Rational TestManager VM14 - Model Based Performance Testing of EJB and DCOM/COM+ Components

Let’s talk about Command Line scripts in more detail

You are not limited to using VU language scripts

Other scripting languages are supported Java Command line (compiled executables,shell scripts, perl scripts, etc) Visual Basic

Presentations on Java & Visual Basic scripting: AT28 - Designing Scalable Performance Test Scripts for Java Environments AT30 - Custom Load Generation with Rational TestManager VM14 - Model Based Performance Testing of EJB and DCOM/COM+ Components

Let’s talk about Command Line scripts in more detail

Command Line Test Scripts – Why Use? Command Line Test Scripts – Why Use? Command Line Test Script Type

Run a setup script at the very start of the test Run a cleanup script at the very end of the test Execute any command in system PATH Simply put, run anything that can be run from the command line

New Types based on Command Line Execution Adapter Use perl scripts as your load test scripts Execute shell scripts throughout the run Use binaries compiled from C, Ada or Smalltalk as load test scripts Launch preVue tests as part of a larger load test under TestManager

Creating a new type allows more control over execution conditions than simply using the command line test script type

Command Line Test Script Type Run a setup script at the very start of the test Run a cleanup script at the very end of the test Execute any command in system PATH Simply put, run anything that can be run from the command line

New Types based on Command Line Execution Adapter Use perl scripts as your load test scripts Execute shell scripts throughout the run Use binaries compiled from C, Ada or Smalltalk as load test scripts Launch preVue tests as part of a larger load test under TestManager

Creating a new type allows more control over execution conditions than simply using the command line test script type

Using The Command Line Test Script Type (1)Using The Command Line Test Script Type (1)

Using The Command Line Test Script Type (2)Using The Command Line Test Script Type (2)

Using The Command Line Test Script Type (3)Using The Command Line Test Script Type (3) Resulting Suite: Resulting Suite:

QuestionsQuestions What is the name of the project we are using?

What is the name of our suite?

What are the name of our user groups?

What is the name of the project we are using?

What is the name of our suite?

What are the name of our user groups?

New Type From Command Line Execution Adapter (1)New Type From Command Line Execution Adapter (1)

Using The New Type In A Suite (1)Using The New Type In A Suite (1)

Using the New Type In A Suite (4)Using the New Type In A Suite (4)

Use Of INFO SERVER For Monitoring (1)Use Of INFO SERVER For Monitoring (1) As discussed in AT03 (Basic Tips and Tricks for Performance

Testing), monitoring of system resources on the local computer and agents is done automatically

Monitoring of computers other than the local computer and agents requires use of the INFO SERVER declaration

Robot automatically inserts the INFO SERVER declaration for the database server into SQL scripts

Robot does not automatically insert INFO SERVER into other types of scripts (Tuxedo, HTTP, Jolt, CORBA, DCOM)

As discussed in AT03 (Basic Tips and Tricks for Performance Testing), monitoring of system resources on the local computer and agents is done automatically

Monitoring of computers other than the local computer and agents requires use of the INFO SERVER declaration

Robot automatically inserts the INFO SERVER declaration for the database server into SQL scripts

Robot does not automatically insert INFO SERVER into other types of scripts (Tuxedo, HTTP, Jolt, CORBA, DCOM)

Use Of INFO SERVER For Monitoring (2)Use Of INFO SERVER For Monitoring (2) To add servers to monitor, add the INFO SERVER declaration to a

script. Which script, you ask? One approach is to have a set of re-usable scripts to initialize

monitoring for each computer of interest These scripts can be inserted in any suites you like

Alternatively, the INFO SERVER declarations can be added directly to the scripts where the server access occurs

To add servers to monitor, add the INFO SERVER declaration to a script. Which script, you ask?

One approach is to have a set of re-usable scripts to initialize monitoring for each computer of interest These scripts can be inserted in any suites you like

Alternatively, the INFO SERVER declarations can be added directly to the scripts where the server access occurs

Use Of INFO SERVER For Monitoring (3)Use Of INFO SERVER For Monitoring (3) During Suite Execution, activate the computer view: During Suite Execution, activate the computer view:

Right-click cells to add/remove stats from graph

Right-click cells to add/remove stats from graph

VU Language Trick To Conserve HardwareVU Language Trick To Conserve Hardware If you don’t need to match on the _response or log all

the data (or only need the first part of the _response ):

Set Max_nrecv_saved to a low number just big enough, such as 1 or 100

• SQL: limits # of rows saved• HTTP, socket: limits # of bytes saved

Applies to all receive commands, not just the “nrecv” flavor

Can be used selectively, such as around large SQL queries, or web pages with lots of images...

If you don’t need to match on the _response or log all the data (or only need the first part of the _response ):

Set Max_nrecv_saved to a low number just big enough, such as 1 or 100

• SQL: limits # of rows saved• HTTP, socket: limits # of bytes saved

Applies to all receive commands, not just the “nrecv” flavor

Can be used selectively, such as around large SQL queries, or web pages with lots of images...

Example: Max_nrecv_savedExample: Max_nrecv_saved Using Max_nrecv_saved to save only the first row of a

large SQL query: Using Max_nrecv_saved to save only the first row of a

large SQL query:

sqlexec ["exec big query"] "select * from books"; push Max_nrecv_saved = 1; sqlnrecv ["retrieve big query"] ALL_ROWS; pop Max_nrecv_saved;

sqlexec ["exec big query"] "select * from books"; push Max_nrecv_saved = 1; sqlnrecv ["retrieve big query"] ALL_ROWS; pop Max_nrecv_saved;

Digging Up Data…Digging Up Data…


Evaluate Results

Manage Test Data

Create Test Suite

Plan Test

Design Test

Tester

(from Actors)

Create Test Scripts

Manage Test Data: Hint OutlineManage Test Data: Hint Outline

Streamlining transfer of datapools to agents

All about the Authentication Datapool What it is Why to modify it How to modify it



Streamlining Transfer Of Datapools (1)Streamlining Transfer Of Datapools (1)

Datapools are automatically copied to Agents at the start of a test run to minimize network traffic during script execution

Huge datapools may cause a long startup delay at beginning of a run when they are copied to agents

Control with system env variable on MasterRT_MASTER_DISABLE_MANDATORY_DP_XFER = 1

Datapools are automatically copied to Agents at the start of a test run to minimize network traffic during script execution

Huge datapools may cause a long startup delay at beginning of a run when they are copied to agents

Control with system env variable on MasterRT_MASTER_DISABLE_MANDATORY_DP_XFER = 1

Streamlining Transfer Of Datapools (2)Streamlining Transfer Of Datapools (2) This turns off automatic copying of

datapools to agents Note: Rational TestManager & Rational

Robot must be restarted to read the new value of this variable

Caution!!! Updates to datapools won’t be used

unless the datapool file is refreshed The best way to refresh the datapool is

to set RT_MASTER_DISABLE_MANDATORY_DP_XFER to 0, restart Rational TestManager and Rational Robot, then run a test. The updated datapools will then be copied to the agents.

This turns off automatic copying of datapools to agents

Note: Rational TestManager & Rational Robot must be restarted to read the new value of this variable

Caution!!! Updates to datapools won’t be used

unless the datapool file is refreshed The best way to refresh the datapool is

to set RT_MASTER_DISABLE_MANDATORY_DP_XFER to 0, restart Rational TestManager and Rational Robot, then run a test. The updated datapools will then be copied to the agents.

Manage Test Data: Hint OutlineManage Test Data: Hint Outline





What Is The Authentication Datapool?What Is The Authentication Datapool? Answer:

A unique datapool that collects secure login information during Rational Robot script recordings

Rational Robot generates entries into this file as the connection is opened during secure login sessions

Contains information about username, password, connection type (ex. SQL) and connection state (Active or Passive)

For secure login purposes ONLY, not a general-use datapool file

Answer: A unique datapool that collects secure login information

during Rational Robot script recordings Rational Robot generates entries into this file as the

connection is opened during secure login sessions Contains information about username, password, connection

type (ex. SQL) and connection state (Active or Passive) For secure login purposes ONLY, not a

general-use datapool file

Why Modify The Authentication Datapool?Why Modify The Authentication Datapool? Why?

If the password is encrypted during recording• The client/server security doesn’t allow a virtual tester to log into the

database multiple times simultaneously• Common problem when recording scripts

against secure SQL & Oracle systems• Passwords have been changed

since the initial recording• Stores original input during recording of the

secure login so data can be referencedduring later test runs

Why? If the password is encrypted during recording

• The client/server security doesn’t allow a virtual tester to log into the database multiple times simultaneously

• Common problem when recording scripts against secure SQL & Oracle systems

• Passwords have been changed since the initial recording

• Stores original input during recording of thesecure login so data can be referencedduring later test runs

Task: Authenticating logon information

Example Situation: Passwords are encrypted during the initial recording of the test run – Rational Suite TestStudio can’t play back the results properly because all virtual users after the first “Mary Smith” are locked out

Task: Authenticating logon information

Example Situation: Passwords are encrypted during the initial recording of the test run – Rational Suite TestStudio can’t play back the results properly because all virtual users after the first “Mary Smith” are locked out

How To Modify The Authentication DatapoolHow To Modify The Authentication Datapool

How To Modify The Authentication DatapoolHow To Modify The Authentication Datapool What is inside the script

#include <VU.h>{sparky = sqlconnect("sparky", "scott", LOOKUP_PWD,

"sparky", "oracle7.3");{ INFO SERVER "sparky"="192.168.241.101"; } /*1*/

The VU script uses the “LOOKUP_PWD” constant instead of the actual password.At playback, the authentication datapool issearched.

What is inside the script

#include <VU.h>{sparky = sqlconnect("sparky", "scott", LOOKUP_PWD,

"sparky", "oracle7.3");{ INFO SERVER "sparky"="192.168.241.101"; } /*1*/

The VU script uses the “LOOKUP_PWD” constant instead of the actual password.At playback, the authentication datapool issearched.

How To Modify The Authentication DatapoolHow To Modify The Authentication Datapool

In Rational TestManager

1) Open the Datapool manager

2) Select RTAuthentication (the Authentication DataPool)

3) Edit values accordingly

How To Modify The Authentication DatapoolHow To Modify The Authentication Datapool Tips:

Use realistic data• Experiment with different values/character lengths• Emulate patterns from current usage

Don’t delete or rename the Authentication Datapool (RTAuthentication)

Don’t modify the values of the preset columns

Tips: Use realistic data

• Experiment with different values/character lengths• Emulate patterns from current usage

Don’t delete or rename the Authentication Datapool (RTAuthentication)

Don’t modify the values of the preset columns



Evaluate Results

Manage Test Data

Create Test Suite

Plan Test

Design Test

Tester

(from Actors)

Create Test Scripts

Hints In Creating Test Suites: OutlineHints In Creating Test Suites: Outline Use of Transactor Distributions Use of Transactor Distributions

Use Of Transactor DistributionsUse Of Transactor Distributions Basic transactor tips were covered in

AT03 Basic Tips and Tricks for Performance Testing

Quick Review: Transactors control the rate at which Rational TestManager starts scenarios (groups of scripts)

As such, transactors control the arrival rate of work to the system under test

Here, we’ll be discussing transactor distributions When to use each one What each one “looks like”

Basic transactor tips were covered in AT03 Basic Tips and Tricks for Performance Testing

Quick Review: Transactors control the rate at which Rational TestManager starts scenarios (groups of scripts)

As such, transactors control the arrival rate of work to the system under test

Here, we’ll be discussing transactor distributions When to use each one What each one “looks like”

Constant Distribution – When To UseConstant Distribution – When To Use Use the constant distribution when input to the system under

test occurs at an exact and unvarying rate

Customer situation: Insurance company with 20 automated document scanners At start of day, scanners are each initialized 1 second apart All forms to be scanned are of the same length Each scanner takes exactly 20 seconds to scan one of these forms Scanned data is added to a customer record in a database

We would like to emulate the 20 scanners in order to verify that the database can handle the workload

Use the constant distribution when input to the system under test occurs at an exact and unvarying rate

Customer situation: Insurance company with 20 automated document scanners At start of day, scanners are each initialized 1 second apart All forms to be scanned are of the same length Each scanner takes exactly 20 seconds to scan one of these forms Scanned data is added to a customer record in a database


Constant Distribution – How To Do ItConstant Distribution – How To Do It

Contains SQL statements that add customer info to database

Contains SQL statements that add customer info to database

Simulatesstaggered initializationof scanners


Constant Distribution – What It Looks LikeConstant Distribution – What It Looks Like If the database processes & verifies addition of each

customer record in exactly one second: If the database processes & verifies addition of each

customer record in exactly one second:

No overlaps, no gaps!

No overlaps, no gaps!

Uniform Distribution – When To UseUniform Distribution – When To Use Use the uniform distribution when input to the system under test

occurs at a known average rate, but the time between each input is not constant

Customer situation: Insurance company with 20 automated document scanners At start of day, scanners are each initialized one second apart Forms to be scanned are of varying length Each form requires a different amount of time to scan

• Average scan time is 20 seconds After a scanner scans a form, it submits a customer record for addition to

a database


Use the uniform distribution when input to the system under test occurs at a known average rate, but the time between each input is not constant

Customer situation: Insurance company with 20 automated document scanners At start of day, scanners are each initialized one second apart Forms to be scanned are of varying length Each form requires a different amount of time to scan

• Average scan time is 20 seconds After a scanner scans a form, it submits a customer record for addition to

a database


Uniform Distribution – How To Do ItUniform Distribution – How To Do It

Contains SQL statements that add customer record to database




Uniform Distribution – What It Looks LikeUniform Distribution – What It Looks Like Assuming the database processes & verifies addition of

each customer record in exactly one second: Assuming the database processes & verifies addition of

each customer record in exactly one second:

Some overlaps, a few gaps!

Some overlaps, a few gaps!

Negative Exponential Distribution – When To UseNegative Exponential Distribution – When To Use Use the negative exponential distribution when input to the system

under test occurs at a variable rate

Customer situation: Insurance company with 20 customer service representatives (CSRs) CSRs enter customer information into application running on PCs This application, just like the automated scanners, adds customer information

as record in a database Incoming customer calls are assigned to CSRs in the order in which the queue

for that CSR was initialized Because of timing of incoming calls, as well as length of each call, the time

between the submission of customer data to the database varies

We would like to emulate the 20 customer service representatives & their desktop PCs in order to verify that the database can handle the workload

Use the negative exponential distribution when input to the system under test occurs at a variable rate

Customer situation: Insurance company with 20 customer service representatives (CSRs) CSRs enter customer information into application running on PCs This application, just like the automated scanners, adds customer information

as record in a database Incoming customer calls are assigned to CSRs in the order in which the queue

for that CSR was initialized Because of timing of incoming calls, as well as length of each call, the time

between the submission of customer data to the database varies

We would like to emulate the 20 customer service representatives & their desktop PCs in order to verify that the database can handle the workload

Negative Exponential Distribution – How To Do ItNegative Exponential Distribution – How To Do It



Simulatesstaggered Initialization of work queues

Simulatesstaggered Initialization of work queues

Negative Exp Distribution – What It Looks LikeNegative Exp Distribution – What It Looks Like Assuming the database processes & verifies addition of

each customer record in exactly one second: Assuming the database processes & verifies addition of

each customer record in exactly one second:

Many overlaps, more gaps!

Many overlaps, more gaps!

QuestionsQuestions What if the database response time increases with number of concurrent updates? In the previous response examples, the database response time for the addition of a customer record was always one second, no matter how many requests were being served concurrently

However, server response times often increase as the number of updates being concurrently handled increases, especially at higher user loads

If database response time = 1 second * number of updates currently in progress

What response times will users experience if they arrive in a constant distribution? negative exponential distribution?

What if the database response time increases with number of concurrent updates? In the previous response examples, the database response time for the addition of a customer record was always one second, no matter how many requests were being served concurrently

However, server response times often increase as the number of updates being concurrently handled increases, especially at higher user loads

If database response time = 1 second * number of updates currently in progress

What response times will users experience if they arrive in a constant distribution? negative exponential distribution?

Answer: Constant DistributionAnswer: Constant Distribution No difference from before No difference from before

Answer: Negative Exponential DistributionAnswer: Negative Exponential Distribution

A drastic difference!!! A drastic difference!!!



Evaluate Results

Manage Test Data

Create Test Suite

Plan Test

Design Test

Tester

(from Actors)

Create Test Scripts

Introduction To cwDBMON Introduction To cwDBMON cwDBMON is a user-

contributed tool

Monitors database(s) during test execution

Provides integration into Rational TestManager reports for results evaluation

cwDBMON is a user-contributed tool

Monitors database(s) during test execution

Provides integration into Rational TestManager reports for results evaluation

Using cwDBMON (1)Using cwDBMON (1) Selecting Parameters to Monitor Selecting Parameters to Monitor

Using cwDBMON (2)Using cwDBMON (2) Test-time monitoring of the database Test-time monitoring of the database

Using cwDBMON (3)Using cwDBMON (3) Integrating DB parameters into Rational

TestManager Integrating DB parameters into Rational

TestManager

Using Of cwDBMON (4)Using Of cwDBMON (4) Response VS Time graph with Oracle <Latch Efficiency> parameter

Response VS Time graph with Oracle <Latch Efficiency> parameter

Oracle Latch Efficiency

Oracle Latch Efficiency

Response timesResponse times

Downloading cwDBMON Downloading cwDBMON Remember, this is a user-contributed tool

Not part of any Rational product

Not supported by Rational Technical Support

But the user community provides help

Download instructions posted atftp://exchange.rational.com/exchange/outgoing/ruc_2002_at_22/cwdbmon.txt

Remember, this is a user-contributed tool

Not part of any Rational product

Not supported by Rational Technical Support

But the user community provides help

Download instructions posted atftp://exchange.rational.com/exchange/outgoing/ruc_2002_at_22/cwdbmon.txt

A Quick Word On EvaluationsA Quick Word On Evaluations In the Comments section:

Please indicate your level of performance testing experience• Beginning, Intermediate, Advanced

The 3 tips you’ve found most useful Format for next year

• Same as this year (separate sessions for Beginning & Intermediate/Advanced)

OR• One unified session for Beginning , Intermediate and AdvancedOR• Other suggestions

– Things you would like to see– Things you didn’t need to see

Any other comments are greatly appreciated

In the Comments section: Please indicate your level of performance testing experience

• Beginning, Intermediate, Advanced The 3 tips you’ve found most useful Format for next year

• Same as this year (separate sessions for Beginning & Intermediate/Advanced)

OR• One unified session for Beginning , Intermediate and AdvancedOR• Other suggestions

– Things you would like to see– Things you didn’t need to see

Any other comments are greatly appreciated

Scott [email protected]

Jeff [email protected]

Scott [email protected]

Jeff [email protected]

Thank You!Thank You!

This presentation will be posted by tomorrow at: http://www.rational.com/ruc

This presentation will be posted by tomorrow at: http://www.rational.com/ruc


©199

8, 1

999,

200

0, 2

001,

200

2 Ra

tiona

l Sof

twar

e - A

ll rig

hts

rese

rved

intermediate_and_adv

Documents

command line test scripts

command line scripts

command line test script

load test scripts

nontraditional test

java command line

command line new types

shell scripts