Low Cost Load and Performance Testing

Example Test

Why Test?

Performance is a direct reflection on your companyYour client does not care about the ‘cause’

Whose Fault Is It?

Cause can come from many sources◦Poor architecture◦Poor configuration◦ Inadequate hardware◦ Insufficient network bandwidth

Why Isn’t Performance Testing Done

Takes timeTakes production or production- like

environmentTakes a special skill setTakes specialized tools

Why Isn’t Performance Testing Done

Can take lots of $$$$$!◦Proprietary tools – can cost 100’s of

thousands of dollars to buy and maintain

So What Are We Supposed to Do?

Option 1 – Do nothingOption 2 – Keep buying hardwareOption 3 – Buy an expensive load and

performance tool. Hire a tool expert to use it.Option 4 – Allocate in-house personnel to

build a custom solution

So What Are We Supposed to Do?

Option 5 – Pray

Low Cost Options

Big tool… little price◦Utilize an industry recognized best-of-

breed tool at a greatly reduced cost◦Pros

Access to functionality that normally costs lots of $$

Good for client server or web applications

◦Cons Proprietary languages. Need ‘tool guru’ Need hardware, bandwidth to run it

Low Cost Options

Open Source◦40+ options - JMeter, OpenSTA,

WebLoad, Grinder◦Pros

Good price Can extend and adapt to your needs Good for simple tests

◦Cons Not as easy to use. Lacks functionality Only for web based products Need hardware, bandwidth to generate load

Low Cost Options

Cloud Based◦ Rapidly growing offering

BrowserMob, LoadStorm◦ Pros

Low price Can easily scale to thousands of users Can completely eliminate the need for hardware

◦ Cons Early in the development phase Only for web-based products Analysis is slowly coming up to speed

Cost Comparison

How Do I Choose?

How do I know what is best for me?

Understand Your True Needs

Performance, Load, Stress, Reliability◦All unique areas of testing

Performance - not to find functionality bugs , but to eliminate system bottlenecks and establish a baseline for future regression tests Takes a white-box approach. The system is inspected

and monitored "from the inside out" and from a variety of angles. Measurements are taken and analyzed and tuning is done accordingly .

Understand Your True Needs

Load - process of testing the system by feeding it the largest tasks it can operate withGoals

expose bugs such as memory management bugs, memory leaks, buffer overflows, etc

Ensure that the application meets the performance baseline established during performance testing.

Understand Your True Needs

Stress - one primary objective, and that is to determine the load under which a system fails, and how it fails. The main purpose is to make sure that when

the system fails, it recovers gracefully. i.e. – no locked sessions, no corrupted data, if load is

removed does it return to normal working condition

Understand Your True Needs

Reliability – determine application performance over a prolonged period of time. Load can vary between very high and very lowThe main purpose is to make sure that the

system performance does not degrade over time. i.e. – slow memory leaks, stale sessions, other

processes interfering with off hour activities

Understand Your True Needs

Understand your user scenarios◦ Analyze traffic patterns

Google Analytics, application logs◦User types

Administrative, buyer, seller, poster, responder, browser, etc…

◦User activity Downloading, uploading, purchasing, saving cart,

submitting,◦User ‘load’

Mix of user type and activity. How many of each

Don’t Go It Alone

May not do load and performance testing for months/years◦ Difficult for most companies to have SME

Reach out to colleagues, user groups, consultants◦ Can help ‘reduce the noise’◦ Guide you through the process

Keep Customers Happy

Performance is something customers notice when it is poor

Don’t give customers a reason to look at other options

Questions/Discussion Points

Questions?

Lawrence Nuanez – Senior Consultant – ProtoTestlnuanez@prototest.com303-703-1510 – ext 109www.ProtoTest.com