http driver tools · jmeter apache jmeter is a gui desktop application designed to load test...

Comparison HTTP Driver Tools

Tool Name

Purpose

Description of Parameter Evaluation Details with the scenarios

Yes

Yes

JMeterApache JMeter is a GUI desktop application designed to load test functional behavior and

measure performance.

Evaluation Parameter

Supportable? (Mark Yes/No accordingly)

Evaluators Description for the rating. Any other comment that he wants to add.

Ability to script dialogue to drive the specified type of traffic.

The evaluation should be based on the ease of scripting a dialogue, whether a GUI is available for the same.

Has a rich and user friendly graphical interface

Tested login page of a web based application, we need not write any scripts the GUI allows us to input the parameters along with the URL. Please check the attachment.

Ability to parameterize the load of the specified type of traffic by (requests initiated per second, max number of simultaneous requests / dialogues)

The evaluation should be based on the flexibility of the tool to drive the specified number of traffic interms of rate, concurrency and whether the load can be changed easily. The number of requests the tool can generate.

It can be used to simulate a heavy load on a server, network or object to test itsstrength or to analyze overall performance under different load types

The number of users accessing the system, was specified as 100, with the number of requests/ user specified as 10. So a total of 1000 requests was simulated. The concurrency rate was specified at 0, with the number of concurrent users as 100, accessing the system at any point of time..

Input.doc

Perf.doc

Yes

Yes

Yes

Ability to generate test result report (Automatically). Ability to customize the test report?

The evaluation should be based on what type of report the tools generate. - Does the tool generate the report automatically? - Is the generated report easy to understand ?- Does the generated report contain the required statistics, graphs, charts etc. - Can the report be customized, so that the user can only see the information he requires ?

JMeter comes with various reports to choose from and these reports can be used to do a graphical analysis of performance

Check the reports attached for the above test

Ability to start and terminate the test based on (Time, Number of Requests/Dialogues completed, others)

Can the tool start and terminate the test on a specified time- Can the tool start and terminate the test based on the number of requests- OR does one have to manually terminate the test

Several load statistics may be chosen with pluggable timers. The timers only specify the delay between requests. However one cannot specify the start time and the stop time for a test.

JMeter has an option for adding timersHowever the timers only specify the delay between requests, and as such one cannot specify the start time and the stop time for a test.Please find attached the snapshot for the different timer options available

Ability to group/sequence multiple test scripts.

Can the tool run multiple test scripts at a time ? Can the tool run multiple scripts in a sequence specified ?Can the tool facilitate grouping the scripts and running them in sequence?

Full multithreading framework allows concurrent sampling by many threads and simultaneous sampling of different functions by separate thread groups.

We can run multiple scripts at the same time, under the same test plan. However I am not sure if they can be run in a sequence or not? I have done the same using multiple thread groups one for the login application and the other for HelloWorld Servlet

Multiple_Threads.doc

Constant_Timer.doc

Timer.doc

Html.doc HTMLImage.doc

Graph1.doc Graph2.doc

Graphall.doc

Yes Same as AboveCost List the cost of the tool NA

Yes

Strengths Mention the strengths of the tool. Yes

Ability to script collection of test results at the completion of the test

Ability to base the test result on a collection of tests executed?

Can define a common test plan and test multiple processes simultaneouslyFreeware - Open Source

Trial Version / Demo / License

Is a trial version or demo available for the tool ? If a trial version is available, - were you able to use the tool effectively - were you satisfied by its performance ? Give more points to the tool which has a trial version. Also base your details on your satisfaction level after trying out the tool. Also mention the licensing requirements in case of open source software and if support is available for the same

It has a simple and user-friendly GUI which encourages you to use it from the start while you’re developing your application.

1. Freeware2. it’s easy to use. It has a simple and intuitive GUI.3. Its open source. You may modify it or add new features--if you have the time and skills

Cons Mention the weakness of the tool. Yes

System requirements Yes

Hardware NA

1. Its open source! There’s no guarantee of support or further development. However, we cannot consider this a real problem with a tool like JMeter, which easily can be replaced by another tool.2. Getting "correct" response times from JMeter forces you to understand how to make a realistic client/server setup. If you’re not careful you might end up measuring some incidental bottleneck, like your own workstation’s capacity.3.Its always advisable not to run JMeter on the same machine running the application to be tested. JMeter may use extensive resources that might affect the other application's performance if they are both run on the same machine.

Mention the system requirements and other pre-requisite s/w OR h/w required?

Requires working JDK1.4 environment

Does the tool come along with a hardware?

Features Give a descriptive list of features. Yes

1. Has a rich graphical interface2. Can be distributed3. Testing performance both on static and dynamic resources4. It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types.5. You can use it to make a graphical analysis of performance6. Remote Distributed Execution7. Highly Extensible


Tool Name Grinder

Purpose

Description of Parameter Evaluation Details with the scenarios

Yes

Yes

Grinder is a GUI desktop application designed to load test functional behavior and measure performance. Grinder uses Jython(a powerful scripting language)

for defining test.






The scripting of dialogue needs to be done by a jython script. Grinder has a GUI based interface but there is no drag and drop facility for scripting.

Tested login page of a web based application. We need to write Jython scripts for executing the tests. Attached is the script used.

Ability to parameterize the load of the specified type of traffic by (requests initiated per second, max number of simultaneous requests / dialogues)


The load can be easily parameterized by mentioning the number of threads in the grinder.properties file. One can also increase the number of processes(workers) by which simultaneous tests can be run. On the Grinder console one can also mention the time gap between samples.

Attached is the grinder.properties file used for this purpose. It has 10 processes and each process will run the 10 tests. Also attached is a screen shot of the console where in the interval between samples is kept at 1000ms.

Grinder_Console.doc

Http_py.doc

Grinder_Properties.doc

NA



Grinder does generate a report automatically that contains the standard response parameters. The generated report is text based and does not contain any graphs, charts. Also the report is not very user friendly. However grinder comes with a statistics package called net.grinder.statistics. Scripts can use this package to report custom statistics and to register additional views of standard and custom statistics. Hence the test report can be customized.

Attached is the report generated by grinder for process 0, where in 10 tests were run.

data_S1B00559-0.log

NA

No

What types of test results are provided that can be used for reporting the test results

The evaluation should be based on whether the following information is provided :- 1) Number of calls run 2) Number of calls failed 3) Length of test 4) average number of concurrent calls

The report generated by grinder gives the following information from console data:- 1) Successful Tests 2) Number of error 3) Mean Time 4)Mean Time Standard Deviation 5) Transaction per sercond 6) Peak Transaction Per Second 7) Mean Response Length 8) Response Bytes Per Second 9) Response Errors 10) Mean time to resolve host 11) Mean time to establish connection 12) Mean time to first byte Following information is generated in the data log :- 1) Thread Number 2) Run 3) Test Number 4) Milliseconds since start5) Test time 6) Errors 7) HTTP response code 8) HTTP response length 9) HTTP response errors 10) Time to resolve host 11) Time to establish connection 12) Time to first byte

Attached is the console report and data log.

Does the tool create a “Report” that can be used to report status to outside organizations

Is the report presentable, containing graphs, charts ?

The report generated by grinder is elementary. It generates a text based report.

grinder-console.data grinder-data.log

Yes

Yes

Yes

Cost NA



In grinder.properties one can mention the number of tests to run, the number of simultaneous processes to run. The console can be used to control the test on time basis. If you want the tests to run for an hour and your sample interval is set to 5 seconds, set the console to collect 720 samples.



It is possible to have different agent processes running scripts with different test numbers (e.g. one process running tests 1 to 5 and another running tests 6 to 10)., reporting to the same console.



The reports created are process wise. But if 10 different processes are running then there will be 10 different reports.

Attached ZIP file contains all the 10 reports for the 10 processes.

List the cost of the tool. Put the cost in column D.

Grinder is a free to use tool. There is no support available for this tool.

grinder-files.zip

Yes Grinder is a free to use tool.

Strengths NA

Cons NA



Mention the strengths of the tool.Put the strengths in column D

1) Grinder is open source software.2) Grinder tests are written in Jython(a Java Python implementation) which is very powerful. 3) Grinder can be setup very quickly and is easy to use.4) Grinder have a Java swing user friendly interface

Mention the weakness of the tool. Put the cons in column D

1) There is no support available for Grinder.2) There is no GUI based interface for defining the scripts (Like in Jmeter)3) The reports created by grinder do not contain graphs, chart.

System requirements NA

Hardware NA No additional hardware requirements.

Features NA

Mention the system requirements and other pre-requisite s/w OR h/w required? Put the requirements in column D

Grinder requires Java 2 Standard Edition 1.4 or above. JSSE 1.0.2, Apache Jakarta Ant 1.5.1, JUnit 3.8.1 and Checkstyle 3.3 are the other optional software requirements for Grinder depending on the purpose for which the tool is being used.

Does the tool come along with a hardware? Put additional hardware requirements in column D

Give a descriptive list of features. Put the features in column D.

1) Grinder tests are written in Jython(a Java Python implementation) which is very powerful.2) Grinder has a Java swing user friendly interface.3) Grinder can carry out load testing, capacity testing, functional testing and stress testing.4) The Grinder can be used to test databases using JDBC5) Jython allows any Java-based API to be used directly from the test script.6) A TCP proxy utility is supplied that can be used to intercept system interaction at the protocol level. It is useful for recording scripts and as a debugging tool.


Tool Name Load Runner (by Mercury Interactive)

Purpose

Description of Parameter

Yes

Yes

Load Runner is a GUI based tool designed to load test functional behavior and measure performance.






There is a GUI based interface to script the tests. There is an option to record the test depending on user interactions. So while defining the test, one can open a browser, put in request and carry on the actions he wants to test. Once the user has recorded the test, the test can be used for load testing.

Ability to parameterize the load of the specified type of traffic by (requests initiated per second, max # of simultaneous requests / dialogues)


The load can be easily parameterized with load runner. In the controller a scenario can be provided for a script. Here one can specify how many requests would get initiated per second, ramp up, ramp down and duration of the test.

Virtual_User.doc

ScheduleBuilder.doc

NA

NA

Yes

Yes



Load runner generates a report for the tests executed. The report contains information in statistical form which is easy to understand. Load runner can generate HTML, Word and crystal report. The user can also select only the details he requires in the report.


The evaluation should be based on whether the following information is provided :- 1) Number of calls run 2) Number of calls failed 3) Length of test 4) average number of concurrent calls

The load runner report gives the following details :1) Pass Tests2) Failed Tests3) HTTP responses4) Hits Per Second5) Throughput6) Transaction Summary7) Average transaction response time 8) Load scheme



The report generated by loadruner represents data in statistical form which is easy to understand.



The load runner can start and stop the tests based on the specified time. One can schedule the tests for a specified hour, where in the tests will get automatically executed. In the scenario one can mention the number of tests he wishes to execute.

Report.doc

scenario_timing_details.doc

multiple_test.doc

Yes

Yes

Cost NA



It is possible to run two test scripts simultaneously. In the scenario one can mention different load, ramp up, duration and ramp down details for different tests.



A single report can be created for multiple tests.


Load runner comes with different type of licenses. The cost of the tool depends on the number of users required to test the application. We called up Mercury Interactive sales team in Bangalore. They quoted $60,000 for 50 concurrent users. One needs to know the proper requirements and then contact the sales team for the exact pricing details.

Report_multiple_test.doc

multiple_test.doc

Yes

Strengths NA

Cons NA



An evaluation version of Load runner is available. The tests run for the evaluation were carried out on a 10 day evaluation version. The evaluation version has some restrictions like the number of simultaneous tests cannot exceed a specified value. For evaluation purpose, the evaluation version is satisfactory.

Mention the strengths of the tool.Put the strengths in column D

1) Load runner has an easy to use GUI based interface. One can create scripts by recording actions.2) Load runner has good reporting features. It creates HTML, Word and crystal reports with easy to understand statistical information.3) Load runner has separate interfaces to Create a test, run a test and analyze the results. This way the tool becomes very easy to use as it does not give the user too many toolbars, tool icons4) The load can be easily parameterized in terms of time and quantum.


1) Load runner is a very costly tool to use. It has licenses depending on the load for which the test is required to run.

SystemRequirement.doc

System requirements NA


Features NA


Attached is the system requirements for Load runner.


Give a descriptive list of features. Put the features in column D.

1) Load runner can drive hundreds and thousands of virtual users executing different business processes to emulate the production conditions a deployed application will face.2) Load runner supports around 40 protocols.3) Load runner's non intrusive, real time performance monitors provide a detailed metrics on all parts of the system under test.4) Load runner has good reporting features. It creates HTML, Word and crystal reports with easy to understand statistical information.

SystemRequirement.doc

Evaluation Details with the scenarios

Attached is the Virtual User screen of the loadrunner. It contains the actions recorded for a test.

Attached is the screen that displays the schedule builder.

Virtual_User.doc

ScheduleBuilder.doc

Attached is a complete Word report generated by loadrunner.

Attached is a screen shot which displays the timing details for a test.

Report.doc

scenario_timing_details.doc

multiple_test.doc

Attached is a screen shot which shows two different scripts running.

Attached is a report which contains consolidated data for the two tests run.

Report_multiple_test.doc

multiple_test.doc


Tool Name

Purpose

Description of Parameter

No

No

Selenium (by ThoughtWorks)

Selenium is a GUI based tool that runs on the user's browser ,just as the users do. It is designed to test the

acceptance-test and functionality of the web applications.


Supportability (Mark Yes/No accordingly)


Evaluation Details with the scenarios



There is no GUI based interface to build the testcase. Selenium Test case are defined in (static) HTML documents as a table of command-records. Each table row has three columns- command, target and value.

Attached is the screen that displays the test case and the test suit that are built using table layout of HTML.

Ability to parameterize the load of the specified type of traffic by (requests initiated per second, max number of simultaneous requests/dialogues)


Selenium tests run directly in a browser, just as real users do.Selenium is meant only for functional testing.

TestRun.doc

Yes

NA

Yes



Selenium provides a separate logconsole where we capture errors, warnings, information of the command that was executed and the debug information for our test case that we have run.

Attached is log console of the selenium were we can find the options to see the errors, warnings, informations and debug.


The evaluation should be based on whether the following information is provided :- 1) Number of calls run2) Number of calls failed3) Length of test4) average number of concurrent calls

The Selenium report gives the following details :- 1) Number of test runs2) Number of runs failed3) Number commands passed4) Number of commands failed5)Number of commands incomplete6) Including the Error mesages, warnings, Information on every command execution and debugs.

Attached is the control panel of the selenium were we can find the details about the test and the commands in the test case.



The report generated by Selenium is elementary. It generates a text based report.

ControlPalel.doc

LogConsole.doc

NA

Yes

Yes

Cost NA



Selenium was a functionality testing tool that typically started by running the TestRunner.html file in the selenium sub-directory. It only provides a command to Pauses the execution of the test script for a specified amount of time. This is useful for debugging a script or pausing to wait for some server side action.

Attached is a screen shot which displays the test case which uses the pause command.



It is possible to execute a sequence of test cases by using the "All" option provided in the selenium control pannel. Using that, all the test cases that are in our test suit will be executed. Also three modes of execution are available "run"," walk" and "step".



We can specify that the test results should be posted to a url where a servlet or other post-handling component knows how to deal with the Selenium results. Selenium can be integrated with an automated build. When the parameter "auto=true" is added to the URL, Selenium will run the entire suite of tests, and then post the results to a handling URL.

Attached is a report which contains the URL that explains how to include our test suit and post our test results to handler.


Selenium is a open-source software and can be downloaded and used without any charge.

TestResult.doc

CommandPause.doc

Yes

Strengths NA

Cons NA



Selenium is a open-source software and can be downloaded from http://www.openqa.org/selenium/download.action

Mention the strengths of the tool. Put the strengths in column D

1) Selenium helps to automate the acceptance tests and allows you to build better tested, reliable and maintainable software.2) Selenium runs in Internet Explorer, Mozilla and Firefox on Windows, Linux, and Macintosh.3) Selenium also supports tests written using a tool-specific document format.4) It is very easy to build the test cases and the test suit because we can build with HTML using a table layout


1) There is no GUI based interface for defining the scripts2) The reports created by selenium do not contain graphs, chart.3) Selenium does not allow you to create threads which is required to perform load test.

NA


System requirements


Since the test cases and test suites are built using HTML and they can also run in any application server where applications reside, no other special s/w or h/w is required for selenium. Various platform where selenium can run: 1)Windows 2)Linux 3)Mancintosh


Features NAGive a descriptive list of features. Put the features in column D.

1) Selenium runs in Internet Explorer, Mozilla and Firefox on Windows, Linux, and Macintosh.2)Selenium can be used for Browser compatability testing where it can test your application to see if it works correctly on different browsers and operating systems. The same script can run on any Selenium platform.3)Selenium can be used for System functional testing to create regression tests to verify application functionality and user acceptance.4) The core of Selenium, also referred to as the browser bot, is written in JavaScript. This allows test scripts to run inside supported browsers. The browser bot is responsible for executing commands received from test scripts which are written in either HTML using a table layout or a supported programming language.5)Selenium gives us in the browser window a good overview of what the results of running a single TestCase or an entire TestSuite are. Through the View DOM and Show Log options, we can zoom in on individual steps and commands.6) There are two modes of operation for Selenium: TestRunner and Driven

http driver tools · jmeter apache jmeter is a gui desktop application designed to load test...

Documents