why & how-performance test
TRANSCRIPT
-
8/10/2019 Why & How-Performance Test
1/25
1
Why & How : Performance Test
For: QAAC
By: Deepak
-
8/10/2019 Why & How-Performance Test
2/25
2
Performance Test References
Investigate Performance Early; Validate Performance Last- by Scott Barber is the Chief Technologist, CEO, and President of PerfTestPlus,
Inc. Reference article: Performance Investigatorin Better Software Magazine(March 06 issue).
To derive a good testing strategy out of scalability andperformance testing, there needs to be testable and
measurable requirements - by Srinivasan Desikan is the director ofquality engineering at Siebel Systems, Inc. Reference article: Scalability andPerformance Testing of Server Software (Website http://www.stickyminds.com
published July 05).
http://www.stickyminds.com/BetterSoftware/magazine.asp?fn=cifea&id=74http://www.stickyminds.com/sitewide.asp?ObjectId=9392&Function=DETAILBROWSE&ObjectType=ART&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28relevance%29%2AK%28simplesite%29%2AF%28performance%29%2A&sidx=0&sopp=10&sitewide.asp?sid=1&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28http://www.stickyminds.com/sitewide.asp?ObjectId=9392&Function=DETAILBROWSE&ObjectType=ART&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28relevance%29%2AK%28simplesite%29%2AF%28performance%29%2A&sidx=0&sopp=10&sitewide.asp?sid=1&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28http://www.stickyminds.com/sitewide.asp?ObjectId=9392&Function=DETAILBROWSE&ObjectType=ART&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28relevance%29%2AK%28simplesite%29%2AF%28performance%29%2A&sidx=0&sopp=10&sitewide.asp?sid=1&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28http://www.stickyminds.com/sitewide.asp?ObjectId=9392&Function=DETAILBROWSE&ObjectType=ART&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28relevance%29%2AK%28simplesite%29%2AF%28performance%29%2A&sidx=0&sopp=10&sitewide.asp?sid=1&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28http://www.stickyminds.com/sitewide.asp?ObjectId=9392&Function=DETAILBROWSE&ObjectType=ART&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28relevance%29%2AK%28simplesite%29%2AF%28performance%29%2A&sidx=0&sopp=10&sitewide.asp?sid=1&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28http://www.stickyminds.com/sitewide.asp?ObjectId=9392&Function=DETAILBROWSE&ObjectType=ART&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28relevance%29%2AK%28simplesite%29%2AF%28performance%29%2A&sidx=0&sopp=10&sitewide.asp?sid=1&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28http://www.stickyminds.com/sitewide.asp?ObjectId=9392&Function=DETAILBROWSE&ObjectType=ART&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28relevance%29%2AK%28simplesite%29%2AF%28performance%29%2A&sidx=0&sopp=10&sitewide.asp?sid=1&sqry=%2AZ%28SM%29%2AJ%28MIXED%29%2AR%28http://www.stickyminds.com/BetterSoftware/magazine.asp?fn=cifea&id=74 -
8/10/2019 Why & How-Performance Test
3/25
3
Table of Contents
Performance Test description and its need
Performance Test Objectives
Performance Testing Engagements
PVS Test Terminology
PVS Test Requirements
PVS Test Process
PVS Tool-set
Transaction Response Time
Response Times Vs Service-level agreements (SLAs)
Monitor Application and Database Server UsagePerformance Testing Best Practices
Benefits of Performance Testing
Summary and Conclusion
-
8/10/2019 Why & How-Performance Test
4/254
What is Performance Testing?
The process of exercising an application by
emulating actual users. This can be used tomonitor how the database, application code andhardware will act under load.
Performance testing verifies that the applicationmeets specified requirements under normalconditions.
-
8/10/2019 Why & How-Performance Test
5/25
5
The Need For Performance Testing
No matter how rich application is functionally, if
it fails to meet the performance expectations(SLA) of customer, the application will bebranded a failure.
-
8/10/2019 Why & How-Performance Test
6/25
6
Performance Test Objectives
Determine business transaction response time
Identify system bottlenecks
- Server resource usage e.g. CPU, Memory and Disk Space- Network Bandwidth(throughput) and Latency(delay)
e.g. Limited network throughput speeds serve to introduce alatency(delay) when transmitting larger amounts of data at specific
location
Determine system (hardware/software) optimalconfiguration
Verify current system capacity and scalability for futuregrowth
- Determine how many users the system can support
- Determine if the application will meet its SLA
-
8/10/2019 Why & How-Performance Test
7/25
7
Performance Test Engagements
Performance Risk Assessment
-Determine if application needs to be PVS tested bycompleting performance risk assessment matrix
Performance Volume Stress(PVS) Testing
- Test Strategy with Action plan
-Script development
- Test execution- Resource monitoring
-Analysis & Recommendations
Load-balancing and Fail-over Test
Network Analysis
-
8/10/2019 Why & How-Performance Test
8/25
8
PVS Test TerminologyTypes of Tests
Load Test: To determine the impact on an applications servicelevel for a specific number of users and user types.
Volume Test : Inject heavy workload (e.g. future volumes) toshow if the system can handle the peak volume of data. Testsserver response time under heavy load and varying user levels,measures round-trip response time and determines maximumnumber of users.
Stress Test: Test the application or server to determine a loadpoint at which the application suffers severe degradation ormalfunction.
Network Application Analysis Test: To capture the network
characteristics of an application and to proactively seekinefficiencies to forecast potential performance and network risk.
Latency: The amount of time it takes a packet to travel fromsource to destination. Together, latency and bandwidth define the
speed and capacity of a network.
-
8/10/2019 Why & How-Performance Test
9/25
9
Other Terminology
Workload: The total user load exercised in a productionenvironment.
Workload Model: A subset of actual workload used forperformance testing (80/20 rule: 80% of the workload is
generated by 20% of the transactions mix).Test Cycle: A group of testing scenarios designed to accomplish aspecific threshold or condition (production level conditions, futureproduction conditions, end-state conditions).
Scenario: A logical unit of work or a transaction sequence toexercise a type of performance test that achieves a performancetest objective.
Script: Automated set of instructions to drive the performance test
during execution.Virtual User: A software process that simulates real userinteractions with the application.
Response Time: The time it takes between initial request and
complete download of response (rendering of entire web page).
-
8/10/2019 Why & How-Performance Test
10/25
10
PVS Test Requirements/Prerequisites
Technical design and architecture, Non-functional businessrequirements and Data flow diagrams.
Hardware/software configuration and access to monitorresource usage.
Establish and document SLAs (e.g. scope and objectives,
response time, availability, business scenarios, acceptancecriteria and project timelines etc).
Stable test environment, test data, user IDs or any otherunique data required for the Performance test types.
Use-cases: Test scripts reflecting how the end-users are
using the system.
-
8/10/2019 Why & How-Performance Test
11/25
11
Perfo
rman
ceTestingPha
ses
Performance Testing Processes
Templates/Documents Needed
Performance Testing Questionnaire
Test Cases
Templates/Documents Needed
Performance Test Strategy
Effort Estimation Guideline
Templates/Documents Needed
Performance Test Report
RequirementGathering
Effort Estimation And
Planning
Scripting And Execution
Result Analysis , Reporting And DefectTracking
Templates/Documents Needed Performance Testing Best Practices
Scripting Standards and Naming
Conventions Guidelines
Workload calculation
-
8/10/2019 Why & How-Performance Test
12/25
12
PVS Test Process
Gather and ReviewChecklist Requirements
Test Strategy/ApproachDocument-Scope and Objectives
-Identify needed test
resources-Milestones
Script Development
-Design test-Develop workload model-Build test bed
-Create script test data
Test Execution-Performance/Load test-Review initial output
-Volume and Endurance test-Stress test
Result Analysis and FinalReport-Executive summary
-Recommendations toimprove application
performance-Results and Analysis
-
8/10/2019 Why & How-Performance Test
13/25
13
Performance Engineering Methodology
Performance Engineering
Understand System (Sys.
Architecture, Workload,
Performance Goals)
Develop Test Assets
(Test Plan, Test Scripts,
Monitoring Strategy etc)
Execute Tests Collect monitoring metrics
Analyze Results andmonitoring metrics
Performance criteria are met
TUNE
Criteria not met
Re-test after tuning
Complete Testing
-
8/10/2019 Why & How-Performance Test
14/25
14
PVS Tool-set
Mercury-HPs LoadRunner, Borland-Segues SilkPerformer,Compuwares QA Load, IBMs RPT, and Empirixs eLoad
-Load injection and scenario design mirrors production
-Round trip transaction times-Monitor application and architectural components
Perfmon/Sitescope
-Performance logs and alerts supports detailed monitoring of theutilization of operating system resources.Web Analytics/Trends/Topaz/Gomez
-Interactive web analytics reporting(monitor production usagepattern)Vantage/Sniffer/Netsense
-Network Management - Analysis for bandwidth usage andlatency issues
-
8/10/2019 Why & How-Performance Test
15/25
15
Multi-Tier Architecture
For instance, its veryimportant to performancetest a complex architecture
which support highlyscalable and robust load-balancing mechanisms at alltiers of the architecture using
a combination of hardwareand software load-balancing.
All tiers can be scaled
individually by buildingclusters of servers for load-balancing and highredundancy.
-
8/10/2019 Why & How-Performance Test
16/25
16
End-to-End Interface Architecture Diagram
PWMCICS
[Mainframe]
JSP MQ
MQ
SeeBeyondAdmin ServerHTTP call
[XML]
MQ Wrapper program
Request[CobolCopybook]
Response[CobolCopybook]
AgentInsured
Home office employee
Policy Request
-
8/10/2019 Why & How-Performance Test
17/25
17
Monitor Server Usage
Timings-Response time-Execution time-Wait and connection times
Resource Consumption
-CPU, Disk Space and Memory usage-Number of connections, active threads, timeouts and Servlets
requested, loaded and active-Web page counts, size and timingsDatabase Performance
-Physical reads & writes
-Access paths and connections-Locking performance- I/O, buffer and wait counts and timing
Additional application specific metrics
f
-
8/10/2019 Why & How-Performance Test
18/25
18
Measurement of Application Business TransactionResponse Time
-
8/10/2019 Why & How-Performance Test
19/25
19
Response Times Vs Service-level agreements
(SLAs)Average Transact ion Response Time
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
ResponseTime
(Seconds)
Cycle 1-1300 users 0.632 0.587 0.07 0.066 0.59 0.658 0.594 0.622 0.595 0.594
Cycle 2-1500 users 0.609 0.892 0.068 0.063 0.533 0.536 0.581 0.611 0.586 0.586
Claims
Login
Corres.
Login
Claims
Enter ID
Corres.
Enter ID
Claims
View
Image
Corres.
View
Image
Claims
Item
Close
Corres.
Item
Close
Claims
Logout
Corres.
Logout
Login Acceptance
Criteria 10 seconds
View Image
Acceptance
Criteria 30
seconds
Enter ID
Acceptance
Criteria 5
seconds
Logout
Acceptance
Criteria 5
seconds
Item Close
Acceptance Cri teria
10 to 60 second s
-
8/10/2019 Why & How-Performance Test
20/25
20
Monitor Database Server
-
8/10/2019 Why & How-Performance Test
21/25
21
Monitor Database Server on Mainframe
Database Server Timings Application AAA
Batch Job / MetricCycle: 1B
3.5mil records09/15 09/22
Cycle: 25.4mil records09/22 90/30
Cycle: 311.6mil records
10/01 10/10
GXX0000
SQL Total Elapsed Time 01:44:03.25 00:55:29.78 06:10:12.24
SQL CPU Time 00:04:27.76 00:01:53.88 00:31:02.91
SQL I/O Elapsed Time 01:15:38.35 00:38:32.57 03:38:11.97
SQL Wait for Read 00:00:13.71 00:00:07.06 00:00:08.20
SQL Lock/Latch Time 00:00:00.00 00:00:00.09 00:00:00.57
Total SQL Elapsed Time 01:57:23.10 01:12:36.65 06:26:08.99
DML Comparisons Application AAA
Batch Job / MetricCycle: 1B
3.5mil records09/15 09/22
Cycle: 25.4mil records09/22 90/30
Cycle: 311.6mil records
10/01 10/10
GXX0000V
Total Cursor Open / Closes 3,235 1,336 8,462Total Fetch 7,035 3,323 25,443Total Select 441,554 170,307 2,284,024Total Insert, Prepare, Delete 0 0 0Locks Suspended / Timeout 0 0 0Max Page Locks Held 1 1 1Total Waits for Lock/Latch 132 620 752Locks Requested 112,028 5,594 2,149,614
-
8/10/2019 Why & How-Performance Test
22/25
22
Performance Testing Best Practices
Performance testing requires a different mind set and skill-set tothat of functional testing.
-Best started early in the development life cycle wheneverpossible.
Understanding business requirements and expectations.
-User activities and behaviors is key to designing suitable tests.
-Ensure tests represent realistic usage of the application.Automated test tools coupled with fast backup and restoremechanisms are essential due to the need to repeat tests multiple
times.System bottlenecks can rapidly become very technical in natureand consume considerable resources and efforts to diagnose.
-Resolution may require considerable re-work and even redesign.
-
8/10/2019 Why & How-Performance Test
23/25
23
Benefits of Performance Testing
The ultimate goal of all performance testing is to optimize performancewhile minimizing costs.
Does the application perform as required?. Will it continue to perform asrequired?.
Will it meet current and future performance requirements for the leastcost?.
Improve new or upgraded applications and third-party services quality,workflow and reporting.
Reduce system downtime by isolating and resolving infrastructurebottlenecks, if any.
Improve quality from a users prospective.
Early identification of major application defects and architectural issuesReduce cost of change, system costs and increase profits.
Cost per Defect
Design Test Production$1x $10x $100x
-
8/10/2019 Why & How-Performance Test
24/25
24
Summary and Conclusion
In this presentation we have covered
Performance Testing and its need to start early in the
development process.
Performance Testing Objectives and Engagements.
PVS Testing Terminology, Requirements, TestProcess, Tool-set, Transaction Response Time,Service-level agreements (SLAs), Application andDatabase Server Usage.
Performance Testing Best Practices.
Benefits of Performance Testing.
-
8/10/2019 Why & How-Performance Test
25/25
25
Questions ?For More Information contact:Deepak Sharma
Email: [email protected]: (860)4032195(O), (919)9397091(C)
mailto:[email protected]:[email protected]