Best Practices for Identifying Web Performance Issues Before Your Customers Do

A Gomez/Dollar Thrifty Web Performance Testing Case Study:

Imad Mouline, CTO, GomezEmanuel Daniele, Sr. Solution Engineer, GomezJim Arrowood, DTAG, Director, Web Development and Architecture

Agenda

Reality Load Overview – Imad Mouline

Dollar Thrifty Case Study – Jim Arrowood

Product Demo – Emanuel Daniele

Internet

Content DeliveryNetworks

MajorISP

Local ISP

Mobile Carrier

3rd Party/Cloud Services

Browsers and devices Users

The Challenge of Delivering Quality Web Experiences

Storage

Load Balancing

Web Servers

App Servers

DB Servers

Mobile Components

Storage

Load Balancing

Web Servers

App Servers

DB Servers

Mobile Components

Users

The Web Application Delivery Chain

Traditional zone of control

Traditional testing: “OK” …user is

NOT happy

Internet

Content DeliveryNetworks

MajorISP

Local ISP

Mobile Carrier

Storage

Load Balancing

Web Servers

App Servers

DB Servers

Mobile Components

Users3rd Party/

Cloud ServicesBrowsers

and devices

The Web Application Delivery Chain

• Network peering problems

• Outages

• Inconsistent geo performance• Bad performance under load• Blocking content delivery• Incorrect geo-targeted content

• Configuration issues• Oversubscribed POP• Poor routing optimization• Low cache hit rate

• Network peering problems

• Bandwidth throttling• Inconsistent

connectivity• Configuration

errors• Application

design issues• Code defects• Insufficient

infrastructure

• Poorly performing JavaScript

• Browser/device incompatibility

• Page size too big

• Too many objects

• Low cache hit rate

• Network resource shortage

• Faulty content transcoding

• SMS routing / latency issues

The Challenge of Ensuring Quality Web Experiences

Zone of customer expectationZone of customer expectationTraditional zone of control

Traditional testing: “OK” …user is

NOT happy

Internet

Content DeliveryNetworks

MajorISP

Local ISP

Mobile Carrier

Storage

Load Balancing

Web Servers

App Servers

DB Servers

Mobile Components

Users3rd Party/

Cloud ServicesBrowsers

and Devices

BackboneHigh volume load (HTTP, Browser)

Find infrastructure breaking points Define capacity headroom

100,000+ consumer- grade desktops

168+ countries

2,500+ ISPs

Major mobile carriers around the globe

Gomez Load Testing: On-Demand Realistic Load Testing from Browser to Data Center

Last Mile Real-world load

Find user experience breaking points Accurately measure response time

100+ commercial-grade nodes & data centers

Gomez Network: The World’s Most Comprehensive Performance and Testing Network

Web Performance Management and Load Testing

100+ locations

Backbone

Cross-Browser Testing 500+ browser/OS combo’s5,000+ supported devices

Virtual Test Bed

Web Performance Management and Load Testing

100,000+ locations

Gomez Last Mile

Reality Load XF – Overview & Demo

SaaS with no investment or maintenance costs and rapid paybackSelf-service, full turnkey solution, or tailored to meet your needs with Gomez Professional Services offerings

Tests from an “Outside-in” customer point of view, with drill down to all web application components

Full desktop browser testing across globally distributed geographies

Expanded Load Generation

• Additional Geographies for Based Load high volume testing

• More Last Mile peer populations

Simplified Test Provisioning

• More easily schedule complex test scenarios

• Save and re-use test scenarios

• Last Mile UI scheduling improvements

Enhanced Reporting

• Detailed reporting for complex scenarios

• Filter results by load origin or time

• Streamlined data and chart exports

New in July 2009

Dollar Thrifty Case Study

Jim Arrowood, DTAGDirector -Web Development and Architecture

8

Who is Dollar Thrifty?

• Driven by the mission “Value Every Time,” the Company’s brands, Dollar Rent A Car and Thrifty Car Rental, serve value-conscious travelers in over 70 countries.

• Dollar and Thrifty have over 600 corporate and franchised locations in the United States and Canada, operating in virtually all of the top U.S. and Canadian airport markets.

• The Company’s approximately 6,400 employees are located mainly in North America, but global service capabilities exist through an expanding international franchise network.

9

Dollar and Thrifty.com

10

The Dollar Thrifty Environment

Dollar.comMiddle-Tier

Thrifty.comMiddle-Tier

Rate EngineMiddle Tier

Reservation Middle Tier

Database

11

The Problem

• In a highly, horizontally scaled environment, development, test, and staging environments rarely 100% match production

• Dollar Thrifty significantly redesigned both of our websites in 2008 leaving question as to how the sites will perform in the peak of the 2009 summer

• Internal load tests using traditional methods in test and staging environments did not provide the needed confidence

12

The Gomez Relationship

• Dollar Thrifty has utilized the Gomez ActiveXF platform for several years– Synthetic Tests of Key Business Processes

• Monitors for response time (Home Page Load & Reservation Process)

• Monitors for successful execution (Home Page & Reservation Process)

13

Gomez Reality Load

• In Early 2009, Dollar Thrifty teamed with Gomez to launch the first external load test of the company’s infrastructure and key software platforms utilizing Gomez Reality Load

14

The Goals

• Regardless of how much load, where is our weakest point?

• Can we handle our previous year’s peak +25% traffic?

15

Identify Core Business Processes

• How do the vast majority of the consumers interact with the site?

• Use 80-20 rule– Shop for Rates– Make Reservation– Modify Reservation– Cancel Reservation

80%

20%

Core Business ProcessesOther Processes

16

Distribute Core Business Processes

• By percentage, how do consumers interact with the site?– Shop for Rates – 70%*– Make Reservation – 15%*– Modify Reservation – 10%*– Cancel Reservation – 5%*

70%

15%

10%5%

Shop for RatesMake ReservationModify ReservationCancel Reservation

*Hypothetical 17

Forecast Visitors

• Based on historical web analytic data and the goal of the test, compute the intended visitor capacity of the site within an hour– 10,000 visitors per hour at peak historically*– If we intend to support peak +25% then our visitor

forecast is 12,500 visitors

*Hypothetical

0

2000

4000

6000

8000

10000

12000

Jan Feb Mar-Apr May-Jun Jul-Aug Sep-Oct Nov-Dec

Vis

ito

rs P

er

Ho

ur

18

Identify the Needed Scripts

• Throughput Tests – Where are we weakest?– One script per core business process with no

“think-time”• Capacity / Load Tests – How much load can we

legitimately handle?– One script per core business process with “think-

time”

19

Develop Needed Scripts

• Gomez Script Recorder allows the client to create of the script, inject look-up data, and replay the script

• RealityLoad utilizes the same scripting engine as ActiveXF

• The base of our scripts existed as we already had them built to execute in the ActiveXF platform– Added logic to pull from look-up data – Added think time

• Once, complete the tests are loaded into the Gomez portal

20

Configure Tests

• Using the Gomez portal, tests are configured and scheduled for execution

21

Execute

• Execute tests at off peak time – We chose 2 AM to 4 AM CST

• Establish war room with key system owners• Establish bridge line with key infrastructure and

monitoring personnel• As tests are executing ensure all internal monitoring

tools in all tiers of the application are capturing as much detail as possible without skewing results– CPU, Memory, Disk IO, Network Throughput, etc.

22

Analyze Results

• Each system owner and infrastructure owner collects key metrics and findings and submits to the test leader

23

Discuss Key Findings

• Thrifty.com front-end servers ran hot while the middle-tier was cold and overpowered

• Dollar.com front-end servers ran cool while the middle-tier was running on target

• Discovered excessive calls for validation to legacy reservation system

• Thrifty.com middle-tier had sticky sessions enabled causing load balancing to become less ineffective

24

Make Corrective Actions & Retest

• Mitigate the discovered issues and risks– We were able make a simple configuration change to

resolve the “sticky sessions” issue– We were able to move Thrifty middle-tier boxes into

the Thrifty.com front-end and Dollar.com middle-tier to better distribute our load with no additional expense

• When time and budget allows, retest to ensure mitigating actions resolve the issue and ensure no additional bottlenecks have appeared

25

Demo

Emanuel Daniele Gomez Reality Load Demo

26

Wrap Up

Questions & Answers Check back on QA Forums

To find out more about Reality Load:

cmason@gomez.commgil@gomez.com

Product Informationhttp://www.gomez.com/products-solutions/products/load-testing/2 Minute Explainerhttp://www.gomez.com/resources/video-library/gomez-reality-load-testing-two-minute-explainer/

Company: Online presence for a popular TV show

• Following episodes of the TV show the web site sees high traffic spikes

• Goal was to achieve 1500 logins per minute

• Load tested DB to improve performance in anticipation of another traffic spike

Ensuring Performance Of Login Process

1

Internet

Content DeliveryNetworks

MajorISP

Local ISP

Mobile Carrier

3rd Party/Cloud Services

Browsers and devices

Storage

Load Balancing

Web Servers

App Servers

DB Servers

Mobile Components

Storage

Load Balancing

Web Servers

App Servers

DB Servers

Mobile Components

Users

Application Bottleneck Causes Immediate Response Time Issue

• As users were added, the response time of step 3 (the login) climbed immediately

• The test bottlenecked at 160 logins per minute (Goal 1500)

• But quickly dropped off as users received server errors

• New login query was not optimized and was bottlenecking the database servers’ CPUs

Application Bottleneck – Re-test

30

• Following the tuning effort the application performance was improved.

• This time the bottleneck was at 1300 logins per minute.

• A bandwidth limit was reached at just under 90 Mbps, resulting in an overall slowdown as users were added.

• This highlights:• The importance of re-testing

following each change.• The fact that applications often

have many bottlenecks, that can only be uncovered one at a time.

Company: Online Retailer

• Several 3rd Parties now involved in serving up key content

• Goal was to validate performance of entire application

Ensuring Performance of All 3rd Party Components

Internet

Content DeliveryNetworks

MajorISP

Local ISP

Mobile Carrier

3rd Party/Cloud Services

Browsers and devices

Storage

Load Balancing

Web Servers

App Servers

DB Servers

Mobile Components

Storage

Load Balancing

Web Servers

App Servers

DB Servers

Mobile Components

Users2

The load increases throughout the test

The transaction rate increases and then falls off as response times climb

Errors are seen, all on a 3rd party object

Response Times Rise Due To 3rd Party Object Error

• 3rd party hardware was insufficient for overall demands on application

• Based on SLAs 3rd party had to improve performance to get paid

Company: Regional Online News Source

• Began testing for the 2008 election season

• Goal was to validate overall performance focusing in 2 key regions

Ensuring Performance in Key Markets

Internet

Content DeliveryNetworks

MajorISP

Local ISP

Mobile Carrier

3rd Party/Cloud Services

Browsers and devices

Storage

Load Balancing

Web Servers

App Servers

DB Servers

Mobile Components

Storage

Load Balancing

Web Servers

App Servers

DB Servers

Mobile Components

Users

3

By traditional test standards the test passed

Page response times stayed under 4 seconds, outside of one brief blip

There was only 1 page error and 11 errors total out of 60000+ transactions

Increase and hold load and not exceed response times of 4 seconds and Success Rate of 99%

No Performance Issues Detected From Data-Center

Last Mile data showing substantial number of measurements greater than 4 seconds

Performance Issues Detected From Real User Desktops

Key geographies for this customer are New York and Pennsylvania

Last Mile Case Study: Primary Geographies

36

The response time never met the 4 second average

goal.

Availability was Less than 99%.

Key geographies for this customer are New York and Pennsylvania.

By these standards the test failed.