Download - Preparing Website for the Holiday Rush
Keynote Benchmark
Making a Site,and Checking it Twice
A Guide to Preparing a Web site for the Holiday Rush
f any retailer needs proof of the importance of load testing, they need only look to the headlines around the launch of the new Apple
iPhone 4. All day on June 15, Apple and AT&T’s online stores went down and out. Tech bloggers spent hours trying to place online
orders, and hours more on hold with customer service. An official announcement from AT&T called it “the busiest online sales day in
AT&T history.” Yet despite the problems, they managed to sell out their first day’s allotment before the day was out.I
2
Retailers dream of having such problems as rolling out a new blockbuster and taking orders for 600,000 units in one day. But you can count on one hand the number of sellers who command such fanatic loyalty and patience. For the rest, site outages, freezes and checkout failure mean lost revenue that will likely never be made up — and those losses mount minute by minute, hour by hour, especially during the peak holiday season, when most retailers rack up most of their annual business. During those critical four to six weeks at the end of the year, when the site breaks, the bottom line breaks, too. So: it’s July or later as you’re reading this. The biggest, most successful online retailers have been prepping their sites for the holidays for six or seven months now. The next tier started in June. Retailers who have not started to stress test their sites could be looking at many sleepless nights come the last months of the year, and potentially serious hits to their 2010 revenues.
ONLINE MAKES THE BOTTOM LINEThere’s no underestimating the importance of
e-commerce to overall retail success,
particularly during the crucial holiday season.
Every year from 2000 through 2007, online
sales posted double-digit growth, handily
outpacing in-store sales.i And last year, as
retail struggled to recover from its worst
holiday ever (in 2008), online holiday sales
grew at four times the rate of total retail
growth — though admittedly modest growth,
at 4 percent for online sales,ii and just 1
percent for total retail.iii (For the year,
e-commerce sales grew 2 percent, while overall
retail sales fell by nearly 3 percent, according
to the U.S. Department of Commerce.)
With online playing such a critical
role in holiday sales, it is impossible to ignore
the real bottom-line costs in lost revenue of
site slowdowns and outages. Site metrics and
sales data show the revenue value of every site
visitor, and site testing can show the actual
cash lost when visitors abandon the site
because of poor performance. As the
accompanying chart shows, using actual
results, a traffic spike 25 percent beyond the
site’s optimum capacity can cost $100,000 in
lost revenue every hour.
Further complicating matters is the
huge unknown factor of mobile for 2010.
With the tremendous and ever-accelerating
proliferation of browser-equipped, app-loaded
smartphones, mobile is sure to play a bigger
role than ever, and likely to exceed even the
most optimistic retailer’s projections. All those
smartphones searching for products,
comparing prices, and making purchases
will be accessing the same back-end
databases and applications as all the PC-based
browsers, putting even more stress on
e-commerce systems.
THE FIRST BILLION-DOLLAR ONLINE DAY?One likely milestone in the upcoming 2010
holiday season will be the first billion-dollar
e-commerce day. Sales broke the $900
million mark on Green Tuesday (December 15
last year), and at least nine shopping days
exceeded $800 million.iv A major east coast
snow storm kept shoppers home-bound and
contributed to brisk online sales. This year,
mobile, with shoppers logging on even as they
are out in the stores, is likely to have an even
greater effect.
FIRST THINGS FIRST: UPDATESGetting a site into the near-final form it will
have for the holiday season is the first step to
making sure it will handle the demand. New
features and functionality — like enhanced
product presentation, reviews, and
Keynote Benchmark
This diagram illustrates actual results of a test of a major retailer’s Web site. The golden area illustrates that the more users who log on beyond the site’s optimum capacity, the more revenue is lost. A spike of 25 percent results in losses in the neighborhood of $100,000 every hour.
personalization options — should be well
under way if not complete by now. Sports
fashion leader Lacoste is one retail marketer
that understands the importance of readying
major site revisions early.
“We’ve been working on a huge
upgrade of our site and have been doing
side-by-side testing with Keynote in order to
measure the current experience against the
new experience,” Lacoste E-Commerce
Director Maryssa Miller told Benchmark.
“Our criteria is to be better in performance
than the current existing experience, even
with all the new functionality. And we want to
make sure that any kinks are worked out prior
to the busy holiday season, obviously.”
“We’re adding larger images with
greater resolution,” Miller continues. “We’re
adding new features to be able to check your
tax and shipping or your total costs much
earlier in the process. We’re also adding social
networking tools — the ability for people to
more easily share products through any of the
major social networks.”
BEST PRACTICE: BUILD PAGES LEANPerformance management starts with how the
pages are built, which often presents a
dilemma for retailers. With more and more
products approaching commodity status and
available at multiple online outlets, site
experience becomes a key differentiator.
Retailers want to create a rich experience for
visitors with interactivity, dramatic product
presentation, perhaps Flash, personalization
or other features to set themselves off from the
competition. But a heavy load of features and
functionality can drag site performance down,
often because of third-party content, and,
instead of making visitors sticky, can drive
them to leaner, faster competitive sites.
“Ideally, it’s best to focus on limiting
your number of third-party content and
domains on each page to roughly six domains
at most,” says Keynote Consulting Manager
Cliff Crocker. “As opposed to what we see for a
lot of retailers, where there’s 20 or 30 different
domains that are killing performance on
their page.”
Crocker also recommends keeping
the number of objects on a page to 40 or 50;
many retailers are packing their pages with
150 or more elements.
“You have to try to limit the time
that’s being spent in the browser,” Crocker
says. “The more JavaScript and the more
functionality a retailer adds to their page,
oftentimes can create very big delays within
the user’s browser.”
FACTOR MOBILE IN FROM THE STARTThe successful retailers this year will have
built mobile into their strategy right from the
start — not just as an afterthought to the
“main” site, but side-by-side with it. Shoppers
carrying smartphones are using them to check
prices, locate products, find deals, look at
reviews and, more and more, to make
purchases. (See “Shoppers, Start Your
Smartphones!” in Benchmark.) Many retailers
were surprised at the amount of mobile traffic
3
Keynote Benchmark
Lacoste is updating its site half a year ahead of the holiday season, and setting a performance bar even higher than last year. Bigger images and social media functionality are among the enhancements being implemented.
they got during the 2009 holiday season. And
there will be millions more smartphones in
the hands of shoppers this year.
“We were very surprised,” says
Lacoste’s Miller. “And I think that’s where we
really realized that we need to focus on
[mobile] for 2010. We already knew we
needed to focus on it, but I think [the 2009
season] was just really additional proof and
we knew that it needed to be a huge part of
our strategy for this year, and then continue to
be part of our strategy going forward.”
Because of the inherent slowness of
cellular networks and devices, mobile sites
need to be even leaner and meaner than wired
Web sites. It takes some hard decision-making
and analysis of what is essential for users
when they are browsing on the go and what it
takes to satisfy them, including their need for
speed. Search results can be confined to return
four or five results, for example, instead of the
40 or 50 that might be delivered on the wired
Web. And perhaps tracking pixels are needed
only on the landing page and cart page,
instead of every page on the site.
“There are some pretty strict rules
around how many elements you want to have
on a mobile page,” Crocker says. “Where we say
40 to 50 over the wired Web, we’re looking at 8
to 10 elements on a mobile page, just because
of all the latency that’s encountered over the
carrier networks. So step one is, you’ve got to
build a dedicated mobile site. There’s no way
you’re going to port what you’re putting up on
the wired Web or pare that down. You’ve really
got to build it from scratch.”
STEP 1: ESTABLISH YOUR BASELINE AND PROJECT TRAFFIC INCLUDING MOBILEKnowing your site’s baseline — how much
traffic it can handle right now — is the logical
starting point to begin understanding how
much work needs to be done. Then,
projections need to be made for how much
traffic and sales are expected for the holiday
season. Typically the purview of the sales and
marketing team, this is where it gets a little
tricky; consumer behavior is difficult to
project, especially in these times of continued
economic turbulence. Whatever the traffic and
sales projections, it’s up to IT and the Web
team to make sure the site can handle any
unexpected surge in site visitors.
“From the technology side, you
should always treat whatever the business tells
you with a grain of salt,” says Keynote Director
of Global Testing Services Donald Foss. “You
may say, ‘I’m going to add 50 percent as
planning-appropriate, and then I’m going to
add another 30 or 50 percent as a technology
buffer — an insurance policy,’ and so it ends
up roughly double, or maybe 250 percent, of
what the marketing department projects.”
Many, if not most, retailers still fail
to account for mobile in their traffic
projections, and that could be a bigger mistake
than ever this year. Visitors logging on from
mobile devices are calling on the same
resources and databases as visitors coming
through PC browsers, and so are adding to
overall site load. Making matters worse is that
those mobile connections are extremely slow;
traffic coming in through mobile actually holds
the connections longer, and consumes more
than its share of resources.
“In planning for traffic, it’s important
to account for the total sum of the traffic from
a holistic standpoint,” Foss says. “It’s
important to make sure you’re actually testing
the mobile site and the regular Web traffic all
at the same time, in the correct proportions
and in the right demographics, to see how
it works.
“Early on, you take a multi-pronged
approach. You test the core Web site by itself.
You test the mobile site by itself, and make sure
they’re both tuned and working correctly. Then
you start the holistic testing, with the full sum of
traffic from the correct sources that you predict.
If 15 to 20 percent of user traffic is coming over
mobile, then we will actually have 15 to 20
percent of the test traffic running mobile-type
connections, and make mobile scripts with
mobile line speed coming into the site.”
For some retail marketers such as
Lacoste, the mobile and wired Web
experiences are tied tightly together, so the
necessity for holistic testing is apparent.
Speaking about their iPhone app,
Lacoste’s Maryssa Miller says, “My favorite
part is that it’s actually connected to the
e-commerce site. I’ve seen other apps where it
is a very separate experience, but ours is very
much integrated, which I think is great
because we’ve found that a lot of people will
save items in their shopping cart and, for
whatever reason, they don’t purchase them.
This way, if they decide they want it, they can
access their shopping cart on the e-commerce
site or directly from the app. That’s a really
usable feature and one of the best things
about it.”
MAKE TESTING A REAL REALITY CHECKIn a very oversimplified nutshell, with load
testing you are throwing users at the site until
it breaks (which is why live-site load testing is
typically done in off-hours). But to be realistic,
accurate, and reliable, a load test has to be far
more sophisticated than simply simulating a
gross volume of visitors showing up, which is
the essence of the often-used but imperfect
“concurrent user” methodology.
A concurrent user approach delivers
a picture of performance that is far from
accurate. Basically, it measures a static
threshold. Here’s 100 thousand users. Does it
break? OK, add another 10 thousand. And
another. The methodology assumes incoming
users will patiently wait their turn to enter,
and that everything will be fine until they do.
But in reality, those additional users are trying
to log on and being turned away, and the
experience for the users already on the site
degrades, often quickly. Testing only for
concurrent users does not accurately mirror
user behaviors and actual site traffic dynamics.
4
Keynote Benchmark
In reality, users are continuously coming and going. Those with slow connections or mobile devices are taking longer. Some who are already familiar with the site are in and out quickly. When the site slows down under the load, more and more users pile on, because it’s taking longer for the previous users to complete their tasks and leave. The load is never static. And users are not merely numbers going in and out. They are individuals with individual thresholds of patience — and that is the critical factor to know when it comes to converting visitors to sales, and minimizing revenue lost due to poor performance. A more accurate representation of site load is created using an “arrival rate”
methodology, which factors the peaks and valleys of traffic and availability based on likely user behaviors and site slowdowns. Hand-in-hand with this methodology is behavior modeling, which factors, among other things, users who are familiar/unfamiliar with the site, their tolerance for delays, and their tenacity in sticking with the site until they accomplish their tasks. “There’s a certain amount of slowness that can be tolerated, and a certain amount that won’t be,” says Foss. “That’s latency tolerance. There’s also tenacity, which means essentially, how important is it for a person to finish the transaction they’re on? How likely are they to continue with it or not?” If they’re checking out, they’re more likely to be patient with order and credit card processing. But if they’re elsewhere on the site, tenacity may be significantly lower.
BEST PRACTICES: METHODOLOGY & MODELINGTo really understand how your site
performance will hold up — or not — under
holiday stress, and to understand what the
experience will be like for users, use an arrival
rate methodology and factor in behavior
models for the many, many types of users and
tasks your site will serve.
Behavior modeling results in
numerous permutations (often thousands)
combining these variables:
• Familiarity: experienced users vs.
newcomers
• Connection speed: super-fast FIOS vs.
super-slow mobile device, and everything in
between
• Latency tolerance: patience of users with
slow site response
5
Keynote Benchmark
Lacoste’s iPhone app is tied directly to their e-commerce site, so registered users can access their cart and make purchases from either place. With mobile sites typically calling on the same backend resources as the wired site, it is important to test the two simultaneously to evaluate overall load-handling capability.
The “concurrent user” model assumes a constant volume of site traffic, which is a poor representation of the ups and downs of actual traffic. Arrival rate methodology, on the other hand, combined with user behavior modeling, mirrors much more closely the patterns and volumes of peak site traffic.
• Interaction speed: complexity of the page to
navigate, and attention level of the user
• Tenacity: willingness of users to stick with a
task through completion
“We’re not just replaying a
transaction multiple times ad nauseam to try
to put load on the site,” Foss says of Keynote’s
LoadPro testing. “There’s a whole lot more
going on behind the scenes. There’s literally
hundreds or thousands of different types of
users that we emulate during a single test to
make sure we’re emulating what a real user
population would be.”
TEST IN THE REAL WORLD — ALL OF ITTo know how your site will perform for users
dispersed across the country or the world,
testing must be done over the Internet, from
the same geographic locations as your users,
not from behind the firewall. There’s simply
no way to simulate the vagaries of Internet
backbones, third-party content feeds, CDN
performance, and signal transmission through
the critical last mile — unless you are at the
end of that mile, with a browser.
With testing agents dispersed where
your users are, you get an accurate picture of
variations in performance, and overcome the
danger of looking at averages. An average
page-load time of three or four seconds may
seem OK, but that kind of average could mean
your page is loading in one second for
someone in New York, but taking six or more
seconds for someone in Chicago. And that is
not likely to be acceptable. The solution is to
test from multiple, geographically dispersed
locations, look at the data, and address any
local or regional bottlenecks.
STAFF UP AND STOCK UP — ON CAFFEINE AND CAPACITYRetailers who put off load testing can look
forward to many sleepless nights in October
and early November as they try to get their
sites up to speed. But even the best-in-class
retailers can count on a sleepless night or
two — particularly Thanksgiving night — as
they stay up to make sure all their hard work
has paid off.
“One of my biggest clients spent the
night at headquarters on Thanksgiving night
last year,” Foss says. “He’s up all night making
sure everything is running perfectly, and goes
home Black Friday morning at 9:00 a.m.”
The holiday shopping season is the
culmination of many hard hours of work for
the IT/Web department. And no matter how
well things are planned, no matter how
rigorously everything is tested, there’s always
the chance that the unexpected will happen
and something will go wrong. So it makes
good sense to have technical personnel on
hand and on call during all the critical
shopping periods to handle any emergencies,
and to have extra computing capacity standing
by just in case it’s needed.
DON’T WAIT ANOTHER MINUTEIf you haven’t started load testing your site
yet, there’s no time to lose. “The most
successful retailers look at holiday readiness
as an ongoing process,” says Crocker. “They
start in January and they test and they
monitor performance and they benchmark
and they trend all the way through until the
end of December.”
And December is just a few short
months away.
6
Keynote Benchmark
.............................................................................................................................................................................© 2010 Keynote Systems, Inc. All rights reserved. The trademarks of Keynote Systems, Inc. include Keynote®, DataPulse®, CustomerScope®, Keynote CE Rankings®, Keynote Customer Experience Rankings®, Perspective®, Keynote Red Alert®, Keynote Traffic Perspective®, Keynote WebEffective®, The Internet Performance Authority®, MyKeynote®, SIGOS®, SITE®, keynote™, The Mobile & Internet Performance Authority™ and all related trademarks, trade names, logos, characters, design and trade dress are trademarks or registered trademarks of Keynote Systems, Inc. in the United States and other countries and may not be used without written permission.
ENDNOTES
i. U.S. Department of Commerce
ii. comScore press release, “comScore Reports $29.1 Billion in U.S. Retail E-Commerce Spending for Full November-December
Holiday Season, UP 4 Percent vs. Year Ago, 1/6/2010
iii. Internet Retailer, “Total holiday retail sales rise 1%, trailing e-retail’s 4% increase
iv. op cit