the user and business impact of server delays, additional bytes, and http chunking in web search...
DESCRIPTION
We took very small percentages of our users and slowed the page down in several waysAdded server side delaysAdded additional page weight to different parts of our pageAdded client side rendering delaysWe then measured how each of these changes impacted metrics like clicks, time to click, repeat site usage, queries per visit, revenue, etc.This talk is to share the results.TRANSCRIPT
Performance Related Changes and their User
ImpactEric Schurman
Principal Development LeadBing
Jake BrutlagDecision Support Engineering Analyst
Experiments• Server Delays (Microsoft and
Google)• Page Weight Increases• Progressive Rendering
Server-side Delays Experiment• Goal
• Determine impact of server delays• Methodology
• Delay before sending results• Different experiments with different
delays• Small number of users• Monitor negative impact
Server Delays Experiment: Results
• Strong negative impacts• Roughly linear changes with increasing delay• Time to Click changed by roughly double the delay
50ms - - - - - -200ms - - - -0.3% -0.4% 500500ms - -0.6% -1.2% -1.0% -0.9% 12001000ms -0.7% -0.9% -2.8% -1.9% -1.6% 19002000ms -1.8% -2.1% -4.3% -4.4% -3.8% 3100
- Means no statistically significant change
Google Web Search Delay Experiments• A series of experiments
on a small % search traffic to measure the impact of latency on user behavior
• Randomly assign users to the experiment and control groups (A/B testing)
• Server-side delay:• Emulates additional
server processing time• May be partially masked
by network connection• Varied type of delay,
magnitude (in ms), and duration (number of weeks)
Search Traffic Impact
Type of DelayDelay (ms)
ExperimentDuration (weeks)
Impact on Average Daily Searches Per User
Pre-header 50 4 Not measurablePre-header 100 4 -0.20%Post-header 200 6 -0.29%Post-header 400 6 -0.59%Post-ads 200 4 -0.30%
• Increase in abandonment heuristic = less satisfaction• Abandonment heuristic measures if a user stops
interacting with search engine before they find what they are looking for
• Active users (users that search more often a priori) are more sensitive
da
ily s
ea
rch
es
pe
r u
ser
rela
tive
to c
on
tro
l
w k1 w k2 w k3 w k4 w k5 w k6
-1%
-0.8
%-0
.6%
-0.4
%-0
.2%
0%
0.2
%2 0 0 m s d e la y4 0 0 m s d e la y
a c tua ltre nd
Im p a c t o f P o s t-he a d e r D e la ys O ve r Tim e
-0.22%
-0.44%
-0.36%
-0.74%
da
ily s
ea
rch
es
pe
r u
ser
rela
tive
to c
on
tro
l
w k3 wk4 w k5 w k6 w k7 w k8 w k9 w k1 0 w k1 1
-1%
-0.8
%-0
.6%
-0.4
%-0
.2%
0%
0.2
%
de
lay
rem
ov
ed
P e rs is te nt Im p a c t o f P o s t-he a d e r D e la y
2 0 0 m s d e la y4 0 0 m s d e la y
a c tua ltre nd
-0.08%
-0.21%
Page Weight Experiment• Goal
• Determine impact of a heavier page.
• Isolate bytes over the wire cost, not layout costs, etc.
• Methodology• Use incompressible HTML
comments• Vary size (from 1.05x to 5x page
size) and location of payload • Experiment with payload in
individual and multiple locations• US-only test – mostly good
broadband
<html>
<head>
<CSS />
<!--Payload-->
</head>
<body>
<answer>...
<!--Payload-->
</answer>
<results>
<result>...
<!--Payload-->
</result>
<result>...
<!--Payload-->
</result>
<result>...
<!--Payload-->
</result>
</results>
...
<!--Payload-->
<script />
<!--Payload-->
</body>
</html>
Page Weight Experiment: Results• Minimal impact for small payloads• Payload at top of page had stronger effect• Performance suffered slightly – would have been
worse if tested in regions with poor connectivity• Click metrics impacted more than Query metrics• Largest experiment (approx 5X control page
size)• Any Clicks: -0.55%• No changes to query metrics
• Results only apply to one GET – not multiple
Progressive Rendering Experiment
• Goal• Determine impact sending visual header before
results.• Methodology
• Build page in phases• Send using HTTP 1.1 Chunked Transfer Encoding
• Application design impacts
Visual Header - Fast to compute
Results - Slower to compute
Progressive Rendering Experiment: ResultsMetric Change
PerformanceFaster across all latency percentiles4-18% faster to download all HTMLRoughly halved time to see visible page change
Time to Click ~9% fasterQuery refinement +2.2%
Clicks overall +0.7%Pagination +2.3%Satisfaction +0.7%
Conclusion• "Speed matters" is not just lip service• Delays under half a second impact business
metrics• The cost of delay increases over time and
persists• Number of bytes in response is less
important than what they are and when they are sent
• Use progressive rendering