synthetic and real world web performance monitoring

EdgeCastPerformance Engineering Group

Third Party MonitoringUse Cases

Nathan Moore, Web Performance [email protected]

2850 OCEAN PARK BLVD, SUITE 110, SANTA MONICA CA

2

Analyze performance trends and come up with ideas to make the CDN faster

© 2013 EDGECAST NETWORKS

Web Performance Group


3

• Network Stack• Routing• Kernel• Application


Performance Optimizations

TCP Optimizations

4

• We monitor as much as we can to find these:

© 2013 EDGECAST NETWORKS2850 OCEAN PARK BLVD, SUITE 110, SANTA

MONICA CA

Performance Optimizations

Needles in haystacks


5

You can’t manage what you don’t measure

-- Deming-- Drucker

-- Traditional


Why monitor?


6

• Performance– Latency– Bandwidth

• Availability– Uptime

• Rate of Change of Internet– Snapshot of Internet as a whole


What can you monitor with Third Party Applications?


7

Web performance drives KPIs


Why care?


8

KPIs:

• Fast websites drive Sales and Engagement• Reliable websites drive Sales and Engagement


Why care?


9© 2013 EDGECAST NETWORKS

Not just a made-up statistic

http://sites.google.com/site/glinden/Home/StanfordDataMining.2006-11-28.ppt?attredirects=0


10

What you can measure, you can optimize

-- Ilya Grigorik


Why monitor?



Simplified Internet Traffic Flow



Categorizing the World



Monitoring the World


14

Synthetic monitoring is needed to dynamically monitor the External

Internet


What are Third Party Monitors for



Monitoring Hooks for Synthetic Monitoring



Synthetic Monitoring

Synthetic monitoring is clean

• Small number of nodes• Well defined network• Highly reliable links between nodes and networks• Shows health of the backbone of the Internet



State of the Internet


18

Real User Monitoring (RUM) is needed to

measure what your end users really see


What are Third Party Monitors For



From Synthetic to Real User Metrics (RUM)

Reporting Server

CDN 1 CDN 2 CDN 3



Real User Data / RUM



Real User Metrics (RUM)

RUM is messy. Rum is the real observed data generated by real users on the real Internet on real web browsers

Combinatorics:• Hundreds of Millions of TCP connections• Tens of Millions of end users • Thousands of ISPs• Tens of backbone providers• One of your web servers

• Data Distribution issues:• Power law distributions => averages lose explanatory power• Quantiles and Cumulative Density Functions (CDF) gain importance• Aggregating data and mining for correlations becomes more difficult



Real User Data / RUM benefits

Data aggregated across country and displayed across world



Use Case 1:What if you only have synthetic monitors?

New platform launch:

EdgeCastTransact



Use Case 1:Launch of Transact

Directive from CEO:

Performance must match the existing matured network!




Transact is a brand new product from EdgeCast, designed to accelerate e-commerce performance on the web.

Featuring full PCI compliance, it has its own dedicated and highly reliable network, its own dedicated servers, and a fully integrated security layer.

The challenge is to deploy a brand new network, and make it perform like an existing, fully matured network.

Third party monitors are the key to accomplish this, and to know when we’ve accomplished it.



Use Case 1: Launch of TransactEstablished monitors across the major third parties to observe actual latencies worldwide




Compared performance between Transact and EdgeCast ADN at all nodesUsed the delta to determine where issues were




Iterated: As issues were resolved, updated with new monitoring data

Repeat process!




Result: Worldwide average performance within 1.5ms

EdgeCast ADN Transact


30

Third party monitoring allows A/B comparison


What are Third Party Monitors For



Use Case 2: RIPE/ATLAS

Using RIPE/ATLAS to evaluate the network for IPv6 launchfor World IPv6 Day, 2012




For World IPv6 Day, we deployed IPv6 across our entire network.

The Challenge:

How does the IPv6 Internet perform? We used RUM data provided by RIPE/ATLAS to find out.




http://atlas.ripe.net/about

RIPE Atlas is a global network of probes that measure Internet connectivity and reachability, providing an unprecedented understanding of the state of the Internet in real time.

http://www.ripe.net

The RIPE NCC is one of five Regional Internet Registries (RIRs) providing Internet resource allocations, registration services and coordination activities that support the operation of the Internet globally.




JSON based API Ability to run multiple tests




We launched a beacon dedicated to IPv6 measurements which tested:

• IPv4 only• IPv6 only• Dual IPv4 & IPv6

To reduce failures, the beacon also checked connectivity to ipv6.google.com




What did we learn from Atlas about our IPv6 performance and availability?




IPv6 Reach Analysis

AS path analysis AS path of failed traces



Use Case 2: RIPE/ATLASVisualization of Country-of-Origin traffic to EdgeCast POP

Ex: Ukraine is served by both POP7 and POP1, and POP7 has higher latency




Availability predictions before launch



Use Case 2: RIPE/ATLASPerformance improvements during pre-launchConclusions: IPv6 still has room for improvement



Use Case 3: Analytics within Third Party Monitors

Using Third Party Monitors to answer real-world questions

Often, we’re asked questions by different groups of the company.

Sales, for instance, always wants to prove that we’re the fastest worldwide.

But what to do when some regions are faster than others?




Sales wanted to know why a prospective client was performing really well in Asia, but not nearly as well in Europe.

AsiaEdgeCast Mystery Competitor X

EuropeEdgeCast Mystery Competitor X




There was a lot of variance within countries

EdgeCast Mystery Competitor X




The scatterplot shows some odd signs of horizontal banding.





Good clue: Looking at the total downloaded bytes instead of response time, the banding becomes really obvious, and wildly different in size





A histogram shows the magnitude of this difference




Conclusion:

Client was returning different data depending on the geoIP lookup of the end user. So end users in different countries were deliberately being returned different objects.

However, as caches, both EdgeCast and Mystery Competitor X were caching the index.html page, so whatever was on that page was cached and returned on every request. New data was only returned once every day, when the caching rule expired, and a fresh request to origin was made.

The extreme difference in size was largely a function of random chance- we got lucky to return smaller objects during this sample time.



Questions?

synthetic and real world web performance monitoring

Technology

santa monica ca3

santa monica ca4

santa monica ca5

santa monica ca6

santa monica ca7

santa monica ca8

santa monica ca9

santa monica ca10