GeoRoute — Geographically

Constrained Routing

Matthias Gorner

November 22, 2010

Abstract

GeoRoute is an overlay network that allows geographically constrained routing throughsource routing and probing with traceroutes. I deploy GeoRoute worldwide on Planet-Lab. My goal is to show that it is possible to develop an overlay network that providesan altered Internet service in which a user can specify a geographic constraint that isrespected when routing packets. I measure the resulting performance of the Internetwhen using these altered routes. The motivation for this work comes from the fact thatdifferent countries apply different communication, inspection, and censorship policies.This makes GeoRoute interesting as a user application as well as a framework to mea-sure geographic properties of the Internet and the impact of the communication policieson the Internet.I evaluate the effectiveness of GeoRoute by estimating the accuracy of the geolocationdata as well as the stability of routes. Though GeoRoute cannot guarantee that packetsdo not traverse countries specified to be avoided, the results indicate that the constraintsare met with high probability, at least on IP level. The results include examples of al-ternative routes avoiding countries as well as the associated performance penalty givenas simulated delay times and as measured TCP throughput.

Contents

1 Introduction 3

2 Related Work 6

2.1 Geographic Properties of and GeoLocation in the Internet . . . . . . . . . 6

2.2 Other Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3 Overview of GeoRoute 8

3.1 Design Decision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3.2 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

4 Possible Applications 14

5 Implementation 16

5.1 Control Plane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

5.1.1 Overlay Nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

5.1.2 Client Side Functionality . . . . . . . . . . . . . . . . . . . . . . . 17

5.1.3 Algorithm to Compute Routes . . . . . . . . . . . . . . . . . . . . 18

5.1.4 Locating Routers with Undns . . . . . . . . . . . . . . . . . . . . . 20

5.2 Data Plane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

5.2.1 Structure of Forwarding Header and NextHop . . . . . . . . . . . . 23

5.2.2 Encapsulating Packets at the Client: ForwHeader, StripForwHeader 24

1

5.2.3 Encapsulating Packets at the Overlay Node Closest to the Server:RevRoute and Network Address Translation . . . . . . . . . . . . . 24

5.2.4 Deployment on PlanetLab . . . . . . . . . . . . . . . . . . . . . . . 25

5.2.5 UDP Sockets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

5.2.6 Implementing the Middlebox Functionality on the Client Machine 26

5.2.7 Packet Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

5.2.8 Packets that Cannot Be Delivered . . . . . . . . . . . . . . . . . . 27

6 Results 28

6.1 Preparation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

6.1.1 Choosing Test Web Servers . . . . . . . . . . . . . . . . . . . . . . 29

6.1.2 Choosing Overlay Nodes . . . . . . . . . . . . . . . . . . . . . . . . 29

6.1.3 Choosing a Client Site and Entry Point . . . . . . . . . . . . . . . 33

6.1.4 Filesize for Bandwidth Measurements . . . . . . . . . . . . . . . . 33

6.1.5 Locating Routers with Undns and Pathologies in First Experiment 33

6.1.6 Manually Locating Routers . . . . . . . . . . . . . . . . . . . . . . 35

6.1.7 Creating a List of Interesting Country Constraints . . . . . . . . . 37

6.2 User Experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

6.3 Virtual Link Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

6.3.1 Virtual Link Properties . . . . . . . . . . . . . . . . . . . . . . . . 41

6.3.2 Stability of Virtual Links . . . . . . . . . . . . . . . . . . . . . . . 41

6.4 Analysis of Rerouting Possibility and Impact . . . . . . . . . . . . . . . . 43

6.4.1 Routes from the U.S. . . . . . . . . . . . . . . . . . . . . . . . . . . 43

6.4.2 Routes between All Test Web Servers . . . . . . . . . . . . . . . . 48

6.5 Measured Overlay Performance . . . . . . . . . . . . . . . . . . . . . . . . 51

7 Discussion 53

8 Acknowledgments 55

2

Chapter 1

Introduction

The Internet is designed to be a transparent network delivering a piece of information (apacket) from source to destination without the end nodes involved in the routing deci-sions. Ideally, the Internet makes the geography invisible allowing the user to access anyresource from anywhere in the same fashion. However, the routing decisions influenceperformance measures such as delay, jitter, and bandwidth. Furthermore the routing de-cisions affect what countries a packet might traverse and therefore which communication,inspection, and censorship policies are applied to a packet. Even though knowledge ofthe routing process or control over it are of interest to the user, the Internet neither hasa built-in mechanism to report the trajectory of a packet to the user, nor does it allowthe user to influence the routing decisions for packets.

GeoRoute addresses these shortcomings. GeoRoute allows the user to have more controlover the routing process and to avoid policy domains in which unwanted policies areapplied to a packet. Policy domains can be, for example, geographical regions such ascountries or Autonomous Systems. In the implementation described here, each user canspecify an individual geographic constraint and GeoRoute attempts to deliver a packetwithout violating this constraint. It reports to the user if no such route can be found.I implement and deploy GeoRoute worldwide on PlanetLab. PlanetLab is a testbedfor globally distributed network applications [PACR03]. Many academic and industrialinstitutions run PlanetLab computers distributed around the world and allow researchersvirtualized access to them.

GeoRoute is an overlay network, i.e., a network running on top of the Internet. Wecall the Internet route between two overlay nodes or an overlay node and an end hosta virtual link. An overlay node has two basic functions: 1. measuring the performanceand geography of virtual links through standard Internet measurement tools such as pingand traceroute; 2. forwarding packets along a path specified by the user (source routing).A client polls these measurements to compute a route through the overlay network. Byattaching an extra header (Forwarding Header) to a packet, the client instructs the

3

overlay network to use the specified route.

The algorithm to compute routes is executed on the client and uses the extra tool undns[SWA] to locate routers in the virtual links from the traceroute data. It assigns a listof countries and a performance quality to each virtual link. After all virtual links notrespecting the user-specified constraint have been filtered out, the best path to a desiredtarget in the graph of virtual links is computed.

I design GeoRoute to work with unmodified standard Transmission Control Protocol(TCP) clients and servers, such as standard web browsers and servers. While packets areforwarded through the overlay network, they carry a Forwarding Header with routinginformation. As standard TCP clients and servers do not understand these headers, theproper encapsulation and decapsulation is also the responsibility of GeoRoute. On theclient side, either a separate middlebox or just a piece of software running on the clientmachine does this and connects the client to the overlay network. Close to the server,GeoRoute’s last overlay node encapsulates and decapsulates packets to connect the clientto the server. The routing decision happens on the client site, the overlay node routesserver responses back to the client along the same route.

To evaluate GeoRoute’s effectiveness, I estimate the geolocation accuracy of routers andthe stability of Internet routes, analyze the new routes that GeoRoute’s algorithm com-putes, and measure the performance of the overlay network along those routes. Thesemeasures are evidence that the results are accurate and it is possible to route packets re-specting a user specified geographic constraint with high probability, at least on InternetProtocol (IP) level. I also test GeoRoute as an application from a user perspective.

The results include interesting examples of alternative routes to avoid countries and theexpected delay inflation when trying to meet a geographic constraint. For example, oftenan alternative route with small delay inflation can be found to avoid either Germany,the Netherlands or the United Kingdom. Avoiding all three countries however, resultsin routes with large delay inflation or cannot be achieved at all.

Even though there has been a lot of research on geographic properties of the Internetand of Internet routes, this work is novel as it investigates the impact of rerouting andits possibilities to avoid geographic regions.

I structure the rest of the report as follows: Chapter 2 is describing previous work onrelated topics such as geographic properties of the Internet.

The overview of GeoRoute in Chapter 3 discusses design decisions as well as the overallarchitecture which is divided into a control plane and a data plane.

After discussing the possible applications of GeoRoute in Chapter 4, Chapter 5 revisitsthe architecture and describes its implementation in detail. In particular, it explains themeasurements of routes and the algorithm to compute new routes (control plane, Section5.1), as well as the mechanism to route through the overlay network and connect it to

4

standard TCP/IP clients and servers (data plane, Section 5.2).

The results in Chapter 6 include both an estimation of GeoRoute’s effectiveness as wellas examples of alternative routes and their delay inflation.

I conclude the report in Chapter 7 with a discussion of fundamental problems as well aspossible biases in the data.

The acknowledgments are in Chapter 8.

5

Chapter 2

Related Work

As there is a lot of work on related topics, I separate work explicitly addressing thegeographic properties of the Internet from other work. This project is novel and differ-ent from previous projects because it is addressing the possibilities and implications ofrerouting to avoid certain countries.

2.1 Geographic Properties of and GeoLocation in the In-ternet

Subramanian, Padmanabhan, and Katz [SPK02] investigate geographic properties ofpacket routes such as circuitousness of Internet paths. This work is based on varioustechniques, e.g. measurements of delays and traceroutes [KBJK+06], identifying thelocation from hostnames and inferring information from dumps of Border Gateway Pro-tocol routing tables and public information about Autonomous Systems. Padmanabhan[PS01] compares these techniques. Wang, et al. [WBF+10] refine these techniques byadding, for example, webspiders crawling web servers to determine their location fromtheir content allowing location up to street-level. There are also various commercialdatabases and services available to map an IP address to a geographic location.

From the perspective of placing servers at central locations in the Internet, mappingIPs to network coordinates has been investigated, for example, in [LPS06]. Networkcoordinates model proximity based on network latency and only vaguely correlate to thegeographic locality or proximity of nodes.

6

2.2 Other Work

Overlay networks have been proposed earlier to circumvent the default IP routing. An-dersen, Balakrishnan, Kaashoek, and Morris [ABKM01] use overlay networks to improveperformance and overcome connectivity shortages.

Savage, et al. [SCH+99] investigate the effects of rerouting on the performance andclaim that “in 30%-80% of the cases, there is an alternate path with significantly supe-rior quality.” Furthermore, Savage, et al. [SAA+99] explain the connection of routinginefficiencies and the Internet’s history and propose a framework to explore alternativenetwork solutions.

Paxson [Pax99] collects statistics on packet delay, duplication, and loss with view on itsimplications on TCP performance. For future work, similar analysis [Pax97a] can beused to investigate the reduced bandwidth of GeoRoute further and to find out how theperformance can be improved.

Paxson [Pax97b] also investigates routing stability and routing pathologies using tracer-outes. This is similar to the analysis in Section 6.3 for the PlanetLab sites which is,however, focusing on the stability of routes in terms of traversing the same countries.

7

Chapter 3

Overview of GeoRoute

This Chapter gives an overview of GeoRoute. After discussing the design decisions inSection 3.1, it explains the architecture of the overlay network in 3.2 which I describe inmore detail later in Chapter 5.

3.1 Design Decision

Overlay Network. Changing major Internet routers at several locations is clearly notfeasible and most routers do not allow the user to influence the routing decisions. In par-ticular, existing options in the IP header, e.g., for source routing, to allow the user controlover the routing process are disabled by most routers in the Internet [FPK+05]. Hence,overlay networks are the only possible mechanism to circumvent the default routing in aglobal testbed, justifying the associated overhead.

I do not investigate changes to the routing protocols because of similar reasons andbecause such changes would allow Internet Service Providers (ISP) rather than individualusers to influence the routing decisions. The routing protocol used between InternetService Providers is the Border Gateway Protocol (BGP). An example of a possiblemodification to BGP is the inclusion of a list of countries to which a route should not bepropagated when announcing an IP prefix.

Link State Protocol Routing Algorithm Centralized on Client. GeoRoute im-plements the routing algorithm centrally on the client. It is based on a link state protocol.Hence the client has to maintain a picture of the complete graph of all overlay nodes.Therefore the complexity of the algorithm scales as n2. This is still feasible if the numberof nodes is small, here around the magnitude of the number of countries in the world. Iadd an extra step to the link state protocol algorithm to filter out all virtual links notcomplying with a user specified geographic constraint. This implementation is describedin Section 5.1.3.

8

The alternative is a distance-vector or path-vector routing protocol modified to propa-gate several routes to the same host for different geographic constraints. Examples ofsuch distributed routing algorithms are Routing Information Protocol (RIP) and Bor-der Gateway Protocol (BGP). These protocols propagate routes through the network.Along the way, information about the network is compressed, for example by leavingout information about other possible routes and about links that are irrelevant to nodesfurther down the network. This compression makes the algorithms scalable to the entireInternet.

However, for what geographic constraints should the algorithm compute routes? Thespace of all geographic constraints grows exponentially in the number of countries. So apriori, it is not clear whether a path-vector routing protocol is feasible if it is supposedto offer the same flexibility as a link state protocol. Here, flexibility means the ability toquickly compute a route for an arbitrary user specified constraint on demand.

A possible modification to a path-vector protocol allowing the same flexibility is to prop-agate several alternative routes for each destination. Given different routes to the samedestination, there is a partial ordering defined by whether the set of countries one routetraverses is a subset of countries another route traverses. If several routes traverse thesame set of countries, they are ordered by latency. A node in the overlay network propa-gates for each destination all routes that are minimal with respect to this partial ordering.These are exactly the routes among which to search for the best possible route given anarbitrary country constraint. The question is how quickly the number of the routes thatare propagated this way grows. The data on routes in Section 6.4 shed light on this issueand indicate that such a modified path-vector protocol is also feasible.

Non-Connection Oriented Virtual Links. When a client and a server connectthrough the Internet, the TCP end-to-end connection could be split at each overlay nodeinto several TCP connections. Split connection would redundantly implement TCP’sfunctionality at each overlay node potentially violating the end-to-end principle unlessperformance improvement results. It would also be closer to a circuit switched network.To model routing through the Internet which is packet switched, I decide to use virtuallinks between overlay nodes that are not connection oriented.

Balakrishnan, Padmanabhan, Seshan, and Katz [BPSK96] compare a regular connectionto a split connection in case of lossy links (wireless). Even though overlay networkssuffer from increased delay, jitter, and packet reordering rather than packet loss, splitconnection might still be a valuable mechanism to improve performance. I will investigatethis in future work.

9

Web Server

client-side overlay network server side

routingdecision

measurevirtual links

normaloperation

routepackets

encapsulatein ForwardingHeaders with

RoutingInformation

routingalgorithm

Web Servermiddlebox

receivelink data

Browser

controlplane

dataplane

rou

tin

gin

form

ati

on

user specifiedconstraints

Figure 3.1: Functions of the control and data plane in GeoRoute.

10

3.2 Architecture

The overlay network is split into a control and a data plane, see Figure 3.1. The controlplane is responsible for performing measurements of the virtual links and computingroutes based on these data and the geographic constraints specified by the user. Theseroutes are communicated to the data plane of the middlebox where this information isused to generate “Forwarding Headers” containing a hop list (see Figure 3.2). The dataplane operates on these Forwarding Headers to route packets. It is also responsible fordecapsulation and encapsulation.

Control Plane. I discuss the control plane in detail in Section 5.1. Overlay nodes per-form the measurements of virtual links between overlay nodes and from overlay nodes toother hosts. Between overlay nodes, measurements are taken regularly every 10 minutes(see Section 6.3.2 for a discussion). A client can request measurements to additionalhosts. For the measurements, the overlay node is calling the standard Internet measure-ment tools ping and traceroute. Ping estimates performance data such as round trip timeby sending several ICMP echo messages to a host and waiting for responses. Traceroutecollects routing information in an indirect way. It sends packets with different time-to-live (TTL) values to a host. Each router decreases the TTL and sends back an errormessage once the TTL has dropped to zero. Traceroute receives these error messages totrace a route hop by hop.

The client side polls this information and uses the router names from the traceroute datato determine the location of the routers by the tool undns [SWA] (see Figure 6.3 andSection 6.1.6 for evaluation). The result is a graph of virtual links carrying performancedata and geographic information. This graph serves as input for the algorithm to computethe routes.

The first step in this algorithm is to filter out all virtual links not complying with a spec-ified user constraint. Then Dijkstra’s Algorithm (see [Dij59] and [CLRS09]) is performedto compute the route with the best performance as described in Section 5.1.3.

Data Plane. I discuss the details of the data plane in Section 5.2. The functions of thedata plane of the overlay network are forwarding packets based on information in theForwarding Header (structure specified in Section 5.2.1), encapsulation and decapsula-tion. The Forwarding Header has a list of hops and an index pointing at the next hopto which the packet needs to be routed. This index is increased at each hop (see Figure3.2). Close to the end nodes however, the data plane attaches and strips these Forward-ing Headers when receiving or delivering the payload from or to the application. To workwith ordinary TCP clients and servers without modification, the entire TCP/IP packetsfrom these clients and servers are encapsulated into Forwarding Headers, see Figure 3.3.

The details of encapsulation and decapsulation into Forwarding Headers are different atthe GeoRoute component close to the client and close to the server, as shown in Figure3.3. At the client, the routing information for the Forwarding Header is looked up in a

11

middleboxHop1Hop2Hop3

Hop1

Hop2

Hop3

ForwHeader

payload middleboxHop1Hop2Hop3

ForwHeader

payload

Figure 3.2: Basic Functionality of the Forwarding Header.

Client

middlebox Hop1 Hop2 Hop3

Server

TCP/IP

ForwHeader

TCP/IP

TCP/IP

en- and decapsulationbased on routing tablesupplied from control planecomputed based on user-constraint

en- and decapsulationto route server responsesback along same route

Figure 3.3: Basic encapsulation and decapsulation of Forwarding Headers in the dataplane.

12

routing table maintained by the control plane (see Section 5.2.2). At the overlay nodeclosest to the server, the data plane maintains enough state (see Section 5.2.3) to routeserver responses back to the client along the same route. Furthermore, the web servershould send its responses back to the overlay node and not to the client directly. Thisis achieved through Network Address Translation at the node closest to the server asexplained in Section 5.2.3 and shown in Figure 5.4.

13

Chapter 4

Possible Applications

The primary application of GeoRoute is to allow the user to circumvent countries inwhich unwanted packet policies apply. Such policies include modifying, dropping andcensoring packets as well as inspecting and analyzing packets.

Arguably encryption is an alternative to rerouting for circumventing some of these poli-cies mentioned above. As the overlay network cannot guarantee that the detected routesthrough the Internet are correct, rerouting cannot replace encryption to circumvent in-spection in some situations.

However, encrypting does not prevent an eavesdropper from still monitoring traffic vol-ume and logging access to websites. Encryption might be useless if there is a policy todrop encrypted packets. In these situations, rerouting has advantages in circumventingthe possibilities of the eavesdropper.

Secondary applications come from the fact that the overlay network is flexible enough toroute each packet along a different route. If this is used, a spy having access to a router inthe middle is able to only see parts of the transmission. As this technique is potentiallyprone to packet reordering, implementing this scheme and measuring the impact on TCPperformance is interesting future work.

This scheme can be further modified such that a spy having access to only one of theroutes would not gain any information at all. The basic idea is to send the key andthe encrypted information along different routes. For example, one can send a packet ofrandom data and another packet of the XOR with the payload to the same destinationhost but along two different routes. The packets converge at the destination host wherethey can be decrypted by XORing them again, see Figure 4.1.

14

Random

Data XOR

XOR

Figure 4.1: Sending XORed packets along different routes.

15

Chapter 5

Implementation

5.1 Control Plane

I implement the control plane in Python, using XML-RPC for communication. I designit to be flexible and able to save a snapshot of the state of the overlay network and thecollected data for later offline analysis, keeping the original ping and traceroute data aspart of the snapshot.

I describe the functions of the control plane implemented in the overlay nodes in Section5.1.1. The client side functions are described in Section 5.1.2.

The algorithm to make the routing decision is the most interesting part of the controlplane. I describe it in Section 5.1.3, with Section 5.1.4 explaining how the routers arelocated.

5.1.1 Overlay Nodes

The key function of the control plane of an overlay node is measurement. Other support-ive functions are finding the other overlay nodes and starting click which is the programrunning the data plane. Click [KMC+00] is a configurable modular router for Linux towhich I add several new elements for this project. I describe click and the new elementsin detail when explaining the data plane in Section 5.2.

Measurements. An overlay node regularly (e.g., every 10 minutes, see discussion inSection 6.3.2) performs a traceroute and a ping (with 10 packets) to every other runningoverlay node and saves them temporarily. It also provides an XML-RPC call for clientsto traceroute and ping a particular destination host. In the current implementation, theoverlay node does not process the output of the command line tools ping and tracerouteand gives it to the client in unparsed text form.

16

Finding Other Overlay Nodes. The control plane of a node maintains a list of allother running overlay nodes that is seeded at the start with a fixed set of nodes. Thelist is updated by querying other nodes regularly (e.g., every 2 minutes) for their listsof running overlay nodes and by adding nodes from which a query is received. Newlydiscovered nodes are first given candidate status until an XML-RPC call to them can bemade successfully. If an overlay node does not respond to several consecutive XML-RPCcalls, the node is removed from the list.

Starting Click. The control plane starts the click router on an overlay node. Therespective python script connects to the output of the click process for debugging pur-poses. Output on stdout can be read remotely through XML-RPC. If click does not startproperly or dies, the output on stderr is written to disk and the overlay node shuts down.

Click can be started with different configuration files for various levels of logging: packetlevel logging is very helpful in debugging, but might clog stdout and introduce jitter badlyaffecting TCP throughput. For better performance however, click can be configured touse minimal logging. The measurements in Section 6.5 are performed with the latterclick configuration.

The monitoring and debugging mechanism is motivated by the fact that the software onmany PlanetLab nodes is outdated or faulty. In particular, the click configuration failson many nodes, as the new vsys scripts are still in the process of being rolled out.

5.1.2 Client Side Functionality

This Section describes the functions of the control plane on the client site. I describe theactual algorithm to compute routes separately in Section 5.1.3.

Compute Routes. If the control plane is queried to compute a new route to a target,it requests measurement data to the new target from all overlay nodes. It then calls thealgorithm with these data (Section 5.1.3) and writes the resulting entry into the routingtable of ForwHeader (see Section 5.2.2) in the data plane.

Find Overlay Nodes and Poll Data. Similarly to the overlay nodes, the software onthe client side maintains a list of running overlay nodes. It regularly polls the collectedtraceroute and ping data of the virtual links between overlay nodes.

Find Entry Point. The user specifies a list of entry points with preferences. Thecontrol plane finds the overlay node with the highest preference among those that arerunning.

Interactive Mode. In interactive mode, the client side polls the list of IP addressesof queued packets from the ForwHeader element in the data plane (see Section 5.2.2).These packets are queued because no entry in the routing table exists for them. Thecontrol plane computes routes for these IP addresses as described above.

17

If no route is found, an error is reported to the user.

To respond quicker to the user, there are several optimizations to accelerate the datacollection in this mode: the number of pings is reduced and the client software stopswaiting for responses from the overlay node until a timeout (30 seconds) passes or morethan 90% of the overlay nodes have replied to the traceroute request. The assumption isthat the remaining 10% of virtual links to the target server are slow outliers that can beignored in the route computation.

The bottle neck in the response time is traceroute. For future implementations, I proposea version of traceroute that probes a route with packets of different TTL in parallel tospeed up the process.

5.1.3 Algorithm to Compute Routes

In this Section, I explain the algorithm to compute the routes respecting a user specifiedconstraint. Section 5.1.4 describes, and Sections 6.1.5 and 6.1.6 evaluate, the process oflocating the routers in the traceroute data.

The input data of the algorithm are:

• entry point

• list of all overlay nodes

• list of targets for which we want to compute routes

• traceroute and ping data between overlay nodes

• traceroute and ping data between overlay nodes and targets

• list of countries that are to be avoided

The algorithm performs these steps:

1. Construct the following undirected graph: Vertices are the overlay nodes and tar-gets. Edges are the virtual links. Assign performance and geographic data to thesevirtual links.

2. Filter out all virtual links whose list of countries intersects the list of countriesspecified by the user

3. Perform Dijkstra’s Algorithm (see [Dij59] and [CLRS09]) on the graph to findthe best route using the metrics described below to evaluate a route. Dijkstra’sAlgorithm is a classic graph search algorithm to solve the shortest path problem.

18

Computing Virtual Links. A virtual link is assigned a set of countries, a delay t(based on average ping Round Trip Time), a jitter σ (equal to the standard deviationof round trip time reported by ping) and a packet loss rate l (from ping). The set ofcountries is determined from the traceroute hops by using undns as described in Section5.1.4.

For a virtual link between two overlay nodes, measurements exist from both sides. Thesemeasurements are averaged for the virtual link. For the countries however, the vir-tual link is assigned the union of countries determined when tracerouting forward andbackward. The jitter is also computed differently: delays tforward, tbackward and jittersσforward, σbackward of both directions are considered to be mean and standard deviationof two sets of statistically independent measurements of a random variable. Hence, thestandard deviation σsym computed when combining the two sets of measurements is givenby:

σsym =

√σ2

forward + σ2backward

2+(tforward − tbackward

2

)2

.

The first term in the sum under the square root just averages the variances (variance issquare of the standard deviation) from each set of measurements. The other contributionto the variance can derived by noticing that ((tforward − tbackward)/2)2 is the same thanthe average of (tforward − tsym)2 and (tbackward − tsym)2.

Evaluating a Route. The quality of a route, i.e. path of virtual links, is estimatedfrom its delay t, jitter σ, packet loss l, and hop count n.

The delay t of a route is just the sum of delays t =∑ti of its virtual links.

The packet loss rate l is computed by multiplying the probabilities l = 1−∏

(1− li) ofthe complementary events.

The resulting jitter σ is computed by adding the variances (i.e. the square of the jitters)of the virtual links σ =

√∑σ2

i .

The penalty Q of a route is given by:

Q = t+ 2 · σ + 2500ms

100%· l + 50ms · n.

A route with a lower penalty is preferred. Penalty is supposed to be a rough inversemeasure for TCP throughput. Packet loss has a very high penalty because TCP’s perfor-mance degrades strongly with packet loss. The factor is chosen to be a magnitude higherthan the round trip time between two far apart countries. Packet loss has a strong effecton TCP’s performance because TCP implementations assume that packet loss indicatescongestion and hence reduce the transmission rate drastically if there is a packet loss[Jac88]. Similarly, jitter can cause late TCP ACKs and decrease throughput. Padhye, etal. [PFTK98] describe a more detailed modeled of TCP throughput.

Errors in the delay measurements can be larger than the round trip times between close

19

PlanetLab nodes, causing unnecessary rerouting unless penalty is given to n.

The factors in the expression for the penalty Q are chosen somewhat arbitrary. However,the routing decision is stable under a wide range of coefficients. The reason is that Q ismostly just dominated by the delay t and the other terms assure that pathological routesare avoided. Section 6.3.1 contains an analysis of delays, jitters, and loss rates of thevirtual links giving evidence to this claim.

Notice that the penalty is not a strictly additive measure, i.e., the penalty of a route isnot the sum of the penalties of its virtual links. Hence Dijkstra’s Algorithm does notnecessarily produce the best route. Heuristically however, the route is almost as good asthe best possible route.

5.1.4 Locating Routers with Undns

The routing algorithm calls undns [SWA] on the hops from a traceroute to determinethe geography of a virtual link. Undns is a program that matches a router name (e.g.ldn-bb1-link.telia.net) against a library of patterns to determine the location of arouter (here London, United Kingdom).

As the first experiment in Section 6.1.5 indicates, undns is incomplete and has strongbiases. To find out which networks are not detected by undns, a wrapper function logsnames of routers whose location is not resolved. Figure 6.3 shows the results that arediscussed in Section 6.1.6.

5.2 Data Plane

I implement the data plane in the click modular router [KMC+00] on Linux. Click is aprogram that acts as a configurable router. The click user can build such a router fromsimple packet processing modules called “elements” in a flexible way. The user specifiessuch a router in an easy to read configuration file as a directed graph. This makes clickmore flexible and easier to use than, e.g. the Netfilter implementation in the Linux kernel.Click is written in C++ and can run in user space as well as kernel space.

The data plane uses UDP to forward encapsulated payload between nodes. Figure 5.1shows all headers of a packet when traveling between two overlay nodes.

I write several new click elements in C++ to implement the functions of the data plane.Figures 5.2 and 5.3 show simplified click configurations of the client and overlay nodes.This Section is structured as follows: I first describe the new click elements for forwardingand for encapsulation and decapsulation (Sections 5.2.1 – 5.2.3). The details of thisprocess are illustrated in Figure 5.4. Then I explain how I deploy click on PlanetLabin Section 5.2.4 and discuss several technical details (Sections 5.2.5 – 5.2.8), the most

20

IP UDP ForwHeader IP TCP

payloadHop1Hop2........

routing from overlay node to

overlay node through the

Internet

routing through the overlay

network

client and server see normal TCP/IP connection

Figure 5.1: Headers of a packet traveling between two overlay nodes.

NextHop

UDP Socket

ForwHeader

StripForwHeader

NetworkDevice

Overlay Network

at last nop

not at last hop

Client

Middlebox

Figure 5.2: Click configuration of middlebox.

NextHop

UDP Socket

VSysTun(NAT)

Overlay Network

at last nop

not at last hop

Server

IPRewriter(NAT)

RevRoute

Figure 5.3: Click configuration of overlay node.

21

Clien

t

TC

P/I

PS

rc:

Cli

en

tD

st:

Serv

er

UD

P/I

PForw

Hea

der

mid

dle

box

Hop

1H

op

2H

op

3

payl

oad

mid

dle

box

Hop1

Hop2

Hop3

Ser

ver

TC

P/I

PS

rc:

Cli

en

tD

st:

Serv

er

payl

oad

TC

P/I

PS

rc:

Hop

3D

st:

Serv

er

payl

oad

TC

P/I

PS

rc:

Serv

er

Dst

: C

lien

tpayl

oad

TC

P/I

PS

rc:

Serv

er

Dst

: H

op

3payl

oad

TC

P/I

PS

rc:

Serv

er

Dst

: C

lien

t

UD

P/I

PForw

Hea

der

Hop

3H

op

2H

op

1m

idd

leb

ox

payl

oad

NA

T a

nd

Rev

Route

at

Hop3

routi

ng t

able

com

pute

d in C

ontr

ol P

lane:

route

to s

erve

r: m

idd

lebox,

hop1, hop2, hop3

Note

th

at

NA

T c

han

ged

Src

to H

op

3

RevR

ou

te m

ain

tain

ed

revers

ed

Hop

Lis

t

Fig

ure

5.4:

Det

ails

ofth

een

caps

ulat

ion

and

deca

psul

atio

n.

22

interesting being packet size in Section 5.2.7.

For clarity, I describe an implementation using a separate middlebox connecting the clientto the overlay network. However, I can implement the middlebox functionality on thesame client machine running the user application software (Section 5.2.6). This makes aseparate appliance to connect the client to the overlay network unnecessary.

5.2.1 Structure of Forwarding Header and NextHop

The basic functionality of the overlay network is routing packets based on informationin the Forwarding Header. I implement the basic forwarding function (see Figure 3.2)in the new click element NextHop that emits packets on different outputs depending onwhether they reached the last hop or not. If they have not reached the last hop, thenext-hop-index of the Forwarding Header is increased and click’s internal destinationannotation of the packet is set.

The structure of the Forwarding Header is as follows:

8 bits 8 bits 8 bits 8 bits

hop-count (n) next-hop- connection optionsindex (i) method (c)

origin IP (0)first-hop IP (1)

. . .

. . .last-hop IP (n-1)

Payload. . .. . .. . .

• hop-count (n) is the number of forwarding hops including the origin where thepacket is injected into the overlay network

• next-hop-index (i) is the index of the IP address of the next-hop. The IP addressesin the Forwarding Header are 0-indexed. A packet in transit between two overlaynodes usually has visited hop i− 1 before and is destined to hop i next.

• The connection method field c determines how the pay load is delivered to an endnode or application when the last hop in the overlay network is reached. In thecurrent implementation, the payload is a TCP/IP packet from a client or server.Two connection methods are defined so far, one for the direction from a client to aserver (c = 0) and one for the reverse direction (c = 1).

23

• Options have not been defined yet, but might be used in future.

5.2.2 Encapsulating Packets at the Client: ForwHeader, StripForwHeader

The middlebox connects a client to the overlay network. It encapsulates the TCP/IPpackets from the client before injecting them into the overlay network using the clickelement ForwHeader. Vice versa, a packet from the overlay network has to be decapsu-lated. This is done by simply stripping the Forwarding Header with StripForwHeader(see Figure 5.2 for location in click configuration).

ForwHeader looks up the information for the Forwarding Header in a routing table main-tained by ForwHeader itself. An entry in the routing table contains a target and then alist of hops, the first hop is the IP of the middlebox, the last hop is the last node in theoverlay network along the packet route. If no route is found, a packet is queued until amatching entry is added to the routing table.

The ForwHeader provides basic functions to communicate with the control plane. It hasfunctions to add and delete routes and to list all IP addresses of packets that are queuedbecause a respective entry is missing in the routing table.

The control plane polls the pending IP addresses, computes the routes, and adds themto ForwHeader’s routing table.

5.2.3 Encapsulating Packets at the Overlay Node Closest to the Server:RevRoute and Network Address Translation

A standard TCP server is unaware of the overlay network. Hence click encapsulatesand decapsulates the Forwarding Header. This is done in RevRoute such that the re-sponse from the server to a packet is sent back along the same route as the packet itself.RevRoute has two input and output ports for the forward and backward direction. Inthe forward direction, Forwarding Headers are stripped and a mapping is created fromthe encapsulated TCP/IP flow (SrcIP, SrcPort, DstIP, DstPort, Proto) to the reversedhop list in the Forwarding Header. In the backward direction, this mapping is appliedto attach a Forwarding Header.

The standard TCP server is supposed to send packets back to the last overlay node andnot to the client directly. I achieve this by Source Network Address Translation usingthe standard click element IPRewriter. The source field of the TCP/IP packets sent tothe TCP server are rewritten to the IP address of the last overlay node, hence the serversends its response back to the overlay node.

Figure 5.3 shows the location of RevRoute and IPRewriter in the click configuration.An example of a forward and backward flow is shown in Figure 5.4.

24

5.2.4 Deployment on PlanetLab

When deploying the software on PlanetLab, I face a major problem: how can click sendout and receive raw TCP/IP packets to other hosts? Software deployed on PlanetLabnodes has only virtualized access to the machine, including virtualized network access.The virtualized Raw Socket implementation of PlanetLab is not suitable. In particular,Raw Sockets do not suppress the TCP RSTs the Linux kernel sends out when receivingreplies to my TCP/IP packets.

Sapan Bhatia and I achieve the ability to send raw TCP/IP packets through two vsysscripts: fd tuntap and vif up. Sapan Bhatia is a member of the PlanetLab support teamat Princeton and the author of vsys. vsys is a privilege allocation tool that allows aPlanetLab user to execute scripts needing raw, unvirtualized root access to the Linuxkernel running on a PlanetLab Node [Bha]. Usually, a Planetlab User has only access toa “slice” that is isolated from other slices and has only restricted rights. The PlanetLabadministrators deploy vsys scripts on the PlanetLab nodes and give individual PlanetLabusers rights to use certain scripts. The PlanetLab users sees files for these scripts in the/vsys directory of his slice and can trigger the execution of the script by reading andwriting to these files.

Sapan Bhatia has rewritten, debugged, and deployed these scripts on PlanetLab while Ihave been testing them. The fd tuntap script allocates a Tun/Tap device and passes thefile descriptor to a user-space application. The vif up script configures such a Tun/Tapdevice, enables Source Network Address Translation optionally and adds entries to therouting table.

I rewrite the KernelTun click element to use the vsys scripts for allocating and configuringa Tun/Tap device. The new click element is called VSysTun.

Using the new VSysTun element, click can send raw TCP/IP packets to a host if thesource IP belongs to an allocated private IP prefix (here 10.6.0.0/24). The source IP isrewritten to the external IP address of the PlanetLab node. Responses to packets arereceived through VSysTun as well.

5.2.5 UDP Sockets

The UDP socket in Figures 5.2 and 5.3 actually corresponds to two UDP sockets in theclick configuration. It is necessary to deploy two UDP sockets, because click UDP Clientand UDP Server sockets have different semantics and both are needed: A UDP ClientSocket determines the destination from the packet annotation to send packets anywhere.In contrast to a UDP Client Sockets, a UDP Server Socket can receive packets fromanywhere.

25

5.2.6 Implementing the Middlebox Functionality on the Client Ma-chine

For clarity, I describe an implementation involving a middlebox running click and con-necting the client to the overlay network. In this Section, I explain how to implementthe middlebox functionality on the same machine as the client software making an extraappliance unnecessary.

For this, I rewrite the click configuration by replacing the network device connecting tothe client machine with a KernelTun click element accessing a Linux Kernel Tun/Tapdevice. The default route in the Linux kernel is set to this KernelTun device so thatclick receives TCP/IP packets from the user application. The user application uses thestandard socket interface to initiate TCP connections.

Since the data plane and control plane need to communicate to the overlay network, thecontrol plane sets up a kernel route to each node in the overlay network via the gatewayto the Internet. It also sets up routes to the name servers.

5.2.7 Packet Size

When encapsulating a TCP/IP packet in another Forwarding, UDP and IP header, theoverhead is 40-122 bytes. Packets that are small enough without overlay overhead mightneed to be fragmented or cannot be delivered anymore.

In practically all situations, the most severe restriction of the Maximum TransmissionUnit (MTU) is 1500 bytes from the Ethernet connection at the end hosts and is usuallyavoided all together by the transport layer choosing an appropriate chunk size. Frag-mentation does usually not happen past this first link. Hence I avoid the problems aboveby simply setting the MTU of the KernelTun device interfacing the application softwareto 1378 bytes to account for possible overheads from the encapsulation (extra IP (20-60bytes), UDP (8 bytes), and Forwarding Header (12-54 bytes, assuming at most 12 hops)).

From a theoretic perspective, this issue deserves further discussion. IPv4 and IPv6 havedifferent mechanisms for dealing with packets larger than the MTU. IPv4 fragmentsand reassembles packets (RFC 815), hence choosing a too large packets size results ina performance penalty. In contrast, IPv6 allows too large packets to be dropped andthe end host has to perform MTU path discovery (RFC 1191) to size its packets. Theperformance penalty in IPv4 is multiplied by the fact that fragmentation and reassemblycould potentially occur at each node on the route if a packet is too large while traversingthe overlay network.

This fragmentations is invisible to the client application or operating system which wouldinstruct the transport layer to use smaller packets. It is invisible because it happens atthe level of the outer IP header and not the inner IP header.

26

Mechanism to avoid this involve translating messages about the MTU or fragmentationhappening at the level of outer IP headers to inner IP headers. The translation of ICMPmessages about MTU has to rewrite IP addresses, adjust the MTU size to account foroverlay overhead and deliver it through the overlay network to the client. Translatingfragmentation means to break up the inner IP packets into fragments of a size accountingfor overhead from overlay network headers.

The control plane can also perform MTU path discovery as part of the measurementsof virtual links. The collected information is used to compute the maximal MTU alonga route and the result is added to the routing table, so that click can inform the clientoperating system or application of the maximal MTU when sending a packet throughthat route.

RFC 1853 (IP in IP Tunneling) also discusses these issues.

5.2.8 Packets that Cannot Be Delivered

When a packet cannot be delivered, for example, because a route complying with theconstraint cannot be computed, future implementation should deliver an ICMP “Desti-nation Unreachable” message to the client application. If there is a server sided error,such a message also needs to be encapsulated and delivered back to the client.

27

Chapter 6

Results

In the first Section, I discuss the preparation necessary to perform the experiments andmeasurements. The next four Sections show results of the measurements, in particularthe possibilities to reroute and the predicted performance impact in Section 6.4 and themeasured overlay performance along these routes in Section 6.5.

The preparation includes choosing countries to be investigated as well as estimatingthe geolocation accuracy of GeoRoute, in particular Figure 6.3 shows the fraction ofparticipating routers that can be located as a measure of accuracy for the followinganalysis.

I structure the results into four parts. In Section 6.2, I test GeoRoute as a possible userapplication with a standard web browser and describe why GeoRoute works well withsome websites and not as well with others.

I analyze the virtual links in Section 6.3. I use one snapshot of the data collected bythe overlay network for statistics across the virtual links and several snapshots collectedover 4 days to estimate the stability of virtual links. These data are further evidence ofGeoRoute’s effectiveness.

The most interesting results are in Section 6.4 and show the possibilities and the impactof rerouting around certain countries by comparing alternative routes and computingstatistics on the delay inflation. This Section lists explicit examples of newly computedroutes, e.g. an alternative route from the U.S. to China crossing the Atlantic instead ofthe Pacific.

Finally, I measure the influence of rerouting on the TCP/IP throughput when usingGeoRoute in Section 6.5.

To summarize, the analysis of the stability of routes in Section 6.3.2 and the high fractionof routers located (see Figure 6.3) indicates that the data are fairly accurate. As discussedin Section 6.2, GeoRoute can be modified to be used as a user application that forwards

28

packets along routes fulfilling a given geographic constraint with high probability. Theoverlay overhead reduces bandwidth to around 50KB/s.

6.1 Preparation

The preparations include choosing countries we want to investigate and web servers andoverlay nodes in those countries (Section 6.1.1–6.1.3). I also determine a good test filesize for measuring bandwidth in Section 6.1.4. In my first experiment, I have discoveredmany pathologies due to the inability of undns to locate some important routers (Section6.1.5). To have accurate data, I manually locate most routers in the traceroutes collectedby the overlay network (Section 6.1.6) and use those locations as basis for all othermeasurements and results. The last part of the preparation (Section 6.1.7) describeswhat different country constraints I investigate.

6.1.1 Choosing Test Web Servers

I setup web servers (implemented in Python, serving static files of various sizes) atdifferent locations for bandwidth measurements. I chose 15 countries based on the numberof Internet users, see Table 6.1. For each country, I try to find two working and accessiblePlanetLab nodes. If there are more than two nodes, I pick some random trying to ensurethat when accessing the test web server the bandwidth is at least 100KB/s. Table 6.2shows the web servers I use for the experiments. The nodes of PlanetLab are highlyunevenly distributed around the world. Whereas there are hundreds of nodes in the U.S.alone, there are none in, e.g. Mexico. Even if there are PlanetLab nodes available ina country, they might be so poorly maintained, that none of them is working, or badconnectivity renders them unusable.

6.1.2 Choosing Overlay Nodes

The selection of overlay nodes is severely restricted by the fact that, by the time of themeasurements, only a fraction of PlanetLab nodes has the necessary scripts deployedand working and most of these are located in the U.S. As a rough guide line for selectingcountries for overlay nodes, I use the number of packet routes that are traversing a certaincountry (see Table 6.3). This number can be seen as a measure of how much importantInternet infrastructure a country contains. The overlay nodes are shown in Table 6.4.During the experiment, the node planetlab5.ie.cuhk.edu.hk fades in and out.

29

Table 6.1: Countries by number of Internet Users, Source: wikipedia.

Country TLD Internet UsersChina .cn 425,000,000United States - 240,000,000Japan .jp 99,150,000India .in 81,000,000Brazil .br 75,943,600Mexico .mx 68,430,000Germany .de 65,200,000Russia .ru 60,000,000United Kingdom .uk 51,450,000France .fr 44,630,000Nigeria .ng 43,985,000South Korea .kr 39,500,000Turkey .tr 35,000,000Italy .it 34,000,000Iran .ir 33,200,000

http://en.wikipedia.org/wiki/List of countries by number of Internet users

30

Table 6.2: The test web servers used for the measurements.

Name Countrypl1.6test.edu.cn Chinaplanetlab-2.sjtu.edu.cn Chinaplanetlab6.cs.cornell.edu U.S.planetlab4.flux.utah.edu U.S.planetlab-02.kusa.ac.jp Japanplanetlab-02.naist.jp Japan

India missingplanetlab1.pop-mg.rnp.br Brazilplanetlab2.pop-mg.rnp.br Brazil

Mexico missingplanet02.hhi.fraunhofer.de Germanyplanet2.prakinf.tu-ilmenau.de Germanyplab-1.sinp.msu.ru Russiaplebt2.essex.ac.uk United Kingdomplanetlab-1.imperial.ac.uk United Kingdompeeramide.irisa.fr Franceplanetlab-1.imag.fr France

Nigeria missingplanetlab3.netmedia.gist.ac.kr Koreaplanet1.ku.edu.tr Turkeyplanetlab2.plab.ege.edu.tr Turkeygschembra4.diit.unict.it Italyplanet2.unipr.it Italy

Iran missingplanetlab-2.elisa.cpsc.ucalgary.ca Canadapl1.planetlab.uvic.ca Canada

31

Table 6.3: Routes through a country using data from Archipelago project[Cai], analyzedin [MGD].

Country TLD RoutesUnited States - 256271Germany .de 75803United Kingdom .uk 59235Japan .jp 41107France .fr 37209Netherlands .nl 28419Sweden .se 23292Denmark .dk 18288Australia .au 14905Canada .ca 12610

Table 6.4: The overlay nodes used for the measurements.

Name Countryplanetlab2.pop-mg.rnp.br Brazilplanetlab-2.elisa.cpsc.ucalgary.ca Canadapl1.planetlab.uvic.ca Canadaplanetlab4.inf.ethz.ch Switzerlandplanet3.prakinf.tu-ilmenau.de Germanyplanet01.hhi.fraunhofer.de Germanyricepl-4.cs.rice.edu U.S.planetlab4.cs.duke.edu U.S.planetlab-01.cs.princeton.edu U.S.planetlab5.ie.cuhk.edu.hk Chinaplanetlab3.di.unito.it Italyplanetlab-02.naist.jp Japanplanet1.jaist.ac.jp Japanplab-1.sinp.msu.ru Russiaplanetlab-1.ssvl.kth.se Sweden

32

6.1.3 Choosing a Client Site and Entry Point

I use the client site (UC Berkeley) and entry point (ricepl-4.cs.rice.edu) chosen here forall bandwidth measurements in Section 6.5.

Client Site. Unfortunately, the client side software cannot be deployed on PlanetLabnodes on time because it requires a mechanism to redirect the default routing for packetsfrom a user application to click so that packets are sent through the overlay network. Dueto this restriction, I test only one client site, namely I choose a computer as client for themeasurements that is connected to the Internet through the network of the University ofCalifornia, Berkeley in the U.S.

Possible future mechanisms to use PlanetLab nodes as client for testing TCP throughputare reimplementing the entire TCP/IP stack in user space without Linux kernel support,e.g. in a click element, or redirecting data from a test application using the socket interfacein the same fashion as described in Section 5.2.6.

Entry Point. An ideal entry point has low network latency to the client site and is in thesame country. Since a PlanetLab node with the right software deployed is not available inBerkeley, I measure round trip times to several nodes and pick three candidates. As testweb server, I pick a random node outside the U.S. (planetlab2.pop-mg.rnp.br). For eachcandidate entry point, I use wget to measure TCP throughput when sending packets tothe server and back through that entry point. Figure 6.1 shows the results. For all otherexperiments, I choose ricepl-4.cs.rice.edu as entry point where the client injects packetsinto the overlay network.

6.1.4 Filesize for Bandwidth Measurements

TCP/IP needs some time to achieve maximal throughput. To determine a good file sizefor testing TCP throughput, I make several measurements for various sizes, see Figure6.2. Unless otherwise indicated, I perform the bandwidth measurements in all followingexperiments with wget downloading a 500KB files. Wget is a standard UNIX commandline tool to download files through HTTP.

6.1.5 Locating Routers with Undns and Pathologies in First Experi-ment

The results of the routing algorithm are inaccurate when relying on undns to locaterouters. Instead of rerouting to avoid countries, the algorithm really produces routesthrough networks that are undetected by undns. The results include many pathologies.For example, when asked to find a route starting from a node in the U.S. and avoiding theU.S., the algorithm responds by rerouting to a node in Canada because that particularvirtual link has no router detected by undns. Looking through the traceroutes of the

33

rice

pl-

4.c

s.ri

ce.e

du

pla

netl

ab4

.cs.

duke

.edu

pla

netl

ab-0

1.c

s.pri

nce

ton.e

du0

10

20

30

40

50

60

70

80

Bandw

idth

in K

B/s

Figure 6.1: Bandwidth when using different entry points measured by downloading250KB files from planetlab2.pop-mg.rnp.br with wget using the overlay network; averageof 10 measurements.

0 200 400 600 800 1000File size in KB

0

50

100

150

200

250

300

Bandw

idth

in K

B/s

Figure 6.2: Average bandwidth achieved when using different file sizes measured bydownloading from planetlab2.pop-mg.rnp.br with wget directly and not through the over-lay network; average of 50 measurements.

34

overlay links manually for several routes, I can find pathological cases where the algorithmis doing the opposite of what it is supposed to: instead of finding a route avoiding acertain set of countries, it computes a route worse than the direct route, hitting more ofthe countries in the set specified to be avoided.

It turns out that undns only recognizes a small fraction of router names and also has avery strong U.S. bias, see Figure 6.3. One possible reason is that undns has not beenupdated for three years.

Commercial databases such as “IP2Location” seem to perform badly when locatingrouters. The free and the commercial version of the IP2Location database have the samequality, the free version being restricted to the IP address space up to 99.255.255.255. Itlocates almost every router in this address space incorrectly. I observe this by manuallycomparing some router names with the result from the IP2Location database.

6.1.6 Manually Locating Routers

No DNSName

15%

UnresolvedLocation

6%

Locationdetectedby undns

27%

Locationdetected manually

51%

Figure 6.3: Fraction of routers that could be located.

In order to obtain accurate data, I manually determine the location for 332 routers:

First, I look at the website given by the last two or three parts of a computer’s domainname. The webpage reveals to what institution a computer belongs.

If it is a university or national science institute, I assume that the country of the com-puter’s location is just given by the Top-level Domain (TLD) country code.

If it is an Internet Service Provider (ISP), the router name often contains entire city

35

http://two.wide.ad.jp/widebb-200912.png

1100 Walnut

http://he.net/HurricaneElectricNetworkMap.pdf

Figure 6.4: Two network maps used to verify the location ofrouters: e.g. ve-61.cisco2.dojima.wide.ad.jp is in Dojima, Japan andgige-g2-17.core1.sto1.he.ne appears close to a router in Frankfurt in tracer-outes such that the network map strongly suggests “sto” stands for Stockholm,Sweden.

36

names, sometimes abbreviated city names, country codes, U.S. state codes, airport codes,acronyms of Internet Exchange points or internal names of the ISP’s sites. To disam-biguate, I try to find network maps (see Figure 6.4) of various ISPs. For example, anetwork has a gateway starting with “ja” suggesting that it connects to Japan. Therespective network map reveals that the gateway really connects to Janet, an educationnetwork in the United Kingdom.

Additional difficulty comes from the fact that some ISPs have merged and the routername has a domain different from the now official name of the ISP. Also, some ISPs donot provide public network maps or have websites not in English. In one case, I try tomatch router names to a network map in Cyrillic.

From the 645 IP addresses reported by the traceroutes collected between the overlaynodes and from the overlay nodes to the web servers, 550 have DNS names, but only177 are located by undns. After my manual location, only 41 routers with DNS namesrespectively 136 routers in total cannot be located, some of them having names such as“unknown.level3.net”. See Figure 6.3.

For future work, I suggest to reimplement undns to guess a naming scheme for an ISPautomatically and verify it by correlating it with other topological data such as the onescollected from traceroute.

Another mechanism to lower the number of routers that cannot be located or that areunresponsive is based on the plausible assumption that a router bracketed on both sidesby two routers in the same country is also located in the same country.

Even though undns lists Hong Kong as a separate country, I count it as part of China inthe data analysis.

6.1.7 Creating a List of Interesting Country Constraints

A country constraint is a set of countries to be avoided and is part of the input to thealgorithm described in Section 5.1.3. To compute the routes for the measurements below,I run the algorithm for each of the following country constraints:

• No constraint (empty set)

• One country from Table 6.1 (top 15 countries by Internet Users)

• One country from Table 6.3 (top 10 countries by routes traversing)

• Two or three countries from Table 6.3

37

6.2 User Experience

To evaluate user experience, I test GeoRoute using the standard browser Mozilla FireFox(Version 3.6).

For this experiment, the browser is running on the same machine as the GeoRoutesoftware connecting to the deployed overlay network. The user specifies various countryconstraints by typing a command of the form set avoid countries([’Netherlands’])into the python command line. Later, a webinterface can be implemented to make theinput of the geographic constraint more user-friendly.

The software configures itself at start and, except for entering the country constraints,requires no further interaction with the user.

When a user accesses a website, the first TCP/IP packet from the browser initiates theGeoRoute software to request ping and traceroute data to the respective web server fromthe overlay nodes and to compute routes to the web server (see description of InteractiveMode in Section 5.1.2). These processes are automatic but can be monitored by the useron the python command line. GeoRoute then source routes the initial and all followingpackets with the same destination in both directions. The routes are cached to acceleratefuture access.

The first time a website is accessed by any user of GeoRoute, route computation causes a20–30 second delay which can be significantly shortened: the majority of the delay timeis due to traceroute and ping. When using traceroute and ping in parallel with differentcommand line options to parallelize the traceroute probing process and sending out pingpackets faster, the measurement of a virtual link takes only 4–8 seconds. In future work,I deploy these simple modifications. I expect the initial delay to be reduced to only 5–10seconds.

If the same user accesses the website again, there is no delay as the route is cached.Furthermore, overlay nodes cache the traceroute and ping measurements, such that thereis only a 1–2 second delay to poll the cached data from all overlay nodes if a differentuser accesses the same website.

Besides the initial delay, the associated overlay overhead makes the website load slower.However, many websites still load at rates fast enough to allow usual navigation andinteraction with a website.

The user experience heavily depends on the website though: dradio.de and pixar.comare good examples where the overlay network hardly degrades the user experience.However, there are examples of websites that are loading slowly through GeoRoute,e.g., www.tagesschau.de served from several different machines in a Content DistributionNetwork.

Problems. As previously mentioned, there is a delay the first time the user accesses

38

a website. This delay is multiplied when websites pull extra information like images,HTML snippets or scripts from many different servers. Even worse, some of these piecesmight in turn load other pieces from yet other servers, making the webpage take minutesto load properly, even requiring the user to hit the reload button in the mean time. Forsome websites, this is worsened by FireFox’s render engine not showing anything untilmany components of a webpage are completely loaded.

Sometimes a route to a host cannot be found. This is inherent to the geography of theoverlay network or even of the Internet itself. In the first case, adding more nodes to theoverlay network allows routes that satisfy a given geographic constraint. In the lattercase, such a route simply does not exist.

If a route cannot be found, GeoRoute produces an error message on the python commandline. This error message can be confusing if it contains a DNS name (e.g., a96-17-70-81.deploy.akamaitechnologies.com) different from the website (here www.tagesschau.de)that the user is trying to access.

For similar reasons, the apparent location of a website can be different from the locationit is actually served from. E.g. when accessing www.tagesschau.de, the DNS record willdirect the browser to an Akamai Content Distribution Network Server in Palo Alto.

Some web servers cannot be pinged or tracerouted so the overlay network cannot estimatewhether the user specified constraint is met. Also, as discussed earlier, undns might failto recognize routers between the overlay network and the web server.

Future improvements. One solution to increase accuracy is to find a better mechanismfor locating routers. Another solution is based on the observation that there are manypotential virtual links to web servers but only few virtual links between the overlay nodes.An organization deploying the overlay network can monitor the virtual links between theoverlay nodes closely and locate the routers manually. To decrease the potential of virtuallinks to web servers traversing certain countries, the overlay software picks an overlaynode closest to the web server guaranteed to be in the same country.

Another future improvement is a user setting to specify how strictly routes have to bechecked, i.e. how accurate the geography of a route has to be determined (e.g. fractionof located routers) before it is allowed.

6.3 Virtual Link Analysis

This Section first analyzes properties of virtual links in detail for one particular snapshotin time and then estimates stability over the course of several days.

39

0 100 200 300 400 500 600 700 800Delay in ms

20

40

60

80

100

120

140

Num

ber

of

Vir

tual Li

nks

0 20 40 60 80 100Jitter in ms

100

200

300

400

500

600

Num

ber

of

Route

s

0.0 0.2 0.4 0.6 0.8 1.0Loss rate

100

200

300

400

Num

ber

of

Vir

tual Li

nks

Figure 6.5: Histogram of delays, jitter, and loss rates of virtual links (round trip time,its standard deviation and loss rate as reported by ping).

40

6.3.1 Virtual Link Properties

To further estimate the accuracy of the geography of routes, I compute the completenessof the traceroutes. On average, a traceroute is 94% complete. Here the average is takenover all virtual links of the overlay network and from the overlay network to a test webserver. I define the completeness as the fraction of routers whose IP is detected to thetotal of all routers that are either detected or listed as *. Traceroute reports a router as* if the IP router does not send a response to traceroute or the response is dropped alongthe route. In this case, it continues probing the next router by sending packets with alarger TTL.

Figure 6.5 shows performance measures such as delay, jitter, and loss rates of the virtuallinks.

Almost all virtual links have a round trip time of less than 400ms. Regarding that thelargest distance of two points on earth is 20000km and the refractory index of glass usedin most fiber optic cable is around 1.5, the expected maximal value for round trip timeis 2 · 20000km

c/1.5 = 200ms. The factor of 2 is due to the circuitousness of Internet paths,processing times, queuing and store-and-forward delays, and satellite links. There is aspike in the histogram for round trip times less than 50ms indicating that the overlaynodes and web servers are not spread evenly around the world but form clusters.

The jitter and the loss rate of all virtual links are fairly low (less than 10ms respectivelyless than what 10 pings can detect). Virtual links with reported loss rates of 100% aremostly due to the host not responding to pings and not to the packets actually beinglost.

6.3.2 Stability of Virtual Links

For this analysis, I use 8 snapshots of ping and traceroute data taken during a period of4 days to see how the properties of virtual links change over time, first with respect tothe route underlying a virtual link and the geographic properties, then with respect tothe performance. Future work will take snapshots more frequently.

Virtual links are mostly stable over days with respect to their geography and their perfor-mance. The traceroutes for more than half of the virtual links list the exact same routerseach time. For another third, even though the exact router names change, the routers arestill in the same countries, see Figure 6.6. This suggests that collecting traceroute dataevery 10 minutes is enough to ensure that user constraints are met with high probability.

Of the routes that change countries, 60% switch around the same time (around November15, 2010 20:00 UTC), switch back the next day, and are stable otherwise.

The other routes that change countries involve planetlab3.di.unito.it and plab-1.sinp.msu.ru.They switch several times during the measured 4 day period. Routes to the latter node

41

Countrieschanged

14%

Samerouters

53%

Samecountries

34%

Figure 6.6: Route stability of virtual links over 4 days.

alternate between traversing Denmark and traversing Germany to connect the EuropeanResearch and Education network GEANT to the Russian network.

0 20 40 60 80 100Standard deviation when delay is measured over several days in ms

0

50

100

150

200

250

300

350

400

Num

ber

of

vir

tual lin

ks

Figure 6.7: Standard deviation of delay over 4 days.

Figure 6.7 shows the standard deviation of the delay measurements taken over 4 days.Whereas Figure 6.5 measures this standard deviation for measurements over a shortperiod of seconds, Figure 6.7 is from a more long term perspective of days. The standarddeviation is still fairly low indicating that queuing delays vary only little despite changingusage patterns of the Internet. The spike around 35ms is probably due to the underlyingroutes actually changing.

42

6.4 Analysis of Rerouting Possibility and Impact

I analyze the possibilities to find routes respecting geographic constraints and the impacton delay.

These data are based on a snapshot of all ping and traceroute measurements betweenthe overlay nodes and from the overlay nodes to the web servers. I run the algorithmdescribed in Section 5.1.3 on the resulting graph to compute alternative routes. I firstanalyze the routes from the U.S. in Section 6.4.1 and then from every test web server toevery other test web server that is not in the same country in Section 6.4.2.

The more countries are added to the geographic constraint, the more overlay nodes andvirtual links violating the constraint have to be discarded. This makes paths between twofixed nodes longer and, at some point, splits the graph into several connected componentssuch that paths between them cannot be found. For example, the overlay node in Italy isseparated from the other overlay nodes when all virtual links passing through Switzerlandare removed. There are probably routes in the Internet leaving Italy and not traversingSwitzerland, but there are not enough overlay nodes to see such a route. For constraintsof up to two countries, this is the only example of a disconnected graph of overlay nodesand web servers. In case of constraints of three countries, several such combinations existsplitting smaller islands from the rest of the graph.

The high connectedness is explained by the fact that connectedness is the transitiveclosure of the relationship defined by admissible virtual links. This means that manyvirtual links can be removed before the graph partitions.

However, the paths in the graph become significantly longer even before the graph par-titions. Figure 6.9 and Figure 6.10 show the resulting delay inflations in case of avoidingone or two countries: rerouting around one country causes delay inflations of only upto 24ms in the average case (average over pairs of web servers). Rerouting to avoid twocountries, however, causes delay inflations which are almost a magnitude higher (163ms).The results below also include the worst case scenario when avoiding one country.

6.4.1 Routes from the U.S.

I first compute routes starting at ricepl-4.cs.rice.edu in the U.S. to every test web serverand for every country constraint described in Section 6.1.7. Table 6.5 lists all the com-puted routes together with the countries which they traverse and the expected delay,jitter, and loss.

For example, the direct route from ricepl-4.cs.rice.edu to plab-1.sinp.msu.ru traversesGermany, Russia, and the U.S. If the user specifies to avoid Germany, GeoRoute takesthe indirect route via planetlab-1.ssvl.kth.se traversing Denmark and Sweden insteadof Germany. The delay inflation is the difference between the direct route and the

43

alternative route, here 189ms− 183ms = 6ms.

U.S. UK

IT

BR

(Map not to scale.)

Figure 6.8: Routes to plebt2.essex.ac.uk and gschembra4.diit.unict.it.

Figure 6.8 shows two more examples. The direct route from ricepl-4.cs.rice.edu toplebt2.essex.ac.uk traverses the Netherlands and Germany. If the user specifies to avoidthe Netherlands, GeoRoute sends packets via planetlab-01.cs.princeton.edu which is di-rectly connected to the UK. Similarly, redirecting through planetlab2.pop-mg.rnp.brwhen routing to gschembra4.diit.unict.it avoids Germany.

Notice that even though there are so many country constraints, the algorithm computedonly 22 alternative routes. Addressing the question about feasibility of routing algorithmsin Section 3.1, this number suggests that a modified path-vector routing protocol can beused for GeoRoute, reducing the amount of data a client has to poll.

Another observation is that either the algorithm fails to find a route for a given countryconstraint or the computed route has at most one redirection hop.

Examples of small delay inflations occur when avoiding either Germany, the Netherlandsor the United Kingdom. The alternative routes to web servers in Europe or Russia avoid-ing one of those three countries often route through one of three other countries and havedelay inflation less than 20ms, for example, the alternative route to peeramide.irisa.fr viaplanetlab-01.cs.princeton.edu.

However, avoiding two or three of these countries sometimes results in no route or a routewith large delay inflation (greater than 300ms), for example a route via planetlab2.pop-mg.rnp.br to peeramide.irisa.fr.

Another interesting example of a small delay inflation of only 20ms are the routes topl1.6test.edu.cn. The direct route is going through the Pacific, whereas the route viaplanetlab5.ie.cuhk.edu.hk through the Atlantic.

The routing algorithm cannot find a route to the web servers in Italy avoiding Switzer-land.

44

Notice that China does not appear in the Countries column of the Table for the directroute to pl1.6test.edu.cn. This is explained by the fact that the traceroutes to the webservers in China are often incomplete, ending at a router in or close to China. It seemsthat the so called Great Firewall of China is blocking router responses to traceroutepackets.

45

Tar

get

Cou

ntri

esde

rive

dm

easu

red

via

Rou

tede

lay

jitte

rlo

ssB

Wσ

BW

Ms

pla

net

lab

1.p

op-m

g.rn

p.b

rbr

us20

8ms

3ms

0%42

KB

/s12

KB

/s29

pla

net

lab

2.p

op-m

g.rn

p.b

rbr

us20

8ms

3ms

0%38

KB

/s13

KB

/s29

pla

net

lab

-2.e

lisa

.cp

sc.u

calg

ary.

caca

us0m

s0m

s10

0%36

KB

/s13

KB

/s29

pl1

.pla

net

lab

.uvic

.ca

caus

86m

s1m

s0%

102K

B/s

67K

B/s

29p

l1.6

test

.ed

u.c

njp

us23

8ms

0ms

0%26

KB

/s5K

B/s

29vi

apl

anet

lab4

.cs.

duke

.edu

us31

1ms

2ms

0%49

KB

/s36

KB

/s28

via

plan

etla

b5.ie

.cuh

k.ed

u.hk

cntw

nlus

258m

s1m

s0%

78K

B/s

33K

B/s

10p

lan

etla

b-2

.sjt

u.e

du

.cn

jpcn

us27

9ms

1ms

0%42

KB

/s13

KB

/s28

via

plan

et1.

jais

t.ac

.jpjp

us28

5ms

1ms

0%72

KB

/s30

KB

/s29

via

plan

etla

b4.c

s.du

ke.e

duus

334m

s2m

s0%

32K

B/s

11K

B/s

27vi

apl

anet

lab5

.ie.c

uhk.

edu.

hkcn

twnl

us28

7ms

1ms

0%27

KB

/s8K

B/s

8p

lan

et02

.hh

i.fr

aun

hof

er.d

ede

us14

6ms

0ms

0%40

KB

/s17

KB

/s30

pla

net

2.p

rakin

f.tu

-ilm

enau

.de

deus

138m

s0m

s0%

44K

B/s

15K

B/s

29p

lan

etla

b6.

cs.c

orn

ell.ed

uus

49m

s0m

s0%

40K

B/s

15K

B/s

29p

lan

etla

b4.

flu

x.u

tah

.ed

uus

41m

s0m

s0%

45K

B/s

14K

B/s

29p

lan

etla

b-1

.im

ag.f

rch

deus

fr15

7ms

0ms

0%44

KB

/s12

KB

/s29

via

plan

etla

b2.p

op-m

g.rn

p.br

brus

esfr

478m

s4m

s0%

39K

B/s

20K

B/s

27vi

apl

anet

lab4

.cs.

duke

.edu

ukus

fr16

3ms

1ms

0%88

KB

/s52

KB

/s28

pee

ram

ide.

iris

a.fr

chde

usfr

155m

s0m

s0%

32K

B/s

11K

B/s

29vi

apl

anet

lab-

01.c

s.pr

ince

ton.

edu

ukus

fr16

1ms

0ms

0%66

KB

/s59

KB

/s28

via

plan

etla

b2.p

op-m

g.rn

p.br

brus

esfr

476m

s5m

s0%

25K

B/s

3KB

/s28

Tab

le6.

5:A

lter

nati

vero

utes

from

the

U.S

.(r

icep

l-4.

cs.r

ice.

edu)

wit

hpr

edic

ted

dela

y,jit

ter,

and

loss

rate

and

mea

sure

dba

ndw

idth

(con

tinu

edon

next

page

).M

sis

the

num

ber

ofba

ndw

idth

mea

sure

men

ts.

46

Tar

get

Cou

ntri

esde

rive

dm

easu

red

via

Rou

tede

lay

jitte

rlo

ssB

Wσ

BW

Ms

gsch

emb

ra4.

diit.

un

ict.

itch

itde

us17

5ms

1ms

0%35

KB

/s13

KB

/s29

via

plan

etla

b2.p

op-m

g.rn

p.br

brch

ites

us51

0ms

5ms

0%55

KB

/s20

KB

/s29

pla

net

2.u

nip

r.it

chit

deus

153m

s0m

s0%

37K

B/s

15K

B/s

29vi

apl

anet

lab2

.pop

-mg.

rnp.

brbr

chit

esus

480m

s4m

s0%

51K

B/s

18K

B/s

28p

lan

etla

b-0

2.ku

sa.a

c.jp

jpus

160m

s0m

s0%

46K

B/s

13K

B/s

29p

lan

etla

b-0

2.n

aist

.jp

jpus

157m

s0m

s0%

40K

B/s

13K

B/s

30p

lan

etla

b3.

net

med

ia.g

ist.

ac.k

rkr

us20

3ms

1ms

0%29

KB

/s8K

B/s

29vi

apl

1.pl

anet

lab.

uvic

.ca

caus

237m

s1m

s0%

68K

B/s

67K

B/s

29p

lab

-1.s

inp

.msu

.ru

deru

us18

3ms

1ms

0%43

KB

/s12

KB

/s29

via

pl1.

plan

etla

b.uv

ic.c

aca

ruus

320m

s2m

s0%

80K

B/s

42K

B/s

28vi

apl

anet

lab-

1.ss

vl.k

th.s

edk

seru

us18

9ms

1ms

0%75

KB

/s42

KB

/s28

via

plan

etla

b5.ie

.cuh

k.ed

u.hk

cntw

nlru

us57

9ms

3ms

0%47

KB

/s14

KB

/s12

pla

net

lab

2.p

lab

.ege

.ed

u.t

rtr

ronl

uscz

dehu

182m

s0m

s0%

36K

B/s

13K

B/s

29vi

apl

anet

lab-

01.c

s.pr

ince

ton.

edu

trus

czde

behu

197m

s1m

s0%

26K

B/s

1KB

/s29

via

plan

etla

b2.p

op-m

g.rn

p.br

brtr

itus

beat

chhu

es52

9ms

4ms

0%47

KB

/s13

KB

/s28

pla

net

1.ku

.ed

u.t

rtr

deus

152m

s1m

s40

%64

KB

/s25

KB

/s21

via

plan

etla

b-02

.nai

st.jp

trjp

us48

0ms

4ms

0%43

KB

/s14

KB

/s21

via

plan

etla

b2.p

op-m

g.rn

p.br

brtr

nluk

us50

1ms

4ms

0%44

KB

/s12

KB

/s21

via

plan

etla

b4.c

s.du

ke.e

dutr

us17

4ms

2ms

40%

41K

B/s

16K

B/s

23p

leb

t2.e

ssex

.ac.

uk

nlde

ukus

146m

s0m

s0%

33K

B/s

11K

B/s

27vi

apl

anet

lab-

01.c

s.pr

ince

ton.

edu

ukus

150m

s1m

s0%

25K

B/s

1KB

/s29

pla

net

lab

-1.im

per

ial.ac

.uk

nlde

ukus

145m

s0m

s0%

103K

B/s

69K

B/s

29vi

apl

anet

lab-

01.c

s.pr

ince

ton.

edu

ukus

149m

s0m

s0%

93K

B/s

56K

B/s

29

47

6.4.2 Routes between All Test Web Servers

In this Section, I consider the graph of all virtual links between overlay nodes and betweenall overlay nodes and test web servers. I compute routes from every test web server toevery other test web server that is in a different country. For a given pair of web servers, Icompare the fastest route (defined as route with the shortest delay) given no constraintswith the fastest route fulfilling a country constraint. The difference is the delay inflation.Unless otherwise indicated, I average the delay inflation over all pairs of test web serverssuch that the two servers are in two different countries and such that neither violates thecountry constraint.

Rerouting around one country. Figure 6.9 shows the average delay inflation dueto rerouting around one country (from Table 6.1 or 6.3) and the fraction of the fastestroutes that need to be recomputed to avoid that country. For some countries, a fairly highfraction of the fastest routes needs to be replaced by slower routes to avoid that country.However, the average delay inflation is still fairly low (at most 24ms). In contrast tothe fairly low delay inflation in the average case, Table 6.6 shows the three largest delayinflations which are up to 411ms.

Table 6.6: The largest delay inflations due to avoiding one country.

Country DelayRoute constraint inflation

planet1.ku.edu.tr → planetlab2.pop-mg.rnp.br nl 411msplanet1.ku.edu.tr → planetlab2.pop-mg.rnp.br uk 411msplanet1.ku.edu.tr → planetlab-2.sjtu.edu.cn ja 397ms

In terms of hop count, the longest route that the algorithm produces has four vir-tual links which means that it is not that circuitous. Given a pair of web servers,the algorithm computes up to five alternative routes to meet the different country con-straints, for example, five alternative routes between planetlab2.plab.ege.edu.tr and plan-etlab3.netmedia.gist.ac.kr. This addresses the question of the feasibility of link-state ver-sus path-vector routing protocol in Section 3.1 again. It indicates that a path-vectorrouting protocol is still feasible.

Rerouting around two countries. Figure 6.10 shows the same data as Figure 6.9but for different country constraints, namely for avoiding a pair of countries from Table6.3. The average delay inflation due to a constraint of two countries can be almost amagnitude higher than in the case of one country, namely up to 163ms.

In three cases of pairs of web servers, the algorithm computes nine alternative routes tomeet the country constraints. One of the cases is the pair of planetlab2.plab.ege.edu.trand planetlab3.netmedia.gist.ac.kr which already forces five alternative routes when avoid-

48

de us dk nl uk ja br cn fr kr0

5

10

15

20

25

Dela

y inflati

on in m

sAverage delay inflation

when rerouting around a country

de us nl dk uk fr br kr ja cn0

5

10

15

20

25

30

35

40

Perc

enta

ge o

f ro

ute

s

Fraction of fastest routestraversing one country

Figure 6.9: Average delay inflation and fraction of necessary reroutes due to avoidingone country.

49

de/uk de/fr de/us dk/us de/nl de/se de/dk de/ja fr/nl dk/nl0

20

40

60

80

100

120

140

160

180

Dela

y inflati

on in m

sAverage delay inflation

when rerouting around two countries

de/us de/nl nl/us de/uk de/fr dk/us de/ja ca/de fr/us au/de0

10

20

30

40

50

60

Perc

enta

ge o

f ro

ute

s

Fraction of fastest routestraversing two countries

Figure 6.10: Average delay inflation and fraction of necessary reroutes due to a countryconstraint avoiding two countries.

50

ing one country only. This is explained by the direct route traversing many countriessuch that country constraints result in many alternatives, some of them rather circuitous.

In terms of hop count, the longest route has six virtual links, and in terms of delay, theworst rerouting scenario causes 591ms delay inflation.

6.5 Measured Overlay Performance

This Section measures how well the GeoRoute overlay network performs using the routesfrom the previous Section 6.4.1 and putting them into practice. The average bandwidthachieved through the overlay network is 50KB/s, compared to 220KB/s when directlyconnecting to, for example, planetlab2.pop-mg.rnp.br. The bandwidth measurements donot seem to correlate with delay inflation or redirection.

The last columns of Table 6.5 show the results for each route. BW is the average from thenumber of measurements indicated in the Ms column, σBW denotes the standard devia-tion of the measurements. Some PlanetLab machines (such as planetlab5.ie.cuhk.edu.hk)fade in and out during the experiment, making less measurements available for theseroutes.

To make the measurements in Table 6.5, I estimate the TCP throughput by downloadinga 500KB file from the respective web server using wget on the client machine. The clientmachine is connected to the overlay network through the UC Berkeley campus network.The packets go from the client machine via the entry point (ricepl-4.cs.rice.edu) and apossible redirection node to the test web server and vice versa.

0 200 400 600 800 1000File size in KB

0

50

100

150

200

250

300

Bandw

idth

in K

B/s

No OverlayOverlay MaximumOverlay Average

Figure 6.11: Bandwidth without and with overlay network.

To find ways how the overlay performance can be improved, I also measure the band-width as a function of file size (see Figure 6.11) and take tcpdumps on both the client

51

machine and the last overlay node for future analysis. Possible reasons for the dimin-ished performance are overhead from extra headers and packet drop, extra delay, andjitter through processing time. Some of these reasons are inherent to overlay networks,whereas others might be avoided in future. Running click on a dedicated machine might,for example, avoid jitter due to scheduling together with other processes. Implementingsome functionality closer to the Linux Kernel might avoid processing time due to contextswitches into user space.

52

Chapter 7

Discussion

This project has implemented an overlay network allowing geographically constrainedrouting and evaluated its effectiveness. It also has results on the geographic propertiesof the Internet listing, for example, delay inflations when rerouting around a certaincountry. I conclude the report with a discussion of fundamental technical issues, possiblebiases of the analysis and the deployment of such an overlay network.

Fundamental Technical Issues. A fundamental problem when determining the geo-graphic trajectory of a packet is the lack of tools for tracing routes below the IP level.This is an inherent technical difficulty. Finding the exact route of a fiber optic cable,or the non-IP network devices along it, is impossible unless one actually digs up thecable or gets hold of a network map. The relevance of this point depends on whethercommunication policies are actually applied at levels lower than IP and how accuratelyIP routers still mirror the geography of the underlying links.

There is evidence that the IP level does not see countries even though a packet is forced totraverse those countries topologically. For example, a traceroute can leave a landlockedcountry, yet not show any router in one of the neighboring countries.

Biases of the Analysis. Even though the Internet is a global network, its origins are inthe U.S. and so are disproportionally many Internet research facilities. The distribution ofPlanetLab nodes clearly shows this, making measurements in some countries impossibleeven though these countries have a significant amount of Internet users (see Table 6.2).

Furthermore, PlanetLab is an academic network mostly connecting to other academicnetworks, giving this analysis a bias. Commercial networks can have different geographicproperties. In light of the academic principles, academic networks (e.g. CENIC) are alsomore likely than commercial networks to publish network maps and allow researchersopen access to network resources, for example, enabling the use of Internet measurementtools or providing network maps. In particular, traceroutes to most university webservers are complete whereas traceroutes to e.g., www.ibm.com or www.microsoft.com

53

are missing all routers towards the end. Finding a network map for, e.g., Comcast ismore difficult than for most academic networks. These findings indicate that GeoRouteperforms differently when deployed on non-academic networks.

I try to avoid introducing my own biases. For example, I choose the countries to beinvestigated based on objective measures, e.g., the number of Internet users in a country.My knowledge of geography greatly varies. I can, for example, easily locate routers fromtheir names in German networks, but I need extra resources to do this for, say, a routerin Japan. Through extensive search of network maps, I try to compensate for this bias.

GeoRoute as User Application. As mentioned in Section 6.2, improvements are nec-essary for GeoRoute to be suitable as a user application. These include improvementsrequiring software engineering such as achieving better bandwidth (see Section 6.5), pro-viding a better user interface, and authentication of overlay nodes as well as research ongeolocation techniques (see Section 6.1.6).

Besides solving such technical issues, an organization deploying such an overlay networkneeds to have access to computers at several locations around the world. Furthermore,it needs to be trusted by the user, otherwise the overlay network might just defeat itspurpose.

A system addressing similar issues than GeoRoute is Tor [DMS04]. Tor allows anonymityand location hiding through rendez-vous points. A plug-in for Tor could add GeoRoute’srerouting mechanism to avoid certain countries.

Security. Another consideration is that rerouting allows users to abuse Internet re-sources and makes network usage unpredictable and hence network management difficultfor Internet Service Providers.

54

Chapter 8

Acknowledgments

I have to thank Sapan Bhatia and S. Caglar Onur from Princeton University for helpingme a lot by modifying and deploying the vsys scripts on PlanetLab to enable my appli-cation.I would like to acknowledge Ashwin Matthew for introducing me to the topic of geo-graphic properties of the Internet and suggesting it for our course project [MGD].I am grateful for the many encouraging conversations with Professor Carlo Sequin.Thanks also to Professor Vern Paxson for agreeing to be the second reader for this re-search report and for his thoughtful comments.Many thanks to Tugba Colak and John Zhu for careful proofreading and helpful com-ments to improve the presentation.

Special thanks to Professor Randy Howard Katz for his time. He has been an exception-ally great and enthusiastic adviser.

55

Bibliography

[ABKM01] David Andersen, Hari Balakrishnan, Frans Kaashoek, and Robert Morris,Resilient Overlay Networks, SIGOPS Oper. Syst. Rev. 35 (2001), no. 5,131–145.

[Bha] Sapan Bhatia, Vsys: A privilege allocation tool, http://www.cs.princeton.edu/~sapanb/vsys/docs/.

[BPSK96] Hari Balakrishnan, Venkata N. Padmanabhan, Srinivasan Seshan, andRandy H. Katz, A comparison of mechanisms for improving TCP perfor-mance over wireless links, SIGCOMM Comput. Commun. Rev. 26 (1996),256–269.

[Cai] Caida, http://www.caida.org/projects/ark/.

[CLRS09] Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and CliffordStein, Introduction to algorithms, third ed., MIT Press, Cambridge, MA,2009. MR 2572804 (2010j:68001)

[Dij59] E. W. Dijkstra, A note on two problems in connexion with graphs, Numer.Math. 1 (1959), 269–271. MR 0107609 (21 #6334)

[DMS04] Roger Dingledine, Nick Mathewson, and Paul Syverson, Tor: the second-generation onion router, Proceedings of the 13th conference on USENIXSecurity Symposium - Volume 13 (Berkeley, CA, USA), SSYM’04, USENIXAssociation, 2004, pp. 21–21.

[FPK+05] R. Fonseca, G. Porter, R. Katz, S. Shenker, and I. Stoica, IP options arenot an option, Tech. report, EECS Department, University of California,Berkeley, 2005.

[Jac88] V. Jacobson, Congestion avoidance and control, SIGCOMM Comput. Com-mun. Rev. 18 (1988), 314–329.

[KBJK+06] Ethan Katz-Bassett, John P. John, Arvind Krishnamurthy, David Wether-all, Thomas Anderson, and Yatin Chawathe, Towards IP geolocation usingdelay and topology measurements, Proceedings of the 6th ACM SIGCOMM

56

conference on Internet measurement (New York, NY, USA), IMC ’06, ACM,2006, pp. 71–84.

[KMC+00] Eddie Kohler, Robert Morris, Benjie Chen, John Jannotti, and M. FransKaashoek, The click modular router, ACM Trans. Comput. Syst. 18 (2000),263–297.

[LPS06] Jonathan Ledlie, Peter Pietzuch, and Margo Seltzer, Stable and accuratenetwork coordinates, International Conference on Distributed ComputingSystems (2006), 74.

[MGD] Ashwin Jacob Mathew, Matthias Goerner, and Ozan Demirlioglu, Geo-graphically constrained routing, Course Project, CS268, Fall 2008, Uni-versity of California, Berkeley, http://math.berkeley.edu/~matthias/cs268_paper.pdf.

[PACR03] Larry Peterson, Tom Anderson, David Culler, and Timothy Roscoe, Ablueprint for introducing disruptive technology into the internet, SIGCOMMComput. Commun. Rev. 33 (2003), 59–64.

[Pax97a] Vern Paxson, Automated packet trace analysis of tcp implementations, SIG-COMM Comput. Commun. Rev. 27 (1997), 167–179.

[Pax97b] , End-to-end routing behavior in the internet, IEEE/ACM Trans.Netw. 5 (1997), 601–615.

[Pax99] Vern Paxson, End-to-end internet packet dynamics, IEEE/ACM Transac-tions on Networking, 1999, pp. 277–292.

[PFTK98] Jitendra Padhye, Victor Firoiu, Don Towsley, and Jim Kurose, Modelingtcp throughput: a simple model and its empirical validation, SIGCOMMComput. Commun. Rev. 28 (1998), 303–314.

[PS01] Venkata N. Padmanabhan and Lakshminarayanan Subramanian, An inves-tigation of geographic mapping techniques for internet hosts, SIGCOMMComput. Commun. Rev. 31 (2001), 173–185.

[SAA+99] Stefan Savage, Thomas Anderson, Amit Aggarwal, David Becker, Neal Card-well, Andy Collins, Eric Hoffman, John Snell, Amin Vahdat, Geoff Voelker,and John Zahorjan, Detour: Informed internet routing and transport, IEEEMicro 19 (1999), 50–59.

[SCH+99] Stefan Savage, Andy Collins, Eric Hoffman, John Snell, and Thomas Ander-son, The end-to-end effects of internet path selection, SIGCOMM Comput.Commun. Rev. 29 (1999), 289–299.

[SPK02] Lakshminarayanan Subramanian, Venkata N. Padmanabhan, and Randy H.Katz, Geographic properties of internet routing, Proceedings of the General

57

Track of the annual conference on USENIX Annual Technical Conference(Berkeley, CA, USA), USENIX Association, 2002, pp. 243–259.

[SWA] Neil Spring, David Wetherall, and Tom Anderson, http://www.scriptroute.org/, Version 0.2.0-pre2.

[WBF+10] Yong Wang, Daniel Burgener, Marcel Flores, Aleksandar Kuzmanovic, andCheng Huang, Towards street-level IP geolocation: A .com approach, OSDI2010, 2010.

58

Appendix:

Man Pages of

new Click Elements

ETHERPRINT(n) ETHERPRINT(n)

NAMEEtherPrint − Click element; pretty-prints Ethernet Headers

SYNOPSISEtherPrint

Ports: 1 input, 1 output

DESCRIPTIONExpects EthernetHeader as input. Prints out Header in a human-readable format.

Processing: agnosticPackage: local (core)

SEE ALSOEtherEncap(n), EnsureEther(n), IPPrint(n)

Click 6/Oct/2010 1

62

FORWHEADER(n) FORWHEADER(n)

NAMEForwHeader − Click element; encapsulates IP packets in Forwarding Header with the hop list being lookedup in a routing table maintained byForwHeader

SYNOPSISForwHeader(HOPS, METHOD, OPTIONS)

Ports: 1 input, 1 outputProcessing: pushPackage: local (core)

DESCRIPTIONencapsulates each packet in the Forwarding Header. The METHOD and OPTIONS 8-bit fields in the For-warding Header will be set with the values of METHOD and OPTIONS. The hop list written into the For-warding Header will be looked up in a routing table maintained byForwHeader. If no entry for the desti-nation IP is found, the packet will be enqueued until an entry for the destination IP is added.

HOPS will be used to initialize the routing table with one row. The first IP is the destination, the remainingIPs the hop list. This is for testing purposes, the preferred way to set up the routing table is through the han-dlers.

EXAMPLESForwHeader(192.168.151.1 192.168.152.1 192.168.175.1,1,10)

will initialize the routing table so that IP packets with destination 192.168.151.1 are encapsulated in a For-warding Header with hop list 192.168.152.1 192.168.175.1.

ELEMENT HANDLERSset_route (write-only)

Sets the hop list for a destination in the routing table. Expects a list of IP addresses, the first one is thedestination, the remaining ones the hop list.

clear (write-only)Clears the routing table

dump_routes (read-only)Lists the routing table. Each row first lists the destination IP, then the hop list.

queued (read-only)Lists the destination IPs of all IP packets which are queued because their destination IP was not in therouting table.

SEE ALSOForwHeaderPrint(n), NextHop(n), RevRoute(n)

Click 6/Oct/2010 1

63

FORWHEADERPRINT(n) FORWHEADERPRINT(n)

NAMEForwHeaderPrint − Click element; pretty-prints forwarding headers

SYNOPSISForwHeaderPrint(ForwHeaderPrint([LABEL])

Ports: 1 input, 1 outputProcessing: agnosticPackage: local (core)

DESCRIPTIONExpects forwarding headers as input. Prints out Forwarding Header in a human-readable format, precededby the LABEL text.

SEE ALSONextHop(n), RevRoute(n), ForwHeader(n)

Click 6/Oct/2010 1

64

NEXTHOP(n) NEXTHOP(n)

NAMENextHop − Click element; increases next-hop index in Forwarding Header and sets destination annotationto the next hop

SYNOPSISNextHop

Ports: 1 input, 2 outputsProcessing: PushPackage: local (core)

DESCRIPTIONIf the next-hop index is not pointing at the last hop, the index will be increased by one and the destinationannotation set to the next hop. The packet will be send to output 0. If the next-hop index is pointing at thelast hop, the packet will be send unmodified to output 1. Input which does not have a valid ForwardingHeader is discarded.

SEE ALSORevRoute(n), ForwHeader(n), ForwHeaderPrint(n)

Click 6/Oct/2010 1

65

REVROUTE(n) REVROUTE(n)

NAMERevRoute − Click element; saves Forwarding Headers for flows, applies the Forwarding Header withreversed route to each reverse flow

SYNOPSISRevRoute

Ports: 2 inputs, 2 outputsProcessing: PushPackage: local (core)

DESCRIPTIONExpects ForwardingHeaders on port 0 and IP packets on port 1. For each packet on port 0, the ForwardingHeader is stripped and the packet emitted on port 0 and if the encapsulated payload was an IP packet amapping (DADDR,DPORT,SADDR,SPORT,PROT O) => Forw-Header(n)(ReverseRoute,ReverseMethod,Options) is installed (ReverseMethod has the 0-bit inverted). IPPackets on port 1 are encapsulated in Forwarding Headers according to the above mapping.

SEE ALSONextHop(n), ForwHeader(n), ForwHeaderPrint(n), IPRewriter(n)

Click 6/Oct/2010 1

66

STRIPFORWHEADER(n) STRIPFORWHEADER(n)

NAMEStripForwHeader − Click element; strips outermost Forwarding Header

SYNOPSISStripForwHeader

Ports: 1 input, 1 outputProcessing: agnosticPackage: local (core)

DESCRIPTIONStrips the outermost Forwarding Header.

Note thatStripForwHeader assumes that the Forwarding Header starts at the beginning of the packet,hence you need to use StripIPHeader and Strip(8) before processing packets iwthStripForwHeader. Alsonote thatStripForwHeader will set the network and transport header annotation, assuming that an IPheader is following.

SEE ALSORevRoute(n), NextHop(n), ForwHeader(n), ForwHeaderPrint(n)

Click 6/Oct/2010 1

67

VSYSTUN(n) VSYSTUN(n)

NAMEVsysTun − Click element; Uses planetlab’s /vsys/fd_tuntap.control to allocate a tun device. Uses the speci-fied file to bring up the device on a planetlab node.

SYNOPSISVsysTun(ADDR/MASK, VSYS_FILE [, keywords HEADROOM, MTU, IGNORE_QUEUE_OVER-FLOWS])

Ports: at most 1 input, 1-2 outputsDrivers: userlevelPackage: local (core)

DESCRIPTIONReads IP packets from and writes IP packets to a tun device reservered through vsys.

Opens a UNIX socket to /vsys/fd_tuntap.control and calls recvmsg.vsys will create a tun device and openit and return to click the network device name and a file descriptor. click will send the device name, the IPaddress and prefix length and "snat=1" to VSYS_FILE.in and read VSYS_FILE.out which is expected toreturn nothing. This will bring up the network device and set its IP and also enable source NAT .

To use it on planetlab, you need to have reserved your own private IP and vsys access to vif_up and fd_tun-tap. After initialization,VsysTun behaves like KernelTun(n).

EXAMPLESVsysTun(10.6.0.1/24, VSYS_FILE "/vsys/vif_up")

will set up a tun device with IP 10.6.0.1. An IP packet send to the device should have source IP 10.6.0.2.

Ke yword arguments are

HEADROOMInteger. The number of bytes left empty before the packet data to leave room for additionalencapsulation headers. Default is 28.

MTU Integer. The interface’s MTU, not including any link headers.KernelTun(n) will refuse to sendpackets larger than the MTU. Default is 1500; not all operating systems allow MTU to be set.

IGNORE_QUEUE_OVERFLOWSBoolean. Iftrue, don’t print more than one error message when there are queue overflows errorwhen sending/receiving packets to/from the tun device (e.g. there was an ENOBUFS error).Default is false.

NOTESThis is specfic for planetlab.

SEE ALSOKernelTun(n)

Click 6/Oct/2010 1

68

Appendix:

List of

Manually Located Routers

Router Name Country

planet01.HHI.FRAUNHOFER.DE Germanyplanet02.HHI.FRAUNHOFER.DE GermanyMSK-M9-AR1.E-ARENA.Ru RussiaGATEWAY-MAN.GIGAPOP.GEN.TX.US U.S.LINK2RICE.GIGAPOP.GEN.TX.US U.S.RICE.GIGAPOP.GEN.TX.US U.S.RICE-MAN.GIGAPOP.GEN.TX.US U.S.clgr1rtr1.canarie.ca Canadatoro1rtr1.canarie.ca Canadaplanetlab-2.elisa.cpsc.ucalgary.ca Canadanode1.dsl.ucalgary.ca Canadapc185.hidden.ucalgary.ca Canadapc187.hidden.ucalgary.ca Canadacsc1cled050.bb.uvic.ca Canadadrc2cled050.bb.uvic.ca Canadaemc1cled050.bb.uvic.ca Canadaemc1corb115.bb.uvic.ca Canadapl1.planetlab.uvic.ca Canadara2so-ge3-1-71.cg.bigpipeinc.com Canadate3-2.ccr01.slc01.atlas.cogentco.com U.S.te4-2.ccr01.slc01.atlas.cogentco.com U.S.te3-4.ccr01.slc02.atlas.cogentco.com U.S.te4-4.ccr01.slc02.atlas.cogentco.com U.S.r-imag-mbnt.grenet.fr Franceplanetlab-1.imag.fr Francepeeramide.irisa.fr Franceirisa-rennes-gi8-7-rennes-rtr-021.noc.renater.fr Franceman-tigre-te1-4-grenoble-rtr-021.noc.renater.fr Francete0-0-0-0-paris2-rtr-001.noc.renater.fr Francete1-4-lyon2-rtr-021.noc.renater.fr Francete4-2-caen-rtr-021.noc.renater.f Francete4-2-rouen-rtr-021.noc.renater.fr Francec7604.ust.hk Chinaeek084.ust.hk Chinaeng-gw1.ust.hk Chinainternet-gw1.ust.hk Chinaisg2kcr1.ust.hk Chinaisg2kcr2.ust.hk Chinaing-gw2.cdc.unict.it Italygschembra4.diit.unict.it Italyc76.jaist.ac.jp Japanct65a-ct65b.jaist.ac.jp Japanct65b-ct65a.jaist.ac.jp Japanplanet1.jaist.ac.jp Japanplanetlab-02.kusa.ac.jp JapanNYC-gate1-G1-0-107.sinet.ad.jp U.S.nyc-gate1-RM-GE-7-2-0-207.sinet.ad.jp U.S.nyc-gate1-RM-P-7-0-0-11.sinet.ad.jp U.S.tokyo1-dc-RM-AE-0-11.sinet.ad.jp Japantokyo1-dc-RM-P-2-3-0-11.sinet.ad.jp Japantokyo2-dc-RM-AE-0-11.sinet.ad.jp Japantokyo2-dc-RM-GE-5-0-0-11.sinet.ad.jp Japan

72

tokyo2-dc-RM-P-6-2-0-11.sinet.ad.jp Japanve-3724.cisco2.dojima.wide.ad.jp Japanve-61.cisco2.dojima.wide.ad.jp Japanve-7.cisco2.dojima.wide.ad.jp Japanpo-1.cisco2.komatsu.wide.ad.jp Japanve-3724.cisco2.komatsu.wide.ad.jp Japanve-62.cisco2.komatsu.wide.ad.jp Japanve-0-0-v3727.hitachi1.kurashiki.wide.ad.jp Japanve-7.hitachi2.nara.wide.ad.jp Japanve-61.cisco2.notemachi.wide.ad.jp Japanve-62.cisco2.notemachi.wide.ad.jp Japanvlan53-cisco2.notemachi.wide.ad.jp Japanjuniper-itc3.naist.jp Japanplanetlab-02.naist.jp Japanwnoc-hitachi2.naist.jp JapanORAN-UVicA.VICTX.BC.net CanadaUVicA-ORAN.VICTX.BC.net CanadaUVicB-Policy1.VICTX.BC.net Canadaae-7-7.ebr1.Atlanta2.Level3.net U.S.ae-63-60.ebr3.Atlanta2.Level3.net U.S.ae-73-70.ebr3.Atlanta2.Level3.net U.S.ge-8-35.car1.Dallas1.Level3.net U.S.ae-71-70.ebr1.Dallas1.Level3.net U.S.ae-62-62.csw1.Frankfurt1.Level3.net Germanyae-72-72.csw2.Frankfurt1.Level3.net Germanyae-82-82.csw3.Frankfurt1.Level3.net Germanyae-92-92.csw4.Frankfurt1.Level3.net Germanyae-41-41.ebr2.Frankfurt1.Level3.net Germanyae-42-42.ebr2.Frankfurt1.Level3.net Germanyae-44-44.ebr2.Frankfurt1.Level3.net Germanyae-1-69.edge4.Frankfurt1.Level3.net Germanyae-2-79.edge4.Frankfurt1.Level3.net Germanyae-3-89.edge4.Frankfurt1.Level3.net Germanyae-4-99.edge4.Frankfurt1.Level3.net Germanyae-1-13.bar1.Houston1.Level3.net U.S.ae-2-5.bar2.Houston1.Level3.net U.S.ae-5-5.car1.Houston1.Level3.net U.S.ge-3-6.car2.Houston1.Level3.net U.S.ge-5-1-114.hsa2.Houston1.Level3.net U.S.ge-5-2.hsa2.Houston1.Level3.net U.S.vlan79.csw2.NewYork1.Level3.net U.S.vlan99.csw4.NewYork1.Level3.net U.S.ae-61-61.ebr1.NewYork1.Level3.net U.S.ae-71-71.ebr1.NewYork1.Level3.net U.S.ae-81-81.ebr1.NewYork1.Level3.net U.S.xe-10-3-0.edge1.NewYork1.Level3.net U.S.ae-4-4.ebr1.NewYork2.Level3.net U.S.ae-71-71.csw2.Washington1.Level3.net U.S.ae-81-81.csw3.Washington1.Level3.net U.S.ae-2-2.ebr1.Washington1.Level3.net U.S.ae-3-3.ebr2.Washington1.Level3.net U.S.ae-72-72.ebr2.Washington1.Level3.net U.S.ae-82-82.ebr2.Washington1.Level3.net U.S.ae-92-92.ebr2.Washington1.Level3.net U.S.

73

xe-0-0-0.r0.ams.asgc.net Netherlandsasgc-i2.r0.chi.asgc.net U.S.asgc-startap.r0.chi.asgc.net U.S.canet4w-asgc.r0.chi.asgc.net U.S.i2-asgc.r0.chi.asgc.net U.S.so-2-0-0.r0.hk.asgc.net Chinaso-4-1-0.r1.tpe.asgc.net Chinaxe-3-0-0.r1.tpe.asgc.net TaiwanCER-0002.gw4.hkg3.asianetcom.net ChinaNEW-0021.gw4.hkg3.asianetcom.net Chinage-0-1-0-0.gw4.hkg3.asianetcom.net Chinage-1-1-0-0.gw4.hkg3.asianetcom.net Chinagi9-0-0.cr2.nrt1.asianetcom.net Japanpo2-1-1.cr2.nrt1.asianetcom.net Japancr1-bb3901.vantx1.bc.net Canadacr1-bb3901.victx1.bc.net Canadahht-21-p3-0-0.cernet.net Chinapo1-40G.ar6.AMS2.gblx.net Netherlandstengigabitethernet1-4.ar5.NYC1.gblx.net U.S.te-6-4-10G.ar3.SJC2.gblx.net U.S.so-6-3-0.rt1.vie.at.geant2.net Austriaso-2-0-0.rt1.sof.bg.geant2.net Belgiumulakbim-lb2-gw.rt1.sof.bg.geant2.net Belgiumso-2-1-0.rt1.pra.cz.geant2.net CzechRepublicnordunet.rt2.cop.dk.geant2.net Denmarknordunet-gw.rt2.cop.dk.geant2.net Denmarkso-6-0-0.rt2.cop.dk.geant2.net Denmarkclara.rt1.mad.es.geant2.net Spainclara-gw.rt1.mad.es.geant2.net Spainso-2-0-0.rt1.mad.es.geant2.net Spainrenater-gw.rt1.par.fr.geant2.net Franceso-2-1-0.rt1.par.fr.geant2.net Franceso-3-0-0.rt1.par.fr.geant2.net Franceso-4-1-0.rt1.par.fr.geant2.net Franceso-2-0-0.rt1.bud.hu.geant2.net Hungaryso-3-0-0.rt1.bud.hu.geant2.net Hungaryas0.rt1.mil.it.geant2.net Italygarr.rt1.mil.it.geant2.net Italygarr-gw.rt1.mil.it.geant2.net Italyas1.rt1.ams.nl.geant2.net Netherlandscanarie.rt1.ams.nl.geant2.net Netherlandscanarie-gw.rt1.ams.nl.geant2.net Netherlandsso-4-0-0.rt1.ams.nl.geant2.net Netherlandsso-0-0-0.rt1.buc.ro.geant2.net Netherlandsulakbim-lb1-gw.rt1.buc.ro.geant2.net Romaniajanet-gw.rt1.lon.uk.geant2.net United Kingdomso-3-0-0.rt1.lon.uk.geant2.net United Kingdomso-4-0-0.rt1.lon.uk.geant2.net United Kingdom10gigabitethernet1-2.core1.atl1.he.net U.S.10gigabitethernet3-3.core1.fra1.he.net Germany10gigabitethernet3-2.core1.mia1.he.net Miami10gigabitethernet1-2.core1.par1.he.net Francegige-g2-17.core1.sto1.he.ne Swedenapanjp-RGE.hkix.net China

74

asgcnet-RGE.hkix.net Chinajucc1-RGE.hkix.net Chinakreonet-RGE.hkix.net Chinai00ams-015-ten1-3.bb.ip-plus.net Netherlandsso-5-0-0.lond-sbr1.ja.net United Kingdomso-6-0-0.read-sbr1.ja.net United Kingdomessex.site.ja.net United Kingdomchltcrs-gw-to-internet2-atl.ncren.net U.S.chltcrs-gw-to-rtpcrs-gw.ncren.net U.S.dep7600-gw2-to-nlr-atl.ncren.net U.S.dep7600-gw2-to-rtp7600-gw.ncren.net U.S.internet2-atl-to-chltcrs-gw.ncren.net U.S.nlr-atl-to-dep7600-gw2.ncren.net U.S.nlr-dc-to-dep7600-gw2.ncren.net U.S.rlgh7600-gw-to-dep7600-gw2.ncren.net U.S.rlgh7600-gw-to-rtp-crs-gw.ncren.net U.S.rtp-crs-gw-to-rtp7600-gw-tg4-2.ncren.net U.S.rtp7600-gw-to-duke7600-gw.ncren.net U.S.rtpcrs-gw-to-chltcrs-gw.ncren.net U.S.rtpcrs-gw-to-internet2-wash.ncren.net U.S.tge-0-2-0-2.atla.layer3.nlr.net U.S.atla-hous-70.layer3.nlr.net U.S.tge-0-1-0-1.chic.layer3.nlr.net U.S.chic-denv-36.layer3.nlr.net U.S.chic-newy-100.layer3.nlr.net U.S.denv-chic-36.layer3.nlr.net U.S.denv-seat-58.layer3.nlr.net U.S.hous-atla-70.layer3.nlr.net U.S.hous-losa-87.layer3.nlr.net U.S.losa-hous-87.layer3.nlr.net U.S.losa-seat-49.layer3.nlr.net U.S.tge-0-5-0-4.211.newy.layer3.nlr.net U.S.tge-0-5-0-4.89.newy.layer3.nlr.net U.S.newy-wash-98.layer3.nlr.net U.S.seat-denv-58.layer3.nlr.net U.S.wash-atla-64.layer3.nlr.net U.S.nl-sar.nordu.net Netherlandsus-man.nordu.net U.S.syr-7600-buf-7600.nysernet.net U.S.syr-7600-nyc-7600.nysernet.net U.S.abilene-1-lo-jmb-702.lsanca.pacificwave.net U.S.nlr-1-lo-jmb-702.lsanca.pacificwave.net U.S.sinet-1-is-jmb-776.lsanca.pacificwave.net U.S.transpac-1-lo-jmb-702.lsanca.pacificwave.net U.S.abilene-1-lo-jmb-706.sttlwa.pacificwave.net U.S.kreonet-1-lo-jmb-706.sttlwa.pacificwave.net U.S.nlr-1-lo-jmb-706.sttlwa.pacificwave.net U.S.Gi4-0-0.ASBNVA1603WIG02.paetec.net U.S.po-6-0-0.PHLAPAFGW26CR02.paetec.net U.S.ge-2-3-0.core02.phlapa02.paetec.net U.S.gi-1-0-311.hse1.phlapa02.paetec.net U.S.miami-saopaulo.core.redclara.net Brazilrnp-br-spau.core.redclara.net Brazilsaopaulo-miami.core.redclara.net Brazil

75

rnp-br-spau.peer.redclara.net Brazilae1-4.RT.TC2.AMS.NL.retn.net Netherlandsxe902-200.RT.TC1.STO.SE.retn.net Swedeng2-1.br1.ams.terremark.net NetherlandsTe2-0-1.br1.lon.terremark.net United KingdomTe2-0-0.br2.lon.terremark.net United Kingdomt0-0-0-4.br1.mia.terremark.net U.S.t0-0-0-1.br2.mia.terremark.net U.S.t0-0-0-5.br2.mia.terremark.net U.S.t9-1.gw1.mia.terremark.net U.S.Te2-0-0.br1.nyc.terremark.net U.S.Te2-0-1.br2.nyc.terremark.net U.S.losa-tokyo-tp2.transpac2.net Japantokyo-losa-tp2.transpac2.net Japanmsk-1-gw.runnet.ru Russiaa1g-cn6-p2p.gw.kth.se Swedenssvl-a1g-p2p.gw.kth.se Swedengw-vlan290.ssvl.kth.se Swedenplanetlab-1.ssvl.kth.se Swedena1sth-kth.sunet.se Swedenc1sth.sunet.se Swedenc1sth-ae0-1001.sunet.se Swedenplebt2.essex.ac.uk United Kingdomdc-rt1-me-core.net.ic.ac.uk United Kingdomplanetlab-1.ic.ac.uk United Kingdomnw-london.lmn.net.uk United KingdomTYO3-IX1-XGE-3-1.sinet.ad.jp Japancsc0g02brb.net.cuhk.edu.hk Chinagige-g2-17.core1.sto1.he.net Swedenrouter980-3.ie.cuhk.edu.hk Chinarouter993-3.ie.cuhk.edu.hk Chinacsc0g04brb.net.cuhk.edu.hk Chinaplanetlab5.ie.cuhk.edu.hk ChinaLMN-LMN2.site.ja.net United Kingdomte3-3-10G.ar3.SJC2.gblx.net U.S.te6-4-10G.ar4.LAX1.gblx.net U.S.

76