Server-assisted Latency Management for Wide-area Distributed Systems

Wonho Kim1, KyoungSoo Park2, and Vivek S. Pai1

1Department of Computer Science, Princeton University2Department of Electrical Engineering, KAIST

AbstractRecently many Internet services employ wide-area

platforms to improve the end-user experience in theWAN. To maintain close control over their remote nodes,the wide-area systems require low-latency disseminationof new updates for system configurations, customer re-quirements, and task lists at runtime. However, we ob-serve that existing data transfer systems focus on re-source efficiency for open client populations, rather thanfocusing on completion latency for a known set of nodes.In examining this problem, we find that optimizing forlatency produces strategies radically different from exist-ing systems, and can dramatically reduce latency acrossa wide range of scenarios.

This paper presents a latency-sensitive file transfersystem, Lsync that can be used as synchronization build-ing block for wide-area systems where latency matters.Lsync performs novel node selection, scheduling, andadaptive policy switching that dynamically chooses thebest strategy using information available at runtime. Ourevaluation results from a PlanetLab deployment showthat Lsync outperforms a wide variety of data transfersystems and achieves significantly higher synchroniza-tion ratio even under frequent file updates.

1 IntroductionLow-latency data dissemination is essential for coordi-nating remote nodes in wide-area distributed systems.The systems need to disseminate new data to remotenodes with minimal latency when distributing new con-figurations, when coordinating task lists among multipleendpoints, or when optimizing system performance un-der dynamically changing network conditions. All ofthe scenarios involve latency-sensitive synchronization,where the enforced synchronization barrier can limitoverall system performance and responsiveness.

The effects of synchronization latency will becomeincreasingly more important as more Internet services

Overlay Multicast CDN/P2P Epidemic Routing

Fast nodes Slow nodes

Figure 1: Slow Nodes in Overlay – Peering strategies in scal-able one-to-many data transfer systems are not favorable toslow nodes.

leverage distributed platforms to accelerate their appli-cations in the WAN. Recent trends show that many com-panies employ distributed caching services [4, 18] andWAN optimization appliances [24], or deploy their ap-plications at the Internet edges close to the end users [12]to improve the user experience. These platforms shouldhandle frequently-changing customer requirements andadapt to dynamic network conditions at runtime. For in-stance, Akamai [2] reports that its management server re-ceives five configuration updates per minute that need tobe propagated to remote CDN nodes immediately [26].If these systems face long synchronization delays, possi-bilities include service disruption, inconsistent behaviorat different replicas visible to end users, or increased ap-plication complexity to try to mask such effects.

The latency is measured as the total completion timeof file transfer to all target remote nodes. In wide-areasystems, it is usual that a number of nodes experiencenetwork performance problems and/or lag far behind up-to-date synchronization state at any given time. We ob-serve these slow nodes typically dominate the comple-tion time, which means that managing their tail latencyis crucial for latency-sensitive synchronization.

Although numerous systems have been proposed forscalable one-to-many data transfers [8, 10, 16, 17, 19,21], they largely ignore the latency issue because re-

1

source efficiency is typically their primary concern forserving an open client population. In an open client pop-ulation, there is no upper bound on the number of clients,so the systems aim to maximize average performance oraggregate throughput in the system.

As a result, those systems are not favorable to slownodes (Figure 1). For instance, many overlay multi-cast systems attempt to place well-provisioned fast nodesclose to the root of the multicast tree while pushing slownodes down the tree. Likewise, the peering strategiesused in CDN/P2P systems make slow nodes have littlechance to download from fast nodes. The random gossip-ing in epidemic routing protocol helps slow nodes peerwith fast nodes, but only for a short-term period. Thesepeering strategies not only produce long completion timebut make the systems highly vulnerable to changes inslow node’s network condition. Despite these disadvan-tages, it is common that existing services rely on one ofthe data transfer schemes for coordinating their remotenodes.

In this paper, we explore general file transfer poli-cies for latency-sensitive synchronization with the goalof minimizing completion time in a closed client popu-lation. This completion time metric drives us to examinenew optimization opportunities that may not be advis-able for systems with open client populations. In partic-ular, we aggressively use spare bandwidth in the originserver to assist nodes that experience transient/persistentperformance problems in the overlay mesh at runtime.The server allocates its bandwidth in a manner favorableto the slow nodes while synchronizing the other nodesthrough existing overlay mesh. This server-assistedsynchronization reduces the tail latency in slow nodeswithout sacrificing scalable data transfers in the overlaymesh, which drastically improves the completion timeand achieves stable file transfer.

For evaluation of our policies, we develop Lsync, alow-latency file transfer system for wide-area distributedsystems. Lsync can be used as synchronization buildingblock for wide-area distributed systems where latencymatters. Lsync continuously disseminates files in thebackground, monitoring file changes and choosing thebest strategy based on information available at runtime.Lsync is designed to be easily pluggable into existingsystems. Users can specify a local directory to be syn-chronized across remote machines, and give Lsync theinformation about target remote nodes. Other systemscan use Lsync by simply dropping files into the directorymonitored by Lsync when the files need latency-sensitivesynchronization.

We evaluate Lsync against a wide variety of data trans-fer systems, including a commercial CDN. Our evalua-tion results from a PlanetLab [1] deployment show thatLsync can drastically reduce latency compared to exist-

Upload

capacity C

Overlay mesh

FilesServer

bi

e2e

ni

Remote

Nodes N

Clients ...

Figure 2: Synchronization Environment – the server has filesto transfer to remote nodes with low completion latency. Theremote nodes construct an overlay mesh for providing servicesto external clients.

ing file transfer systems, often needing only a few sec-onds for synchronizing hundreds of nodes. When wegenerate a 1-hour workload similar to the frequent con-figuration updates in Akamai CDN, Lsync achieves sig-nificantly higher synchronization ratio than the alterna-tives throughout the experiment.

The rest of this paper is organized into following sec-tions. In Section 2, we describe the model and assump-tions for our problem. Based on the model, we discusshow to allocate server’s bandwidth to slow nodes in Sec-tion 3. In Section 4, we describe how to divide nodesbetween server and overlay mesh in a way to minimizethe total latency. We describe the implementation in Sec-tion 5 and evaluate it on PlanetLab in Section 6.

2 Synchronization Environment

In this section, we describe the basic operational modelused for Lsync, along with the assumptions we makeabout its usage.

Operational model We assume one dedicated serverthat coordinates a set of remote nodes, N, geographicallydistributed in the WAN, as shown in Figure 2. The man-agement server has an uplink capacity of C, and can com-municate with each remote node ni with bandwidth bi

e2e.The C value is configurable, and represents the maximumbandwidth that can be used for remote synchronization.The bi

e2e values are updated using a history-based adap-tation technique, described in Section 4.5, to adjust tovariations in available bandwidth. The server knows theamount of new data by detecting file changes in the back-ground as described in Section 5.

Many distributed systems construct an overlay meshamong their remote nodes to use scalable data transfersystems. If an overlay mesh is available and certain con-ditions are met, Lsync leverages it for fast synchroniza-tion. To make Lsync easily pluggable into existing sys-tems, we do not require modification of a given over-lay’s behavior. Instead, Lsync characterizes the overlay

2

mesh using a black-box approach to estimate its startuplatency. Based on the estimation, Lsync determines howto adjust workload across the server and the overlay.Target environments This model is appropriate for ourtarget environments, where a large number of nodes aregeographically distributed without heavy concentrationsin any single datacenter.1 Our target environments alsorequire that we exclude IP multicast, due to the problemsstemming from lack of widespread deployment and co-ordination across different ASes.

Examples of current systems where our intended ap-proach may be applicable include distributed cachingservices [4, 18], WAN optimization appliances [24], dis-tributed computation systems [25], edge computing plat-forms [12], wide-area testbeds (PlanetLab, OneLab), andmanaged P2P systems [20]. In addition, with most ma-jor switch/router vendors announcing support for pro-grammable blades for their next-generation networks,virtually every large network will soon be capable of be-ing a distributed service platform.Target remote nodes In Lsync, users can specify targetremote nodes in various ways. For global updates, userswill configure Lsync to optimize the latency for updat-ing all remote nodes. For partial updates, it is possibleto select a subset of specific remote nodes to synchro-nize. Users can also specify a certain fraction of nodesthat need to be synchronized fast for satisfying certainconsistency models or incremental rollouts of new con-figurations in the system. We name the parameter targetsynchronization ratio denoted by r, for the rest of the pa-per. This enables Lsync to use intelligent node selection,which is particularly effective for disseminating frequentupdates.

3 Server Bandwidth AllocationLsync’s file transfer policy combines multiple factorsthat contribute to the overall time reduction. We quantifythe benefits separately, so we describe steps separatelyin the paper. After adjusting workload across server andoverlay, Lsync exploits the server’s spare bandwidth tospeed up synchronizing the overlay mesh. In this sec-tion, we begin by examining the effects of the server’sbandwidth allocation policies on completion latency.

Figure 3 shows an example of the timeline when theserver transfers files to a set of target remote nodes,Ntarget ⊂ N. The server detects a new file fnew at timet0. Each horizontal bar corresponds to a remote node,ni ∈ Ntarget , that has variable-sized unsynchronized dataf iprev remaining from previous transfers. The areas of

f iprev and fnew represent the sizes of the files, | f i

prev| and

1The intra-datacenter bandwidths are sufficient to make the syn-chronization latency less of an issue in that environment.

0t cmpT

i

pendt

i

prevfnewf

in

1+in

2+in

1−in

2−in

i

eeb 2

Ntarget

t

Cbi

ee <∑ 2

Figure 3: Timeline of Synchronization – The completiontime Tcmp is determined by the node with the latest finish timeamong the target nodes Ntarget .

| fnew| , respectively. The height of the bar, bie2e, is the

end-to-end bandwidth from the server to ni that starts itstransfer after a pending time t i

pend . At any given timet, the sum of the heights of the bars should not exceedthe server’s upload capacity C in order to avoid self-congestion and associated problems stemming from theserver overload [3].

The completion time, Tcmp, is calculated as

Tcmp = t0 +maxi

(t ipend +

| f iprev|+ | fnew|

bie2e

)(1)

for ni ∈ Ntarget . There are two variables that the servercan control transparently to the remote nodes. The servercan determine t i

pend that ni should wait before starting itstransfer. The server can also select nodes for Ntarget if atarget synchronization ratio is given. From the perspec-tive of the server, controlling these two variables corre-sponds to node scheduling and node selection policiesin the server, respectively. The basic intuition behindthe policies is that we could reduce the latency by giv-ing low t i

pend to slow nodes and carefully selecting nodesfor Ntarget . In the following sections, we compare dif-ferent policies using real measurements on PlanetLab toquantify their effects on latency.

3.1 Node SchedulingWe begin by examining how we schedule transfers whenthe number of transfers exceeds the outbound capacity ofthe server. This is the problem of minimizing makespan,which is NP-hard [7]. We compare two basic schedulingheuristics, Fast First and Slow First. The Fast First issimilar to Shortest Remaining Processing Time (SRPT)scheduling in that the server’s resource is allocated tothe node who has the shortest expected completion time.SRPT is known to be optimal for minimizing mean re-sponse time [6]. The Slow First is the opposite of theFast First policy, and selects the nodes with the longestexpected completion time when the server has availablebandwidth.

3

0

100

200

300

400

500

0 0.2 0.4 0.6 0.8 1

Co

mp

leti

on

Tim

e (

s)

Fraction of Syncd Nodes

Fast First

Slow First

Pruned Slow First

Figure 4: Node Scheduling and Node Selection – PrunedSlow First captures both the initial speed advantage of FastFirst, as well as the total overall advantage of Slow First.

We compare the two schedulers on PlanetLab nodes.We implement the policies in a dedicated server that has100 Mbps upload capacity. Then we generate two files –fnew (1 MB), and fprev (10 MB) that has been in the pro-cess of synchronization on the nodes, from 1% to 99%complete. We measure the time to synchronize all livePlanetLab nodes (559 nodes at the time of the experi-ments) with either of the two scheduling modes enabled.

The result of the measurement is shown in Figure 4.Fast First synchronizes most nodes faster than Slow First,but at high target ratios, Fast First performs much worse.The reason for the difference is somewhat obvious: nearthe end of the transfers in Fast First, only slow nodesremain, and the server’s uplink becomes underutilized.This underutilization occurs for the final 175 seconds inFast First, but only for 0.5 seconds in Slow First. We alsoevaluated Random scheduling, which selects a randomnode to allocate available bandwidth. From 10 repeatedexperiments, we found that Random scheduling yieldscompletion times between Fast First and Slow First, butgenerally closer to Slow First for all target ratios.

This result implies that slow nodes dominate the com-pletion time in the WAN and that the Slow First schedul-ing can mask their effects on latency. Another implica-tion of the result is that offline optimization will providelittle benefit in the scenario. Note that the server’s band-width is underutilized only for 0.5 seconds in Slow First.This means that no scheduler can reduce the latency bymore than 0.5 seconds in the setting. In addition, ourevaluation results show that runtime adaptation has a sig-nificant impact on latency, which offline schedulers can-not provide.

3.2 Node SelectionIn this section, we examine the effect of node selection.In Lsync, users can specify their requirements in the formof target synchronization ratio, a fraction of nodes thatneed to be synchronized fast. If the ratio is given, Lsyncattempts to find a subset of nodes that could further re-duce latency.

0

0.1

0.2

0.3

0.4

0.5

0 5000 10000 15000 20000

Fra

cti

on

of

Syn

cd

No

des

Time (s)

Upper Bound

Fast First

Pruned Slow First

Slow First

Figure 5: Synchronizing Frequent Updates – While FastFirst synchronizes quickly at first, Pruned Slow First actuallyreaches the upper bound more quickly.

We show that integrating node selection with SlowFirst scheduling can blend the best behaviors of SlowFirst and Fast First. Given target ratio r, we first sort allnodes in an increasing order of the estimated remainingsynchronization time. We then pick Ns · r nodes whereNs denotes the number of nodes in N, and use SlowFirst scheduling for the selected nodes. The remainingNs · (1− r) nodes are synchronized using Slow First aswell after the selected nodes are finished. As a result, thecompletion time does not suffer either from the slow syn-chronization in the beginning (Slow First) or the long tailat the end (Fast First). We name the integrated schemePruned Slow First for comparison. Figure 4 shows thatPruned Slow First outperforms the other scheduling poli-cies across all target ratios.

We examine a dynamic file update scenario where newfiles are frequently added to the server, which is commonin large-scale distributed services. For an in-depth anal-ysis, we use simulations for the experiment. We generate2000 nodes with bandwidths drawn from the distributionof the inter-node bandwidths on PlanetLab.

We study how these frequent updates affect the syn-chronization process. Given information about availableresources, we can determine how much change the servercan afford to propagate. We define update rate, u, as theamount of new content to be synchronized per unit time(one second). Then, Ns ·u is the minimum bandwidth re-quired to synchronize all Ns nodes with u rate of change.The upper bound on the achievable synchronization ratiois C

Ns·u where C is the server’s upload capacity. C is set to100 Mbps in the experiment.

Figure 5 shows each policy’s performance under fre-quent updates. As before, the server has two files, 1 MBand 10 MB that are in the process of synchronization,and new files are constantly added to the server with anupdate rate 100 Kbps. The upper bound is 0.5 in thissetting, and no policy can reach beyond this limit.

We see that the completion time drastically changesas we use different policies. In particular, the synchro-nization ratio of Slow First drops over time and reaches

4

0.1

1

10

100

1000

10000

100000

0 0.1 0.2 0.3 0.4 0.5

Co

mp

leti

on

Tim

e (

s)

Target Sync Ratio

Slow First

Fast First

Pruned Slow First

Figure 6: Synchronization Latency for Frequent Updates –While Slow First leads to failure, integrating node selectionwith the Slow First scheduling reduces latency for all targetratios (y-axis is in log-scale).

0 near 6800 seconds because it gives high priority tothe nodes that are far behind up-to-date synchronizationstate. Fast First synchronizes nodes quickly in the be-ginning, but asymptotically approaches the upper boundsince the remaining slower nodes make little forwardprogress given the rate of change.

To examine the performance of Pruned Slow First, weset r to 0.49, which is slightly below the upper bound 0.5in this setting. In Figure 5, Pruned Slow First is worsethan Fast First in the beginning, but it reaches its targetratio much earlier than the other policies. Pruned SlowFirst reduces the completion time by 56% compared withFast First, showing that the best policy can provide sig-nificant latency gains, while the worst policy, Slow First,actually leads to failure in this scenario. In Figure 6, weshow the completion time of each policy for a range offeasible r values. The Pruned Slow First outperforms theother policies for every target ratio (the y-axis is in log-scale).

We showed that the Slow First scheduling helps reducecompletion latency, and integrating node selection withthe Slow First scheduling can further reduce latency par-ticularly when handling frequent updates. Based on theobservations, we extend our discussion to examine howto leverage overlay mesh for latency-sensitive synchro-nization in the next section.

4 Leveraging Overlay MeshWith a better understanding of the bandwidth allocationpolicies in the server, we focus on understanding howto leverage scalable one-to-many data transfer systemswhich we collectively name CDN/P2P systems for therest of the paper. Many large-scale distributed servicesconstruct an overlay mesh for scalable data transfer toexternal clients, and often use the overlay mesh for in-ternal data dissemination to remote nodes as well [26].In this section, we explore how to leverage the overlaymesh to reduce synchronization latency without chang-ing its behaviors.

ServerPs Pd

α copies

ni

Overlay mesh (CDN/P2P)

Tprop

Re2ebe2ei

,

Rcdn

Bcdn

Rcdn

Bcdn

Peer lookups, probing,

exchange of control information,

network delays, etc

File f

Figure 7: Startup Latency in CDN/P2P – To leverage a givenoverlay system, Lsync estimates the startup latency for fetch-ing a new file f from the server and propagating to the remotenodes ni in the overlay.

4.1 Startup Latency in Overlay MeshTo leverage a given overlay mesh, Lsync needs to pre-dict startup latency for distributing a new file that is notcached on the remote nodes in the overlay. However, itis difficult to predict the accurate latency since CDN/P2Psystems typically have diverse peering strategies and dy-namic routing mechanisms. To allow easy integrationwith existing systems, Lsync uses a black-box approachto characterizing a given overlay mesh to estimate itsstartup latency.

Figure 7 presents a general model showing how a newfile f in the server is propagated to a remote node ni viaoverlay mesh. Ps is a peer node contacting the serverto fetch f , and Pd is a peer node that ni contacts. If fis already cached in Pd , ni will receive it directly fromPd . However, in latency-sensitive synchronization, alltarget nodes attempt to fetch f as soon as it is avail-able in the server. In case f should be fetched fromthe server, the CDN/P2P system selects node Ps to con-tact the server. Depending on the system’s configuration,multiple copies of the file can be fetched into the over-lay mesh. The fetched file is propagated from Ps to Pdpossibly through some intermediate overlay nodes, anddelivered to ni. Tprop is the propagation delay from Ps toPd . In addition to the network delay between peer nodes,Tprop also includes other overheads such as peer lookupsand exchange of control messages, which are system-specific. The bandwidth and RTT between CDN/P2Ppeer nodes are Bcdn and Rcdn respectively.

For the simplicity of the model, we begin with anideal assumption that nodes in CDN/P2P are uniformlydistributed, and thus the bandwidth and RTT betweenneighboring nodes are constants, Bcdn and Rcdn. How-ever, we will adjust the parameter values later to accountfor their variations on real deployment in the WAN. Thismodel is not tied to a particular CDN/P2P system or anyspecific algorithms such as peer selection and requestredirection. We apply the model to different types ofdeployed CDN/P2P systems in Section 6.2. We show

5

that the model captures the salient characteristics of thesesystems, and that Lsync can adjust its transfers to utilizeeach of these systems.

4.2 Completion Time EstimationTo schedule workload across server and overlay mesh,Lsync first estimates the expected completion time in theoverlay mesh. The setup cost, δ , measures the first-bytelatency for fetching newly-created content through theoverlay mesh. Specifically, δ is defined as

δ = 2 ·Rcdn +Tprop (2)

The overall overlay completion time, Tcdn, can be cal-culated as follows: To distribute f to ni, the server in-forms ni of the new content availability, taking time Re2e.Then, ni contacts Pd , and starts receiving the file withdelay δ . Delivering the entire file to ni with bandwidthBcdn requires time fs

Bcdnwhere fs denotes the size of f .

The total time, Tcdn is then the sum,

Tcdn = Re2e +δ +fs

Bcdn(3)

Note that Tcdn does not depend on r because all targetnodes fetch f simultaneously via the overlay. In compar-ison, the end-to-end completion time, Te2e(r), for trans-ferring f to remote nodes using Pruned Slow First is

Te2e(r) = Re2e +Ns · r · fs

C(4)

where C is the server’s upload capacity.

4.3 Selective Use of Overlay MeshIf a given CDN/P2P system has high startup latency,it will outperform end-to-end transfers only when itsbandwidth efficiency can outweigh the cost. After theserver estimates Tcdn and Te2e(r), the server can dynam-ically choose between end-to-end transfers and the over-lay mesh to get better latency. In this section, we examinethe conditions that such a selective use of overlay meshcan provide benefits in a real deployment.

To get a more accurate estimation of the completiontime, we extend Tcdn in (3) to reflect the fact that thebandwidth distribution of a CDN/P2P system is not uni-form. Rather than modeling each node’s bandwidth sep-arately, we use the minimum bandwidth value for the topNs · r nodes, yielding

Tcdn(r) = Re2e +δr +

fs

Brcdn

(5)

Brcdn is the Ns · r th largest Bcdn, and δ r is the Ns · r th

smallest setup cost. As we increase r, by definition, Brcdn

will monotonically decrease, and δ r will monotonicallyincrease.

0

0.5

1

1.5

2

2.5

0 0.1 0.2 0.3 0.4 0.5

Co

mp

leti

on

Tim

e (

s)

Target Sync Ratio

CDN

Pruned Slow First

(a) Small file (5 KB)

0

30

60

90

120

150

0 0.1 0.2 0.3 0.4 0.5

Co

mp

leti

on

Tim

e (

s)

Target Sync Ratio

(b) Large file (5 MB)

Figure 8: End-to-End Connections vs. Overlay Mesh – Forsmall file, the latency of overlay mesh is hampered by the longsetup time, but its efficient bandwidth usage outweighs the costfor large file.

Now we can compare Tcdn(r) with Te2e(r) and thenuse either end-to-end connections or the overlay mesh asappropriate. The decision process can be formally de-fined as following. The server uses end-to-end connec-tions when either of the following conditions is met.

fs < δr ·

C ·Brcdn

Ns · r ·Brcdn−C

(6)

Brcdn <

CNs · r

(7)

From the above test conditions, we can draw the follow-ing general guidelines. Using end-to-end connections isbetter when (1) the file size fs is small, (2) the overlaysetup cost δ r is large, (3) the server’s upload capacityC is high, (4) the target synchronization ratio r is low,(5) the client population Ns is small, or (6) the overlaybandwidth Br

cdn is significantly smaller than the server’sbandwidth.

We examine the potential benefits of the scheme bycomparing end-to-end transfers with CoBlitz CDN [19]on PlanetLab. Beyond PlanetLab, CoBlitz has been usedin a number of commercial trial services [11], and webelieve CoBlitz represents one of the typical CDN ser-vices currently available. For end-to-end transfers, weuse Pruned Slow First policy for allocating the server’sbandwidth.

Figure 8 shows the latency for synchronizing a smallfile (5 KB) and a large file (5 MB). For a small file, usingend-to-end transfers shows better performance than theCDN because the completion time of the CDN is ham-pered by its setup cost. When transferring a large file, theCDN shows better performance since its efficient band-width usage outweighs the setup cost in the CDN.

Wide-area systems typically disseminate files of vary-ing sizes. For instance, Akamai management server hasfile transfers spanning 1 KB to 100 MB. The measure-ment results in Figure 8 imply that Lsync will need dif-ferent strategies for different file sizes to address thetradeoffs between startup latency and bandwidth effi-ciency of the overlay mesh.

6

0

20

40

60

80

100

120

0 0.2 0.4 0.6 0.8 1

Co

mp

leti

on

Tim

e (

s)

Target Sync Ratio

Lsync (+10% E2E)

Lsync (-10% E2E)

Lsync

Figure 9: Optimality of the Division – The split between over-lay and E2E is not improved by moving some nodes to the othermechanism, suggesting that Lsync’s split is close to optimal.

4.4 Using Spare Bandwidth in ServerWhen nodes are being served by the overlay mesh, theorigin server load is greatly reduced, leading to sparebandwidth that can then be used to serve some nodes by-passing the overlay. Given r, Lsync divides remote nodesinto two groups. One group directly contacts the originserver, and the other group downloads the file via theoverlay mesh. Lsync adds nodes with the worst overlayperformance to the end-to-end group until the expectedend-to-end completion time matches the overlay comple-tion time.

More formally, Lsync calculates re2e, the ratio ofnodes to place in the end-to-end group. After re2e is de-termined, Lsync selects the Ns · (r− re2e) nodes havingthe fastest overlay connections. From (5), the overlaycompletion time is estimated as Tcdn(r− re2e). The re-maining Ns · re2e nodes will use end-to-end connections.To estimate the spare bandwidth in the server, we con-sider CDN load factor α , which represents how manycopies are fetched from the origin server into the over-lay mesh. Different systems have different load factors,and the CoBlitz fetches 9 copies from the origin server.As the origin server is supposed to send α copies of f tothe overlay group, Lsync should reserve Bload bandwidthwhich is α· fs

Tcdn(r−re2e)for overlay traffic, yielding a value

of (C−Bload) for the server’s spare bandwidth. Usingthis spare bandwidth, the end-to-end completion time is

Te2e(re2e) = Re2e +Ns · re2e · fs

C−Bload(8)

Then, Lsync calculates re2e that makes the two groupscomplete at the same time.

We simulate synchronizing PlanetLab nodes using thebandwidths and setup costs measured in all PlanetLabnodes. Figure 9 shows a sensitivity analysis of re2e, withtwo other simulations for slightly higher and lower val-ues of re2e, in sending a 5 MB file. Lsync outperformsboth for all target ratios, suggesting that it is choosing theoptimal balance of the two groups.

4.5 Adaptive Switching in Remote NodesTo mitigate the effect of real-world bandwidth fluctua-tions, we add a dynamic adaptation technique to Lsync.The main observation behind the technique is that minorvariations in performance do not matter for most nodes,since most nodes will not be the bottleneck nodes in thetransfer. However, when a node that is close to beingthe slowest in the overlay group becomes slower, it risksbecoming the bottleneck during file transfer. The lowerbound on the overlay bandwidth is Br−re2e

cdn .When the origin server informs the remote nodes of

new content availability, the server sends Br−re2ecdn and

Tcdn(r− re2e). After Tcdn(r− re2e) passes, if a node isnot finished, it compares the current overlay performancewith Br−re2e

cdn . If the current performance is significantlylower than the expected lower bound, the node stopsdownloading from the overlay mesh, and directly goes tothe origin server to download the remaining data of thefile. In our evaluation, we configure the node to switchto the origin server when its overlay performance dropsbelow 75% of its expected value. Our evaluation resultsshow that using adaptive switching improves the comple-tion time while lowering variations.

5 Implementation

Lsync is a daemon that performs two functions – in onesetting, it operates in the background on the server todetect file changes, manage histories, and plan the syn-chronization process. In its second setting, it runs on allremote nodes to coordinate the synchronization process.Both modes of operation are implemented in the same bi-nary, which is created from 6,586 lines of C code. Lsyncdaemon implements all the techniques described in theprevious sections.

Since Lsync is intended to be easily deployable, it op-erates entirely in user space. Using Linux’s inotify mech-anism, the daemon specifies files and directories that areto be watched for changes – any change results in anevent being generated, which eliminates the need to con-stantly poll all files for changes. When the Lsync daemonstarts, it performs a per-chunk checksum of all files in allof the directories it has been told to watch using Rabin’sfingerprinting algorithm [23] and SHA-1 hashes. OnceLsync is told a file has been changed, it recomputes thechecksums to determine what parts of the file have beenchanged. If new files are created, Lsync receives a no-tification that the directory itself has changed, and thedirectory is searched to see if files have been created ordeleted.

Once Lsync detects changes, it writes the changes intoa log file, along with other identifying information. Thislog file is sent to the chosen remote Lsync daemons, ei-ther by direct transfer, or by copying the log file to a Web-

7

Systems δ r Brcdn

Division RatioE2E Overlay

CoBlitz 1.4 1.2 0.24 0.76Coral 7.6 0.6 0.52 0.48BitTorrent 29.7 4.7 0.30 0.70

Table 1: Division of Nodes between E2E and overlaymesh. r is 0.5, and file size is 5 MB. We also testedsmall files (up to 30 KB), but E2E outperformed all thesesystems. δ r is in seconds, and Br

cdn is in Mbps.

accessible directory and informing the remote daemonsto grab the file using a CDN/P2P system. Once the re-mote daemon receives a log file, it applies the necessarychanges to its local copies of the file.

6 EvaluationIn this section, we evaluate Lsync and its underlyingpolicies on PlanetLab. Each experiment is repeated 10times with each setting, and 95th percentile confidenceintervals are provided where appropriate.

6.1 SettingsWe deploy Lsync on all live PlanetLab nodes (528 nodesat the time of our experiments), and run a dedicatedorigin server with 100 Mbps outbound capacity. Theserver has a 2.4 GHz dual-core Intel processor with 4 MBcache, and runs Apache 2.2.6 on Linux 2.6.23. We mea-sure latency for a set of target synchronization ratios in-cluding 0.05, 0.25, 0.5, 0.75, and 0.98. We use a maxi-mum synchronization level of 0.98 to account for nodesthat may become unreachable during our experiments.Lsync attempts to achieve the given target ratio fast whileleaving other available nodes synchronized via overlay inthe background.

6.2 Startup Latency in CDN/P2P SystemsCDN/P2P designers typically expect that the steady stateof the CDN/P2P system is that the content is alreadypulled from the origin, and is being served to clients overa much longer lifetime with high cache hit ratio. How-ever, for synchronization, remote nodes typically requestchanges that are not in their overlay mesh. Therefore,Lsync monitors the performance of first fetching contentfrom the origin server, which was not a major issue inexisting CDN/P2P systems.

Using our black-box model described in Section 4.1,we measured parameters, δ r and Br

cdn, for two runningCDN systems and one P2P system that we deploy onPlanetLab. Table 1 shows the setup costs and bandwidthsof the three systems. Each system was characterized bysimply fetching new content from the remote nodes, andmeasuring each node’s first-byte latency and bandwidth.

The three systems show interesting differences in behav-ior. BitTorrent spends more time getting the content ini-tially, but then has higher bandwidth. The CDN systemsshow relatively low setup costs because they were de-signed for delivering web objects to clients rather thansharing large files.

The table also shows how Lsync would allocate nodesbetween overlay transfers and end-to-end transfers fora synchronization level of 0.5. For small file transfers,Lsync opts to use end-to-end connections rather thanany of the systems. For larger transfers, the fraction as-signed to direct end-to-end transfers is determined by thetradeoff between latency and bandwidth. For CoBlitz(with the lowest latency) and BitTorrent (with the high-est bandwidth), Lsync assigns most nodes to the over-lay group. For Coral, with slightly higher latency andlower bandwidth, Lsync assigns nodes roughly equallybetween overlay and E2E. We select CoBlitz for the restof our experiments, mostly due to its low latency.

6.3 Comparison with Other SystemsWe compare Lsync with different types of data trans-fer systems. We transfer our CoBlitz web proxy exe-cutable file (600 KB) to all PlanetLab nodes using eachof the systems, and measure completion times (Fig-ure 10). Each system is designed for robust broadcast(epidemic routing), simple cloning (Rsync), bandwidthefficiency (CDN), and high throughput and fairness (P2Psystems). We evaluate the performance of the systemswhen they are used for latency-sensitive synchronizationin the WAN.

Rsync Rsync [27] is an end-to-end file synchronizationtool widely used for cloning files across remote ma-chines. It uses delta encoding to minimize the amount oftransferred data for changes in existing files. In this ex-periment, however, the server synchronizes a newly gen-erated file not available in remote nodes, so the amountof the transferred data is the same as in the other testedsystems. The Rsync server relies only on end-to-endconnections to remote nodes for synchronizing the file,and does not use any policies in allocating server’s band-width. Therefore, the result of Rsync represents the per-formance of end-to-end transfers with no policy applied.P2P systems We use BitTorrent [9] and BitTyrant [21]as examples of P2P systems. Although BitTorrent has ahigh setup cost (Table 1), it performs better than end-to-end copying tools (Rsync) for target ratio of 0.5 becausethe majority of nodes have high throughput. However,BitTorrent ends up with very long completion time (424seconds) and high variations (standard deviation 193) forthe target ratio of 0.98. This is because, with BitTor-rent, slow nodes have little chance to download from peernodes with good network conditions, which makes the

8

0

20

40

60

80

100

120

140

160

180

Tim

e (

s)

Epidemic Routing

Epidemic Routing (best)

Rsync

BitTyrant

BitTorrent

Lsync-CoBlitz

CloudFront

Lsync-CloudFront

(a) Target Sync Ratio 0.50

0

100

200

300

400

500

600

700

Tim

e (

s)

BitTyrant

BitTorrent

Rsync

Epidemic Routing

CloudFront

Epidemic Routing (best)

Lsync-CoBlitz

Lsync-CloudFront

(b) Target Sync Ratio 0.98

Figure 10: Comparison with Other Systems – We compare Lsync with various data transfer systems in terms of the latency forsynchronizing CoBlitz web proxy executable file (600 KB).

slow nodes finish more slowly than in other systems. Wesee the similar trend with BitTyrant.Epidemic routing We also implemented epidemic rout-ing (or gossip) protocol [8] to measure its latency in theWAN. In the protocol, each node picks a random subsetof remote nodes periodically, and exchanges file chunks.For a conservative evaluation, we assume that every nodehas the full membership information to avoid member-ship management, which is known to be the main over-head of the protocol [15]. There are two key parametersin the gossip protocol: how often the nodes perform gos-sip (step interval) and how many peers to gossip with(fanout). These two parameters directly impact on theprotocol’s performance, but it is hard to tune them be-cause of their sensitivity to network conditions. To findthe best configuration for our experiments, we tried arange of values for step interval (0.5, 1, 5, and 10 sec-onds) and fanout (1, 5, 10, 50, and 100 nodes). Then wepicked a configuration that generated the shortest latency.Since we use our own implementation of the protocol,we first compared our implementation with publishedperformance results of the protocol in a similar setting.CREW [13] used the gossip protocol for rapid file dis-semination in the WAN and outperformed Bullet [17]and SplitStream [10] in terms of completion time in theevaluation. Our implementation showed a comparablelatency (141 seconds) to CREW’s result (200 seconds)under the same setting (60 nodes, 600 KB file, 200 Kbpsbandwidth).2

“Epidemic Routing (best)” in Figure 10 represents theresults with the best configuration that we found, (1 sec-ond interval and 10 fanout), and “Epidemic Routing”shows the average performance of all configurations thatwe tested (excluding cases with 30 minutes timeout).The result with the tuned configuration always outper-

2As the topology is not specified, we randomly selected 60 Plan-etLab nodes, and averaged over 10 repeated experiments. We used auser-level traffic shaper, Trickle [14], for setting bandwidth on the se-lected PlanetLab nodes.

forms the average case. Both of the results show inter-esting patterns in completion time. The protocol worksworse than both Rsync and P2P systems at low target ra-tio because file chunks are disseminated only during gos-sip rounds. The file data is disseminated relatively slowin the beginning. However, the gossip protocol outper-forms the other systems at target ratio of 0.98. Unlike inP2P systems, the slow nodes have better chances to peerwith fast nodes during file transfer. As a result, they donot become bottleneck in overall completion time. Therandom peering policy in the gossip protocol helps slownodes to catch up with other nodes, but the protocol istypically more optimized for robust dissemination thanlatency.Commercial CDNs Since our CDN is deployed onPlanetLab, one may think its performance is adverselyaffected by some of overloaded PlanetLab nodes. Wemeasure synchronization latency using a commercialCDN, Amazon CloudFront [4]. CloudFront is faster thanthe other systems for the low target ratio due to its well-provisioned CDN nodes. However, we observe that somenodes are still slow in fetching new file from the CDN.Specifically, 11% of PlanetLab nodes spend more than20 seconds downloading the file from CloudFront, andthe slowest nodes are in Egypt, Tunisia, Argentina, andAustralia. As the file is not cached in the CDN, theslow nodes should fetch the file possibly through mul-tiple intermediate nodes in the overlay. This internal be-havior depends on system-specific routing mechanisms,which is not visible outside of the overlay. We im-plemented another version of Lsync, Lsync-CloudFront,which incorporates CloudFront as its underlying overlaymesh. Lsync-CloudFront estimates the overlay’s startuplatency, and focuses the server’s resource on the bot-tleneck nodes at runtime. We find that the slow nodescan be served better from the server than via the well-provisioned overlay for uncached files, leading to thebest performance among the tested systems.

9

0

0.2

0.4

0.6

0.8

1

0 20 40 60 80 100

CD

F

Completion Time (s)

Lsync

Overlay

Rsync

(a) 10 KB

0

0.2

0.4

0.6

0.8

1

0 20 40 60 80 100

CD

F

Completion Time (s)

(b) 100 KB

0

0.2

0.4

0.6

0.8

1

0 20 40 60 80 100

CD

F

Completion Time (s)

(c) 1 MB

Figure 11: Distribution of Completion Times – For all file sizes, Lsync outperforms the other systems because Lsync adjusts itsfile transfer policies based on file size as well as network conditions.

0

0.2

0.4

0.6

0.8

1

0 10 20 30 40 50 60

Syn

c R

ati

o

Time (minute)

Lsync Overlay Rsync

Figure 12: Frequently Added Files – Lsync makes most nodesfully synchronized during the entire period of the experiment.

6.4 Frequently Added FilesIn large-scale distributed services, file updates occur fre-quently, and new files can be added to the server beforethe previous updates are fully synchronized across nodes.To evaluate Lsync under frequent updates, we gener-ate a 1-hour workload based on the reported workloadof Akamai CDN’s configuration updates [26]. A newfile is added to the server every 10 seconds, and the filesize is drawn from the reported distribution in Akamai.We compare Lsync with CoBlitz and Rsync. The twosystems represent alternative approaches: using overlaymesh only (CoBlitz) and using end-to-end transfers only(Rsync).

When a new file is added, we measure how many re-mote nodes are fully synchronized with all previous filesand compute the average of the synchronization ratiosover one minute. Figure 12 shows the results over thetested 1-hour period. The ratio temporarily drops for sev-eral large file transfers, but Lsync makes 90% of nodesremain fully synchronized for 72% of the tested periodwhile the other approaches do not reach the synchroniza-tion ratio 0.9. Figure 11 shows the distributions of thecompletion latency for 10 KB, 100 KB, and 1 MB files.

6.5 Lsync Contributing FactorsLsync combines various techniques discussed in the pa-per, including node scheduling, node selection, workload

0

50

100

150

200

250

300

0.05 0.25 0.50 0.75 0.98

Tim

e (

s)

Target Sync Ratio

No Policy

Slow First Scheduling

Fast Node Selection

Node Selection + Scheduling

Using Overlay

Lsync

Figure 13: Lsync Contributing Factors – We see that eachcomponent in Lsync contributes to the overall time reduction.

division across server and overlay, and adaptive switch-ing in remote nodes. We examine the contribution ofeach factor individually. We transfer CoBlitz web proxyexecutable file (600 KB) to all remote nodes as before.The results of these tests are shown in Figure 13. Vari-ations of Lsync are shown with no scheduling or nodeselection (No Policy), with only node scheduling (SlowFirst Scheduling), with only node selection (Fast NodeSelection), with only overlay mesh (Using Overlay), andwith all factors enabled (Lsync).

At a high level, we see that the individual contributionsare significant, reducing the synchronization latency bya factor of 4-5 versus having no intelligence in the sys-tem. We see that performing scheduling improves thecompletion time for every target ratio, but that intelli-gent node selection is more critical at lower ratios. Thisresult makes sense, since finding the fast nodes is moreimportant when only a small fraction of the nodes areneeded. However, when the ratio is high and even slowernodes are being included in the synchronization process,scheduling is needed to mask the effects of the slownodes on the latency. The CDN is slow at low targetratios, and becomes comparable to the best end-to-endsynchronization latency at high ratios due to its scalablefile transfers. However, it shows the worst latency withhigh variations at the target ratio of 0.98 because somenodes experience performance problems in the overlay.

10

0

0.2

0.4

0.6

0.8

1

0.05 0.25 0.50 0.75 0.98

No

rma

lized

Rati

o

Target Sync Ratio

Overlay E2E

Figure 14: Division of Nodes in Lsync – We see that the frac-tion of nodes served by overlay mesh changes across target ra-tios, and that the fraction is not monotonically changing withtarget ratio.

0

100

200

300

400

0 20 40 60 80 100 120

Un

sy

nc

hro

niz

ed

N

od

es

Time (s)

Overlay

E2E

Adaptive switching in 12 nodes

Figure 15: Adaptive Switching in Lsync – At 80 seconds, 12nodes dynamically switch to end-to-end connections and finishdownloading from the origin server.

Lsync combines all the factors in a manner to reduce thelatency for all target ratios.

6.6 Nodes Division and Adaptive SwitchingTo see how Lsync adjusts workload across server andoverlay mesh, we further analyze how nodes are dividedinto the overlay group and the end-to-end group. Thenodes in the groups are selected so that both groups areexpected to finish file transfer at the same time. In Fig-ure 14, we plot the normalized sizes of the two groupsduring a large file (5MB) transfer. As the target synchro-nization ratio increases, a smaller fraction of nodes areserved using end-to-end transfers. However, for the tar-get ratio of 0.98, the overlay group’s estimated comple-tion time increases because of nodes with slow overlayconnectivity. Therefore, the ratio for the origin serverincreases compared to the case of target ratio of 0.75.

Lsync makes adjustment at runtime, as can be seen inFigure 15, where we plot the number of pending nodesduring file synchronization. The target synchronizationratio is 0.98, and re2e, the ratio of nodes for end-to-endconnections, is 0.29. The two groups start downloading a5 MB file through the overlay mesh and end-to-end con-nections respectively. At 80 seconds, however, 12 nodesin the overlay group detect that they are having unex-

0

0.1

0.2

0.3

0.4

0.05 0.25 0.50 0.75 0.98

Co

eff

icie

nt

of

Vari

an

ce

Target Sync Ratio

w/o Switching

w/ Switching

Figure 16: Stable File Transfers in Lsync – Adaptive switch-ing in Lsync lowers variance of the latency.

pected overlay performance problems, and could nega-tively affect the completion time. The nodes dynami-cally switch to the end-to-end connections, and directlydownload the remaining content from the origin server.This behavior explains the small bump near 80 seconds– when these nodes switch from the overlay group to theend-to-end group, the number of unsynchronized end-to-end nodes increases.

During 10 repeated experiments, an average of 27nodes dynamically switched to end-to-end connections.By assisting a few nodes in trouble, Lsync reduces thecompletion time by 16%. In addition to the reduced com-pletion time, the adaptive switching in Lsync providesstable file transfers because it masks unpredictable vari-ations in the overlay performance at runtime. Figure 16plots the variations of the completion times with/withoutadaptive switching in Lsync. The vertical bars plot thecoefficient of variance (normalized deviation) of comple-tion times during repeated experiments. For every targetratio, Lsync shows smaller variations compared to theversion with the switching disabled.

7 Related Work

CDNs and P2P systems [9, 16, 18, 19, 21] scalably dis-tribute the content to a large number of clients. Whilethese systems achieve bandwidth efficiency and load re-duction at the origin server, they typically sacrifice start-up time and total synchronization latency for smallernode groups. Likewise, peer-assisted swarm transfer sys-tems [20] manage server’s bandwidth using a global opti-mization, but they address bandwidth efficiency in multi-swarm environments, not latency.

Our work on managing latency in distributed systemsis related in spirit to partial barrier [3] that is a relaxedsynchronization barrier for loosely coupled networkedsystems. The proposed primitive provides dynamic kneedetection in the node arrival process, and allows applica-tions to release the barrier early before slow nodes arrive.Mantri [5] uses similar techniques to improve job com-pletion time in Map-Reduce clusters.

11

Lsync aggressively uses available resources becauseit would otherwise remain unused. Another example isdsync [22] that aggressively draws data from multiplesources with varying performance. dsync schedules theresources based on the estimated cost and benefit of op-erations on each resource, and makes locally-optimal de-cisions, whereas Lsync tries to perform globally-optimalscheduling to reduce overall latency.

Gossip-based broadcast [8, 15] provides scalable androbust event dissemination to a large number of nodes.Our measurements demonstrate that the random peer-ing strategy in the protocol helps reduce latency becauseslow nodes are likely to find nodes with the disseminateddata after most fast nodes are synchronized. Lsync showsbetter latency than the gossip protocol because it targetsthe slow nodes from the beginning of file transfers, pre-venting the slow nodes from being the bottleneck.

8 ConclusionLow-latency data dissemination is important for coordi-nating remote nodes in large-scale wide-area distributedsystems, but the latency has been largely ignored in de-signing one-to-many file transfer systems. We found thatthe latency is highly variable in heterogeneous networkenvironments, and many file transfer systems are subop-timal for the metric. To solve this problem, we have iden-tified the sources of the latency and described techniquesto reduce their impact on the latency (node schedul-ing, node selection, workload adjustment, and dynamicswitching). We have presented Lsync that integrates allthe techniques in a manner to minimize latency basedon information available at runtime. In addition to stand-alone application, we expect that Lsync is useful to manywide-area distributed systems that need to coordinate thebehavior of remote nodes with minimal latency.

9 AcknowledgmentsWe would like to thank our shepherd, Frank Dabek, aswell as the anonymous reviewers. We also thank MichaelGolightly, Sunghwan Ihm, and Anirudh Badam for use-ful discussion and their insightful comments on earlierdrafts of this paper. This research was partially supportedby the NSF Awards CNS-0615237, CNS-0916204, andKCA (Korea Communications Agency) in South Korea,KCA-2012-11913-05004.

References[1] PlanetLab. http://www.planet-lab.org/.

[2] Akamai Inc. www.akamai.com/.

[3] J. Albrecht, C. Tuttle, A. C. Snoeren, and A. Vahdat. Loose syn-chronization for large-scale networked systems. In Proceedingsof USENIX ATC, 2006.

[4] Amazon CloudFront. http://aws.amazon.com/

cloudfront/.

[5] G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Stoica, Y. Lu,B. Saha, and E. Harris. Reining in the outliers in Map-Reduceclusters using Mantri. In Proceedings of USENIX OSDI, 2010.

[6] N. Bansal and M. Harchol-Balter. Analysis of SRPT scheduling:investigating unfairness. In Proceedings of ACM SIGMETRICS,2001.

[7] O. Beaumont, N. Bonichon, and L. Eyraud-Dubois. Schedulingdivisible workloads on heterogeneous platforms under boundedmulti-port model. In IEEE IPDPS, 2008.

[8] K. P. Birman, M. Hayden, O. Ozkasap, Z. Xiao, M. Budiu, andY. Minsky. Bimodal multicast. ACM Transactions on ComputerSystems, May 1999.

[9] BitTorrent. http://bittorrent.com/.

[10] M. Castro, P. Druschel, A.-M. Kermarrec, A. Nandi, A. Row-stron, and A. Singh. SplitStream: High-bandwidth multicast incooperative environments. In Proceedings of ACM SOSP, 2003.

[11] CoBlitz Inc. http://www.verivue.com/.

[12] A. Davis, J. Parikh, and W. E. Weihl. EdgeComputing: Extendingenterprise applications to the edge of the internet. In Proceedingsof ACM WWW, 2004.

[13] M. Deshpande, B. Xing, I. Lazardis, B. Hore, N. Venkata-subramanian, and S. Mehrotra. CREW: A gossip-based flash-dissemination system. In IEEE ICDCS, 2006.

[14] M. A. Eriksen. Trickle: A userland bandwidth shaper for unix-like systems. In Proceedings of USENIX ATC, 2005.

[15] P. T. Eugster, R. Guerraoui, S. B. Handurukande, P. Kouznetsov,and A.-M. Kermarrec. Lightweight probabilistic broadcast. ACMTransactions on Computer Systems, November 2003.

[16] M. J. Freedman, E. Freudenthal, and D. Mazieres. Democratizingcontent publication with Coral. In Proceedings of USENIX NSDI,2004.

[17] D. Kostic, A. Rodriguez, J. Albrecht, and A. Vahdat. Bullet: Highbandwidth data dissemination using an overlay mesh. In Proceed-ings of ACM SOSP, 2003.

[18] E. Nygren, R. K. Sitaraman, and J. Sun. The Akamai network: Aplatform for high-performance internet applications. ACM Oper-ating Systems Review, 2010.

[19] K. Park and V. S. Pai. Scale and performance in the CoBlitzlarge-file distribution service. In Proceedings of USENIX NSDI,2006.

[20] R. S. Peterson, B. Wong, and E. G. Sirer. A content propagationmetric for efficient content distribution. In Proceedings of ACMSIGCOMM, 2011.

[21] M. Piatek, T. Isdal, T. Anderson, A. Krishnamurthy, andA. Venkataramani. Do incentives build robustness in BitTorrent?In Proceedings of USENIX NSDI, 2007.

[22] H. Pucha, M. Kaminsky, D. G. Andersen, and M. A. Kozuch.Adaptive file transfers for diverse environments. In Proceedingsof USENIX ATC, 2008.

[23] M. O. Rabin. Fingerprinting by random polynomials. TechnicalReport TR-15-81, Harvard University, 1981.

[24] Riverbed Technology Inc. http://www.riverbed.com/.

[25] SETI@home. http://setiathome.berkeley.edu/.

[26] A. Sherman, P. A. Lisiecki, A. Berkheimer, and J. Wein. ACMS:The Akamai configuration management system. In Proceedingsof USENIX NSDI, 2005.

[27] A. Tridgell. Efficient Algorithms for Sorting and Synchronization.PhD thesis, The Australian National University, 1999.

12