Reverse Engineering the Internet

Neil Spring

This is an unpublished thesis proposal. For the HotNets paper of the same name, go tohttp://www.cs.washington.edu/homes/nspring/papers/reverse.pdf

Abstract

Understanding the structure and design of the Internet isis increasingly important as we seek to improve its reli-ability and robustness. At the same time, as the networkgrows in scale and diversity, a complete and accurateview is increasingly hard to come by. In this paper, wepresent a framework for classifying network measure-ment tools by how they can contribute to network map-ping and whole-Internet analysis. We describe tech-niques to accommodate the scale and heterogeneity ofthe network. While most network measurement toolsfocus on performance and pathologies, we focus on de-veloping tools to understand structure, design, and rout-ing policy.

1 Introduction

Measuring and understanding the Internet has becomeincreasingly important as we rely on it more and its vul-nerabilities and limitations are exposed. Understandingthe design of the Internet is important for researchersinterested in routing protocols [10, 53], multicast stud-ies [78], selfish routing schemes [84, 6], and denial ofservice traceback and response [59, 74, 85, 87, 80]. Un-derstanding the structure is also important from a con-sumer and regulatory standpoint, where an independentview of the reliability, performance, and provisioningof the Internet can help shape public policy [28, 67].

While administrative tools can be used to measurean individual network for management [34, 33], it isdifficult to discover the structure of the Internet in gen-eral, or even simply a handful of specific ISP net-works. While some ISPs publish high-level topologies,and a few research networks even publish some de-tails, what few maps exist may be inaccurate and are oflimited value in research. Because no single organiza-tion “owns” the Internet, research on the global Internetinfrastructure suffers [8], while it demands a new setof tools that can cope with its heterogeneity and ever-increasing scale [27, 36].

Heterogeneity permeates every aspect of the Inter-

net, making it difficult to extrapolate where any par-ticular problem occurs, who is responsible, and whatshould be done to fix it. There are thousands of distinctadministrative domains (“autonomous systems”) coop-erating to run the network, each with its own choiceof hardware, choice of protocols, set of service-levelagreements (SLAs), approach to traffic engineering,etc. However, each has only a limited view of the net-work and can rarely evaluate the far-reaching effects ofa change [62]. Heterogeneity also means that variousexceptional, buggy behaviors can complicate otherwiseelegant analyses [69, 4].

Existing tools provide a partial view of the globalpicture. They often measure end-to-end (path) proper-ties but these path properties can be hard to composein a map [76, 100, 83, 68]. Some measure hop-by-hop properties, but send too many packets to be used atscale [58, 45, 30, 75]. While recent projects have com-posed many, many end-to-end measurements of lossand latency with “tomographic” analyses to infer hop-wise properties [70, 13], it is unclear whether proper-ties that have traditionally required more traffic, suchas bandwidth and congestion, can also be inferred in alightweight manner.

In this thesis proposal, I assert that these problemsof heterogeneity and scale can be addressed to “reverseengineer” the core of the Internet accurately and effi-ciently. The goal of this reverse engineering is to pro-vide sufficient detail in an annotated map of the net-work to evaluate potential changes: new links, new pro-tocols, new queue management, etc. With accurate,measured data, one might even look for correlationsbetween properties – such as whether traffic engineer-ing primarily avoids congested links or whether routingpolicy arrangements are symmetric. This thesis is aboutdemonstrating that such a map can be constructed with-out new infrastructure, not about the resulting map. Weuse “core” above to avoid detailed external mapping ofedge networks, such as corporate or campus networks,which are less useful for network simulation, are lessinteresting in measurement as more traffic traverses thecore, and are more likely to perceive measurement traf-fic as a possible attack. The primary challenges in this

1

mapping are in recovering attributes of the network thatare currently only observed by network administrators– network topology, routing policy and traffic loads.

The rest of this paper is organized as follows. In Sec-tion 2, we present an overview of the Internet archi-tecture and introduce terms we will use in the rest ofthe paper. Section 3 presents a framework for networkmeasurement projects based on their potential contri-bution to network mapping. Section 4 describes recentprojects in this framework, along with brief discussionsof potential short-term research directions. Section 5briefly describes our recent work that begins to addressthe problems of discovering Internet structure, policy,and pathologies. Section 6 describes a component thatremains to complete the picture, lightweight hop-wiseavailable bandwidth estimation. Finally, we concludein Section 7.

2 Internet Primer

The Internet is a collection of networks run by differ-ent Internet Service Providers (ISPs). Each network ismade up ofroutersconnected bylinks. In this paper, wetreat all links as IP-level links, and ignore the physicaltopology underlying the network-level hop. Networksthat connect to each other are said topeer, those lo-cations where they connect arepeering pointsand theconnections arepeering links, which are otherwise or-dinary links except that they connect routers owned bytwo different ISPs. Each ISP is made up of one or moreautonomous systems(ASes) representing an adminis-trative domain. Large ISPs, such as AT&T, have differ-ent autonomous systems, one for each continent. EachAS runs tworouting protocols, one interior gatewayprotocol for its own internal network and one exteriorgateway protocol for communicating with its neighborsand the rest of the Internet. Finally, each ISP has severalpoints of presence (POPs) that represent physical loca-tions, roughly one per city, and the connections betweenPOPs constitute thebackboneor coreof the network.

ISPs typically design their network to be robust tofailures using redundant links and redundant routers.This arrangement of routers and links isnetwork de-sign. ISPs also attempt to balance traffic across the ca-pacity they do have, keeping utilization (and thereforeloss and queueing) low. The balancing of traffic and thegeneral optimization of routes to improve performanceare calledtraffic engineering.

The Internet Protocol (IP) is designed to carry dataacross different data link types like Ethernet, 802.11wireless, ATM, etc. As such, it makes minimal assump-tions about the reliability and speed of those underly-ing links, and provides minimal service beyond end-to-

end connectivity. This means that the IP service modelallows packet loss, delay (reordering), corruption, andeven duplication. The Transmission Control Protocol(TCP) corrects for these faults to provide a reliable,error-free, byte-stream oriented transport protocol forapplications to use. Because it detects and correctsfaults, as well as adapts to the capacity of the network,TCP’s adaptive behavior can be observed to measurethese properties.

There are two primary performance metrics of thenetwork, latency and bandwidth.Latencyis the basetransmission time of the smallest unit of data. Whenmeasured alone, it is the time a very small (40- to 64-byte) packet takes to transit the network. When mea-sured with bandwidth, it is the time for the first bit,in other words they-intercept if transmission time isa function of packet size. Latency is comprised ofqueuing delay, which varies with load, andtransmis-sion delay, which is fixed.Bandwidthis the additionaltime taken per bit, or the slope of that function relatingpacket size to time, not a physical property of the trans-mission medium.Capacityis the maximum possible bitrate offered by a link, whileavailable bandwidthis theunused, idle capacity of the link.1

In each packet sent through the Internet, a headerfield called the time-to-live (TTL) prevents the packetfrom cycling about the network in an endless loopshould routing be temporarily inconsistent. Routersdecrement the TTL field when forwarding a packetalong a path; if the field reaches zero, an error messageis returned to the source. This is a basic primitive ulti-mately used by network measurement tools to addressindividual hops along a path.Tracerouteis one suchtool to discover the routers along a path; we occasion-ally use traceroute as a verb to represent executing thetool to discover the path to some other host.

3 Contextual Framework

In this paper, we are interested in tools that allow anoutsider to infer the behavior of the network with suf-ficient precision to annotate a map. Researchers havedeveloped many tools to discover network propertiesto varying degrees of precision. Figure 1 shows aconceptual hierarchy of the precision of existing tools.The width of the lower segments implies a density oftools that infer many properties of nodes and end-to-endpaths, fewer that observe hops and trees, virtually nonethat measure maps, and none that synthesize measure-ments to annotate the map. We present the context for

1An alternate definition of available bandwidth, sometimes used,is the bandwidth a new flow could expect to achieve over a link, whichmay be greater than the idle bandwidth as existing traffic is displaced.

2

Node

Node pair (path)

Hop (link)

Tree

Map

Synthesis

Detection

Precision

Scale: many destinations

Scale: many sources

Support

Composition

Figure 1: Hierarchy of network measurement precisionand scale. The precision and scope of understanding isat left in the pyramid, while the challenges to overcomeare at right.

measurement in this order, with increasing detail on theproblem solving approaches as we near the top of thepyramid. Promotion from one level to the next often re-quires solving difficult problems of support, detection,precision, scale, and composition.

Figure 1 alludes to Maslow’s hierarchy of needs fromthe field of psychology [64]. In 1943, Abraham Maslowstudied sane adults (as opposed to asylum patients, chil-dren, or animals), and based on these studies formu-lated a hierarchy of human needs to explain humanmotivation. This hierarchy is comprised (from lowestto highest) of the physiological (food), security (avoiddanger), love (acceptance), esteem (approval), and self-actualization (realizing potential). The hierarchy waslater revised to include somewhat more detailed levels,but the insight is the same: humans satisfy needs in es-sentially this order, seeking food before safety beforeapproval.

In understanding network design and performance,we face a similar hierarchy: understanding the globalpicture is an evolution from tools that first detect prop-erties along some path, then focus on a link at a time,and ultimately permit properties to be measured atscale, forming a tree or combination of trees to forma map. To understand a new property, we need first tounderstand node (end-host or intermediate router) be-havior, then to use that knowledge in building a tool toestimate a path property. After developing a methodol-ogy for measuring a path, extensions and engineeringcan measure hop-wise properties, perhaps recognizingthe site of some anomaly or measuring the bandwidthof individual links. Such tools often require further en-gineering to run at scale – perhaps trading accuracy forefficiency. We now sketch each of these levels with se-

lected examples.Node measurement tools, such as tbit [69], nmap [38]

and fsd [39], discover the properties of individual nodes(end hosts or routers) in the network. Such tools arefundamental, and measuring node properties is implicitin the design of more advanced tools. For example,traceroute implicitly measures the responsiveness of anode when it returns an ICMP error response. Alsoforming a foundation for more advanced tools, node lo-cation services like IDMaps [37] or GeoTrack [71] ex-pose geographic properties of routers and end-nodes,which is useful for rendering on a map, estimatingpath latencies, or understanding geographic propertiesof links [56, 92].

Node-pair (or end-to-end path) measurement toolsare similarly diverse. TCP-based tools, such as the ac-tive (probing-based) Sting [11, 83] and Treno [65], aswell as the passive (trace-analysis based) tcpanaly[76]and TRAT [100], generally look at end-to-end paths,often characterizing several properties at a time. Pas-sive measurements of conversations between pairs ofmachines are often used to estimate a property with-out precisely locating it in the network, for example,the measurements of reordering by Bellardo and Sav-age [12] or loss by Paxson [76]. Unfortunately, the re-sults of these measurements are difficult to use in deter-mining where problems occur. One approach comparesmany paths that do and do not traverse a peering pointto determine whether peering points are bottlenecks [2],but this precision is insufficient for map annotation.

Tools that can pinpoint an attribute to a particularhop (IP level link) often build upon node-pair tools us-ing limited time-to-live fields in the IP header to solicitICMP time-exceeded errors. Such tools include trace-route [46] and its variants tcptraceroute [96], fft [66],etc., to measure connectivity and round-trip latency,and pathchar [45] and its variants pchar [58], clink [30],etc., that measure bandwidth. Alternatives to solicit-ing time-exceeded messages include finding paths di-rectly to intermediate routers that are prefixes of anend-to-end path as used by cing [4], or using limitedTTL in a one-way measurement in a “tailgating” tech-nique [43, 54].

Tree-analysis tools can either compose a set of hop-based measurements to different destinations or com-pose a set of path measurements using inference orclustering techniques. Padmanabhan,et al., composeinference using path measurements of TCP transac-tions with traceroute in their study of loss in the net-work [70]. They find that loss generally occurs on seg-ments close to clients, though their vantage point of awell-provisioned server may bias the result. Other stud-ies discover the properties of branches in the tree with-out looking at the underlying IP-level topology, such as

3

MINC [1], and Coates,et al. [26]. Such projects useshared behavior along segments of an (often multicast)tree to infer a particular property. The simple examplethat describes the intuition of multicast inference is loss– when loss is rare, one can assume that multiple sta-tions that miss a packet share a (lossy) segment of themulticast tree.

Map-analysis tools compose tree-analyses from sev-eral vantage points to understand the network as awhole, rather than simply the network as observedfrom a single point. Such mapping approaches includeearly work by Pansiot and Grad [72], followed by Skit-ter [25], Mercator [40] and Lumeta [18, 24]. The use-fulness of these connectivity maps has been limited toinferring relatively abstract graph-theoretic propertiesof degree distribution [32, 15], small-world-ness [16],resilience, expansion, distortion [93], etc. However,these metrics alone have changed the focus of topologygeneration from structural [99] to degree-distribution-based [48]. That is, rather than model the Internet as thehierarchical collection of ISPs and points-of-presence(POPs, roughly one per city per ISP), recent effortshave attempted to match the properties of the measuredgraph – a demonstration that real, measured data canshape research in surprising ways.

The synthesis of mapped attributes to annotate a mapis essentially unexplored. While recent work has com-bined properties like geographic location with networkpaths [92] to understand how network paths traversethe globe, and other work renders Internet maps color-coded by ISP or address space [18, 25], work that bringstogether network attributes to annotate a map using ex-ternal measurements is in its infancy. Demonstratingthe ability to fill this vacuum using external measure-ment is the overall direction of this work.

Summarizing the hierarchy, to traverse from onelevel to the next requires solving important and oftendifficult engineering problems. Promoting from a path-wise tool, such as bprobe, to a hop-by-hop tool, suchas pathchar, requires much more than simply settinga TTL field: it requires many packets and manipula-tion of aggregate statistics to estimate per-link band-widths [30]. Promoting such an invasive tool further toone that can be used at scale is a particular (and open)challenge.

4 Previous Approaches and Op-portunities

We summarize existing research projects and their toolsin Table 1. While many properties have been studied tomeasure their prevalence along paths, few have beenstudied to discover where in the Internet they occur.

We draw a distinction between tools that measure prop-erties useful for the engineering and construction of anetwork, and those tools that measure the deliverableperformance attributes. Our focus will be on the devel-opment of new tools to measure the former category,often using tools and techniques from the latter.

In Table 1, parentheses represent tools that have yetto be built. Hop-tools to precisely locate pathologieslike reordering and duplication would be both usefuland challenging. Equally challenging is to extend thoseand existing hop-measurement tools to run at scale andto help complete the annotated map of the Internet, fill-ing the Tree and Map columns of the table.

4.1 Node Analysis Techniques

As we observe in Table 1, node measurement tools areincomplete in our ability to remotely measure failureand (processor) load. Our understanding of node failureis limited to internal studies like one from Sprint [44],that observe internal routing protocol traffic to detectlink and router failure. They found that failures weregenerally short and occurred more often during sched-uled maintenance intervals. To measure a failure rateexternally, one would likely start by measuring a mapof the network and repeatedly measuring paths to ob-serve if they detour around a possible failure. Thistechnique could be augmented with active probing ofrouters, perhaps looking for reset IP identifier counters,a sign that the in-memory state of a router had beencleared through reboot. In this way, the analysis of oc-casional failures to expose topology through BGP orother routing protocol dynamics [5] can be inverted tofind the location of failures given a topology.

To detect load (a node’s equivalent of network linkcongestion) an unpublished report by Cardwell andSavage suggests a method. They used the rate of changeof IP identifiers as a relative estimate of Web server load– faster climb indicates that more packets have beengenerated, suggesting additional load on a Web server.Such an approach may not directly estimate a router’shost processor load, as many of its activities (recom-puting routing tables, collecting statistics) may not in-volve packet generation. We also know that routers takeoccasional “coffee breaks” where they are unrespon-sive while handling periodic maintenance tasks [73],but whether this can be applied to understand the pro-cessing load of a router is unclear.

4.2 Path Analysis Techniques

Detecting properties along paths has been well-studied.Vern Paxson’stcpanaly [76] is notable by its cover-age of many different network behaviors by using “pas-

4

Nod

eP

ath

Hop

Tre

eM

apN

etw

ork

Man

agem

ent,

Eng

inee

ring,

and

Str

uctu

reLo

catio

nG

eoT

rack

[71]

,N

etG

eo[1

9]–

––

–

Ow

ner

Who

is,B

GP

(us)

––

––

Con

nect

ivity

–pi

ngtr

acer

oute

BG

Pin

fere

nce

[5],

clus

terin

g[5

2]M

erca

tor

[40]

,Lum

eta

[18]

(x)

Rou

ting

Pol

icy

––

–(

)(x

)D

eman

dM

atrix

––

––

inte

rnal

only

:ne

tflow

-bas

ed[3

3],

tom

ogra

vity

[101

],(

)M

edia

type

––

()

()

Net

wor

kP

erfo

rman

ce,B

ehav

ior,

and

Pat

holo

gies

Impl

emen

tatio

nnm

ap[3

8],t

bit[

69]

––

––

Loss

–tc

pana

ly[7

6],s

ting

[83]

pcha

r[5

8](x

)M

INC

[1],

Venk

at[7

0]in

tern

alon

ly:

SN

MP

Late

ncy/

Del

ay:

fsd

[39]

RT

T–

ping

,kin

g[4

1]tr

acer

oute

,pch

ar(

)(

)O

ne-w

ay(O

TT

)–

tcpa

naly

cing

[4]

()

(())

Con

gest

ion

()

((x)

)M

INC

[1],

()

(late

ncy

var.)

Ban

dwid

th:

Cap

acity

–bp

robe

[22]

,pat

hrat

e[2

9]ne

ttim

er[5

4],p

athc

har

Kat

abi[

50]

()

Ava

ilabl

e–

tcpa

naly

,pat

hloa

d[4

7],

chirp

s[8

2]((

))(

)(

)

Reo

rder

ing

–tc

pana

ly,s

ting-

ii[1

1]((

x))

()

()

Dup

licat

ion

–tc

pana

ly(

)(

)(

)C

orru

ptio

n–

Sto

ne[9

1]((

x))

()

“Per

form

ance

”–

iper

f[95

],tr

eno

[65]

,T

RAT

[100

],O

lym

pics

[9],

SP

AN

D[8

6],N

g[6

8]

b-fin

d[2

]–

–

Fai

lure

()

Dah

lin[2

3](

)(

)in

tern

alon

ly:

Spr

int[

44](

)A

pplic

atio

nm

ix–

MC

I[94

],C

AID

A[2

0]

Tabl

e1:

Cla

ssifi

catio

nof

tool

sby

prop

erty

and

prec

isio

n–

som

eto

ols

may

inte

rpre

tth

esa

me

info

rmat

ion

topr

ovid

em

any

met

rics

(suc

has

pcha

r.)T

heir

cate

goriz

atio

nab

ove

shou

ldbe

inte

rpre

ted

asa

cove

ring

ofsp

ace,

nota

nen

umer

atio

nof

the

feat

ures

ofm

easu

rem

entt

ools

and

proj

ects

.A

‘–’r

epre

sent

sa

cate

gory

that

does

n’ta

pply

,suc

has

the

band

wid

thof

ano

de.

A“(

)”re

pres

ents

aca

tego

ryth

atm

aypr

ove

usef

ulin

unde

rsta

ndin

g,bu

thas

been

un-

(or

unde

r-)

expl

ored

.A

“(x)

”re

pres

ents

aca

tego

ryin

whi

chw

eha

veal

read

ym

ade

sign

ifica

ntpr

ogre

ss.

A“(

())

”su

gges

tsan

ambi

tious

buta

chie

vabl

edi

rect

ion.

Bla

nkce

llsm

ayex

istt

oth

ele

ft,w

here

path

orho

pm

easu

rem

ents

are

redu

ndan

twith

proj

ects

that

unde

rsta

ndth

em

apor

atr

ee.

5

sive analysis” of packets that are already traversing thenetwork combined with an understanding of TCP’s re-action to various network behaviors. Other tools gener-ate their own traffic and are considered “active.” Thisdistinction between active and passive techniques is notparticularly important in the context of path analysis; anactive technique can be made passive through patience(a technique used by early versions of nettimer [54])and a passive technique can be made active by gener-ating traffic. However, active techniques have greaterpotential for promotion to analyze per-hop behaviorsbecause actively-generated traffic can be given differ-ent TTL values or be sent to intermediate routers. Ac-tive techniques also have the potential to measure thenetwork as a whole rather than just the traffic passing aparticular site.

Whether passive or active, path-analysis techniquescan contribute to our goal of mapping the network inthree important ways. First, the techniques provide anend-to-end check on the validity of hop-by-hop tools.Second, they provide detection of rare properties, likeloss, reordering, and duplication, that can allow mea-surements of, say, hop-by-hop loss to be skipped if apath has no end-to-end loss. Third, they provide in-sight into how a property can be measured, allowinghop-wise tools to be built as extensions.

4.3 Hop Analysis Techniques

We now start to describe techniques of particular util-ity for reverse engineering an annotated map of thenetwork. Hop-analysis techniques discover propertiesof individual IP-level links in the network; composingmany of these hop measurements brings us closer toa network map. Network measurements that addressproperties of IP-level network links are typically basedon TTL-limited probing, the same technique as at theheart of traceroute [46].

Bandwidth measurement tools such aspathchar [45], clink [30], and pchar [58]measure the serialization delay of variably-sizedpackets. Serialization delay is the time it takes a routerinterface to write each bit on a wire. These techniquesare vulnerable to cross traffic, so require many packetsto find samples that experience no queueing (onlyservice time) in the network. Longer paths requiremore packets, not just to measure the extra links, butbecause the likelihood of experiencing no queueingis reduced. This makes a comprehensive study ofbandwidth at scale difficult. Further, these techniquesreturn erroneous results when intermediate, data-linklayer switches are present [79, 75]. The ACCSIGtool [75] preserves the same probe traffic of variablepacket sizes, but observes the delay variation pattern to

avoid the need for minima filtering and the overhead itentails.

Recent approaches are more robust and deal withan absence of router cooperation using a “tailgating”technique. As with the previous tools, “tailgating” ap-proaches also measure the serialization delay of a largepacket, but observe bit delay by the effect it has on a“tailgating” packet that proceeds unobstructed throughthe rest of the network after the large packet’s TTL ex-pires. Tailgating has the advantage that it is possibleto collect one-way timings, without conflating forwardand reverse path properties, which is a significant ad-vantage in that only the properties of the (known, sin-gle) forward path are involved. Further, tailgating doesnot require routers to generate ICMP messages. Thisenables the study of shared route segments includingswitched network paths (shared network segments be-low the IP layer and not otherwise visible) using meth-ods from MINC, below. Using tailgating probes fortopology inference has not been explored. Packet quar-tets [75] and cartouche probing [43, 42] improve uponthe approach using more probe packets in a train. Theseapproaches are less vulnerable to cross traffic on linksleading up to the one being measured.

Another innovative tool analyzing hop properties iscing [4], which uses ICMP timestamps to measure per-link delays. The challenges for cing primarily involveclock synchronization, but a second requirement is thatthe ICMP timestamp requests traverse a prefix of thepath to the destination being studied. This means thatcing must ensure that the path to an intermediate routeris a prefix of the path being studied using additionaltraceroutes. However, the advantage is that the proper-ties of the return path back to the source and the forwardpath along to the ultimate destination are factored out.

4.3.1 Opportunities

Using NTP with tailgating approaches. NTP hasthe potential to provide more precise timestamps thanthe ICMP timestamp message or IP timestamp option.NTP may synchronize end-system clocks to microsec-ond resolution, and provide access to such a value, an-notated with its precision and parent in the time syn-chronization tree, while ICMP timestamp messages aremillisecond fields that may be updated less frequently.NTP has the advantage that remote sites do not need torun a special server, like that used by netperf [49] oriperf [95]. Interestingly, if NTP proves useful, it maybe an interesting case of an application-layer protocolhaving more utility than one designed into the network-layer control plane.

6

Measuring and validating hop latency. Latency es-timation is inevitably conflated with clock synchroniza-tion and understanding the relative skew, relative offset,and patterns of jumps in clocks. Paxson’s work in [77]lays the groundwork for coping with individual prob-lems on the clocks of hosts. Anagnostakis,et al. [4]dealt with the more numerous and less well-behavedclocks of routers with techniques to discard clock mea-surements that violated too many assumptions of stablebehavior.

The techniques used in cing [4] assume symmet-ric paths, which makes clock normalization possible.However, the localization techniques of GeoPing andGeoTrack [92] suggest a different method, using geo-graphic latency. It is possible that combining the twotechniques would yield a better approach, or at the veryleast, help to validate cing’s estimates of link delay.

Per-hop available bandwidth Merging the tech-niques that measure end-to-end available bandwidthwith those of hop-wise capacity tools may permit de-velopment of a hop-wise available bandwidth estimator.We describe this in more detail in Section 6.

4.4 Tree Analysis Techniques

Tree analyses can be reached in two ways: the composi-tion of hop-based measurement tools from a single site,or the use of path-measurement tools for inference ofshared properties and shared segments. The latter ap-proach is typically termedtomographyby reference tothe medical imaging procedures CT (computerized to-mography) and PET (positron emission tomography),surprisingly not because both attempt to infer internalstructure by external measurement, but because of sim-ilarities in the mathematical formulation of the prob-lems.

Early work in tree analysis discovered shared proper-ties using multicast trees. Multicast distribution of indi-vidual probe packets allow analyses like MINC [1] andMINT [13] to assign events such as loss and queueingto branches of the tree. When a probe packet is lost ordelayed, all recipients of that packet along that branchsee the loss or delay. Such work could be combinedwith multicast traceroute [35] to discover an IP-levelmulticast topology to annotate.

Padmanabhan,et al. [70] show that a similar anal-ysis can be supported by unicast tools in a “passive”analysis. They study the TCP transactions that accessthe busy Microsoft Web site, use traceroute to deter-mine which path is taken back to the client, then useBayesian inference and Gibbs sampling to infer whichlink along the path was the site of persistent loss. Previ-ous techniques, such as the multicast-based techniques

above, actively generated traffic to measure loss rates;the authors’ emphasis on passive analysis serves to dis-tinguish the work from [31] and [13], but is not particu-larly useful. Each provides a combination of inferencetechniques needed to make unicast probing effective.

Akella, et al. [2] use a similar approach to deter-mine where bottlenecks occur. They measure the per-formance of a series of adaptive UDP transactions, de-termine whether they see a bottleneck in the network(path-based detection) then analyze whether those pathstraverse a peering point. Though currently a work inprogress, with this analysis, they hope to determinewhether peering points are often bottlenecks for net-work traffic.

The tree-measurement based “tomography” we de-scribe here (wherepathmeasurements yield link prop-erties as in [70, 1]) is the reverse of “tomography” as ap-plied to traffic matrices (wherelink measurements yieldpath properties, as in [101, 97, 21]). Surprisingly, thedissimilarity ends there: both problems are addressedusing the same methodology by Liang and Yu [57].

4.4.1 Opportunities

Efficient tree bandwidth measurement. Bandwidth(capacity) measurement takes many packets, so the goalis to keep it efficient. Multiple paths may traverse thesame early links, and measuring these links only oncecan save traffic. Second, accuracy could be traded awayfor efficiency if estimates for faraway links do not needto be as accurate.

Tree reordering. While Padmanabhan [70] studiesloss by observing retransmissions, the same analysiscould discover sites of forward path reordering. Anal-ysis of the stream of TCP acknowledgements exposeswhen packets were received out of order: when aTCP receiver sees an out of order packet, it immedi-ately sends an acknowledgement of the last in-sequencepacket received. This behavior of duplicate acknowl-edgements is just as observable as the retransmitted seg-ments used to infer loss, and can be used to estimatewhere reordering occurs.

Web server-based measurement. Padmanab-han [70], Balakrishnan [9], and others have made useof the implicit conversations of a busy Web server tomeasure path properties. Use of a Web server or setof servers as measurement sources may simplify theproblem of destinations that are otherwise invisible tothe network because they are behind firewalls. Onecould imagine extending the single-server tomographicanalyses to use several sites to the behavior of the

7

network as a whole, though this analysis may not betractable.

4.5 External Mapping Techniques

In this subsection, we focus on Internet mapping tech-niques and the different approaches to measurement atthe scale of the Internet. These techniques are relevantbecause they solve the challenges necessary to builda global picture of the network. In this section, wefirst contrast the work of Pansiot and Grad, Govindanand Tangmunarunkit (Mercator), Burch and Cheswick(Lumeta), and claffy, Monk and McRobb (Skitter). Im-ages from three of these are in Figure 2. Then, we de-scribe some open problems in network mapping.

Pansiot and Grad [72] measured a network topologyusing traceroute so that they could characterize mul-ticast protocol proposals. In their study, the topologyof the network influences the design of multicast treesin that many routers may not participate in forward-ing, and many routers may simply forward a multicastpacket as if it were a unicast packet, perhaps requir-ing less routing state. Pansiot and Grad collected twodata sets. First, 12 sources were used to traceroute to1,270 hosts, and 1 source (their own) to 5,000 hosts thathad previously communicated with their department.To handle the problem of scale, Pansiot and Grad opti-mized their measurement tool, modifying traceroute intwo ways. First, they did not probe three times per hop,but instead returned after the first successful response(retrying if no response was received). This has the po-tential to reduce the overhead of traceroute-based map-ping by two-thirds. Second, they configured tracerouteto start probing some number of hops into the trace,avoiding repeated, redundant probes to nearby routers.To provide an accurate map, they pioneered work onalias resolution2: the process of recognizing which IPinterface addresses belong to the same router.

Burch, Cheswick, and Branigan [18, 24] use a dif-ferent approach, increasing the number of destinationsto ninety thousand, but using only a single source host.This work is primarily concerned with understandingsmall components, tracking change over time, and visu-alizing the overall network, and has resulted in Lumeta,a company specializing in mapping networks.

Govindan and Tanmunarunkit [40] in the Merca-tor project added “informed random address probing,”in which traceroute destinations were chosen by con-sulting the global Internet routing table. Govindan,like Burch, uses a single source, but have many vir-tual sources by using source routing – a technique that“bounces” probes off remote routers. Like Pansiot,

2Pansiot and Grad termed themsynonyms; we prefer the Mercatorterminology.

Govindan uses the source-address based alias resolu-tion methodology, but enhances it using source-routedprobes and repeated tests. Mercator’s maps were vali-dated by extracting components that belong to researchISPs and comparing those to the real maps.

CAIDA’s Skitter [25, 15] project increases the num-ber of destinations by picking Web servers (originally29,000, though this has increased over time). They usesix DNS servers as measurement sources to get a repre-sentative picture of the network. Like Burch, the Skitterproject maintains a history of previous measurements tocharacterize the change in the network.

Different mapping projects approach efficiency indifferent ways, summarized in Table 2. Most map-ping projects choose a small set of destinations, whichis a form of sampling that may not be well justified.Different approaches solve the alias resolution prob-lem for accuracy in Table 3. Previous approaches haveused source-address-based alias resolution or ignoredthe problem. These differences may result from theirdifferent goals, summarized in Table 4. For example,those projects looking at evolution over long time scaleshave ignored the alias resolution problem,

These Internet mapping techniques focus specificallyon the connectivity between routers. Structural viewsare limited to the opaque pictures like those in Figure 2.The whole Internet is a consistent goal, but samplingapproaches are needed to make this feasible – fromchoosing limited destinations (Pansiot and Grad, Skit-ter) to choosing a single source (Lumeta) to choosing torun for weeks (Mercator).

The challenges for wide area Internet map construc-tion are several; we list the three most important here.First, without many trees as vantage points, the obser-vations of the network are biased in favor of high de-tail “bushyness” near measurement sources, and the ap-pearance of long chains further away [55]. This needfor many sources stresses the scalability of a measure-ment. As a further consequence, alias resolution be-comes more important as different vantage points seedifferent interfaces. However, unreachable routers thatdo not respond to alias resolution probes, the secondchallenge, thwart efforts to build an accurate map [89].Finally, anonymous routers that do not even send time-exceeded messages to TTL-limited probes threaten tolimit the ability to construct a map at all [98].

4.5.1 Opportunities

Focus on a smaller network. One approach to scal-ing tools to map networks is avoid the scaling problemas much as possible by focusing only on a sub-graph.This is the Rocketfuel approach, described in Section 5.

8

Figure 2: Internet map visualizations from different projects. At left is Mercator’s view of an ISP named Cable andWireless. At right is Burch and Cheswick’s map of the Internet from 1999; Cable and Wireless is the green star in theupper right. For a larger picture, see [17]. These maps show two very different structures for the same network. Thedifficulty of interpreting this data, let alone verifying its accuracy, prevents its use.

Project Approach to EfficiencyPansiot and Grad [72] Few destinations (1200 to 5000), tuned traceroute.

Lumeta [18] Single source (more tree than map)Mercator [40] Run for three weeks.Skitter [25] Limit destinations to active Web servers.

Table 2: Different mapping projects achieve efficiency in mapping the Internet in different ways.

Map by DNS scanning. The DNS names associatedwith router interfaces include a lot of information abouttheir location and connectivity. For example, Abilenerouters are named likesttlng-dnvrng.abilene.ucaid.edu, implying a point-to-point link between Seattle andDenver. Adjacent IP addresses are likely to belong tothe same network link, as IP network addresses are as-signed first to network links, then IP addresses to theirinterfaces. The DNS approach has the opportunity toobserve otherwise unseen interface addresses, usefulfor ensuring a complete map. The questions raisedby this approach are in which ISPs can be accuratelymapped using this approach – for example, will net-works that use MPLS or other switched links exten-sively confound the analysis or be much easier to un-derstand? One challenge will be in discovering andunderstanding peering relationships, because the DNSnames associated with peering link interface addresseshave less structure.

4.6 Map annotation and synthesis ofproperties

Connectivity alone can help to answer questions aboutthe robustness of the network to partition and the staterequired by routing protocols and for multicast forward-ing. However, it tells us little about performance be-cause the capacity of links can vary over 6 orders ofmagnitude (33 kilobits per second modem to 40 gigabitper second OC-768 links), and a single network hopmay as easily represent crossing a machine room ascrossing an ocean.

Annotating a map with node properties does not ap-pear to pose a significant challenge. For example, DNSnames recover geography [71], and router role [89].DNS with address space information from BGP alsogives the AS that manages a router [63]. More challeng-ing is adapting link-measurement tools to run at scale,as described in the previous section on tree measure-ment.

9

Project Approach to AccuracyPansiot and Grad Basic source-address alias resolution

Mercator Source routed probing and retried alias resolutionLumeta No alias resolution required - single sourceSkitter Several sources, but no alias resolution described

Table 3: Different mapping projects achieve accuracy in mapping the Internet in different ways.

Desired mapSingle snapshot Tracked across time

SourcesFew Pansiot & Grad Lumeta

Many Mercator Skitter

Table 4: Different mapping projects along two axes, how many sources of traffic, and whether or not an evolutionaryview is desired.

Broadcast: x.y.z.3Network: x.y.z.0/30

x.y.z.1 x.y.z.2R1 R2

Figure 3: Address allocation of a point-to-point net-work. Shaded boxes represent routers, adjacent circlesinterfaces, and the line between them a point-to-pointlink. A /30 prefix (the first 30 bits of a 32-bit IP ad-dress are significant) leaves two addresses for use, tworeserved as broadcast and network addresses.

4.6.1 Opportunities

Inferring network policy. A map of connectivity an-notated with link weights is sufficient for modelingpaths taken through an individual network. This al-lows us to recognize not just the connections betweenrouters, but an aspect of their configuration to handlethe workload. Additional characterization of the policyacross pairs of ISPs proves more difficult, but it may bepossible to classify policies into a handful of categoriesthat represent the inter-ISP relationship. Our work onrecovering policy is described in Section 5.

Media type inference. Underlying each IP-level hopis a data-link layer topology, such as an Ethernet, awireless link, a ring, or a point-to-point link. If we cantell what data-link network is used, we can simplify themeasured topology from the clique of IP-level connec-tivity measured by traceroute to the concentrated star orring topology of a shared network when it exists. Eachhas particular behaviors that may expose just what typeof network is used. For example, artifacts that influ-ence measurements of bandwidth in [79, 75] suggest

Network: x.y.z.0/29Broadcast: x.y.z.7

x.y.z.1 x.y.z.2

x.y.z.3

x.y.z.4

x.y.z.5

R1 R2

R3

R4R5

Figure 4: Address allocation of a multiple-access net-work. A /29 prefix (the first 29 bits of a 32-bit IP ad-dress are significant) leaves six addresses for use, tworeserved as broadcast and network addresses. Thosethat end with 3 and 4 would have been unavailable ifused in a point-to-point network.

a method to identify switched networks. A switchednetwork is one in which an intermediate node partic-ipates in forwarding a packet at a time, but does notparticipate in IP – this means that it delays, buffers, andeven may drop traffic, but is not otherwise visible be-cause it does not respond to traffic. Such a switch canbe detected when when bandwidth measured by vari-able packet size approaches (such as pathchar) is halfof a common value (eg. 50 Mbits instead of 100) orless than the delivered performance measured by a toollike pathload or iperf. Perhaps wireless links may beinferred by observation of the inter-frame spacing usedin collision avoidance.

Interpreting DNS names and IP address allocation

10

provides an alternative to hunting telltale performanceattributes. Many router interfaces have names that im-ply they belong to SONET rings, gigabit Ethernet links,or T-3 links. Even those that don’t are likely to haveIP addresses allocated from ranges that make it obvi-ous whether they belong to multiple-access networks.Each network is given a (small) prefix, and typicallythe two addresses at the top and bottom of the range arereserved as broadcast and network addresses. Point-to-point links are given /30 prefixes, which reserves thoseaddresses that end in 00b or 11b, leaving those that endwith 01b or 10b available for both ends of the link. Thisis shown in Figure 3. Multiple-access networks will beallocated larger prefixes, meaning that some addresseson multiple access networks should end in, for exam-ple 011b (3) or 100b (4). An example allocation formultiple-access networks is shown in Figure 4. An in-terface address that ends in either 00b or 11b is likely tobelong to a multiple access network of some sort; ad-jacent addresses are likely to connect to the same net-work.

5 Methodology

In this section, we summarize our recent work that en-ables accurate and efficient annotated map construction.While my proposed thesis focuses on discovering de-sign and configuration, we also describe supporting per-formance and debugging tools that more generally dis-cover attributes required to evaluate network protocolsin simulation.

The Rocketfuel [89] project focused hundreds ofpublic traceroute servers on the task of mapping a hand-ful of large ISP networks. The goal was to measure veryaccurate maps using the expectation that using manymore vantage points would expose many more paths.Our insight was to combine the information availablethrough DNS and BGP to guide the mapping processand interpret the result. We also innovated in the de-sign of an alias resolution procedure that can recognizewhich IP addresses (assigned to interfaces) belong tothe same router; a task of crucial importance for anaccurate map. The experiences gained on this projectshape our later work. First, alias resolution is difficultand error-prone, but necessary for an accurate result.Second, large scale network measurements inevitablytrigger security alarms resulting in some complaintsfrom remote system administrators, so good citizenshipand ties with local system administrators are important.Third, filtering out measurement errors is problematic– it is not always obvious when traceroute measures afalse link – so robust tools that can detect signs of inac-curacy and avoid giving false results are important.

In follow on work [60], we combined the measuredmaps with the original traceroutes collected to modelthe internal routing policy of the ISPs. The insight isthat given a map, the observed, chosen path is shorter(has lower cost) than alternate paths. By assemblingrules of this form, we can use linear programming toinfer a cost for each link that is consistent with ob-served routing. Three insights make this possible. First,many constraints are redundant – there are very manyalternate paths from one point to another, only a few ofthese are reasonable contenders. Second, the ingress-to-egress measurements are incomplete for each pair ofPOPs, however, each sub-path of a chosen path is alsoa chosen path, improving overall completeness. Third,some noise is present in the data as occasionally, per-haps due to router failure, an alternate path is actuallytaken, so constraint hierarchies [14] are used to dealwith this error. The final solution that assigns weightsto links is not unique, so the actual link-cost settings arelikely to differ from those that are inferred. However,the weights provide a simple, compact description ofobserved routing.

We proceeded further to try to understand inter-domain policy, or how traffic is managed when itcrosses from one ISP to the next [88]. When study-ing global Internet routing policy, we made simplify-ing assumptions about the topology and used the geog-raphy to understand the indirectness of chosen routes.That is, when paths are diverted or special cased, traf-fic engineering is indicated because the direct routeis avoided – the ISP must have some reason to avoidthe direct, simplest path. Using these analyses, wefound that we could infer the relationship between ISPs,for example, whether they use “early” or “late” exitrouting. The combination of Rocketfuel with internaland external policy completes a picture of network de-sign and observable configuration; what remains undis-covered are the specific choices of configuration val-ues, the goals achieved by these configuration choices,and unused backup connections that carry no traffic.However, topology and policy have been sufficient toparameterize recent studies of traffic engineering ap-proaches [7, 102].

The need for robust, flexible tools for network mea-surement motivated the design of Scriptroute [90],which is an environment for distributed network mea-surement. The Scriptroute approach combines mobile,untrusted code with a security policy that restricts “un-safe” network behaviors, such as port-scanning and de-nial of service attacks. Using Scriptroute, we havemodified network mapping tools for robustness to rout-ing changes (rockettrace), integrated Rocketfuel’s aliasresolution techniques with the DNS to handle unre-sponsive routers (ally), and implemented efficient dis-

11

tributed mapping techniques (reverse path tree). Scrip-troute allows us to be ambitious in the scale of our net-work mapping without stressing the public tracerouteserver infrastructure used in Rocketfuel.

With this architecture for flexible measurement,we have constructed tools to measure and locate thepathologies previously studied only on a whole path ba-sis [61]. We term this approach “Internet diagnosis,”and the techniques allow a hop-by-hop measurementof congestion by observing variation in queueing de-lay, and of forward per-hop loss and reordering usingIP identifiers. While the goal of the study was to enableusers to assign blame for poor perceived performance,these lightweight techniques, integrated with networkmapping, help to expose the heterogeneity of the net-work for simulation and evaluation. Although thesetools do not directly expose network design, they forma basis for understanding factors involved that guide theevolution of networks.

6 Near-future work

A long-term goal of this network measurement work isto provide a measured network for simulation studies.This measured network should be as real, demandingaccuracy in measurement, and as complete (both in de-tail and annotations) as possible. In the short-term, ourgoal is to measure the properties needed to explain andunderstand the network’s design, considering perfor-mance only as it helps explain the network’s design. Acomplete picture of a network’s design requires at leasta glimpse into the workload it supports. This workloadis expressed as atraffic matrix, the amount of trafficthat traverses the network from each source to each des-tination. The challenge is recovering this informationwithout using management tools like netflow or SNMP,which report link utilization to administrators. In thissection, we outline possible approaches to construct ameasurement tool that recovers link utilization, or itscomplement, available (idle) bandwidth.

There are two potential techniques to estimate linkutilization, or conversely, unused or available band-width. A measurement of utilization coupled with ca-pacities makes it possible to use the methods in [97,101] to infer a traffic matrix over a routing inferredby [60]. The traffic matrix is relevant because the effec-tiveness of a network is evaluated in terms of its abilityto handle its workload, and it is reasonable to expectthat the network has been tuned with its workload inmind.

The first approach to measure available bandwidth isto load links to force them to queue traffic; the avail-able bandwidth is that which could be consumed with-

out lengthening the queue. This approach is that ofpathload [47], which should support easy extension tobe used in a per-hop basis. Pathload adaptively deter-mines how much additional data must be sent through abottleneck to cause queueing. Combining this with thetailgating technique of nettimer [54] would allow load-ing links before the bottleneck without loading a path’sbottleneck link. However, the need to load links makesit impossible to measure the available capacity on linksafter a bottleneck. For purposes of ISP mapping, thismay be a real problem if the access to the ISP is a bot-tleneck link, at least relative to the network backbone.

The second approach is a measurement of the distri-bution of queueing delay on the link. The Delphi [82]approach uses a series of “packet chirps” to observecross traffic that modifies the spacing of chirped pack-ets. One approach to estimating the available capacityon hops before the end-to-end bottleneck would be tosend TTL-limited packet chirps. Alternately, one couldmeasure the distribution of queuing delay directly, us-ing approaches from cing [4]. Any measurements offorward path delay greater than the minimum suggestqueueing along the path. Ideally, cross traffic even afterthe bottleneck should have an effect on the distributionof delay samples, which may allow a reasonable guessas to its utilization using Little’s formula. Little’s for-mula relates the length of the queue to the arrival rateand the expected wait time [51]. Alouf,et al. [3] useLittle’s formula to develop a methodology to estimatethe cross traffic sharing a bottleneck link (as well asthe queue capacity). This approach was evaluated onlyin simulation, but appears promising. The most accu-rate moment-based estimator presented in [3] relatesthe measured loss probability (PL), measured averagequeue delay (R), and known link capacity (µ) with uti-lization (ρ) using the formula:

R =1

µ(1− ρ)− PL log(PL/(1− ρ(1− PL)))

µ(1− ρ)(1− PL) log(ρ)(1)

An interesting engineering question is how (andwhether) to combine capacity measurement with avail-able bandwidth estimation. Such optimizations may beof particular importance in building scalable tools.

The development of two independent techniquesbased on different methodologies (loading the link untilqueueing and measuring queue delay distributions) hasthe potential to support cross-validation. A second val-idation strategy would be to compare with the Abilenenetwork statistics available athttp://www.abilene.iu.edu/noc.html. Unfortunately, these are somewhat di-gested for graphical presentation, so a different accesspath may be needed for real-time comparison betweenthe tool’s measurement and the SNMP-retrieved utiliza-

12

tion statistics.By measuring link utilization to recover a traffic

matrix, a complete picture of network engineeringcan be recovered from the outside. Vardi introducesthe problem of recovering traffic matrices from linkloads [97], including methods for handling multi-pathrouting. However, Vardi assumes a Poisson arrival pro-cess, which motivates the work of Caoet al. [21], whouse trace data to correct the Poisson assumption andvalidate using a small network. Recent work [101] hassought to make such methods practical and robust to er-ror by integrating a gravity model. A gravity modelrelates the amount of traffic between pairs of nodesas the product of their (destination-independent) traf-fic volume, in much the same way as the gravitationalforce between bodies is proportional to the product oftheir masses. This tomographic gravity model providesa methodology to scale the analysis. Measured link uti-lizations combined with a little guesswork based on thedistribution of address space (say, there are more ma-chines in San Francisco than Sacramento) to parame-terize the gravity model seem promising for recoveryof a full traffic matrix.

The major limitation to the effectiveness of thesemeasurements will likely be scaling to the very fast linkspeeds in the backbone of the Internet. The transmis-sion time of a large (1500 byte) vs. small (40 byte)packet across a gigabit network is only 12µsec, strain-ing the precision of commodity clocks. This means thatmeasurements requiring precise timing are likely to bedwarfed by noise without specialized hardware. Theapproaches for estimating link capacity assume that this12µsec difference can be measured accurately. Withoutan accurate link capacity measure, the available band-width measure may not in the end give link utilization,even though its use of loss rate and queueing delay es-timation make it less susceptible to imprecise clocks.

While the limited detail and accuracy of external in-ference will prevent such tools from replacing internalnetwork management tools, we hope that measured datawill prove useful to the development of new protocolsand the evaluation of the Internet’s robustness.

7 Conclusions and (distant) futurework

In this paper, we have described the techniques of net-work measurement with an eye toward inferring its in-ternal structure and design. While the prevalence ofmany properties is well-studied, developing techniquesto discover where they occur with any precision is chal-lenging. Adapting these measurement tools to workat the scale of the Internet requires careful design and

sometimes new approaches.In this thesis proposal, my focus is the external un-

derstanding of the design and management of networksthat make up the Internet. Future work in measurementcan complete the construction of an annotated map thatallows large parts of the Internet to parameterize a sim-ulation. Periodic measurement can construct an evolv-ing history of the design and performance of these net-works, with the potential to characterize the long termeffects of conspicuous failures like the Baltimore tunnelfire [81].

The measurements themselves enable new studiesinto how ISPs can better manage their networks, per-haps using new algorithms to set link policy. A mo-tivating goal of Rocketfuel was to recover topologiesupon which we could evaluate robust interior gatewayrouting protocols. Further afield, researchers and de-velopers can test new protocols or designs on individ-ual network topologies and workloads to demonstrateto the network operator that they solve a real problem,and quantify the expected benefit of deployment.

References

[1] A. Adams, T. Bu, R. Caceres, N. G. Duffield,T. Friedman, J. Horowitz, F. LoPresti, S. B.Moon, V. Paxson, and D. Towsley. The use ofend-to-end multicast measurements for charac-terizing internal network behavior.IEEE Com-munications Magazine, 2000.

[2] A. Akella, S. Seshan, and A. Shaikh. An empiri-cal evaluation of wide-area Internet bottlenecks.In ACM SIGMETRICS, June 2003. (poster).

[3] S. Alouf, P. Nain, and D. Towsley. Inferring net-work characteristics via moment-based estima-tors. InIEEE INFOCOM, Apr. 2001.

[4] K. G. Anagnostakis, M. B. Greenwald, and R. S.Ryger. cing: Measuring network-internal delaysusing only existing infrastructure. InIEEE IN-FOCOM, 2003.

[5] D. G. Andersen, N. Feamster, S. Bauer, andH. Balakrishnan. Topology inference from BGProuting dynamics. InACM SIGCOMM InternetMeasurement Workshop, Nov. 2002.

[6] D. Anderson, H. Balakrishnan, M. F. Kaashoek,and R. Morris. Resilient overlay networks. InSOSP, Oct. 2002.

[7] D. Applegate and E. Cohen. Making intra-domain routing robust to changing and uncer-tain traffic demands: Understanding fundamen-

13

tal tradeoffs. InACM SIGCOMM, Aug. 2003.(to appear).

[8] R. Atkinson and S. Floyd. IAB concerns and rec-ommendations regarding Internet research andevolution. Internet draft: draft-iab-research-funding-00.txt, Feb. 2003.

[9] H. Balakrishnan, V. N. Padmanabhan, S. Seshan,M. Stemm, and R. H. Katz. TCP behavior ofa busy Internet server: Analysis and improve-ments. InIEEE INFOCOM, Mar. 1998.

[10] A. Basu and J. Riecke. Stability issues in OSPFrouting. InACM SIGCOMM, Aug. 2001.

[11] J. Bellardo and S. Savage. Measuring packet re-ordering. InACM SIGCOMM Internet Measure-ment Workshop, Nov. 2002.

[12] J. C. R. Bennett, C. Partridge, and N. Schect-man. Packet reordering is not pathological net-work behavior.IEEE/ACM Transactions on Net-working, 7(6), Dec. 1999.

[13] A. Bestavros, J. Byers, and K. Harfoush. In-ference and labeling of metric-induced networktopologies. InIEEE INFOCOM, June 2002.

[14] A. Borning, B. Freeman-Benson, and M. Wilson.Constraint hierarchies.Lisp and Symbolic Com-putation, 5(3), 1992.

[15] A. Broido and kc claffy. Internet topology: con-nectivity of IP graphs. InSPIE Int’l symposiumon Convergence of IT and Communication, Aug.2001.

[16] T. Bu and D. Towsley. On distinguishing be-tween Internet power law topology generators.In IEEE INFOCOM, Apr. 2002.

[17] H. Burch and B. Cheswick. Burch/cheswick mapof the Internet. http://research.lumeta.com/ches/map/gallery/isp-ss.gif, June 1999.

[18] H. Burch and B. Cheswick. Mapping the In-ternet. IEEE Computer, 32(4):97–98, 102, Apr.1999.

[19] CAIDA. NetGeo - the Internet geographicdatabase.http://www.caida.org/tools/utilities/netgeo/.

[20] CAIDA. Traffic workload overview. http://www.caida.org/Learn/Flow/tcpudp.html,June 1999.

[21] J. Cao, D. Davis, S. VanderWiel, and B. Yu.Time-varying network tomography: Router linkdata.Journal of the American Statistical Associ-ation, 95(452):1063–1075, Dec. 2000.

[22] R. L. Carter and M. E. Crovella. Dynamic serverselection using bandwidth probing in wide-areanetworks. InIEEE INFOCOM, 1997.

[23] B. Chandra, M. Dahlin, L. Gao, and A. Nayate.End-to-end WAN service availability. InUSITS,Mar. 2001.

[24] B. Cheswick, H. Burch, and S. Branigan. Map-ping and visualizing the Internet. InUSENIX An-nual Technical Conference, 2000.

[25] k. claffy, T. E. Monk, and D. McRobb. Internettomography. InNature, Jan. 1999.

[26] M. Coates, R. Castro, and R. Nowak. Maximumlikelihood network topology identification fromedge-based unicast measurements. InACM SIG-METRICS, 2002.

[27] Computer Science and TelecommunicationsBoard, National Research Council.LookingOver the Fence at Networks: A Neighbor’sView of Networking Research. The NationalAcademies Press, 2001.

[28] Computer Science and TelecommunicationsBoard, National Research Council.The Inter-net Under Crisis Conditions: Learning fromSeptember 11. The National Academies Press,2003.

[29] C. Dovrolis, P. Ramanathan, and D. Moore.What do packet dispersion techniques measure?In IEEE INFOCOM, Apr. 2001.

[30] A. B. Downey. Using pathchar to estimate In-ternet link characteristics. InACM SIGCOMM,Sept. 1999.

[31] N. G. Duffield, F. LoPresti, V. Paxson, andD. Towsley. Inferring link loss using striped uni-cast probes. InIEEE INFOCOM, 2001.

[32] M. Faloutsos, P. Faloutsos, and C. Faloutsos. Onpower-law relationships of the Internet topology.In ACM SIGCOMM, 1999.

[33] A. Feldmann, A. Greenberg, C. Lund, andN. Reingold. Deriving traffic demands for op-erational IP networks: Methodology and experi-ence. InACM SIGCOMM, Sept. 2000.

14

[34] A. Feldmann, A. Greenberg, C. Lund, N. Rein-gold, and J. Rexford. Netscope: Traffic engineer-ing for IP networks. IEEE Network Magazine,Mar. 2000.

[35] W. Fenner and S. Casner.mtrace . Inter-net Draft: draft-idmr-traceroute-ipm-04.txt, Feb.1999.

[36] S. Floyd and V. Paxson. Difficulties in sumu-lating the Internet.IEEE/ACM Transactions onNetworking, Feb. 2001.

[37] P. Francis, S. Jamin, C. Jin, Y. Jin, D. Raz,Y. Shavitt, and L. Zhang. IDMaps: Aglobal Internet host distance estimation service.IEEE/ACM Transactions on Networking, Oct.2001.

[38] Fyodor. NMAP: The network mapper.http://www.insecure.org/nmap/.

[39] R. Govindan and V. Paxson. Estimating routerICMP generation delays. InPassive & ActiveMeasurement (PAM), Mar. 2002.

[40] R. Govindan and H. Tangmunarunkit. Heuristicsfor Internet map discovery. InIEEE INFOCOM,2000.

[41] K. P. Gummadi, S. Saroiu, and S. D. Gribble.King: Estimating latency between arbitrary In-ternet end hosts. InACM SIGCOMM InternetMeasurement Workshop, Nov. 2002.

[42] K. Harfoush, A. Bestavros, and J. Byers. Mea-suring bottleneck bandwidth of targeted pathsegments. Technical Report BUCS-2001-016,Boston University CS, July 2001.

[43] K. Harfoush, A. Bestavros, and J. Byers. Mea-suring bottleneck bandwidth of targeted pathsegments. InIEEE INFOCOM, Apr. 2003.

[44] G. Iannaccone, C. Chuah, R. Mortier, S. Bhat-tacharyya, and C. Diot. Analysis of link failuresin an IP backbone. InACM SIGCOMM InternetMeasurement Workshop, Nov. 2002.

[45] V. Jacobson. Pathchar. ftp://ftp.ee.lbl.gov/pathchar.

[46] V. Jacobson. Traceroute.ftp://ftp.ee.lbl.gov/traceroute.tar.Z.

[47] M. Jain and C. Dovrolis. End-to-end availablebandwidth: measurement methodology, dynam-ics, and relation with TCP throughput. InACMSIGCOMM, Aug. 2002.

[48] C. Jin, Q. Chen, and S. Jamin. Inet: In-ternet topology generator. Technical ReportCSE-TR-433-00, University of Michigan, EECSdept., 2000. http://topology.eecs.umich.edu/inet/inet-2.0.pdf.

[49] R. Jones. Netperf.http://www.netperf.org/.

[50] D. Katabi, I. Bazzi, and X. Yang. A passive ap-proach for detecting shared bottlenecks. InIEEEInt’l Converence on Computer Communicationsand Networks (ICCCN), 2001.

[51] L. Kleinrock. Queueing Systems, volume 1. JohnWiley & Sons, 1975.

[52] B. Krishnamurthy and J. Wang. On network-aware clustering of Web clients. InACM SIG-COMM, Sept. 2000.

[53] C. Labovitz, A. Ahuja, A. Bose, and F. Jahanian.Delayed Internet routing convergence. InACMSIGCOMM, Sept. 2000.

[54] K. Lai and M. Baker. Nettimer: A tool formeasuring bottleneck link bandwidth. InUSITS,Mar. 2001.

[55] A. Lakhina, J. Byers, M. Crovella, and P. Xie.Sampling biases in IP topology measurements.In IEEE INFOCOM, Apr. 2003.

[56] A. Lakina, J. W. Byers, M. Crovella, andI. Matta. On the geographic location of Inter-net resources. InACM SIGCOMM Internet Mea-surement Workshop, 2002.

[57] G. Liang and B. Yu. Pseudo likelihood estima-tion in network tomography. InIEEE INFO-COM, Apr. 2003.

[58] B. Mah. Estimating bandwidth and other net-work properties. InInternet Statistics and Met-rics Analysis Workshop, Dec. 2000.

[59] R. Mahajan, S. M. Bellovin, S. Floyd, J. Ioanni-dis, V. Paxson, and S. Shenker. Controlling high-bandwidth aggregates in the network (extendedversion).http://www.aciri.org/pushback/, July2001.

[60] R. Mahajan, N. Spring, D. Wetherall, and T. An-derson. Inferring link weights using end-to-end measurements. InACM SIGCOMM InternetMeasurement Workshop, Nov. 2002.

[61] R. Mahajan, N. Spring, D. Wetherall, and T. An-derson. User-level Internet path diagnosis. InSOSP, 2003. (In submission).

15

[62] R. Mahajan, D. Wetherall, and T. Anderson.Understanding BGP misconfiguration. InACMSIGCOMM, Aug. 2002.

[63] Z. M. Mao, J. Rexford, J. Wang, and R. Katz.Towards an accurate as-level traceroute tool. InACM SIGCOMM, Aug. 2003. (to appear).

[64] A. H. Maslow. A theory of human motivation.Psychological Review, (50), 1943.

[65] M. Mathis. Diagnosing Internet congestion witha transport layer performance tool. InINET’96,June 1996.

[66] N. McCarthy. fft. http://www.mainnerve.com/fft/.

[67] The national strategy to secure cyberspace,Feb. 2003.http://www.whitehouse.gov/pcipb/cyberspace strategy.pdf.

[68] T. S. E. Ng, Y. Chu, S. G. Rao, K. Sripanid-kulchai, and H. Zhang. Measurement-based op-timization techniques for bandwidth-demandingpeer-to-peer systems. InIEEE INFOCOM, Apr.2003.

[69] J. Padhye and S. Floyd. Identifying the TCPbehavior of Web servers. InACM SIGCOMM,Aug. 2001.

[70] V. N. Padmanabhan, L. Qiu, and H. J. Wang. Pas-sive network tomography using bayesian infer-ence. InACM SIGCOMM Internet MeasurementWorkshop, Nov. 2002.

[71] V. N. Padmanabhan and L. Subramanian. Aninvestigation of geographic mapping techniquesfor Internet hosts. InACM SIGCOMM, Aug.2001.

[72] J.-J. Pansiot and D. Grad. On routes and multi-cast trees in the Internet.ACM Computer Com-munication Review, 28(1):41–50, Jan. 1998.

[73] K. Papagiannaki, S. Moon, C. Fraleigh, P. Thi-ran, F. Tobagi, and C. Diot. Analysis of mea-sured single-hop delay from an operational back-bone network. InIEEE INFOCOM, June 2002.

[74] K. Park and H. Lee. On the effectiveness ofroute-based packet filtering for distributed DoSattack prevention in power-law internets. InACM SIGCOMM, Aug. 2001.

[75] A. Pasztor and D. Veitch. Active probing us-ing packet quartets. InACM SIGCOMM InternetMeasurement Workshop, Nov. 2002.

[76] V. Paxson. End-to-end Internet packet dynamics.In ACM SIGCOMM, Sept. 1997.

[77] V. Paxson. On calibrating measurements ofpacket transit times. InACM SIGMETRICS,1998.

[78] G. Philips, S. Shenker, and H. Tangmunarunkit.Scaling of multicast trees: Comments on theChuang-Sirbu scaling law. InACM SIGCOMM,Aug. 1999.

[79] R. S. Prasad, C. Dovrolis, and B. A. Mah. The ef-fect of layer-2 switches on pathchar-like tools. InACM SIGCOMM Internet Measurement Work-shop, Nov. 2002.

[80] P. Radoslavov, H. Tangmunarunkit, H. Yu,R. Govindan, S. Shenker, and D. Estrin. Oncharacterizing network topologies and analyzingtheir impact on protocol design. Technical Re-port CS-00-731, USC, 2000.

[81] A. Ratner. Train derailment severs commu-nications. The Baltimore Sun, July 2001.http://www.sunspot.net/news/local/bal-te.bz.fiber20jul20,0,1743066.story.

[82] V. Ribeiro, M. Coates, R. Riedi, S. Sarvotham,B. Hendricks, and R. Baraniuk. Multifractalcross-traffic estimation. InITC Specialist Sem-inar on IP Traffic Measurement, Modeling andManagement, Sept. 2000.

[83] S. Savage. Sting: a TCP-based network mea-surement tool. InUSITS, Oct. 1999.

[84] S. Savage, A. Collins, E. Hoffman, J. Snell, andT. Anderson. The end-to-end effects of Internetpath selection. InACM SIGCOMM, Aug. 1999.

[85] S. Savage, D. Wetherall, A. Karlin, and T. Ander-son. Practical network support for IP traceback.In ACM SIGCOMM, Aug. 2000.

[86] S. Seshan, M. Stemm, and R. H. Katz. SPAND:Shared passive network performance discovery.In USITS, Dec. 1997.

[87] A. C. Snoeren, C. Partridge, L. A. Sanchez, C. E.Jones, F. Tchakountio, S. T. Kent, and W. T.Strayer. Hash-based IP traceback. InACM SIG-COMM, Aug. 2001.

[88] N. Spring, R. Mahajan, and T. Anderson. Quan-tifying the causes of path inflation. InACM SIG-COMM, Aug. 2003. (to appear).

16

[89] N. Spring, R. Mahajan, and D. Wetherall. Mea-suring ISP topologies with Rocketfuel. InACMSIGCOMM, Aug. 2002.

[90] N. Spring, D. Wetherall, and T. Anderson. Scrip-troute: A public Internet measurement facility.In USITS, Mar. 2003.

[91] J. Stone and C. Partridge. When the CRC andTCP checksum disagree. InACM SIGCOMM,Aug. 2000.

[92] L. Subramanian, V. N. Padmanabhan, and R. H.Katz. Geographic properties of Internet routing.In USENIX Annual Technical Conference, June2002.

[93] H. Tangmunarunkit, R. Govindan, S. Jamin,S. Shenker, and W. Willinger. Network topolo-gies, power laws, and hierarchy. Technical Re-port CS-01-746, USC, 2001.

[94] K. Thompson, G. J. Miller, and R. Wilder. Wide-area Internet traffic patterns and characteristics.IEEE Network, 11(6):10–23, Nov. 1997.

[95] A. Tirumala, F. Qin, J. Dugan, and J. Fer-guson. Iperf. http://dast.nlanr.net/Projects/Iperf/, May 2002.

[96] M. C. Toren. tcptraceroute.http://michael.toren.net/code/tcptraceroute/.

[97] Y. Vardi. Network tomography: Estimatingsource-destination traffic intensities from linkdata. Journal of the American Statistical Asso-ciation, 91(433):365–377, Mar. 1996.

[98] B. Yao, R. Viswanathan, F. Chang, andD. Waddington. Topology inference in the pres-ence of anonymous routers. InIEEE INFOCOM,Apr. 2003.

[99] E. W. Zegura, K. Calvert, and S. Bhattacharjee.How to model an internetwork. InIEEE INFO-COM, 1996.

[100] Y. Zhang, L. Breslau, V. Paxson, and S. Shenker.On the characteristics and origins of Internetflow rates. InACM SIGCOMM, 2002.

[101] Y. Zhang, M. Roughan, N. Duffield, andA. Greenberg. Fast accurate computation oflarge-scale IP traffic matrices from link loads. InACM SIGMETRICS, June 2003.

[102] Y. Zhang, M. Roughan, C. Lund, and D. Donoho.An information-theoretic approach to traffic ma-trix estimation. InACM SIGCOMM, Aug. 2003.(to appear).

17