Network Management through Packet Annotation

Arne Baste David Chu Dilip Joseph George Porter

May 9, 2005

1 Introduction and Overview

Today’s networks support an increasingly sophisticated set of applications and network services. Atthe same time they must operate despite the sharp rise in “junk traffic”. This junk traffic includesworm attacks, port scans, and other malicious malware[22]. Additionally, new applications springup that radically change the host-to-host communication patterns, such as peer-to-peer systemsand VoIP. Operating throughout this traffic are traditional and important network services likeDNS and NFS. What is needed to ensure their proper operation is a protection mechanism thatensures that traffic surges, malicious, and “junk” traffic do not crowd them out.

To that end, we propose a distributed system of Inspection and Action points in the networkcalled iBoxes. These iBoxes operate on packet flows that arrive to a single input port to leave viaa single output port. The do not modify the path that packets take. Thus, iBoxes are orthogonalin functionality to routers and switches. Instead, they observe traffic flowing through them. Theseobservations can be cross-layer, in that they can include features of the IP, TCP or UDP, andapplication layers. iBoxes can record their observations into a new protocol layer called the Anno-tation Layer (or A-Layer). This A-Layer consists of “piggybacking” certain packet annotationson live traffic streams flowing through the iBoxes. This frees us from introducing new packets intothe network. iBoxes downstream are able to intercept the annotated packet stream and recoverthe data that they contain. Before sending them onwards, the annotations can be removed. Thisensures that endhosts receive packets exactly the way they were sent from the originating host.iBoxes do not modify the route that packets take in the network. Additionally, any changes thatare made to packets within the network are encoded in such a way that they can be undone. Thisbehavior lessens the impact of iBoxes to the end-to-end argument.

The A-Layer allows iBoxes to communicate their observations with one another. These dis-tributed observations allow for greater network visibility, which give iBoxes a vantage point tounderstand network behavior as a whole. iBoxes are also action points, meaning that they canrestrict, slow down, or fence bandwidth between competing information. Coupled with the greatervisibility gleaned from the A-Layer, this will allow for powerful service protection abilities.

This document describes the design of the A-Layer. We first present a motivating example ofthe A-Layer based on a DNS failure that hit the Berkeley network. We then outline related work.Next, we present a taxonomy of the types of annotations we envision utilizing in the A-Layer.Following that we describe how the task on network management can make use of iBoxes. Wethen present the design and security aspects of the A-Layer . Finally, we compare “piggybacking”annotations with new packet injection and develop an analytical model to study piggyback flows.

1

Figure 1: iBox placement to detect DNS surge

2 Motivating Example: Crippled DNS

In this Section, we describe how the December 2004 loss of service experienced at Berkeley couldhave been mitigated by an annotation layer approach, and in so doing, illustrate the concepts wewill present below. First we consider the case of an externally generated DoS attack against theDNS service and then we examine the alternate case of the misconfigured spam appliance.

Figure 1 shows the network topology, with iBoxes placed at the Internet, Access, and Serveredges of the network (labeled II , IA, and IS respectively). Through protocol-aware packet inspec-tion and statistics collection, IS detects an unusual increase in latency between the arrival of DNSqueries and their response. As other services are within normal ranges, it can pinpoint the problemto the DNS server. It exchanges information with II via the annotation layer to determine if thenumber of external DNS queries is high. If so, II negotiates with IS to slow these requests to alevel that insures that internally generated requests receive adequate service. Alternatively, IS canload balance requests to the Primary and Secondary DNS servers, redirecting external requests tothe latter while local requests are sent to the former. Such a strategy only works if indeed it is theserver rather than the server network segment that is the performance bottleneck.

Another theory for our DNS failure is that our third party spam appliance generated a largenumber of DNS requests to verify mail domain validity during an email flood. An unusual increase inemail traffic positively correlates with increased DNS latencies, thereby affecting the performanceof web access and network file system access in a completely unexpected way. Yet even in thisdifficult case for human diagnosis, the iBoxes can detect such correlations, pinpoint the email surgeas the root cause of the problem, and slow email delivery to regain control of the DNS service.

The scenario illustrates some points about network protection. Slowing email to restore DNSavailability is a policy decision that must be determined in advance, or presented to the networkadministrators for their adjudication. Observability also affects the network topology design. OuriBox placement makes it difficult to observe/act on the traffic between the spam appliance and themail and DNS servers. Had the network been designed so that orthogonal services were placed indifferent segments visible to iBoxes, the infrastructure could then detect the unusual number ofDNS queries originating at the spam appliance. Given sufficient protocol awareness, one possibleaction is to bypass the spam appliance altogether. A better alternative is to spawn a new DNSserver in response to the increased demand for domain name lookups, while redirecting to it someof the traffic load. Our administrators essentially did this by hand: our surge problems went away

2

when they dedicated a new DNS server to the spam appliance.

3 Related Work

Network management is a well-studied area. One of the most common protocols for managingnetwork devices is SNMP[24]. SNMP primarily enables network operators to obtain statistics andconfiguration information from switches and routers in a device-independed format. Additionally,SNMP allows commands to be sent to devices as well as “traps”, or notifications, to asynchronouslybe sent from a device when an error or exceptional condition occurs. Several products have beendeveloped to manage SNMP networks, including HP Openviews[25]. Openviews unifies a varietyof devices through one interface, and can be extended via numerous plugins to enable networkplanning and expansion.

Cisco’s NetFlows[21] is a system to centralize the collection of per-flow state kept in Ciscorouters. Much of the per-flow statistics gathering we propose in this document are based onthe statistics that Cisco routers keep. Whereas NetFlows enables a central monitoring point toperiodically tap into this data in an offline manner, our A-Layer proposal aims to encode this datainto the packet stream, allowing other iBoxes on the path to make use of it for traffic managementpurposes.

The Active Networks[27, 28, 26] effort was designed to extend the function of network devicesto enable new network services. Often these proposed services included more configurable routingand network-level multicast, however most proposals included network management applications.Our approach is different in that our A-Layer is invisible to end-hosts, and although we modifypacket contents to hold A-Layer payloads, those payloads do not modify the path or packets northe end-to-end semantics of each connection.

[23] proposes a network accountantability framework that uses explicit packet ‘obituaries’ totell a host in which Autonomous System (AS) its packets were dropped. Our system can be usedfor providing packet obituaries through annotations. Moreover, it operates at a finer granularitythan individual ASes and is not limited to providing only packet drop information.

4 Annotation Taxonomy

We now consider the types of annotations that the A-Layer will support. Recall that iBoxes areable to perform deep packet inspection on traffic flowing through them. This means that they cananalyze network, transport, and some application layer portions of the packet. The annotationsresiding in the A-Layer can represent classifications of individual protocols, or of combinationsof multiple protocols. Additionally, they can consist of “out-of-band” annotations that containiBox to iBox data not affiliated with one particular packet. We divide the annotations into threecategories:

4.1 Per-flow annotations

Per-flow annotations are markings attached to each packet in a flow that contain information aboutthe flow as a whole. These include:

1. The flow start time

3

Figure 2: Three-part A-Layer annotation taxonomy

2. The number of packets/bytes sent in this flow so far

3. The minimum/maximum packet lengths of this flow

4. The number of packets dropped or lost in this flow

The information listed above is currently captured by Cisco’s Netflows[21] system. By anno-tating packets with this Netflows information, iBoxes downstream could infer path and sessionstatistics. Possible uses include protecting flows that have suffered packet losses and protecting“old” flows during times when the majority of network traffic is recently spawned worm traffic.

4.2 Per-packet annotations

Per-packet information can be further subdivided into two parts. The first part represents featuresof the packet such as the 802.1q VLAN tags. The second part represents application protocolfeatures that are usually only available in the first few packets of the flow such as HTTP URLs andCookies. We summarize some of the flow information below:

1. 802.1q VLAN tags

2. HTTP Cookie

3. HTTP URL

4. DNS name

5. SSH version number, protocol

6. SAN LUNs and block IDs

4

7. NFS volumes and mounts

Part of a solution to the failing DNS example in Section 2 would have been to annotate packetsbased on 1) what iBox first saw the packet, and 2) whether the requested name was in the Berkeleydomain. In the former case, the lack of annotations would have indicated both that the surge inDNS requests were not coming in from outside the network via the ISP link, nor was it comingfrom the clients on the access network. In the latter case, the majority of resolved names wouldhave been for non-Berkeley names, since the resolutions were driven by the To and From fields ofprimarily spam-email. Although neither of these facts point to a malfunctioning spam appliance,they provide much needed visibility that could have drastically reduced the time-to-detect of theproblem.

4.3 Out-of-band annotations

In addition to carrying data that is computed directly as a function of the packet’s contents, theA-Layer can contain out-of-band information that iBoxes need to exchange with each other in adistributed way. While the iBoxes could send this information along in a separate data stream, thereare certain advantages to “piggybacking” the data onto streams heading in the desired direction.Principally, by annotating packets iBoxes can communicate with very heavily utilized parts of thenetwork without injecting new packets. Normally during periods of heavy congestion, it becomesincreasingly difficult to maintain sessions. However, by annotating packets, iBoxes should be ableto deliver data to congested parts of the network.

We envision that out-of-band data will consist of:

1. SNMP control data (both commands and measurements)

2. The ID of iBoxes along the path that have seen this packet

3. The number of flows seen by an iBox at a particular part of the network

4. The number of bytes seen by an iBox at a particular part of the network

5. Information on protocol mixtures seen at various parts of the network

4.3.1 Knowledge Provided by the A-Layer

Based on individual items from the above taxonomy, as well as combinations of items, we can gleansubstantial information about network flows travelling through the network.

Consider an iBox A in the network that is observing several flows transiting through it. If someof those flows originate from another part of the network that is iBox enabled, then A will have newcontext that is difficult or impossible to reconstruct locally. For example, when a packet arriveson an interface to the iBox, it can lookup context for that packet, and know how old the flow is,what type of packet drop rate this flow has seen in its lifetime, and what part of the network thispacket originates from (even in networks where address spaces are not assigned hierarchially, suchas wireless LANs). Depending on the type of application, the iBox would also have informationabout the flow such as what type of DNS request it represents. In the case of web traffic, the iBoxwould know whether the request represents a premium webpage or a value-added portion of the site.It would have this information even if it didn’t have protocol-specific decode knowledge. As long

5

as another iBox in the path was able to extract that information and encode it as an annotation,iBox A can benefit from that knowledge. Identifying which annotations will prove the most usefulto network operators is important ongoing work.

5 System Management Architecture with iBoxes

The primary goal of the A-Layer is to provide extensive diagnostic and management capabilities.Before we discuss the embedding of iBoxes in the network, let us recall the typical AS networktopologies. A typical ISP network consists of multiple routers each directly connected to two ormore subnets. To forward various address prefixes appropriately, administrators either manuallysetup forwarding rules or employ intra-domain routing protocols such as OSPF and RIP. Theserouting protocols propagate local information to establish domain-wide address prefix forwardingrules.

In contrast, a switched enterprise network consists of few routers, and relies on a distributiontier to serve clients, servers, and external access to ISPs. Switching decisions are accomplishedeither by broadcast ARP request/responses (e.g. for clients) or by manual mapping of Ethernetaddresses to switch ports (e.g. for servers).

These admittedily simplified topology descriptions do not nearly give justice to the degree ofvariation present in existing networks. Yet despite these extremes in topology between ISP networksand enterprise networks, we shall describe an A-Layer that is equallly applicable to both.

5.1 iBoxes Within the Network

For ISP-style networks with substantial numbers of routers, we imagine positioning iBoxes at theingress and egress points of the network, with additional iBoxes positioned at the boundaries ofmeaningful zones of interest. For example, these include the borders of OSPF areas or at theentrance/exit to a particular geographic region. For switched enterprise networks, iBoxes naturallyalign themselves at hierarchy boundaries. For example, in the network topology shown in Figure 1,separate iBoxes in front of the clients, servers and external internet are initial placement candidates.

In both the ISP and Enterprise network, each iBox resembles a 2 port line card with an IPaddress, similar to currently available network monitoring cards or traffic shappers [18]. TheiBox only reacts to packets that are addressed to it in the A-Layer annotation, irrespective of theIP-layer destination address. At the extreme, network device makers may come to incorporateA-Layer functionality into each port on their devices. We note that most modern switches containan N+1th management port accessible by an IP address. An A-Layer compatible interface mightreside on this port. These may provide lightweight iBox functionality i.e. a basic set of generallyuseful annotations.1

The remainder of this document focuses on iBox deployment within the enterprise network.Enterprise networks are service-oriented in the sense that they focus on providing messaging ser-vices, corporate web portals, data storage, etc, to clients, whereas ISPs are packet-oriented and areless concerned with the particular content of the data they deliver. As such, enterprises are morelikely to employ iBoxes for service-specific annotations. In addition, Labovitz, et al. [19] indicatethat ISPs have much more free bandwidth than Enterprise networks. It is typically the enterprise

1In fact, we observe that VLAN tags supported by modern switches do resemble a form of annotation, albeitlimited and function-specific.

6

network that is resource constrained. Enterprises are more likely to appreciate the availability ofA-Layer enabled applications (in the face of massive data flows) and economy of the A-Layer design(in processing and bandwidth overhead).

5.2 Discovery

Network administrators derive minimal utility from iBoxes in isolation. Each iBox needs to learnthe identity of other iBoxes in the network. Since network administrators deploy iBoxes, we mayreasonably assume that iBoxes are long-lived, stable, and managed network elements. This leadsus to suggest that an iBox is required to register through a central registry upon introduction intothe network. Indeed, our iBox authentication process already requires periodic interfacing with acentral registry (7), and we leverage it to provide a list of iBoxes in the network upon request.

Equally important, each iBox needs to learn how to route to other iBoxes. Let us recall thatrouting at the A-Layer does not occur in the traditional sense of packet routing. Rather, routingpredominately occurs passively by annotating existing packets (We will describe situations whereinjected packets are necessary in Section 5.3). This means that an iBox must look at charactisticsof the packet stream, e.g. srcIP and dstIP, and annotate those packets likely to be routed to otheriBoxes. An iBox X’s discovery process of iBox Y is consequently that of learning which (srcIP,dstIP) tuples will likely traverse through Y. One option is to allow network admininistrators tostatically and manually setup each iBox with the appropriate entries to reach all other iBoxes. Thisis attractive when the network contains few iBoxes and iBoxes cleanly partition the network intoaggregatable IP address ranges. For example, in Figure 1, iBox IA resides in front of the clients,all of which are assigned addresses in the range 10.0.0.1 - 10.0.0.100. Then for either iBox IS or II

to communicate with iBox IA, the administrator can configure IS and II to annotate those packetswith dstIP in the appropriate range. Next, we describe an automated discovery protocol, AutoDisc,which achieves this same functionality without administrative intervention.

To alleviate the chore of operator configuration, an iBoxparticipates in the AutoDiscprotocol.This gossip protocol disseminates changes in iBox to iBox reachability quickly while maintaining alow overhead in the stable state. In particular, discovery is fast. It is inspired by the Trickle protocol[20]. Upon introduction into the network, an iBox X immediately proceeds to annotate a certainpercentage pmax of all packets, including in the annotation a HELLO message and its own iBoxIPaddress. A reasonable value of pmax is 1 to 0.5. When another iBox Y receives these annotatedpackets, Y knows that it can reach X on the link which the annotated packets came in on (Thisis very similar to how swtiches currently forward IP packets after caching ARP request/responses.Y then appends its own IP address and HELLO message annotation and IP address and forwardsthe packet along the appropriate egress link. We discuss the specifics of how these annotationsare stacked in Section 6. Y also begins to generate HELLO messages of its own, also applied topmax of all packets. As these annotated HELLO packets circulate the network, each iBox learns,relative to its topological position, which (srcIP, dstIP) pairs reach which iBoxes. After a while,these HELLO messages do not communicate any new information to the iBoxes. As the rate ofinformative updates subside, iBoxes decrease the rate at which they add HELLO annotations,eventually adding HELLO annotations to only pmin percent of packets. pmin should be 0.1 to 0.01.iBoxesexpire these iBox path entries, such that after a timeout an iBox that fails to hear a routeto a particular iBox it previously heard from evicts that path.

AutoDisc is responsive to changes, yet of a manageable overhead. A disadvantage of AutoDiscisthat the percentage of packets annotated, pmax to pmin, will be dropped by end hosts that receive

7

Figure 3: End to end delivery of an annotated message

them. However, this percentage is appropriately small as to cause minimal disruption to traffic.However, after this discovery process, an iBox will learn which ports do not reach subsequentiBoxes and hence require removal of annotations for hosts to process. As a tradeoff to annotatinghost-generated packets, an iBoxmay choose to generate annotated dummy packets with srcIP anddstIP matching those of host-generated packets. This allows the network to not lose any legitimatetraffic in exchange for introducing a small amount of dummy packet traffic.

5.3 Issuing Queries and Commands, Receiving Responses and Acks

Having installed iBoxes in the network, the network administrator accesses the iBox networkthrough A-Layer compatible programs on a regular end host. This may be accomplished througheither per service proxies or through services written natively for the A-Layer. Figure 3 illustratesthese options. In the following examples, we work through the design of two services adapted touse the A-Layer.

Example: Per flow attribute annotations The A-Layer is well suited for annotations derivedfrom the flow in which the packet belongs. Example attributes are flow start/end time, flow octetcount, min/max TTL, min/max packet length, etc. The network operator community has begunto standardize on these attributes in the IPFIX Working Group, and associated series of InternetDrafts [18].

Packets annotated with these flow attributes allow in-network real-time decisions on the servic-ing of various packets. For example, a network operator may first request a view of the current flowlifetimes of all packets headed towards the web servers. For the moment, let us consider the casewhere the operator has direct physical access to an iBox. Here we ask the question: How does one

8

iBox disseminate a query to the iBox network? The same technique is applied to disseminate a com-mand. Query dissemination is actually straightforward given the the AutoDiscprotocol describedabove. The initiator iBox begins to mark a limited number of packets known to be destined foreach of the other iBoxes. In the example Figure 1, iBox IA marks packets known to be heading foriBoxes II and IS . For II , it would mark packets with dstIP not in the subnet 10.0.0.0/255.255.0.0.For IS , it would mark packets with dstIP 10.0.0.101 - 10.0.0.255. It would specifically not markpackets with dstIP address 10.0.0.1 - 10.0.0.100. The annotations for II and IS would in effect say:“Send to me network flow signitures and lifteimes of all flows destined for the webserver”. WhenII and IS receive these annotations, they would begin saving the requisite flow information andforward them on to IA (again, also by annotating packets known to be heading towards IA).

We now revisit the issue of establishing the initial communication with the initiator iBox inthe case that the network operator is not directly connected to the initiator iBox. We address thisfirst mile/last mile problem by allowing end hosts to mimic A-Layer functionality. To do this, theoperator’s client machine begins to annotate and send out dummy packets into the network. Thesedummy packets contain no IP payload and their dstIPs are immaterial as long as they traversethrough at least one iBox. An iBox which receives these dummy packets processes the annotationsbut does not forward on the packet. In the case of our current example, the network operator’shost A sends the annotated dummy packet to IA, which processes it accordingly. This is verysimilar to communicating to a known management port with today’s network elements. Since it isultimately the network operator host that wishes to view the flow lifetime information, the initiatoriBox X records the originator of the query and forwards responses from II and IS also by annotated(dummy) packets.

This first mile/last mile suffers from the same problem of network congestion that existingsolutions (e.g. SNMP over UDP) experience. However, we observe that the host need only succeedin one attempt to the closest iBox in order to successfully transmit a message. After this, theinitiator iBox is free to annotate packets of all/any other srcIP (e.g. host B, C, etc.). Lastly, wenote that these iBox-hosts participate in the security protocol (Section 7) just like regular iBoxes.

Example: SNMP annotationsThe natural next operation for a network operator or automated network monitor is to issue

commands to the network. We leverage SNMP, a widely deployed and supported network manage-ment protocol. The goal of the iBox network is to deliver SNMP datagrams, traditionally deliveredover UDP, to network elements. Figure 3 shows the steps involved in the end-to-end delivery. Justas in previous example, we employ annotated packets to contact the iBox closest to the networkelement. In the case the network element understands the A-Layer, we have no further target - therouter or switch simply returns the response in the form of annotations of packets flowing in thereverse direction.

Yet we maintain compatibility with the wide number of existing non A-Layer speaking networkelements by use of a last mile proxy. Each iBox, upon receiving an SNMP request annotation, canlocally test whether it is the last iBox on the path to the network element. This is done with theroute discovery information learned from AutoDisc described earlier. If it is not the last hop iBox,it forwards on the request. If it is the last hop iBox, it initiates a normal SNMP UDP requestand awaits a normal UDP response from the network element. Upon reception of the response,the iBox wraps the responses into annotation units and awaits to apply these to packets flowingtowards the initiator iBox.

9

Figure 4: A-Layer field structure

5.4 Passive vs. Active packets

In our discussion so far, we have mentioned several instances in which iBoxes communicate viaannotations on existing flows. We call this passive communication. We have also discussed instanceswhere it is necessary to either send annotated dummy packets (when connecting with an initiatoriBox) or proxy to send regular UDP or TCP packets (when connecting on the last mile with thelegacy network element). We call this active communication. Our design strives to use passivecommunication to the extent possible, for it provides availability even in the face of massive trafficsurges. Yet an inherent limitation of passive opportunistic communication is that latency is notoptimal. We can easily remedy this if we permit a mix of passive and active packets. When thenetwork is underutilized, we can use active packets to expedite responses without congesting thenetwork. However when the network is saturated, we view passive piggy-backed communication asan expedient means to “get through.”

6 Design of the A-Layer

Conceptually, the A-Layer lies between the network (IP) and transport (TCP) protocols. Thus, itcan be considered a “Layer 3.5” protocol. However, we chose to implement the annotation operationby adding the field to the end of the packet. The reason for this is to avoid unnecessary copying iniBoxes. In our scheme, new annotations are added to the end of the packet each time the packet isannotated. If we chose to include A-Layer annotations after the network field, then the transportportion of the packet would need to be copied each time. Section 6.3 demonstrates our choice tomake stacking annotations a cheap operation, involving modifying only a few fields.

6.1 Structure of the Annotation

An A-Layer annotation is a set of 36 bytes added to the end of the packet. This region is dividedinto the following seven fields:

Prior protocol: When a new annotation is added to a packet, the value of the packet’s IPProtofield is copied here. This allows the field to be restored when the annotation is removed.

Type: Annotations can be of multiple types (see Section 4). This field represents the type ofannotation. Lacking a standard for the assignment of type values, this field can be defined strictlyon a per-AS level.

Authentication Field: The authentication field consists of the lower 80 bits of a secure,

10

keyed H-MAC to prevent endhosts and malicious intermediaries from introducing fraudulent A-Layer annotations. For more information see Section 7.

Sequence Number: The sequence number field is used both to detect A-Layer loss as wellas to enhance security. In the former case, each iBox keeps a counter for every other iBox asto what the current sequence number is. Each time an A-Layer annotation is sent to anotheriBox, the sequence number is monotonically increased. Thus, sequence numbers are per iBox pairs.Additionally, they enhance security by ensuring that replay attacks are not possible, since differentsequence numbers change the value of the authentication field even if the contents of the A-Layerare otherwise the same.

Destination Address: This field can either contain the address of an iBox for which this anno-tation is destined, or it can contain the wildcard address. Annotations with a wildcard destinationaddress are examined by any iBox on the path that is interested in them.

Source Address: This field contains the address of the iBox that initially set this annotation.Payload: This field varies depending on the type of the annotation.

6.2 Method for Annotating and De-annotating

For an iBox to add an annotation A, of length Al, to a packet P , it performs the following operations:Annotation

1. P ’s length is increased by Al bytes by adding Al to the IP header’s length field

2. The annotation A is copied to the end of the packet in the newly created space

3. The value of the IP header’s protocol field is copied to the first byte of the annotation

4. A special value of 200 is copied into the IP header’s protocol field. This signifies that thispacket is annotated

5. The IP header’s checksum field is recomputed

To de-annotate a packet, the following actions are performed:De-annotation

1. The IP header’s protocol field is replaced with the first byte of the outermost annotation.This restores it to its original value

2. The IP header’s length field is reduced by Al

3. The IP header’s checksum is recomputed

6.3 Stacking Annotations

A side effect of the annotation scheme outlined in Section 6.2 is that multiple annotations may beadded to one packet automatically. Each time an annotation is added, the information required to“undo” the packet marking are stored in the annotation itself. As annotations are removed fromthe packet, this process is reversed until the original packet remains.

In order to ensure that no one iBox can reserve all the available annotation space, we requirethat iBoxes can only apply a fixed number of annotations at one point.

11

Figure 5: A-Layer IP Packet Annotation

12

Figure 6: A-Layer address format

6.4 iBoxAddress Space

iBoxes can address each other using A-Layer addresses. An A-Layer address is a 32-bit unsignedvalue that has a two-level hierarchy. The upper 16-bits represent the AS number the iBox residesin. The lower 16-bits represent the ID of the iBox. This address is assigned to the iBox using anout-of-bound mechanism such as extended fields in DHCP or statically at initialization time.

7 Security

7.1 Goals and Challenges

The need for security in the Annotation Layer is important and we have defined our main securitygoals as the following (in prioritized order):

1. First and foremost, provide a scheme that guarantees that only genuine AL units are inter-preted as AL units by any iBox(i.e. prevent spoofing of AL units).

2. Do this in a fashion so that both issuing and authentication of a genuine AL unit by an iBoxis computationally fast (i.e. we can not allow this to be a bottleneck).

3. Do this in a fashion so that we introduce minimal overhead (i.e. reduce the length of thesecurity related fields in the AL unit to a minimum).

4. Try to comply with existing security standards as much as possible. This will ease our owndesign efforts and build confidence around the scheme.

As the first security goal describes, our main goal is to provide a scheme that allows authenti-cation (and not encryption) of all the AL units that will constitute the communication channel inthe A-Layer. There are several techniques that are used to authenticate messages; below we willdescribe three well-known ones:

1. Encrypting a message digest (the digest is the output of a secure hash algorithm with themessage as the input) with a shared secret key using conventional encryption. Other partieswill then verify it by deriving the same digest and checking it using the same shared secretkey. This scheme requires secretly distribution of the shared secret key in advance.

2. Signing a message digest (the digest is the output of a secure hash algorithm with the messageas the input) with your own private key. Other parties will then verify the signature withyour public key. This requires an existing PKI.

3. Concatenate a secret value to your message and then run it through a secure hash algorithm(in some fashion) to produce a Message Authentication Code (MAC). Other parties sharingthe same secret value will simply just do the same in order to verify the MAC. This schemerequires secretly distribution of the shared secret value in advance.

13

All methods need to run the message through a secure hash algorithm, but methods 1 and 2also requires you to run the digest through (processing consuming) encryption functions. Givena choice between these three methods and guided by security goal two we choose the secret valuemethod.

But, in order to implement this scheme we have to build some extra infrastructure that willsupport a secure exchange of a new secret value on a periodic basis. This is motivated by severalreasons. First, given the amount and propagation of iBoxes, we are exposed to losing control ofan iBox and thus compromising the shared secret value. Second, the enormous amount of packetsflowing through an iBox (and thus potentially AL units) in a given period of time will force usto have an extremely large sequence number space. If we were not to implement a large sequencenumber space we would open ourselves up to replay-attacks each time we would start reusing oldsequence numbers (as long as we were still using the same secret value). Last, but not least, whenimplementing the secret value scheme, it is recommended to change the secret value often in orderto reduce the chances for a malicious third party to derive the secret value in a brute-force manner.

Given these goals and challenges we will, in the remainder of this section, describe a schemethat meets them all. First, in Section 7.2, we will do a brief security analysis using the STRIDEThreat Model framework. In Section 7.3 we describe the AL Security Infrastructure that is neededin order to support a secure and fast authentication scheme using secret value authentication.Section 7.4 takes a look at the details around the use of secret value authentication, introduces anacknowledged standard called HMAC and tries to justify our choices of the different parametersaffecting the level of security. A closer look at the processing overhead introduced with this schemeis taken in Section 7.5.

7.2 STRIDE Framework

The STRIDE Threat Model[12] is a framework that can help us identify vulnerabilities and potentialattacks in our security scheme. We will in this section describe our scheme within this framework.STRIDE is an acronym for the taxonomy of security threats a security scheme may face and consistsof the following six threat categories.

Spoofing identity / Tampering with dataThis is our number one priority. We have to make sure that no entity outside our security

infrastructure is able to create forged AL units. This is accomplished by making sure that thesecret value is kept secret and that we are able to securely replace it before we run out of sequencenumbers. But even if we do this correctly we can never avoid the situation where an attacker canguess the correct Authentication Field, thus we will have to mitigate this particular threat to anacceptable level. This is all further discussed in Section 7.4.

RepudiationWe are not able to meet this threat through our security scheme. Given that we are using a

shared (symmetric) secret value it is possible for an iBox to spoof an AL unit so that it looks likeit is coming from another iBox. As a consequence we cannot avoid non-repudiation issues, as wecant prove which iBox really issued the AL unit. Thus, we simply have to trust the iBoxes.

Information disclosureWe decided that the A-Layer should not provide confidentiality due to the substantial process-

ing overhead that would be introduced with encryption. Also, we did not see the big need forconfidentiality in a network management environment like ours. All information that is availablein the headers and the payload of the AL units is thus disclosed. That said, it is not a problem for

14

any protocol using the A-Layer to encrypt its messages. In such a case, it is important to rememberthat the AL unit headers still will be exposed to the public.

Denial of serviceAfter dealing with spoofing of AL units, this is probably the biggest threat against our scheme.

It is easy to envision that malicious attackers could overload an iBox by forcing it to authenticatetoo many AL units. Also, if DDoS attacks are launched against other entities in the securityinfrastructure that the iBox relies on (in order to update to a new secret value) our whole securityscheme could be paralyzed. So dealing with this aspect is very important and it is something thathas to be considered thoroughly in the final design of the iBox and the other entities in our securityinfrastructure. Some of these threats and techniques to mitigate them are discussed in Section 7.3and Section 7.5.

Elevation of privilegeTo avoid this threat we have to make sure that an iBox is able to withstand intrusion. This

way we don’t elevate an outsiders privilege by giving him access to the secret value that is stored inan iBox (or by other means give the intruder an opportunity to trick the iBox into authenticatingspecific AL units upon request).

7.3 Annotation Layer Security Infrastructure

Given the challenges described above, we have come up with a scheme in which we securely replacethe shared secret value at all the iBoxes once every 24 hours. This will allow us to reduce thesequence number space to a finite number and it will also contain the loss of a shared secret valueto between 24 and 48 hours. Also, by changing the shared secret value every day we make it harderfor a malicious attacker to derive the secret value since he or she then has to start over again every24 hours. To accomplish this scheme we have to rely on existing methods that are used in standardPKIs, and we also choose do the distribution of the new secret value in a centralized fashion (usingexisting directory servers). We assume that a PKI is in place to ensure a secure distribution channelfor new shared secret values. The infrastructure that is needed in order to support our scheme iscalled the Annotation Layer Security Infrastructure. In the remainder of this subsection we willdescribe the different entities that are involved in this Annotation Layer Security Infrastructureand their security behavior.

7.3.1 AL Certificate Authority

This entity serves as a standard Certificate Authority. It can be implemented as a certificate-signingapplication that runs on an off-line secured machine. This application will sign X.500 certificatesfrom each of the entities involved in the AL infrastructure (either iBoxes or AL Security Servers)with its own private key. Each of the entities involved in the AL infrastructure will have the publickey of the AL Certificate Authority installed (upon delivery) in their software for verification ofother entity’s certificates (and thus their public keys). It is assumed that the transfer of theunsigned certificate to the AL Certificate Authority happens in an off-line secure manner and thatthe origin of that unsigned certificate can be verified (E.g. the system administrator brings theunsigned certificate from one iBox to the AL Certificate Authority on an USB-disk).

15

Figure 7: The flow of certificates in the AL security infrastructure

16

Figure 8: The behavior and interplay of the AL Security Server and the iBoxes

7.3.2 AL Security Server

This server has one main purpose; to help out in making sure that all iBoxes securely get hold ofa new secret value each day in a verifiable manner. In the initial phase this server has to get holdof the public keys for all the iBoxes in the AL infrastructure, by querying the DNS server for allof the different iBoxes ’ certificates. It will also publish its own certificate to the DNS server. TheAL Security Server is an independent server with an internal synchronized clock that determinesits mode of operation based on where in a 24 hour cycle it is. A point here is to put the changingof secret values time to the most inactive time of the day. This point in time will mark the startof our relative numbering of the hours in the 24 hour cycle. We also consider more than one ALSecurity Server to increase redundancy. We will need to let one of them be master and pick thesecret value, while the rest to be slaves.

7.3.3 iBox

The iBoxes must also participate in the Annotation Layer Security Infrastructure in order to gethold of their new shared secret value every 24 hours. Under normal circumstances, the iBoxeswill contact a local directory server once a day and query it for their own record (e.g. queryit for servertier.ibox.berkeley.edu) and then verify and extract the signed and encrypted messagecontaining the new secret value. In the initial phase all the iBoxes also need to post their ownpublic key (i.e. certificate) to the directory server and retrieve all the other entities’ certificates.

7.3.4 Behavior and Interplay – Best Case Scenario

The interplay (that leads to the distribution of a new shared secret value) between the AL SecurityServer and all the iBoxes is best described by Figure 8.

Due to the lack of synchrony between all the iBoxes we have to introduce overlapping timeperiods around 00:00 that lasts for two minutes (this means that no iBox should be more thanone minute out of synchrony with the correct time). In this overlapping time period, any iBoxwill verify an AL unit one extra time (with either the new secret value if it hasn’t changed yet,or with the old secret value if it just changed) if the first authentication failed. This could be a

17

vulnerability to attackers (due to the double processing of invalid AL units), and thus we want tominimize this time period as much as possible.

7.3.5 Temporary Unavailability of the Directory Service

There are two scenarios in which our updating scheme (within a 24 hour period) will fail given anunavailability in the AL Security Server or the directory service:

1. The AL Security Server stops working or it is unable to upload all of its records (and thusunable to update the counter) to the directory server in the given time period.

2. The directory servers goes down during the updating hour (01:00-02:00) so that one or moreiBox can not get hold of the new secret value (or read the global counter).

If only the first case is happening all the iBoxes will read that the global counter is on the samelevel as last time and thus infer that the AL Security Server is down or that it could not uploadall the records to the directory server. Based on this, all the iBoxes will know that they shouldnot change to a new secret value since not all the new records are available in the directory server.We will show that for a certain period of time it is permissible for iBoxes to be left without a newsecret value. By rate limiting the number of AL units from one iBox to another to maximum 4 permillisecond (on average), we can work around this problem. Since we have reserved 32 bits for thesequence number we have almost 4.3 billion numbers, and if we do not send more than 345,600,000AL units a day from one iBox to another iBox (i.e. per connection per direction) we can run fortwelve consecutive days before we run out of sequence numbers. This means that we can go on fortwelve days without needing to change from an old secret value to a new one without compromisingsecurity. Twelve days is a long enough period of time that we can assume that the AL SecurityServer will be fixed and that it also will be able to upload its new records to the directory serverwithin the time period. So in terms of protocol behavior, the iBoxes will try to update the secretvalue each day, but if the new records are not available they will just resume on their sequencenumbers and try again the next day. The AL Security Server (if it works) will simply repeat itsown behavior for the next 24 hours regardless of whether or not it successfully connected to thedirectory server.

In the second case, some iBoxes will not be able to infer whether or not all the records areupdated because they can not communicate with the directory server. They also can not inferwhether or not everyone else experienced the same (e.g. all the iBoxes that contacted the directoryserver before a directory server DDoS attack would have their records and think that everything isokay). The solution here is to try again and again (in the time period from 02:00 through 23:59)until they are able to establish contact with the directory server and download the secret value. Ifthey are not able to do so in the 22 hour window that is available, they will simply be cut off fromissuing and reading AL units (they can still remove them). They will be cut off from the momentall the other iBoxes changes to a new secret value, until the particular iBox is able to contact thedirectory server again. We will keep a copy of the previous day’s records in the directory server toallow iBoxes that have been “left in the blind” to recover.

7.4 The Authentication Field

As shown in Figure 4, we see that we have a 32 bit Sequence Number field (to prevent replayattacks) and a 80 bit Authentication Field that authenticates the whole AL unit. These fields

18

are our main defense against malicious entities trying to spoof our AL units. Briefly speaking, weguarantee authenticity of an AL unit by running a function over the secret value, the header fields,and the payload. The resulting output is then put in the Authentication Field and the AL unit issent off. The receiving iBox can authenticate the AL unit by performing the same function andcomparing the result with the Authentication Field, and the security relies on the assumption thatno one else (besides the entities that holds the secret value) can produce such correct AuthenticationFields.

MechanismTo accomplish this we will use a function called HMAC-SHA1-80. This function is simply

the HMAC construction (described in RFC 2104) using the SHA1 as an underlying secure hashalgorithm and then truncating the (originally 160 bit) output to 80 bit.

The HMAC construction is adapted by the IPSEC working group of the IETF[13] and is con-sidered a secure way of constructing a Message Authentication Code (MAC) using a secure hashalgorithm and a secret value. The rationale for developing HMAC is the observation that securehash algorithm isn’t keyed primitives, and that simply concatenating a secret value in front of themessage and then running this through a secure hash algorithm isn’t good enough. Further, giventhat the underlying secure hash algorithm is secure, it can be showed that the HMAC constructionproduces secure MACs[13].

We use a secret value of length 80 bit (10 bytes) that consists of a random bit pattern. Accordingto FIPS PUB 198[14] the secret value should not be less than half the length of the output of thesecure hash algorithm. Our secret value is exactly half the length of the original output, so we arejust complying with that requirement.

Analysis of the strength of our authentication schemeGiven that we plan to deploy our Annotation Layer in high speed networks we have to take the

following two threats into considerations:

1. Attackers try to spoof as many AL units as they can onto every IP packet they get hold off bysimply trying to guess the correct Authentication Field (without further analysis one couldimagine that this could work due to the enormous packet throughput an iBox faces).

2. Attackers are trying to derive the secret value based on all the available AL units that arefloating around in a brute-force-manner (this could also be possible on first thoughts since wearen’t using encryption and thus all our plain texts are known to the public)

The only two factors that influence the first threat is the length of the Authentication Fieldand the throughput of AL units that one iBox observes. To understand that the first threat is nota problem, assume that a malicious attacker would be able to force an iBox to authenticate onemillion AL units (with a randomly selected, but not repeated, Authentication Field) per second.Then the attacker would have a chance of 7.1 x 10−14 % to guess a correct Authentication Fieldin the course of twenty-four hours. On the other hand, if our Authentication Field were only fivebytes (40 bits) such an attacker would have a change of 7.9 % in the course of twenty-four hours,something which would be unacceptable.

The second threat is the interesting one and its impact relies on the following factors:

1. The strength of the SHA1 algorithm

2. The availability of plain texts

19

Figure 9: Worst-case scenario analysis of a potential brute-force attack

3. The length (and randomness) of the secret value

4. The computational power that the attacker(s) possesses

According to RSA Security[15] (dated March 11, 2005) they state that there has been discovereda potential weakness in SHA1, but they also write that the security of MACs and message digestsis not affected by these attacks. So in our case the SHA1 is still a strong secure hash algorithm.

Since we are not using encryption we can not hide our plain texts. This gives the attacker apossibility of running a brute-force-attack against us by simply trying out one secret value afteranother and see whether or not he gets the correct Authentication Field.

We have a secret value of length 80 bits and we assume that new secret values are generated ina completely random fashion.

To get an understanding of how successful such a brute-force attack could be we will have toestimate the computational power that a potential attacker would possess. This is hard, so toarrive at a number we have taken a starting point at the famous EFF DES Cracker machine. Thismachine (built in 1998 at a cost of $250,000) could process 88,804,000,000 keys/sec[16] and witha plain text of 88 bytes/key this machine had a DES processing capacity of 7,452,728 MB/sec. Ina speed comparison study between crypto algorithms (in software) we can infer that SHA1 hasa throughput that is 3.19 times higher than DES[17] . So under the assumption that the sameis true in hardware we could imagine a SHA1 Cracker that could process at a rate of 23,774,202MB/sec. Given that our authentication scheme uses a HMAC-SHA1-80 (that needs two runs withthe SHA1 algorithm) with a two 64 bytes block input we could further imagine a HMAC-SHA1-80Cracker that could test almost 100 billion secret values a second, or test one secret value every 0.01nanosecond. (These calculations assume that the additional overhead of the HMAC constructionis negligible.) This processing power will be what we in our worst-case scenario analysis assumethat a malicious third party could obtain. Our worst-case scenario is showed in the Figure 9.

As you can see from this analysis an attacker will at the present only have a 0.00000071 %chance to derive the secret value in a 24 hour period (or once every 383,348 years). We have alsoanalyzed the strength 20 years into the future (under the assumption that the computational powerof the attacker doubles every 18 months) and the probability to succeed in the future with such anattack is still low, about 0.0074 %. Under these assumptions it is fair to say that our authenticationscheme is highly secure against the two types of attacks described.

7.5 Feasibility Study

We will try to answer the question concerning the on-the-fly processing overhead (that the iBoxesface for each AL unit) that is introduced with our security scheme. And we will also come upwith some protocol behavior that will reduce our vulnerabilities for spoofing attacks. As a point

20

of reference, [17] shows that 68 MB/sec throughput for SHA1 on a Pentium4 2.1 GHz is possible.This equals about one authentication each 3.59 s (under the same assumptions as in the previoussection) or about 275,000 authentications per second.

To understand what HMAC-SHA1-80 processing capacity the iBox should possess, let us con-sider the following scenario: Assume that we need to support the throughput of a 1 gigabit link( 1,000,000,000 bits per second or 125,000,000 bytes per second) and that we only receive IP pack-ets of length 424 bytes (64 bytes original length + 10 AL units of length 36 bytes), each of whichhas 10 AL units annotated to it. Then the iBox would need to be able to authenticate almost3,000,000 AL units per second, and in addition it needs to calculate the Authentication Field of itsown outgoing AL units. If we assume that it never needs to send out more than 300,000 AL unitsper second (that equals maximum allowed sending rate to 75 other iBoxes) we arrive at a totalof 3,300,000 authentication operations that needs to be performed each second. That is a scaleup from the software example above of a factor of 12. This sounds feasible, but might result in aspecial made expensive hardware unit (with parallel SHA1 processing units) that is specialized inperforming a two blocks SHA1 function under the HMAC-SHA1-80 construction.

The main point of this section is simply to show that it is feasible to support our authenticationscheme within one iBox, but that it might involve some additional cost in acquiring a specializedhardware unit.

In an attempt to force an iBox to waste resources on authenticating false AL units it is not hardto imagine a malicious entity that will spoof up to 10 AL units on every IP packet it gets it handson. To counter this we will remove (without authenticating) all the remaining AL units in an ALstack when we discover the first false AL unit in a stack. (Some people might say that this couldbe a way for an attacker to force us to drop correct AL units, but if this entity was in a positionto spoof one or more AL units, he could might as well have removed the correct AL units in thefirst place.) This iBox behavior will limit an attacker to force an iBox to, at most, authenticateone false AL unit per packet, even though we enable stacking of up to 10 AL units per IP packet.

8 Big versus Many

Annotating a packet increases its size. The following question naturally arises : Won’t it be betterto simply send a new packet instead of adding an annotation to an existing packet?. In this section,we utilize publicly available router/switch benchmark data to explain the tradeoff between largerpackets and more number of packets.

In our design, a single annotation (called AL-unit) increases the size of a packet by 36 bytes.A larger packet takes more time to flow through the internal circuits of the router/switch as wellas through the links between them. But sending a new packet involves operating system overheadslike allocating memory for the new packet and creating the IP header. In the case of annotations,the existing IP header of the packet is used. Each new packet also experiences per-packet next hoplookup and switching overheads at the router/switch. This overhead is avoided by annotations.

One may argue that if the per-packet overhead is lesser than the overhead due to increased packetsize, it does not make sense to annotate packets – It is advantageous to simply send new packetscontaining the annotation information. Sending separate packets containing the annotations is aviable option only when the annotations are not functions of the packets being annotated. However,new packets are not practical if the annotation is a function of the packet contents. Hence, theA-Layer cannot be avoided.

21

Figure 10: Percentage throughput attained for different packet sizes.

The question remains: Is the per-packet overhead greater than the overhead caused by increasedpacket size? The answer depends on the specific router/switch vendor and model. We try tofind an answer by analyzing data gathered from various public router/switch benchmark studiespublished in the Network Magazine [3], Network World Fusion [2] magazine and the Light Reading[9] website. Router/switch vendors are reluctant to participate in public benchmarking tests. Hencethe amount of benchmarking data available is limited.

In a study of Internet core routers conducted by Data Comm and European Network Labora-tories (ENL, Paris) [4], packets of different sizes (60 bytes, 340 bytes and 1504 bytes) were fed toa Juniper M40 router. In the case of 60-byte packets, the router was able to forward only 99.40 %of the offered traffic without packet losses. The 340 and 1504 byte packets were forwarded at linerate without any loss. In another core router test[8], the Cisco 12416 core router attained 100%throughput on a packet stream with packet sizes following the commonly observed Internet Mixpacket distribution – 40 bytes (56 percent of all traffic), 1500 bytes (23 percent); 576 bytes (17percent); and 52 bytes (4 percent). For a packet stream consisting entirely of 40 byte packets, therouter attained only 52% throughput. Juniper M160, another core router, attained throughputsof 92% and 90% for the 40 byte packets and the Internet mix respectively. This shows that perpacket overhead can be a bottle neck for some Internet core routers.

Similar behavior was observed in the benchmark test of Adtran’s low cost Netvanta 4305 router[7]. The router attained the maximum possible throughput while forwarding 256 and 1518 byteEthernet frames. For 64 byte frames, the router achieved only 67% of the maximum possiblethroughput.

Figure 10 summarizes the results of a comparison test of five 10 Gigabit switches conducted bythe Network World Global Test Alliance in 2003 [5]. The switches were tested with 3 different framesizes - 64 bytes, 256 bytes and 1518 bytes. The Force10 switch was able to achieve the maximumtheoretical throughput with all three frame sizes. The behavior of the other switches was mixed.For example, the Avaya and HP switches attained higher throughput for 256 byte frames than forthe 64 byte frames. However, in both cases, 1518 byte frames resulted in lower throughput. Ina separate benchmark test of the Cisco Catalyst 3550 midrange switch [6], 100% throughput was

22

acheived for all frame sizes.The benchmarking results indicate that per-packet processing overhead is more prominent in

the case of routers than for switches. This can be attributed to the inherent complexity of a routercompared to a switch. Hence, when we are dealing with routers, annotating existing packets isa better alternative to injecting new packets. In the case of switches, we speculate that there isno advantage in annotating over injecting new packets. However, there is no disadvantage either.The switches we studied offer similar throughput for all the different packet sizes. Hence, addingannotations is not a bad choice. This is a reassuring conclusion, especially for the scenario in whichwe have no choice but to annotate. Let us now develop an analytical model to tell us under whatnetwork conditions, annotations are favored over injecting new packets.

9 Analyzing Piggyback Flows

By annotating existing packets, we are creating a new communication channel on top of existingflows. The dynamics of these piggyback channels are deeply influenced by the complex interactionstaking place within the network at any given time. For example, the amount of available piggybackbandwidth is a function of the router/switch capabilities as well as the size and number of packetsflowing across each link in the network. In this section, we motivate and formulate a simple modelto capture the dynamics of piggyback channels.

9.1 Goal

The goal of our analytical model is to answer questions about the feasibility and benefits of piggy-back channels for a particular network. Specifically, we wish to answer the following question:

For a particular network topology and configuration, given the current state of packet flows oneach link, what is the maximum piggyback bandwidth available between any two routers/switches inthe network?

In addition to answering the above question, the model should tell us how many and which pack-ets to annotate for achieving a specified amount of piggyback bandwidth between two routers/switches.Creating a communication channel by injecting new standalone packets into the network is an al-ternative to annotating existing packets. The model should be able to tell us under what networkconditions, it is more beneficial to inject new packets instead of annotating existing packets. Toannotate or not to annotate; that is the question.

9.2 Piggyback Flows as an Integer Program

We tried two different approaches to modeling the piggyback flow problem. The first approachwas to reduce it to the standard MAXFLOW problem. Due to the large number of parametersinvolved – router capacities, link capacities, per-packet overheads, per-byte overheads, etc – wewere unsuccessful in formulating a MAXFLOW problem without loss of vital information. In thesecond approach, we successfully formulated the piggyback problem as a simple Integer Program(IP). IP is NP-complete. So for very large network models, involving thousands of constraints andvariables, we relax the model to a Linear Program(LP) without loss of accuracy. We now describeour IP formulation.

23

Consider a network with N routers/switches. We discretize time and consider the behaviorof the network in a particular snapshot of length t. The load on a router/switch is quantified bythe number of CPU units required to forward and process the packets arriving at it. To simplifypresentation, we limit that packets enter the network from the outside only at a single ingressrouter/switch s. We consider traffic flow only in a single direction. We wish to find the maximumavailable piggyback bandwidth from s to a single router/switch d. The model can be easily enhancedto include reverse traffic and multiple ingress points and destinations.

9.2.1 Constants

The model involves a large number of constant parameters. These are listed and explained in Table9.2.1.

Table 1: Constants used in the IP

Constant DescriptionM l

ij Number of packets of size l flowing from i to j in the chosen snapshot. Please note that lincludes the size of headers.

MTU The Maximum Transmission Unit of the network. Packets with size greater than MTUwill be fragmented.

H Size of a packet header.ngbrsi Neighboring routers/switches of i.Uk

i Processing units required to add an annotation of size of k at router/switch i.V l

i Processing units required to create a new packet of size l bytes of data at router/switch i.Please note that each packet has an additional H bytes of header information, giving it atotal size of H + l.

Xki Processing units to extract an annotation of size k at router/switch i.

Y li Processing units to extract data from packet of size l.

oi Size independent per-packet overhead at router i.fi Per-byte processing overhead at router i.Ci Maximum processing capacity of router/switch i.lij Maximum number of bytes that can be handled on the link from i to j in a snapshot.

9.2.2 Variables

Variables in our model represent the number of annotations and newly injected packets of differentsizes. They are listed in Table 9.2.2.

9.2.3 Objective Function

Our objective is to maximize the flow F from s to d such that the capacity constraints of eachrouter/switch and each link in the network are respected. The flow consists of the data in newpackets in addition to the piggybacked annotations.

The total flow arriving at d is:

24

Table 2: Variables used in the IP

Variable Descriptionakl

ij Number of annotations of size k piggybacked on existing packets of size l flowing from ito j.

nkij Number of new packets of data size k flowing from i to j (header of size H not included

in k).

F =∑

j∈ngbrsd

MTU∑k=1

k ×[(MTU∑l=1

akljd) + nk

jd

](1)

The objective thus becomes:

Maximize F

9.2.4 Constraints

We now list out the constraints associated with the model and explain their role. We wish to avoidfragmentation of packets. This implies that all packets should have a total size less than the MTUof the network. The following set of constraints enforces that annotations can be piggybacked ona packet only if sufficient space exists in the packet.

For each link ij and each k and l,

aklij = 0 if k + l > MTU (2)

The next set of constraints represent the check that the total number of annotations on size lpackets on a particular link ij does not exceed the actual number of size l packets already flowingon the link.

MTU∑k=1

aklij ≤ M l

ij (3)

We now introduce the constraints imposed by the requirements of conservation of packets andannotations at each router. The preexisting flow of packets, represented by M l

ij already obey thepacket conservation requirement. The following set of constraints ensures that new packets of sizek injected into the network obey conservation at each router i 6= s, d.∑

j∈ngbrsi

nkij =

∑j∈ngbrsi

nkji (4)

We also need to conserve the number of annotations at each router. The following set ofconstraints ensures that the number of annotations of size k piggybacked on packets of size l areconserved at each router i 6= s, d. ∑

j∈ngbrsi

aklij =

∑j∈ngbrsi

aklji (5)

25

The next sets of constraints ensure that the processing capacity of a router/switch is neverexceeded. First, we consider the constraints on routers/switches other than s and d. The processingoverhead at a router can be broken into two parts: i. per-byte processing overhead and ii. per-packetprocessing overhead. The total per-byte processing overhead at a router i has three components– pre-existing packets that were not annotated, pre-existing packets that were annotated, newlyinjected packets. It is calculated as follows:

A1(i) =∑

j∈ngbrsi

[ MTU∑l=1

(M lij−

MTU∑k=1

aklij )×fi×l+

MTU∑l=1

MTU∑k=1

aklij×fi×(l+k)+

MTU∑l=1

nlij×fi×(l+H)

](6)

The total per-packet overhead is calculated as follows:

A2(i) = oi ×∑

j∈ngbrsi

MTU∑l=1

(nlij + M l

ij) (7)

A1(i) and A2(i) capture the overheads associated with receiving as well as forwarding packets.The capacity constraint at router i combines the sum of A1 and A2 to give the following

constraint:

A1(i) + A2(i) ≤ Ci (8)

The processing capacity constraints at the ingress and destination routers are treated separatelyas they involve the overheads of annotating packets and creating new ones, as well as that ofextracting information from received packets2. Let us first consider the ingress router s. Theprocessing overhead at s consists of four components. We have already seen two of these components- A1(s) and A2(s), the per-byte and per-packet processing overheads. The additional overheadsat s result from the overheads in adding annotations and creating new packets. The processingrequired for annotating packets at s is the following:

A3(s) =MTU∑k=1

MTU∑l=1

[ ∑j∈ngbrss

aklsj

]× Uk

s (9)

The processing required for creating new packets at s is the following:

A4(s) =MTU∑l=1

[ ∑j∈ngbrss

nlsj

]× V l

s (10)

The following constraint enforces that the total processing overhead at s is less than its maxi-mum processing capacity.

A1(s) + A2(s) + A3(s) + A4(s) ≤ Cs (11)

The constraints associated with the destination d are similar to those at s. At d, routers haveto expend processing capacity for extracting the data from annotated packets and new packets.The processing overhead associated with extracting data from annotated packets is given below:

A5(d) =MTU∑k=1

MTU∑l=1

[ ∑j∈ngbrsd

akljd

]×Xk

d (12)

2We envision routers/switches incorporating iBox functionality

26

Figure 11: XML network configuration and corresponding IP in lp-format.

The processing overhead associated with extracting data from new packets is the following:

A6(d) =MTU∑l=1

[ ∑j∈ngbrsd

nljd

]× Y l

d (13)

The following constraint enforces that the total processing overhead at d is less than its maxi-mum processing capacity.

A1(d) + A2(d) + A5(d) + A6(d) ≤ Cd (14)

For the case of A1(d), A2(d), A5(d) and A6(d), packets and annotations flowing into d, i.e. nkjd,

M ljd and akl

jd are used instead of nkdj , M l

dj and akldj . We do this due to the unavailability of variables

to capture the number of packets leaving the network at d.The last set of constraints is associated with each link ij. The total number of bytes flowing

through the link in a particular snapshot should not exceed the capacity of the link. This constraintis captured as follows:

MTU∑l=1

[ MTU∑k=1

aklij × k + M l

ij × l + nlij × (l + H)

]≤ lij (15)

9.3 Solving the Model

We use lpsolve [10], an Open source (Mixed-Integer) Linear Programming system to solve the IP.lpsolve uses the popular Branch and Bound heuristic [11] to solve IPs. The size of the model beingsolved is limited only by the amount of system memory available. The input to lpsolve consists ofa text file in the lp-file format. The parameters of the network under consideration are specifiedin an XML file. We wrote a program to convert the XML description of the network into thecorresponding IP in lp-file format. Figure 9.3 shows snippets of the XML network description fileand the corresponding lp-format file.

27

Figure 12: A simple distribution network.

9.4 The Real World

Our model depends on the parameters earlier listed in Table 9.2.1. Some of the parameters likeMTU and header size are trivially known. With some effort, the number of packets of different sizesflowing on each link in the network can also be determined. The parameters which are extremelydifficult to estimate are the per-packet and per-byte processing overheads. We were unable to findthese numbers in the router/switch technical specification documents. Our experiments to estimatethese parameters met with limited success due to our inability to generate traffic at speeds capableof stressing today’s gigabit routers and due to the unavailability of fine-grained measurement hookswithin the routers. We are continuing our efforts to find good estimates for all the model parameters.Till better estimates are available, we use ad-hoc parameter values in illustrating how we apply themodel to a small network.

9.5 Example: A Simple Network

Consider the simple network shown in Figure 12. The squares represent switches/routers. Circlesdenote switches or hubs beyond which packets leave the network and are sent to end hosts. To keepthe model small, we consider only four different existing packet sizes - 40, 500, 1000 and 1500 bytes.The number of packets (in thousands) of each size flowing on a particular link is shown beside itin the format N40:N500:N1000:N1500. We consider a single annotation size of 20 bytes. The sizes ofthe newly injected packets are constrained to 20, 40 and 60 bytes. The values of other parametersused in the model are given in Table 9.5.

The IP contains 119 variables and 126 constraints. The absolute value of the maximum attain-able bandwidth is not a useful measure, as the parameter values are fixed in an ad-hoc fashion.However, the behavior of piggyback flows as we vary different parameters is interesting.

We first show the results of varying the per-packet overhead o at the ingress router s in Figure13. The figure indicates that annotations are clearly preferred over new packet injection as theper-packet overhead increases. As long as the per-packet overhead is less than 60 times the per-

28

Table 3: Parameter values used in the IP model for the network in Figure 12Parameter Value Parameter ValueMTU 1500 bytes H 20 bytesU20

i 0.5 ∀i X20i 0.5 ∀i

V li 0.5 ×l∀i Y l

i 0.5 ×l∀ioi 0.01 ∀i fi 0.001 ∀iCi 1000000 ∀i lij 100000 ∀i, j

0

200

400

600

800

1000

1200

1400

1600

0 50 100 150 200 250 300 350 400 450

Num

ber o

f ann

otat

ions

/new

pac

kets

Pig

gyba

ck fl

ow fr

om s

ourc

e to

des

tinat

ion

Ratio of per-packet overhead to per-byte overhead, i.e. o:f

annot 40annot 500

annot 1000new 60

Piggyback flow

Figure 13: Piggyback flow behavior with varia-tion in per-packet overhead.

0

200

400

600

800

1000

1200

1400

1600

0.1 0.11 0.12 0.13 0.14 0.15 0.16 0.17 0.18

Num

ber o

f ann

otat

ions

/new

pac

kets

Pig

gyba

ck fl

ow fr

om s

ourc

e to

des

tinat

ion

Ratio of per-byte overhead to per-packet overhead, i.e. f:o

annot 40annot 500

annot 1000new 60

Piggyback flow

Figure 14: Piggyback flow behavior with varia-tion in per-byte overhead.

byte overhead, router s is unsaturated and allows the maximum number of annotations and newpackets, subject to link capacity constraints. Beyond this point, The number of new packets injectedsteadily drops, while the number of annotations of all three sizes remain at the same maximal level.When the per-packet overhead becomes 410 times the per-byte overhead, the number of annotationssharply falls to 0. It is interesting that annotations on larger sized packets fall to 0 earlier. Thegraph only shows new packets of a single size, 60 – the model always forced the number of 20 and40 byte packets to 0. We see similar trends in Figure 14 where we vary the per-byte overhead f atrouter s , while keeping the per-packet overhead o constant.

Figure 15 shows the number of annotations, number of newly created packets and total availablebandwidth as we vary the per-packet annotation cost at the ingress router. Initially the number ofstandalone packets decreases even as we increase the per-packet annotation cost. This shows thatthe system clearly favors annotations and tries to maintain the number of annotations by takingsome processing power away from new packet creation. After the annotation cost becomes 20 timesthe packet creation cost, the number of annotations falls drastically. The number of annotationson 500 and 1000 byte packets goes to 0, while the number of annotations on 40 byte packets fallsto one-third of its original value. At the same time, the number of new packets shoots up to itsmaximal value. The remaining number of annotations on 40 byte packets slowly goes to 0 withfurther increase in annotation cost. This figure indicates that there is a sharp pivotal point afterwhich new packet injection is crucial to achieving maximum bandwidth.

Figure 16 shows the piggyback flow behavior as the per-packet deannotation cost at the desti-

29

0

200

400

600

800

1000

1200

1400

1600

0 20 40 60 80 100 120 140 160 180 200

Num

ber o

f ann

otat

ions

/new

pac

kets

Pig

gyba

ck fl

ow fr

om s

ourc

e to

des

tinat

ion

Ratio of per-packet annotation cost to packet creation overhead, i.e. U:V

annot 40annot 500

annot 1000new 60

Piggyback flow

Figure 15: Piggyback flow behavior with varia-tion in annotation overhead.

0

200

400

600

800

1000

1200

1400

1600

0 2000 4000 6000 8000 10000 12000 14000 16000

Num

ber o

f ann

otat

ions

/new

pac

kets

Pig

gyba

ck fl

ow fr

om s

ourc

e to

des

tinat

ion

Ratio of per-packet deannotation cost to standalone packet interpretation, i.e. X:Y

annot 40annot 500

annot 1000new 60

Piggyback flow

Figure 16: Piggyback flow behavior with varia-tion in deannotation overhead.

nation router d is varied. In this case, the number of new packets remains constant throughout.The number of annotations on 500 and 1000 byte packets rapidly fall off to zero as the ratio ofthe deannotation cost to the new packet interpretation cost goes over 800. The decrease in thenumber of annotations on 40 byte packets is much more gradual. Both figures 15 and 16 allude tothe desirability of annotating smaller size packets over bigger ones.

9.6 Discussion

The example network we considered in this section is small and simple. Some of the observedbehavior is not surprising. However some aspects of piggyback flows, like the presence of a pivotalpoint in Figure 15 are not obvious. Our model brings out the conditions under which annotationsare favored over new packet injection and vice versa. By giving an estimate of the maximumpiggyback bandwidth attainable, the model also tells us how many packets should be annotated orhow many new packets should be injected in order to achieve a desired level of bandwidth. Biggernetworks will have more complex interactions. In our example, we assumed equal capacities for eachlink and router. In a real network, routers and switches are of different types and manufacturedby different vendors. We hope that our model will be able to accurately capture complex networkdynamics and aid the network admins in making a good decision about how many and what packetsto annotate. The caveat here is that the model is useless without accurate parameters. Accuratelydetermining the value of various router/switch overheads is difficult, but not impossible for themanufacturers themselves.

10 Conclusion

The rise of malicious “junk traffic”, coupled with the increased complexity of network services andenterprise networks, has driven the need for better network management primitives. Because ofcomplex interactions, network operators require more visibility into network traffic and behavior.We have proposed an architecture for an Annotation Layer the enables inspection-and-action pointscalled iBoxes to communicate per-flow and per-session network measurements in a distributed way.

30

iBoxes will reside at key points in the network–at the ISP edge, in front of servers, and near accessnetworks. By observing traffic at the per-flow granularity, and by exchanging those measurements,network operators will be able to track traffic surges and “hot” parts of the network. This will enablethem to more quickly identify malicious behavior as well as misconfigured or novel applications.We hope this contributes to more reliable and dependable networks.

References

[1] R. Katz, G. Porter, I. Stoica, S. Shenker, M. Tsai, “COPS: Quality of Service vs. Any Serviceat All”, IWQoS 2005, Passau, Germany (June 2005)

[2] Network World Fusion, http://www.nwfusion.com

[3] Network Magazine, http://www.networkmagazine.com

[4] The Lone Router, Network Magazine http://www.networkmagazine.com/shared/article-/showArticle.jhtml;jsessionid=WLYGZIBT2OIFUQSNDBCCKH0CJUMEKJVN?articleId=-8711005&pgno=1

[5] Testing 10Gig Ethernet Switches, Network World Fusion, http://www.nwfusion.com/reviews-/2003/020310gbe.html#fig1

[6] Cisco’s Catalyst 3550 midrange switch, Network World Fusion, http://www.nwfusion.com-/reviews/2002/0325rev.html

[7] Adtran serves up newest low-cost router, Network World Fusion, http://www.nwfusion.com-/reviews/2004/112204rev.html

[8] Internet Core Router Test, Light Reading, http://www.lightreading.com/document.asp-?doc id=4009

[9] Light Reading, http://www.lightreading.com

[10] lpsolve, http://groups.yahoo.com/group/lp solve

[11] Optimization Guide: Integer Programming, http://www-fp.mcs.anl.gov/otc/Guide/OptWeb-/discrete/integerprog/section2 1 1.html

[12] Evaluating Security Threats, http://winfx.msdn.microsoft.com/library-/default.asp?url=/library/en-us/wd adonet/html/408d9ca3-e394-4ae8-9dc2-bd8c7681613d.asp

[13] Message Authentication using Hash Functions The HMAC Construction, Mihir Bellare, RanCanetti, and Hugo Krawczyk. CryptoBytes (RSA Labs), Vol. 2, No. 1, Spring 1996.

[14] FIPS Pub 198: The Keyed-Hash Message Authentication Code (HMAC). http://csrc.nist.gov-/publications/fips/fips198/fips-198a.pdf

[15] Hash Function Update Due to Potential Weakness Found in SHA-1. James Randall, RSALaboratories. Mar. 11, 2005. http://www.rsasecurity.com/rsalabs/node.asp?id=2834

[16] Brute force attacks on cryptographic keys. http://www.cl.cam.ac.uk/users/rnc1/brute.html

31

[17] Speed Comparison of Popular Crypto Algorithms. http://www.eskimo.com/ weidai-/benchmarks.html

[18] Endace network monitoring products, 2005. http://www.endace.com/networkMCards.htm.

[19] F. Jahanian C. Labovitz, G. R. Malan. Internet routing instability. In IEEE/ACM Trans. OnNetworking, volume 6, 1997.

[20] Philip Levis, Neil Patel, David Culler, and Scott Shenker. Trickle: A self-regulating algorithmfor code propagation and maintenance in wireless sensor networks. In First USENIX/ACMSymposium on Networked Systems Design and Implementation.

[21] Cisco Netflows. http://www.cisco.com/warp/public/732/tech/nmp/netflow/.

[22] Ruoming Pang, Vinod Yegneswaran, Paul Barford, Vern Paxson, and Larry Peterson. Charac-teristics of internet background radiation. In IMC ’04: Proceedings of the 4th ACM SIGCOMMconference on Internet measurement, pages 27–40, New York, NY, USA, 2004. ACM Press.

[23] K. Argyraki, P. Maniatis, D. Cheriton, S. Shenker, “ Providing Packet Obituaries”, HotNetsIII, San Diego (November 2004)

[24] J.D. Case, M. Fedor, M.L. Schoffstall, and J. Davin. Simple Network Management Protocol(SNMP). RFC 1098, April 1989. Obsoleted by RFC 1157.

[25] HP Openview http://www.managementsoftware.hp.com/

[26] Beverly Schwartz, Alden W. Jackson, W. Timothy Strayer, Wenyi Zhou, R. Dennis Rockwell,and Craig Partbridge. Smart packets: applying active networks to network management. ACMTransactions on Computer Systems, 18(1):67–88, 2000.

[27] David L. Tennenhouse, Jonathan M. Smith, W. David Sincoskie, David J. Wetherall, andGary J. Minden. A survey of active network research. IEEE Communications Magazine, 35(1):80–86, 1997.

[28] David Wetherall. Active network vision and reality: lessions form a capsule-based system. InSymposium on Operating Systems Principles, pages 64–79, 1999.

32