SOAR: Simple Opportunistic Adaptive RoutingProtocol for Wireless Mesh Networks

Eric Rozner Jayesh Seshadri Yogita Ashok Mehta Lili QiuUniversity of Texas at Austin VMware Google University of Texas at Austin

erozner@cs.utexas.edu jseshadr@vmware.com yogitamehta@gmail.com lili@cs.utexas.edu

✦

Abstract —Multihop wireless mesh networks are becoming a new at-tractive communication paradigm owing to their low cost and easeof deployment. Routing protocols are critical to the performance andreliability of wireless mesh networks. Traditional routing protocols sendtraffic along predetermined paths and face difficulties in coping withunreliable and unpredictable wireless medium. In this paper, we proposea Simple Opportunistic Adaptive Routing protocol (SOAR) to explic-itly support multiple simultaneous flows in wireless mesh networks.SOAR incorporates the following four major components to achieve highthroughput and fairness: (i) adaptive forwarding path selection to lever-age path diversity while minimizing duplicate transmissions, (ii) prioritytimer-based forwarding to let only the best forwarding node forwardthe packet, (iii) local loss recovery to efficiently detect and retransmitlost packets, and (iv) adaptive rate control to determine an appropriatesending rate according to the current network conditions. We implementSOAR in both NS-2 simulation and an 18-node wireless mesh testbed.Our extensive evaluation shows that SOAR significantly outperformstraditional routing and a seminal opportunistic routing protocol, ExOR,under a wide range of scenarios.

Index Terms —C.2.1.k [Communication/Networking and InformationTechnology]: Network Architecture and Design – Wireless communica-tion; C.2.2.d [Communication/Networking and Information Technology]:Network Protocols – routing protocols.

1 INTRODUCTION

Multihop wireless mesh networks are becoming a newattractive communication paradigm. Many cities acrossthe world have deployed or are planning to deploythem [34], [33], [30] to provide Internet access to resi-dents and local businesses. Routing protocol design iscritical to the performance and reliability of wirelessmesh networks.

A natural approach to routing traffic in wireless meshnetworks is to adopt techniques similar to those inwire-line networks, which select a best path for eachsource-destination pair (according to some metric) andsend traffic along the pre-determined path. Most of theexisting routing protocols, such as DSR [16], AODV [26],DSDV [25], and LQSR [8], fall into this category, whichwe refer to as traditional routing.

This research is supported in part by by National Science Foundation grantsCNS-0546755 and CNS-0627020. We thank Szymon Chachulski, MichaelJennings, Sachin Katti and Dina Katabi for providing ExOR source code.Jayesh Seshadri and Yogita Ashok Mehta worked on this project while theywere students at University of Texas at Austin.

Recent studies [2], [3], [35] show that traditional rout-ing faces difficulties in coping with unreliable and unpre-dictable wireless medium. Motivated by these observa-tions, researchers [2], [3], [35] developed opportunisticrouting protocols for wireless mesh networks. Oppor-tunistic routing exploits the broadcast nature of the wire-less medium and does not commit to a particular routebefore data transmission. Instead, the sender broadcastsits data; among the nodes that hear the transmission, theone closest to the destination is selected to forward thedata. In this way, opportunistic routing can effectivelycombine multiple weak links into a strong link and takeadvantage of transmissions that reach unexpectedly nearor unexpectedly far.

In this paper, we present the design and imple-mentation of a Simple Opportunistic Adaptive Routingprotocol (SOAR), a new addition to the opportunisticrouting protocol design space. Different from the ex-isting opportunistic routing protocols, SOAR explicitlysupports multiple simultaneous flows by strategicallyselecting forwarding nodes and employing adaptive ratecontrol. To demonstrate its effectiveness and feasibility,we implement SOAR in both the NS-2 simulator [23]and an 18-node wireless testbed. Using extensive evalu-ation, we show that SOAR can significantly out-performtraditional routing and a seminal opportunistic routingprotocol, ExOR [2], under a wide range of scenarios.

The rest of the paper is organized as follows. In Sec-tion 2, we survey related work and motivate opportunis-tic routing. In Section 3, we describe design challengesof opportunistic routing and present the SOAR routingprotocol. We evaluate SOAR using NS-2 simulationsin Section 4. We present testbed implementation andevaluation in Section 5. Finally, we conclude in Section 6.

2 BACKGROUND

In this section, we first review traditional routing pro-tocols for wireless networks. Then, we explain the po-tential benefits of opportunistic routing and introduceexisting opportunistic routing protocols.

2.1 Traditional Routing Protocols

Routing has been an active area in wireless networkingresearch. Most of the original work in this area targetedhigh-mobility scenarios such as battlefield networks.Therefore, the focus was on establishing and maintainingroutes under frequent and unpredictable changes innetwork connectivity. A number of on-demand routingprotocols have been proposed for this purpose, as exem-plified by DSR [16] and AODV [26], where packets arerouted along paths with the shortest hop count.

Recently, wireless mesh networks [17], [28], [30] haveemerged as a new dominant application of multihopwireless networks. Nodes in such networks have littleor no mobility and often are not constrained by shortbattery-life or limited computational power. Therefore,improving network performance becomes the primaryfocus. Researchers have found that the hop-count metric,as used in DSR and AODV, does not provide good per-formance since not all hops are equal. To address this is-sue, various link-quality metrics have been proposed [1],[7], [9], [10], [12], [13]. These metrics quantify the qualityof links using link loss rate, packet transmission time, orsignal-to-noise ratios. These works are complementary toopportunistic routing by offering metrics for comparingdifferent routes. As with other opportunistic routingprotocols, SOAR uses ETX [7] as the underlying routingmetric, but it is easy for SOAR to support any alternativerouting metric.

In addition to routing metric design, there is anotherthread of works that optimize routing in wireless meshnetworks by casting it as a linear program (e.g., [14],[31]) or non-linear program (e.g., [21]).

2.2 Benefits of Opportunistic Routing

More recently, researchers have proposed opportunis-tic routing for mesh networks. Opportunistic routingdiffers from traditional routing in that it exploits thebroadcast nature of wireless medium and defers routeselection after packet transmissions. This can cope wellwith unreliable and unpredictable wireless links. Thereare two major benefits in opportunistic routing. First, itcan combine multiple weak links into one strong link.Second, it takes advantage of unexpectedly short orunexpectedly long transmissions. We further illustratethe benefits using the following two examples.

A

B

C

F

D

E

G

20%

20%

20%

20%

20%

100%

100%

100%

100%

100%

Fig. 1. Opportunistic routing can take advantage ofmultiple weak links.

As shown in Figure 1, a source has weak wirelessconnectivity to each of the five intermediate nodes, with

a delivery rate of 20%. For ease of illustration, assumeindependent loss rates on each link (This assumptionis not required for our protocol). All the intermediatenodes have 100% delivery rate to the destination. Undera traditional routing protocol, we have to pick one of thefive intermediate nodes as the relay node. Therefore, onaverage, a packet needs to be transmitted five times inorder to reach the next hop. Then, the packet needs tobe forwarded once to reach the destination. Altogether,six transmissions are required to deliver the packetend-to-end. In comparison, in opportunistic routing wecan treat the five intermediate nodes as one unit thatcooperatively forwards the packet to the destination. Thecombined link has a success rate of 1−(1−0.2)5 = 0.672.Therefore, on average, only 1/0.67=1.487 transmissionsare required to reach at least one of the five intermedi-ate nodes, and another transmission is required for anintermediate node to forward. Therefore, opportunisticrouting achieves 2.5 times the throughput of traditionalrouting.

A B C D

Fig. 2. Opportunistic routing can maximize the progresseach transmission makes.

Second, a traditional routing protocol has to trade offbetween link quality and the amount of progress eachtransmission makes. For example, consider the networkshown in Figure 2, A sends data to D along the pathA − B − C − D. If B is used as the next hop and thequality of link A − B is good, then no retransmissionsare required to deliver the packet to B. But the progressmade is small. Alternatively, if C is chosen as the nexthop, a large progress is made if the packet reaches C.However if the quality of link A − C is poor, multipletransmissions are required to deliver the packet to C. Incomparison, opportunistic routing does not commit to Bor C before transmissions. Among the nodes that receivethe packet, we choose the one closest to the destinationto forward. In this way, we can opportunistically lever-age transmissions that are either unexpectedly short orunexpectedly long, thereby achieving high throughput.

2.3 Prior Opportunistic Routing Protocols

ExOR [2] is a seminal opportunistic routing protocol.In ExOR, senders broadcast a batch of packets (10-100packets per batch). Each packet contains a list of nodesthat can potentially forward it. In order to maximizethe progress of each transmission, the forwarding nodesrelay data packets in the order of their proximity tothe destination (Throughout the rest of this paper, prox-imity is measured by the ETX metric [7], not in termsof physical distance). To minimize redundant transmis-sions, ExOR uses a batch map, which records the list ofpackets each node has received; every forwarding nodeonly forwards data that has not been acknowledged bythe nodes closer to the destination. ExOR imposes stricttiming constraints among the forwarders to facilitate

coordination in the relay process. Only one forwardercan be active at a time, and spatial reuse of the wirelessspectrum is reduced. Furthermore, ExOR’s forwardingpaths can easily diverge, i.e., nodes on the differentforwarding paths may not hear from each other andcause duplicate forwarding. These difficulties make itunclear how well ExOR supports multiple simultaneousflows.

MORE [3] applies network coding to opportunisticrouting in a clever way. Since random coding can ef-fectively generate linearly independent coded packetswith a high probability, the forwarding nodes in MOREdo not need to coordinate which packets are forwardedby which nodes. However, MORE selects forwardingnodes in a similar way as ExOR and does not preventdiverging paths, which can lead to waste of resourceusage. Furthermore, it does not rate-limit the initialtransmission of packet batches, and may cause degrada-tion in aggregate network performance and fairness. Webelieve that the prevention of diverging paths and rate-limiting are necessary for efficiently supporting multipleflows under opportunistic routing protocols.

ROMER [35], another opportunistic routing protocol,tries to forward the packets simultaneously along multi-ple paths. It incorporates a credit based scheme to limitthe number of transmissions that a packet is allowedbefore reaching the destination. Even with the credit-based scheme, there is still significant overhead since apacket is allowed to be forwarded by multiple nodes ateach hop. Also, setting the credit is non-trivial and staticcredit has difficulties in coping with different topologies.As a result, ROMER only supports a single flow.

Zhong et al. [36] show that the routing metric used toselect and prioritize forwarding nodes is important. Theydevelop a new routing metric, called EAX, to account forinter-candidate communication in opportunistic routing.

In addition to opportunistic routing protocols for meshnetworks, researchers have also designed opportunisticrouting protocols for ad-hoc and sensor networks. Forexample, [5] and [32] both dynamically select forwardingnodes based on recent link quality. However, in both pro-tocols, only one forwarding node is selected before trans-missions, and they cannot take advantage of transmis-sions reaching nodes other than the previously selectedforwarder. [4] balances the energy consumption rates ofdifferent nodes in a sensor network by opportunisticallyincorporating forwarders’ energy consumption.

Our previous workshop paper [29] presents an ini-tial design of an opportunistic routing protocol andpreliminary simulation results in some toy topologies.This paper is built on our previous work, but differs inthe following important ways. First, we make severalsignificant new optimizations, which include (i) devel-oping a rate control scheme for opportunistic routing,which adapts the sending rate according to link lossrate; (ii) adapting the number of forwarding nodesaccording to link quality, (iii) adapting retransmissiontimeout according to network delay, and (iv) cross-flow

ACKs to increase the efficiency of receiver feedbackin the presence of multiple flows. These optimizationsare critical to ensure SOAR performs well in a widerange of scenarios. Second, we extensively evaluate theperformance of SOAR by simulating in a large rangeof synthetic topologies and traffic demands. Third, weimplement SOAR in a wireless testbed, and demonstrateits feasibility and effectiveness in a real network. Indoing so, we add comparisons against ExOR, a seminalopportunistic routing protocol.

3 THE SOAR ROUTING PROTOCOL

In this section, we first describe design challenges ofopportunistic routing protocols. Then we present anoverview and the protocol details of SOAR.

3.1 Design Challenges

The goal of opportunistic routing is to maximize theprogress each transmission makes without causing du-plicate (re)transmissions or incurring significant coordi-nation overhead. In order to achieve this goal, severalimportant design issues should be addressed:

Forwarding node selection: While opportunistic rout-ing defers the final route selection after data transmis-sions, the candidate forwarding nodes should still be se-lected in advance. This is necessary because the numberof duplicate transmissions and coordination overheadtend to increase with the number of forwarding nodes.Without judicious forwarding node selection, the over-head of opportunistic routing might offset its benefits.

Avoid duplicate transmissions: When multiple nodesoverhear a transmission, we want to ensure that onlythe node closest to the destination forwards it. The bestforwarding node should be selected in a cheap anddistributed way.

Loss recovery: In opportunistic routing, each nodebroadcasts data packets, and broadcast packets are vul-nerable to packet losses and corruption since the MAClayer offers no reliability support for broadcast. There-fore it is important for opportunistic routing protocolsto efficiently detect and recover packet losses.

Rate control: Determining an appropriate sendingrate is important for opportunistic routing. Without ratecontrol, a flow may send too many packets on the firstfew hops which cannot be forwarded on the subsequenthops. Due to wireless interference, such transmissionstake away available bandwidth from the subsequenthops and significantly degrade performance [20].

To address the above challenges, SOAR consists of thefollowing four major components: (i) adaptive forward-ing path selection to leverage path diversity while avoid-ing diverging paths, (ii) priority timer-based forwardingto allow only the best forwarding node to forward thepacket, (iii) local loss recovery to efficiently detect andretransmit lost packets, and (iv) adaptive rate control todetermine an appropriate sending rate according to thecurrent network condition.

3.2 Overview

As ExOR and MORE, SOAR is a proactive link staterouting protocol. Every node periodically measures anddisseminates link quality in terms of ETX. Based onthis information, a sender selects the default path anda list of (next-hop) forwarding nodes that are eligiblefor forwarding the data. It then broadcasts a data packetincluding this information. Upon hearing the transmis-sion, the nodes not on the forwarding list simply discardthe packet. Nodes on the forwarding list store the packetand set forwarding timers based on their proximity tothe destination. A node closer to the destination usesa smaller timer and forwards the packet earlier. Uponhearing this transmission, other nodes will remove thecorresponding packet from their queues to avoid du-plicate transmissions. Like all the existing opportunisticrouting protocols, SOAR broadcasts data packets at afixed PHY data rate. How to perform automatic rateadaptation in opportunistic routing is an interestingtopic by itself, and we plan to investigate as part of ourfuture work.

3.3 Adaptive Forwarding Path Selection

To support opportunistic routing, each node maintainsa routing table of the following format: (destination,default path, forwardList), where the default path is theshortest path from the current node to the destinationand the forwarding list includes a list of next-hop nodesthat are eligible to forward the transmission.

3.3.1 Default Path SelectionIn order to compute the default path and forwardinglists, every node measures and maintains the networktopology, as in existing wireless mesh routing protocols(e.g., Srcr [7], LQSR [8], and ExOR [2]). Several routingmetrics have been proposed in the literature to assignlink weights based on link quality. We use the ETX met-ric, a state-of-art routing metric proposed by De Couto etal. [7]. A link’s ETX metric measures the expected num-ber of transmissions (including retransmissions) requiredto reliably send a packet across the link. Let pf and pr

denote the loss probabilities of the link in the forwardand reverse directions, respectively. Each node measuresthe loss rate of its links to and from its neighbors (i.e., pf

and pr) by broadcasting one probe packet every secondand counting the number of probes received in the last10 seconds. Then, the link’s ETX metric is calculatedas 1

(1−pf )×(1−pr) , assuming independent packet losses.

Each node maintains an exponentially weighted movingaverage of ETX samples. The default path is the shortestpath between the source and destination in terms ofETX. The ETX routing metric has been shown effectivein selecting good quality routes [7], [8].

3.3.2 Forwarding Node SelectionForwarding node selection is critical to the performanceof SOAR. In order to leverage path diversity while

avoiding duplicate transmissions, SOAR relaxes the ac-tual route that data traverses to be along or near thedefault path. Different from traditional routing, SOARleverages path diversity by using more flexible routes:nodes other than the next hop can forward the data.Differing from existing opportunistic routing protocols,SOAR constrains the nodes involved in routing a packetto be near the default path, as shown in Figure 3. Thisprevents routes from diverging and minimizes duplicatetransmissions. Moreover, this forwarding node selectionalso simplifies coordination since all the nodes involvedare close to nodes on the default path and can hear eachother with a reasonably high probability. Therefore, wecan use overheard transmissions to coordinate betweenforwarding nodes in a cheap and distributed way.

A B

Fig. 3. The actual route in SOAR involves nodes onor near the default path. In the figure, the nodes in theshaded region participate in forwarding packets from A toB.

Our forwarding list selection algorithm consists oftwo steps. First, a sender selects an initial forwardinglist based on the default path. Then it further limitsthe number of forwarding nodes to minimize duplicatetransmissions. These steps are taken by a sender on eachpacket, allowing for the forwarding list to quickly adaptto network conditions. Also, a sender in this case refersto either a flow source or a forwarder within the flow(i.e., the forwarding list is updated at each network hop).

Selecting initial forwarding lists: When node i ison the default path, i selects the forwarding nodes thatsatisfy the following conditions:(C1) The forwarding node’s ETX to the destination islower than i’s ETX to the destination.(C2) The forwarding node’s ETX to i is within a thresh-old.The first constraint ensures that the packet makesprogress. The second constraint ensures that i hears theforwarding node’s transmissions with a high probabilityto avoid duplicate retransmissions.

Selecting the correct threshold is nontrivial and using aconstant threshold can be ineffective. Consider a 4-nodenetwork, where there is a flow from node A to D witha default path A − B − D. Whether node C should beincluded as a forwarding node not only depends on theabsolute delivery rates of links A−C and C−D, but alsodepends on how they compare with the delivery rate oflinks on the default paths. For example, suppose A−B,B−D, and C−D all have 50% delivery rate. If A−C hasdelivery rate of 5%, it is not helpful to include C as aforwarding node since in most cases when a packet doesnot reach B it does not reach C either. If the deliveryrate of A − B changes to 5%, now it makes sense to

include C as a forwarding node. This example suggeststhat we should adapt the threshold according to the linkto the default next hop. So we set the threshold to γ ×

ETX(i, nexthop) and use γ = 4.0 in our evaluation. Theintuition is that the links among the forwarding nodesand between the forwarding nodes and i should not betoo much worse than the link between i and nexthop(We also try other values of γ between 2 and 6, and theydo not cause significant performance difference).

A

B1 B2 B3

C1 C2 C3 C4 C5 C6

F

D1 D2 D11 D12

Fig. 4. Careful forwarding node selection is necessary toprevent routes from diverging.

SOAR also allows nodes not on the default path toforward packets. These nodes could select their for-warding nodes in the same way as the nodes on thedefault path. However, this could potentially result indiverging forwarding paths and duplicate transmissions.For example, in Figure 4 node A wants to send traffic tonode F . A selects B1, B2, and B3; B1 picks C1 andC2, while B3 selects C5 and C6 as forwarding nodes.Then if B1 and B3 do not hear each other’s forwarding(since the link between them has non-zero loss rate),a copy of the packet is forwarded to C1, C2, C5, andC6. Since C1 and C6 cannot hear each other, the packetwill further diverge and yield redundant transmissions.Therefore not only should we ensure that forwardingnodes make progress and have sufficiently good linkquality from node i, but also we want the selectedforwarding nodes to be adjacent to the default path andevery pair of forwarding nodes has sufficiently goodlink quality between them to avoid diverging paths.This leads to the following two additional constraintsin selecting forwarding nodes:(C3) Each forwarding node is close to at least one nodeon the default path (i.e., with ETX below a threshold).(C4) The ETX of a link between any pair of forwardingnodes is within a threshold.These constraints ensure forwarding nodes have goodconnectivity amongst themselves and to nodes on thedefault path. To achieve this, we filter out the nodesthat do not satisfy (C3), and then successively add theremaining nodes to the forwarding list in an increasingorder of ETX to the destination, continuously ensuringthat (C4) is satisfied. In Section 5.4.1, we found thatadjacent forwarders in ExOR do not always satisfy (C4),which can lead to a partition amongst the forwardingnodes.

Further pruning forwarding nodes: To further reducecoordination overhead and duplicate traffic, we bound

the maximum number of forwarding nodes within aspecified threshold (Our evaluation uses 5 as the upperbound). Meanwhile we may not need to include allthe nodes if the nodes closest to the destination aresufficiently reliable. For example, if the links from A toall five neighbors are 100% in Figure 1, we only needto use one of the neighbors as the forwarding node.This observation suggests that we should only includeenough forwarding nodes (in an increasing order of theirETX to the destination) to bound the loss rate of thevirtual link, which consists of the physical links betweenthe sender and all the forwarding nodes selected so far.

Therefore we start from the forwarding node closestto the destination and iteratively add a node to the finalforwarding list until either the loss of a virtual link isbelow a threshold or the specified maximum number offorwarding nodes is reached. We add forwarding nodesin an increasing order of their ETX to the destinationbecause only when the nodes closest to the destinationhave a sufficiently reliable virtual link can we prune theremaining forwarding nodes. In the other order, evenwhen the forwarding nodes included so far have suffi-ciently reliable links, there is still a benefit of includingadditional forwarding nodes if they are closer to thedestination than those included so far. Finally, if the lossrate of the virtual link is higher than the loss threshold,we replace the last forwarding node added so far withthe forwarding node that has lowest ETX to the currentnode. This reduces the virtual link loss rate and increasesthe chance that the packet makes progress beyond thecurrent node.

3.4 Priority-based Forwarding

We use priority-based forwarding to maximize theprogress each transmission makes. A sender transmits apacket, which specifies a list of forwarding nodes sortedin a non-decreasing order of ETX towards the destina-tion. The forwarding list and its priority are specifiedon a per packet basis. This makes it easy to adapt totopology changes – upon topology changes, a sender cansimply specify a different set of forwarding nodes in adifferent order, and this change takes effect immediately.

Each node hearing the packet first checks if it isincluded in the forwarding list. If not, it discards thepacket. Otherwise, it sets its forwarding timer as follows.The i-th forwarding node on the list sets its forwardingtimer to (i − 1) ∗ δ, where i starts from 1. In thisway, the node with lower ETX towards the destinationforwards the packet earlier, and other nodes hearingits forwarding will cancel their forwarding timer andremove the packet from their queue, thereby avoidingduplicate forwarding.

In order for priority-based forwarding to work well,δ should be large enough to ensure that the transmis-sion from a higher priority node precedes that from alower priority node even under variable queueing andcontention delay. To achieve this, we limit the maximum

number of packets queued on the wireless card to 3. Weuse δ = 45ms, which is appropriate for bulk transfer,targeted by all opportunistic routing protocols, includingSOAR.

3.5 Local Recovery

SOAR uses per-hop network-layer ACKs and retrans-missions to provide best effort reliability. Each node onthe forwarding path will keep retransmitting a packetuntil either the packet is acknowledged by a nodecloser to the destination or the maximum retry count isreached. As with any ACK-retransmission scheme, weneed to address the following issues in our design: (i)when to send ACKs, (ii) what information is containedin ACKs, and (iii) when to retransmit. We will describeour approach to each of these issues.

3.5.1 When to send ACKs?Sending one ACK for every packet yields significantoverhead, and may reduce throughput over a reliablelink. To reduce the overhead of ACKs, SOAR uses acombination of piggyback ACKs and ACK compression asfollows.

A node piggybacks its ACK to a data packet wheneverpossible. Moreover, an ACK piggyback occurs just beforethe data packet is sent to the wireless card to allowthe ACK to carry the latest information about whichpackets have been received. A node considers a packetas received if it receives the packet itself or it hears anACK for this packet from a node closer to the destinationthan itself.

Piggyback ACKs are effective when there are enoughdata packets to transmit. When a node does not havemuch data to send, it should also send stand-alone ACKsto provide timely feedback. To reduce the overhead ofACKs, SOAR uses ACK compression, where an ACKis scheduled either when K data packets have beenreceived or an ACK timer expires. As a further optimiza-tion, before a stand-alone ACK is sent to the wirelesscard, if another data packet coming from the same flowarrives and is within P packets away from the ACK, wecancel the stand-alone ACK since the data packet cancarry the ACK. Our evaluation uses an ACK timer of30 ms, K = 10, and P = 2. Our results show that thecombination of piggyback ACKs and ACK compressionis effective in reducing feedback overhead.

3.5.2 What to send in ACKs?SOAR uses cumulative/selective ACKs to minimize theimpact of ACK loss. The ACKs for each flow containtwo fields: (i) the starting sequence number of the outof order ACKs (start) and (ii) a bit-map of out of orderACKs. (We use a fixed length bit-map with 256 bits). Allthe packets up to start are assumed to be received, andi-th position in the bitmap is 1 if and only if start + i-thpacket is received. We update start so that the largestreceived packet is no more than start+256. This implies

that it is possible that a node may not have receivedall packets up to start even though it assumes so. Thelikelihood of such occurrence is low since the bit-mapsize of 256 bits is generally large enough. Moreover,SOAR is designed to provide best-effort reliability andleaves the upper-layer to ensure full reliability if needed.

When there are multiple flows, a simple approach isto ACK each flow separately. However this would havehigher ACK delay and potentially result in unnecessaryretransmissions. To address this issue, SOAR uses cross-flow ACKs. Whenever a node sends an ACK (regardlessof whether it is a piggyback ACK or stand-alone ACK),it includes not only the ACK for the current flow butalso ACKs for some other flows.

Now the questions are how many flows to ACK eachtime and which flows to include in the ACK. We usethe following constraints to limit the size of cross-flowACKs. (i) The final packet size (including payload) iswithin MTU. (ii) The maximum number of flows to beACKed is no more than a specified number of flows.The limit is set to 4, including the current flow, inour evaluation. (iii) Senders only include an ACK foranother flow when that flow has new packets to beacknowledged since the last transmission of an ACKfor that flow. The last constraint not only reduces cross-flow ACK overhead in the normal situation, but alsoautomatically times out flows that become inactive orchange route (We use the last constraint to conservativelysend cross-flow ACKs. When the previous ACK for flowi is lost, the cross-flow ACK does not include an ACK forflow i, since flow i will generate its own ACKs anyway).When picking which flows to ACK, SOAR favors theflows that have the largest number of new packets toacknowledge since their last ACKs.

3.5.3 When to retransmit data?

In SOAR, each node estimates its retransmission timeout(RTO) in a similar way as done in TCP. Specifically, forevery packet that has not been retransmitted, a nodemeasures the time difference between when the packetis transmitted and when the corresponding ACK (inSOAR) is received. Let T denote the measured round-trip time of the current packet over a virtual link. Thenthe node updates its RTO as shown in Figure 5. RTO isinitialized to 30 ms. Our evaluation uses K = 4, α = 1/8,and β = 1/4 as in TCP [24].

if (T is the first RTT measurement)SRTT = T;RTTVAR = T/2;RTO = SRTT + K*RTTVAR;

elseRTTV AR = (1 − β) × RTTV AR + β × |SRTT − T |;SRTT = (1 − α) × SRTT + α × T ;RTO = SRTT + K ∗ RTTV AR;

end

Fig. 5. Estimation of RTO.

As in TCP, after each retransmission, the timeout ofthe packet is increased exponentially. We update RTO asRTO(i) = RTO(i− 1) ∗ 1.5 where RTO(i) is the timeout

of i-th retransmission. We use 1.5 instead of a commonlyused 2 to reduce the retransmission time, while quicklyadapting to the current network condition.

3.6 Rate Control

Another important design issue is how fast a flow sourceshould transmit data. For both a single flow and multiplesimultaneous flows, not controlling the senders’ datarates can severely degrade network throughput whenthey send more than what the path can support. Thisoccurs because any additional traffic reduces the capacityof the bottleneck links due to wireless interference. Formultiple simultaneous flows, rate control is also useful toimprove fairness and avoid starvation, as shown in [11],[27], [19]. While traditionally rate control is typicallyconsidered the responsibility of a transport layer proto-col, previous works [11], [27], [19] have shown that theexisting congestion control at the transport layer is noteffective for multihop wireless networks. A joint opti-mization of rate limiting and routing is more promising.This is also the approach taken by SOAR. In SOAR eachflow uses end-to-end ACKs from its destination node tocontrol its source’s sending rate.

Sending end-to-end ACKs: A destination sends anend-to-end ACK after receiving every K unique packetsor when a timer expires. The format of an end-to-end ACK is similar to a per-hop ACK in Section 3.5.2.The only difference is an end-to-end ACK contains anadditional totalReceived field, indicating the cumulativenumber of unique packets the destination has receivedso far. This explicit counter helps the source to determinean appropriate sending rate and is more accurate thanusing an ACK bit-map.

Different from per-hop ACKs, end-to-end ACKs areforwarded beyond the immediate neighbors. Since end-to-end ACKs can traverse multiple hops, they are sentless frequently than per-hop ACKs. Our evaluation usesK = 20 and a timeout of 100ms. To reduce the delayin transmitting an end-to-end ACK, the flow destinationtransmits end-to-end ACKs along the shortest path viaMAC-layer unicast to the flow source.

Updating sending rate: In SOAR, a flow source hasa limit on the maximum number of packets sent duringthe current time interval, denoted as Window. Duringtransfers, the sender uses the totalReceived field in theend-to-end ACKs from its destination to update Windowat the beginning of each time interval. As shown inFigure 6, when the number of packets acknowledgedin the previous interval (ACKed(i − 1)) is smaller thanthe minimum window (Min), Window is set to Min.Otherwise, Window is set to ACKed(i− 1) plus a smallincrement (Incr). The rationale is that the network cansupport at least ACKed(i−1) since this is the number ofpackets successfully delivered in the previous interval,and we increment it by Incr to allow the rate to growwhen the network condition improves. Finally, whena sender is unable to send as many packets as were

acknowledged in the previous interval (due to MACscheduling), a Credit is accrued to Window in the suc-cessive interval. Figure 6 shows the pseudo-code. Whilemany rate-control solutions are possible, we choose thisfor its simplicity and effectiveness.

In Interval i:if (ACKed(i − 1) ≤ Min)

Window(i) = Min;else

Window(i) = ACKed(i − 1) + Incr ;

if (Sent(i − 1) < ACKed(i − 2))Credit(i − 1) = ACKed(i − 2) - Sent(i − 1);

elseCredit(i − 1) = 0;

Window(i) = Window(i) + Credit(i − 1);

Fig. 6. SOAR rate control algorithm. (Our evaluation uses200 ms intervals, Min = 1, and Incr = 1.)

4 SIMULATION RESULTS

We evaluate SOAR using NS-2 simulations and testbedexperiments. The former lets us evaluate under a broaderrange of scenarios, while the latter provides a morerealistic setting. In this section, we use NS-2 simulationsto compare SOAR with traditional shortest path routing.In the next section, we will further compare SOAR withtraditional shortest path routing and ExOR using testbedexperiments. Below, we first describe our evaluationmethodology and then present performance results. Ourresults show that SOAR significantly improves goodputand fairness of bulk transfer over traditional routing.

4.1 Evaluation Methodology

We implement SOAR in NS-2 (version 2.29) [23]. We usean ETX-based shortest-path routing protocol as a baselinecomparison. In our ETX implementation, each nodesends one broadcast probe per second, and its neighborsmeasure link loss rates by counting the number of probesreceived over a 10-second time window.

To capture wireless medium losses in real networks,we augment NS-2 to generate packet losses by drop-ping packets received at the MAC layer according to aBernoulli distribution.

Evaluation scenarios: Our evaluation uses 802.11a anddisables RTS/CTS, which is the default setting in mostwireless networks. As in ExOR, we disable auto-rate andset the data rate to 6 Mbps. We generate 6 Mbps CBRtraffic and use a 1000-byte packet size. We use a warm-up period of 500 seconds to let the ETX metrics converge,and then measure network performance over the next110 seconds.

Performance metrics: We quantify the performanceusing two metrics: (i) the average end-to-end goodput ofall flows, where all flows are routed using either shortestpath or SOAR, and (ii) fairness using the traditional Jain’sfairness index [15]. The goodput (i.e., total number of non-duplicate received bits per second) is measured over a

110-second transfer. Jain’s fairness index is defined as(∑

xi)2/(n ∗

∑xi

2), where xi is the goodput of flow iand n is the total number of flows in the network. Forboth metrics, higher values indicate better performanceand fairness.

4.2 Evaluation Results

We evaluate SOAR using a range of network topologiesand traffic demands. First, we study the performance ofa single flow over simple canonical topologies, and thenevaluate the performance of multiple flows using gridand random network topologies.

4.2.1 Single FlowFirst, we use diamond and linear chain topologies toevaluate the performance under a single flow.

0

500

1000

1500

2000

1 0.8 0.6 0.4 0.2

Goodput (Kbps)

p1

SOARShortest Path

0

500

1000

1500

2000

1 0.8 0.6 0.4 0.2

Goodput (Kbps)

p1

SOARShortest Path

(a) 2 intermediate nodes (b) 6 intermediate nodes

Fig. 7. Diamond Topology.Diamond topologies: In the diamond topologies (sim-

ilar to Figure 1), the delivery rate from the source toeach intermediate node (denoted as p1) is varied from0.1 to 1 and all the other links (including the links fromthe intermediate nodes to the source) have delivery rateof 1 (i.e., no loss). Note that there are contention lossesin addition to the above inherent wireless link losses.This is true for all the other evaluation as well. Figure 7compares the goodput of SOAR with the shortest pathrouting for 2 and 6 intermediate nodes. SOAR performssignificantly better for all the values of loss rates. Im-provement ranges from 18.37% to 578.62%. For smallervalues of p1, having more intermediate nodes givesmore opportunism for forwarding packets and resultsin higher goodput improvement. For larger values of p1,we restrict the number of forwarding nodes selected (asdiscussed in Section 3.3.2), so we see similar improve-ment for different numbers of intermediate nodes. Whenp1 = 1, we see SOAR slightly out-performs shortest pathrouting. The performance gain comes from two factors:(i) even when p1 = 1, there are still collision losses, whichSOAR can help to recover via opportunistic routing,and (ii) SOAR uses broadcast transmissions and has noACK overhead, while shortest path routing uses unicasttransmissions and incurs ACK overhead.

Linear chain topologies: Next we use linear chaintopologies to evaluate the effectiveness of SOAR inleveraging unexpectedly long transmissions. The one-hop links are perfect, i.e. the delivery rate of the one-hop link in forward and reverse direction is 1. The two-hop link has p2 delivery rate. In the case of asymmetric

0

1000

2000

3000

4000

5000

0 0.2 0.4 0.6 0.8 1

Goodput (Kbps)

p2

SOAR-2 hopsShortest Path-2 hops

SOAR-4 hopsShortest Path-4 hops

SOAR-6 hopsShortest Path-6 hops

(a) Symmetric losses

0

500

1000

1500

2000

2500

3000

3500

4000

4500

0 0.2 0.4 0.6 0.8 1

Goodput (Kbps)

p2

SOAR-2 hopsShortest Path-2 hops

SOAR-4 hopsShortest Path-4 hops

SOAR-6 hopsShortest Path-6 hops

(b) Asymmetric losses

Fig. 8. Linear Topology - goodput for 2-hop, 4-hop, and6-hop linear chains.

loss rates, p2 is the delivery rate of the two-hop linksin the forward direction (df ), and the delivery rate ofthe reverse direction (dr) is 1. In the case of symmetricloss rates, p2=d2

f =d2r to ensure the bi-directional delivery

rate of p2. Figure 8 compares SOAR with shortest pathrouting for linear chain topologies with symmetric andasymmetric loss rates. We vary p2 from 0.0 to 1.0. Theimprovement is largest (as high as 157.1%) when p2 hasa moderate delivery rate (around 0.5). This is becausewhen p2 is low, there are few long transmissions forSOAR to take advantage of, and when p2 is too high, theshortest path routing already directly uses the reliable“two-hop” path, which is close to optimal in this case.Note that when p2 = 1, SOAR slightly out-performsshortest path routing since SOAR helps to recover colli-sion losses and has no ACK overhead.

We see larger improvement under symmetric lossesthan asymmetric losses. This is because for a fixed p2,df is higher under symmetric losses than asymmetriclosses, thereby providing more opportunities for pack-ets to make progress. Furthermore, dr does not affectSOAR significantly because cumulative/selective ACKsmitigate the effect of ACK losses.

Figure 9 compares SOAR with shortest path routingby varying the number of hops between the source anddestination with p2 = 0.5. Compared to shortest pathrouting, SOAR yields an improvement of 37.2% to ashigh as 256.9%. A 3.5-fold increase occurs in the 8-hop topology, where SOAR has more p2 links to takeadvantage of. We again observe a higher improvementunder symmetric losses for the similar reason as above.

0

500

1000

1500

2000

2500

3000

3500

2 3 4 5 6 7 8

Goodput (Kbps)

Number of hops

SOARShortest Path

(a) Symmetric losses

0

500

1000

1500

2000

2500

3000

2 3 4 5 6 7 8

Goodput (Kbps)

Number of hops

SOARShortest Path

(b) Asymmetric losses

Fig. 9. Linear Topology - goodput comparison acrossdifferent numbers of hops under p2=0.5.

4.2.2 Multiple Flows

Next we evaluate the performance of multiple flowsby randomly choosing source and destination pairs ineither grid or random topologies. For each scenario,we report average goodput and average fairness overfive different runs. In addition, we also report the av-erage improvement in goodput and Jain’s fairness in-dex, which are calculated as: mean( GoodputSOAR

Goodputshortest− 1)

and mean( FairnessSOAR

Fairnessshortest− 1), respectively. In all the

figures, the length of the error bars in the graphs showthe standard deviation. The improvement in (absolute)average goodput and fairness differs from average im-provement in goodput and fairness in that the formeris dominated by the runs with large absolute valueswhereas the latter weighs all the runs equally. Reportingboth metrics allow us to get a more complete pictureon where the improvement comes from. As a concreteexample, consider SOAR’s throughput are 4Mbps in thefirst run and 2Mbps in the second run, whereas thecorresponding values for shortest path are 4Mbps and1Mbps. The average goodput for SOAR is 6Mbps and forshortest path is 5Mbps, which gives 20% improvement in(absolute) average goodput. In comparison, the averageimprovement over the two runs are 50%, since 0%improvement for the first run and 100% improvementfor the second run.

Grid topology: Figure 10 shows the goodput resultsfor different numbers of random flows in a 5x5 gridtopology, where delivery rates of one-hop and two-hoplinks, denoted as p1 and p2, are 1 and 0.5, respectively.SOAR out-performs shortest path routing – its averageimprovement in goodput ranges from 19.9% to 127%.The only exception occurs in 10 simultaneous flows,

200

180

160

140

120

100

80

60

40

20

0

-201210864321

Percentage Improvement

Number of flows

(a) Average improvement

2000

1500

1000

500

01210864321

Goodput (Kbps)

Number of flows

SOARShortest Path

(b) Average goodput

Fig. 10. Goodput results for random flows in a 5x5 grid.

140

120

100

80

60

40

20

0

-201210864321

Fairness Index

Number of flows

(a) Average improvement

1

0.8

0.6

0.4

0.2

01210864321

Fairness Index

Number of flows

SOARShortest Path

(b) Average fairness

Fig. 11. Fairness results for random flows in a 5x5 grid.

where the average goodput of shortest path routing isslightly more than SOAR. However SOAR has 14.4% av-erage improvement for 10 flows. This is because amongthe five runs, one run has several source/destinationpairs that are one hop away. Shortest path routingfavors these flows and starves the other flows, henceit has higher average goodput. However, SOAR usesrate-limiting to give fair share across the flows. Thisfact is evident in Figure 11, which shows that SOARconsistently has a significantly higher fairness index.

Random topology: Next we evaluate random flowsusing 25 nodes in 400m x 400m random topologies. To

0

50

100

150

200

250

300

350

400

121086421

Percentage Improvement

Number of flows

(a) Average improvement

2000

1500

1000

500

0121086421

Goodput (Kbps)

Number of flows

SOARShortest Path

(b) Average goodput

Fig. 12. Goodput results for random 25 nodes placed ina 400 m x 400 m grid.

200

180

160

140

120

100

80

60

40

20

0

-20121086421

Fairness Index

Number of flows

(a) Average improvement

1

0.8

0.6

0.4

0.2

0121086421

Fairness Index

Number of flows

SOARShortest Path

(b) Average fairness

Fig. 13. Fairness results for random 25 nodes placed ina 400 m x 400 m grid.

create each topology, we first randomly place a nodeand then iteratively add nodes as long as they’re intransmission range of an existing node. Transmissionrange is 50 m and the interference range is 100 m. Theloss rates on the links vary from 0% to 80%. The averageneighbors per node is 6.76, which is consistent withprevious evaluations [18]. Figures 12 and 13 show the av-erage goodput and fairness indices of 10 random runs asthe number of flows varies. SOAR out-performs shortestpath routing in both metrics. The average improvement

in goodput ranges from 65.6% to 199.4%, and the averageimprovement in Jain’s fairness index is as high as 144.4%.

4.3 SummaryOur evaluation shows that SOAR yields significantimprovement over shortest path routing. Furthermore,SOAR effectively supports multiple simultaneous flowsby improving both goodput and fairness. The benefit ofSOAR is highest under networks with medium link lossrates.

5 TESTBED EVALUATION

We implement SOAR in a wireless testbed, and compareit with a traditional shortest-path routing protocol (eitherLQSR or SPP, see below) and a seminal opportunisticrouting protocol (ExOR). Furthermore, we disable therate control in SOAR and compare it against each ofthe above protocols. In this section, we first present ourimplementation, and then describe evaluation method-ology and performance results.

Note that MORE [3] is a new opportunistic routingprotocol proposed recently. As shown in [3], its per-formance under multiple flows is very close to thatof ExOR. Given MORE’s similar performance to ExOR,we expect its performance difference from SOAR undermultiple flows to be similar to the difference betweenSOAR and ExOR.

5.1 ImplementationWe implement SOAR in Microsoft Windows XP Profes-sional. Both SOAR and LQSR are implemented in theMicrosoft Research Mesh Connectivity Layer (MCL) [22],which is a 2.5 layer device that resides as a driver for thewireless card. We configure both LQSR and SOAR to usethe ETX metric in the testbed. We use the default MCLETX implementation to periodically obtain topology andlink quality information. Probe packets are sent onceper second, and nodes broadcast their link-state onceevery ten seconds. We set the initial loss rate on a newlydiscovered link to 80%, and provide sufficient warmupperiod for the metrics to converge.

We use the ExOR and SPP implementation in [3].Both are implemented in the Click modular router [6],which runs in Linux. To ensure that the performance ofExOR in Linux and SOAR in Windows is comparable,we compare their performance over a set of controlledtopologies, including 1-hop, 2-hop, and 3-hop lineartopologies. As shown in Table 1, the performance dif-ference due to different implementation platforms isnegligible. We also compare the performance of LQSRand SPP over all the runs. We find that the two shortestpath protocols perform within 10% of each other, furthervalidating that the difference in operating systems doesnot account for the difference observed between SOAR,ExOR and shortest path. Therefore the difference inthe following performance numbers of SOAR versusExOR is due to efficiency of these protocols rather thandifferent implementation platforms.

Hops p1 p2 SOAR ExOR1 1 0 4.093 4.1382 1 0 2.035 2.0293 1 0.5 1.894 1.885

TABLE 1Average goodput (Mbps) of one flow in controlled lineartopologies, where p1 and p2 denote the delivery rates of

one-hop and two-hop links, respectively.

Fig. 14. Node placement in the testbed topology.

5.2 Testbed Description

Our testbed consists of 18 DELL dimensions 1100 PCs,located on two adjacent floors of an office building (seeFigure 14). Each machine has a 2.66 GHz Intel Celeron DProcessor 330 with 512 MB of memory and is equippedwith a 802.11 a/b/g NetGear WAG511 wireless card. Toavoid interference with the campus 802.11b/g network,we set the nodes to use 802.11a. Our building has noother 802.11a users. The RTS/CTS handshake is disabled,which is the default setting of the drivers. We disableauto-rate and set the data rate to 6Mbps. Each node usesthe lowest transmission power so that we can get up to 6hops in the network. We keep other parameters, such asthe number of unicast retransmission attempts, to theirdefault values.

5.3 Evaluation Methodology

We conduct experiments in our testbed as follows. Eachof our experiments consists of a warm-up period anda data transfer period. During the warm-up period,we upload the routing protocol’s driver to all nodes(either SOAR, LQSR, SPP or ExOR) and let the routingmetrics converge. We use 10 minutes as the warm upperiod. During the data transfer period, we evaluate theperformance of each protocol with a varying number ofnetwork flows. As in the evaluation of previous routingprotocols for mesh networks (e.g., ETX [7], ExOR [2],LQSR [8]), the source and destination nodes of each floware randomly selected. Transfers start in parallel and werun a 1-minute transfer for each pair. We augment thettcp tool included with MCL to generate CBR trafficat 6 Mbps and measure the goodput of each flow.The ExOR and SPP implementations also generate CBRtraffic and measure goodput via Click. All measurements

and transfers are run during the nights and weekendsin order to control the topology as much as possible.

As previously mentioned, we find the performanceof the two shortest path protocols to be within 10% ofeach other. Therefore, when we present the performanceof the shortest path, we present the maximum betweenLQSR and SPP. We take great care to ensure that con-sistent topologies are used by the Windows protocolsand the Linux protocols. The consistency between theprotocols within a given operating system is easy toachieve because they both run on the same operatingsystem and temporal variation in topologies is small andeasily smoothed out over multiple random runs. Ensur-ing consistency across operation systems is harder toguarantee since SOAR and LQSR run on Windows andExOR and SPP run on Linux. Even though on both plat-forms we use an equivalent transmission power level,which results in similar received signal strength, westill observe substantial difference in the delivery ratesof the links across the platforms. To ensure consistenttopologies, before experiments we measure the deliveryratio of every link in the network. Whenever the links inWindows have higher delivery rates than those in Linuxby 0.4, we use MCL’s artificialdrop functionality [22]to generate additional artificial loss to compensate forthe difference in their delivery rates (We do not imposeartificial losses to the links under Linux even if they aresignificantly better to favor ExOR). With the imposedartificial losses, the delivery rates of links in Windowsare similar to those in Linux – the average difference ofall entries in the connectivity matrices under Windowsand Linux is 0.0206, where an entry in the connectivitymatrix (i, j) denotes the delivery rate from node i tonode j.

5.4 Evaluation Results

We compare the performance of SOAR, ExOR and short-est path. In all cases, we vary the number of simul-taneously active flows. For each scenario, we conduct30 random runs, where each run consists of a differentset of source/destination pairs. We report the averagegoodput and average fairness index, along with averageimprovement in goodput and fairness index. The lengthof error bars in the graphs corresponds to standarddeviation.

5.4.1 Protocol Results

Figure 15(a) shows the average goodput improvement ofSOAR and ExOR over shortest path (i.e., averaging thepercentage of goodput improvement across 30 randompairs), and Figure 15(b) shows the average goodput. Aswe can see, in all cases SOAR out-performs shortest pathand ExOR. The improvement of SOAR over shortestpath ranges from 11.2% (1 flow) to 85.3% (8 flows).The improvement of SOAR over ExOR ranges from7.1% (1 flow) to 83.1% (8 flows). Generally, SOAR hasa larger difference in the average goodput for larger

-100

0

100

200

300

400

500

600

700

1 2 4 6 8 10 12

Perc

enta

ge Im

pro

vem

ent

Number of Flows

ExORSOAR

(a) Average goodput improvement over shortest path

0

1000

2000

3000

4000

5000

1 2 4 6 8 10 12

Goodput (K

bps)

Number of Flows

Shortest PathExORSOAR

(b) Average goodput

Fig. 15. Protocol goodput results in the testbed.

sets of flows. This is because SOAR judiciously selectsforwarding nodes to limit resource consumption andrate limits the source; these features are especially im-portant under a larger number of flows. We see theaverage goodput of ExOR is comparable to that ofshortest path in our testbed. We find in a very smallnumber of instances ExOR yields close to 0 goodput.After taking a closer look, we find that ExOR selectsnodes as forwarders if they forward at least 10% ofthe total transmissions in simulation but it does notguarantee the selected forwarding nodes are connected.Disconnection among forwarding nodes can result in se-rious performance degradation because the data packetscannot make progress across the partition. Even thoughthe occurrence of such disconnection is very low in adense network, including our testbed, the forwarding listselection in ExOR is inadequate for general topologies.SOAR explicitly ensures forwarding nodes are connectedand achieves reasonable goodput in all cases. We alsofind that the strict timing constraints imposed on ExOR’sforwarders inhibits spatial reuse on multi-hop flows,which can limit the effectiveness of opportunistic trans-missions. Figure 15(a) shows the improvement of SOARover shortest path is consistently high. While this graphshows ExOR can provide improvement over shortestpath, this improvement is still less than SOAR’s.

To further understand the performance benefit ofSOAR across different numbers of flows, we study thelink loss characteristics in our testbed. We classify a linkas a binary link if its loss rate is ≤ 20% or ≥ 80%, and clas-sify a link as a non-binary link if its loss rate is between20% and 80%. The number of non-binary links in thenetwork directly affects the performance improvement

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.2 0.4 0.6 0.8 1

Cum

ulat

ive

frac

tion

Link Loss Rate

100 Kbps CBR200 Kbps CBR500 Kbps CBR

1000 Kbps CBRNo Traffic

Fig. 16. CDF of link loss rates with and without traffic inour testbed using 802.11a.

of SOAR. Figure 16 shows the cumulative distribution(CDF) of loss rates across different links in our testbedas we vary the amount of traffic. As it shows, thereare only 15% non-binary links without traffic. We thenmeasure link loss rates when there are 10 simultaneousflows in the network and each flow generates CBR trafficvarying from 100 Kbps to 1000 Kbps. We observe thatinjecting traffic turns many reliable links into lossy links,thereby creating more non-binary links. For example,when there are 10 random flows, each with 200 Kbps,almost 55% links are non-binary. The larger number ofnon-binary links creates more opportunism and enlargesthe performance benefits of SOAR. ExOR has a toughertime realizing these benefits because the fine-grainedtiming constraints are hard to enforce on forwarders overmany flows. This matches the performance results inFigure 15.

-20

0

20

40

60

80

100

120

140

1 2 3 5 8 10 12

Perc

enta

ge Im

pro

vem

ent

Number of Flows

ExORSOAR

(a) Average improvement over shortest path in fairness index

0

0.2

0.4

0.6

0.8

1

1 2 3 5 8 10 12

Jain

’s F

airness Index

Number of Flows

Shortest PathExORSOAR

(b) Average fairness index

Fig. 17. Protocol fairness results in the testbed.

Next we study the fairness index. Figure 17(a) shows

the improvement of Jain’s Fairness Index as we varythe number of simultaneous flows in the network, andFigure 17(b) further shows the average Jain’s FairnessIndex for each number of flows. We see that SOAR gen-erally provides better fairness than ExOR and shortestpath. For instance, the average fairness of SOAR in theeight flow cases is about 60% higher than the averagefairness of shortest path and about 50% higher than theaverage fairness of ExOR. The rate control in SOAR oftenprevents the starvation of flows and provides betteropportunities for equal medium access across differentflows.

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 1000 2000 3000 4000 5000

Cumulative Fraction

Goodput (Kbps)

SOARExORShortest Path

(a) 4 flows

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 1000 2000 3000 4000 5000

Cumulative Fraction

Goodput (Kbps)

SOARExORShortest Path

(b) 8 flows

Fig. 18. CDFs of protocol goodput over all flows under 4or 8 simultaneous flows in the testbed.

Figures 18(a) and (b) show CDFs of the goodput forall the flows included over 30 random runs under 4 and8 simultaneous flows, respectively. In all the scenarios,SOAR out-performs ExOR and shortest path routing. Inthe 4-flow case, the median per-flow goodput of SOAR is72% higher than the median goodput of ExOR (300 Kbpsvs. 174 Kbps). The 90th percentile goodput improves by17% (951 Kbps vs. 807 Kbps). In the 8-flow case, themedian goodput of SOAR is 2.63 times higher than themedian goodput for ExOR (100 Kbps vs. 38 Kbps). The90th percentile goodput of SOAR again nearly doublesthat of ExOR (1339 Kbps vs. 681 Kbps). In addition, weobserve that 25% of the flows in ExOR get starved (i.e.,throughput ≤ 10 Kbps), and SOAR reduces the numberof starved flows to 10%.

Next, Figure 19 plots goodput as we vary the numberof nodes while fixing the number of flows to 4. As wecan see, increasing the number of nodes significantly in-creases SOAR’s throughput, since it has more forwardersto take advantage of path diversity. In comparison, theperformance improvement for shortest path and ExORis much smaller.

Finally, Figure 20 further compares the performance ofExOR and shortest path with two versions of SOAR: one

0

1000

2000

3000

4000

5000

8 12 16 18

Goodput (K

bps)

Number of Nodes

Shortest PathExORSOAR

Fig. 19. Goodput under a varying number of nodes whenthe number of flows is fixed to 4 in the testbed.

enabling the rate limiting in Figure 6 and one disablingthe rate limiting (SOAR-nrl). Since the comparison wasconducted at a later time, the topology in use was dif-ferent from the one used in Figure 15. As before, SOARout-performs ExOR, which in turn out-performs shortestpath. Comparing SOAR with SOAR-nrl, we observe thatSOAR-nrl performs slightly better under the 1-flow case,since SOAR’s rate limiting is a little conservative. How-ever, SOAR out-performs SOAR-nrl under more flowsas the importance of rate limiting increases with moreflows. In addition, SOAR-nrl out-performs ExOR andshortest path due to better selection of forwarding pathsand avoiding strict timing constraints on the forwarders.SOAR-nrl improves ExOR by up to 13%, and improvesshortest path by 9-27%. As we would expect, SOARyields better fairness than all the other schemes, whileSOAR-nrl provides similar fairness as ExOR and shortestpath. We omit the figure in the interest of brevity. Theseresults suggest that both routing and rate limiting designin SOAR are useful and they each help to contribute tothe final performance gain.

0

500

1000

1500

2000

2500

3000

3500

1 2 4 6 8 10 12

Go

od

pu

t (K

bp

s)

Number of Flows

Shortest PathExORSOARSOAR-nrl

Fig. 20. Comparing SOAR without rate limit in thetestbed.

6 CONCLUSION

In this paper, we develop SOAR, a novel opportunisticrouting protocol. SOAR effectively realizes opportunisticforwarding by judiciously selecting forwarding nodesand employing priority-based timers. It further incorpo-rates adaptive rate control to dynamically adjust send-ing rates according to network conditions and recoverslost packets using efficient local feedback and recovery.

The combination of these techniques enables SOAR toachieve high efficiency and effectively support multiplebulk transfer flows. Our simulations and testbed exper-iments demonstrate the effectiveness of SOAR.

Through the design and implementation of SOAR, wegain the following insights. First, judiciously selectingforwarding nodes and avoiding diverging paths are im-portant especially for supporting multiple simultaneousflows. Second, rate control is important to the perfor-mance of a routing protocol. It improves both aggregategoodput and fairness among multiple competing flows.The joint design of routing and rate limiting as in SOARis useful, and may be useful to the design of otheropportunistic routing protocols. Finally, the default pathselection algorithm is important to the performance ofopportunistic routing. In this paper, we use the ETXmetric to select the default path, but SOAR can easilyaccommodate other default path selection metrics. Theperformance of SOAR could further improve with en-hanced default path selection, which we plan to investi-gate as part of our future work.

REFERENCES

[1] B. Awerbuch, D. Holmer, and H. Rubens. Provably securecompetitive routing against proactive Byzantine adversaries viareinforcement learning. In JHU Tech Report Version 1, May 2003.

[2] S. Biswas and R. Morris. ExOR: opportunistic multi-hop routingfor wireless networks. In Proc. of ACM SIGCOMM, Aug. 2005.

[3] S. Chachulski, M. Jennings, S. Katti, and D. Katabi. Tradingstructure for randomness in wireless opportunistic routing. InProc. of ACM SIGCOMM, Aug. 2007.

[4] C. Chen, D. Aksoy, and T. Demir. Processed data collection usingopportunistic routing in location aware wireless sensor networks.In Proc. of International Conference on Mobile Data Management, May2006.

[5] R. Choudhury and N. Vaidya. MAC-layer anycasting in wirelessad hoc networks. In Proc. of Hot Topics in Networks (HotNets), Nov.2003.

[6] Click. http://pdos.csail.mit.edu/click/.[7] D. D. Couto, D. Aguayo, J. Bicket, and R. Morris. A high-

throughput path metric for multi-hop wireless routing. In Proc.of ACM MOBICOM, Sept. 2003.

[8] R. Draves, J. Padhye, and B. Zill. Comparison of routing metricsfor multi-hop wireless networks. In Proc. of ACM SIGCOMM,Aug. 2004.

[9] R. Draves, J. Padhye, and B. Zill. Routing in multi-radio, multi-hop wireless mesh networks. In Proc. of ACM MOBICOM, Sept.- Oct. 2004.

[10] R. Dube, C. Rais, K.-E. Wang, and S. Tripathi. Signal stabilitybased adaptive routing (SSA) for ad-hoc mobile networks. InIEEE Personal Comm, Feb. 1997.

[11] Z. Fu, H. Luo, P. Zerfos, S. Lu, L. Zhang, and M. Gerla. The impactof of multihop wireless channel on TCP performance. IEEE Trans.on Mobile Computing, Mar. 2005.

[12] T. Goff, N. Abu-Aahazaleh, D. Phatak, and R. Kahvecioglu. Pre-emptive routing in ad hoc networks. In Proc. of ACM MOBICOM,Jul. 2001.

[13] Y. C. Hu and D. B. Johnson. Design and demonstration of liveaudio and video over multi-hop wireless networks. In Proc. ofMILCOM, Oct. 2002.

[14] K. Jain, J. Padhye, V. N. Padmanabhan, and L. Qiu. Impact ofinterference on multi-hop wireless network performance. In Proc.ACM MOBICOM ’2003, Sept. 2003.

[15] R. Jain, D. Chiu, and W. Hawe. A quantitative measure of fairnessand discrimination for resource allocation in shared computersystems. DEC Research Report TR-301, Sep 1984.

[16] D. B. Johnson, D. A. Maltz, and J. Broch. DSR: The dynamicsource routing protocol for multihop wireless ad hoc networks.In Ad Hoc Networking, 2001.

[17] R. Karrer, A. Sabharwal, and E. Knightly. Enabling large-scalewireless broadband: The case for TAPs. In Proc. of HotNets, Nov.2003.

[18] L. Kleinrock and J. Silvester. Optimum transmission radii forpacket radio networks or why six is a magic number. In NTC,Dec. 1978.

[19] Y. Li, L. Qiu, Y. Zhang, R. Mahajan, and E. Rozner. Predictableperformance optimization for wireless networks. SIGCOMMComput. Commun. Rev., 38(4):413–426, 2008.

[20] Y. Li, L. Qiu, Y. Zhang, R. Mahajan, Z. Zhong, G. Deshpande,and E. Rozner. Effects of interference on throughput of wirelessmesh networks: Pathologies and a preliminary solution. In Proc.of HotNets-VI, Nov. 2007.

[21] R. Madan, S. Cui, S. Lall, and A. Goldsmith. Cross-layer designfor lifetime maximization in interference-limited wireless ad hocnetworks. IEEE trans. Wireless Communications, 2006.

[22] Mesh connectivity layer. http://research.microsoft.com/mesh/#software.[23] The network simulator – ns-2. http://www.isi.edu/nsnam/ns/.[24] V. Paxson and M. Allman. Computing tcp’s re-

transmission timer. IETF Internet DRAFT, 2000.http://www3.ietf.org/proceedings/00jul/I-D/paxson-tcp-rto-01.txt.

[25] C. E. Perkins and P. Bhagwat. Highly dynamic destination-sequenced distance-vector routing (dsdv) for mobile computers.In Proc. of ACM SIGCOMM, Aug.-Sept. 1994.

[26] C. E. Perkins and E. M. Royer. Ad hoc on-demand distance vectorrouting. In Proc. of the Workshop on Mobile Computing Systems andApplications, Feb. 1999.

[27] S. Rangwala, R. Gummadi, R. Govindan, and K. Psounis.Interference-aware fair rate control in wireless sensor networks.In Proc. of ACM SIGCOMM, Sept. 2006.

[28] MIT Roofnet. http://www.pdos.lcs.mit.edu/roofnet/.[29] E. Rozner, J. Seshadri, Y. Mehta, and L. Qiu. Simple opportunistic

routing for wireless mesh networks. In Second IEEE Workshop onWireless Mesh Networks, Sept. 2006.

[30] Seattle wireless. http://www.seattlewireless.net.[31] W. Wang, X. Liu, and D. Krishnaswamy. Robust routing and

scheduling in wireless mesh networks. In Proc. of IEEE SECON,2007.

[32] C. Westphal. Opportunistic routing in dynamic ad hoc networks:The oprah protocol. In Proc. of IEEE MASS, Oct. 2006.

[33] City-wide Wi-Fi rolls out in UK. http://news.bbc.co.uk/2/hi/technology/4578114.stm.

[34] Cities unleash free Wi-Fi. http://www.wired.com/gadgets/wireless/news/2005/10/68999.

[35] Y. Yuan, H. Yuan, S. H. Wong, S. Lu, and W. Arbaugh. ROMER:resilient opportunistic mesh routing for wireless mesh networks.In Proc. of IEEE WiMESH, Sept. 2005.

[36] Z. Zhong and S. Nelakuditi. On the efficacy of opportunisticrouting. In Proc. of IEEE SECON, Jun. 2007.

Eric Rozner is a Ph.D. candidate in the Depart-ment of Computer Sciences at the Universityof Texas at Austin. He received B.S. degreefrom University of Wisconsin at Madison in 2005.His research interests are Internet and wirelessnetworking.

Jayesh Seshadri received his M.A. in ComputerSciences from the University of Texas at Austinin 2007. He is a Software Engineer at VMware,Inc, Palo Alto, CA, with a specific focus on dis-tributed systems management. He also has aB.E. in Computer Science and Engineering fromAnna University, India.

Yogita Ashok Mehta received her Masters de-gree from University of Texas at Austin in 2007.She received her Bachelor’s in Engineering fromUniversity of Mumbai in 2005. She is currentlyworking as Software Engineer at Google Inc.

Lili Qiu received her Ph.D. from Cornell Uni-versity in 2001. She is an Assistant Professorin the Department of Computer Sciences at theUniversity of Texas at Austin. Her research areais computer networks, with special focuses onwireless network management, content distri-bution, Internet measurement, and overlay net-works. She is a recipient of NSF career award(2006), and a senior IEEE member. She was aresearcher at Microsoft Research during 2001-2004.