[ieee 27th international conference on distributed computing systems (icdcs '07) - toronto, on,...

Protocol Design for Dynamic Delaunay Triangulation �

Dong-Young Lee and Simon S. LamDepartment of Computer SciencesThe University of Texas at Austin

�dylee, lam�@cs.utexas.edu

Abstract

Delaunay triangulation (DT) is a useful geometric struc-ture for networking applications. In this paper we investi-gate the design of join, leave, and maintenance protocols toconstruct and maintain a distributed DT dynamically. Wedefine a distributed DT and present a necessary and suf-ficient condition for a distributed DT to be correct. Thiscondition is used as a guide for protocol design. We presentjoin and leave protocols as well as correctness proofs forserial joins and leaves. In addition, to handle concurrentjoins and leaves as well as node failures, we present a main-tenance protocol. An accuracy metric is defined for a dis-tributed DT. Experimental results show that our join, leaveand maintenance protocols are scalable, and they achievehigh accuracy for systems under churn and with node fail-ures. We also present application protocols for greedy rout-ing, clustering, broadcast, and multicast within a radius,and discuss and prove their correctness.

1. IntroductionWith almost a hundred years of history, DT [1] and

Voronoi diagram [2] have been widely used for many ap-plications in different fields of science and engineering, in-cluding computer science. A triangulation in a 2D spacemeans, for a given set of nodes, constructing edges betweenpairs of nodes such that the edges form a non-overlappingset of triangles that cover the convex hull of the nodes. DTin a 2D space is usually defined as a triangulation such thatthe circumcircle of each triangle does not include any nodeother than the vertexes of the triangle. DT can be similarlygeneralized for higher dimensions.

An interesting property of DT is that it connects a nodeto other nodes that surround the node. This property may beuseful in simulation-type applications, including distributedvirtual reality systems and multiplayer on-line games, sincean entity in a simulation usually interacts with other en-tities around it. For example, a molecule interacts withother molecules around it, and a character in on-line games

�Research sponsored by National Science Foundation ANI-0319168and CNS-0434515.

mostly interacts with other characters around it. Further-more, we also design a protocol to multicast a messagewithin a given radius from the source node, which will beuseful for many simulation-type applications such as multi-player on-line games.

Another property of DT in networking context is thatgreedy routing always succeeds on a DT [3]. In greedy rout-ing, a node forwards a message to one of its neighbors that isclosest to a given destination node. Note that greedy routingon an arbitrary graph is prone to the risk of being trappedat a local optimum, i.e., routing stops at a non-destinationnode that is closer to the destination than any of its neigh-bors. However, on a DT it is guaranteed that greedy rout-ing always succeeds to find the destination node. Note thatgreedy routing does not always find a shortest route. How-ever, the quality of the greedy route is often very good, sincethe length of an optimal route between a pair of nodes on aDT is within a constant time of the direct distance [4].

While our approach is more system-oriented comparedto previous work, our protocols are also based on a rigoroustheoretical foundation. In a distributed DT, each node in asystem keeps a set of its neighbor nodes. We specify a dis-tributed DT by the neighbor sets of all nodes. A distributedDT is correct when it is equivalent to its corresponding cen-tralized DT. That is, a distributed DT is correct when eachnode has the same set of neighbors as on the correspond-ing centralized DT. (We will define a distributed DT andits correctness more carefully in section 2.) In section 3,we identify a necessary and sufficient condition to achievecorrectness. We use this condition as a guide for design-ing join, leave, and maintenance protocols for constructingand maintaining a distributed DT. Our join and leave pro-tocols are proved to be correct in the following sense: If adistributed DT is correct when a new node joins or an exist-ing node leaves and there is no other concurrent join, leaveor failure then, at the end of protocol execution, the result-ing distributed DT is correct. Thus if a sequence of joinsand leaves occur serially (i.e., one finishes before anotherstarts), the distributed DT is correct whenever protocol exe-cution finishes.

In practice, nodes may join and leave concurrently. Fur-thermore, nodes may fail at any time, immediately breaking

27th International Conference on Distributed Computing Systems (ICDCS'07)0-7695-2837-3/07 $20.00 © 2007

correctness of the distributed DT. Our maintenance proto-col has been designed to address such scenarios. We donot have a convergence proof for the maintenance protocol.However, in every one of a large number of experimentsconducted to date, our maintenance protocol converged to acorrect DT some time after a long period of system churn,during which nodes join and leave (also fail) concurrentlyand frequently.

Note that even in the case of serial joins and leaves, cor-rectness of a distributed DT is, strictly speaking, broken assoon as a node joins or leaves, and it is recovered only at theend of protocol execution. Therefore a correct distributedDT is impossible to achieve continually. We observe thatsome applications can benefit from an incorrect distributedDT as long as it is sufficiently “accurate.” Thus the accu-racy of a distributed DT over a long duration of time is amore useful metric in practice than the notion of conver-gence to correctness. We will define an accuracy metric fora distributed DT, and show that our protocols achieve highaccuracy under different scenarios of system churn.

In addition to protocols to construct and maintain a dis-tributed DT, we present several application protocols, in-cluding greedy routing, clustering, broadcast, and multicastwithin a radius. As we discussed earlier, it is known thatgreedy routing from a node to another node on a DT al-ways succeeds. Then we prove that greedy routing can alsobe used to locate an existing node that is closest to a givenpoint (or a node that is not in the system yet). As an ap-plication of the protocol to find the closest existing node,we present a node clustering protocol. Given a set of nodesand an upper bound on the radius of a cluster, the cluster-ing protocol partitions nodes into clusters of radii within thegiven upper bound. In the protocol, each cluster has a cen-ter node and the center nodes form a distributed DT. Similarapproaches to clustering are found in prior work, based on arandom graph of clusters [11] or a complete graph of clus-ters [12]. Note that greedy routing on a random graph is notguaranteed to succeed and a complete graph may result inlimited scalability.

Our broadcast protocol is based on the reverse path ofgreedy routing, and is named GRPB (greedy reverse pathbroadcast). GRPB does not require any knowledge ofglobal triangulation or per-session state. A node deter-mines its next-hop nodes to forward a broadcast messagesolely using local information, namely the coordinates ofits neighbor nodes and the source node.

We observe that the distance from a source node to eachhop in GRPB monotonically increases, since the distanceto a destination node decreases in greedy routing. There-fore our protocol to multicast within a given radius easilyfollows. RadGRPM (radius greedy reverse path multicast)is basically the same as GRPB, except that it additionallychecks whether the next-hop nodes are within the radius

from the source node. RadGRPM also keeps the advan-tage of GRPB that it does not require any global informationor per-session state. RadGRPM is simple and is useful forsimulation-type applications. For example, an explosion ofa bomb in a battlefield simulation will affect entities withinsome range and will be observed within a longer range.

Experimental results show that our protocols are scal-able, and work very well under system churn, i.e., whenconcurrent joins and leaves occur frequently. Even withungraceful node failures, which inevitably result in an in-correct distributed DT, the maintenance protocol recoverscorrectness some time after churn and failures stop.

The organization of this paper is as follows. In section 2,we introduce concepts and definitions of distributed DT andalso present application protocols. In section 3, we present acorrectness condition for a distributed DT. Then we presentour join, leave and maintenance protocols, discuss their cor-rectness, and introduce an accuracy metric. In section 4, ex-perimental results are presented. We discuss related work insection 5 and conclude in section 6.

2. Distributed Delaunay triangulation

In this section we introduce DT, Voronoi diagram anddistributed DT. Consider a set of nodes. Conceptually,nodes are points in a Euclidean space. The results andprotocols in this paper are for �-dimensional spaces, where� � �. Most previous results on distributed DT in the liter-ature are limited to 2D [5, 8, 9] and 3D [6] spaces. The onlyprior work on distributed dynamic DT for �-dimensionalspaces is by Simon et al. [7]. See section 5 for more details.

We first define Voronoi diagram of a set of given nodesand then define DT as the dual of the Voronoi diagram.Note that there is another way of directly defining DT usingcircumcircles of triangles (or circum-hyperspheres of sim-plexes in higher dimensions), as was briefly introduced inthe introduction. Since the properties of DT of interest to uscome from Voronoi diagram, we believe that this approachis appropriate in our context. Lastly, we define distributedDT. In a distributed DT, each node maintains a set of itsneighbor nodes. We define a distributed DT by the neigh-bor sets of all nodes.

In the second part of this section, applications of DT arediscussed. An important and well-known property of DTis that a simple greedy routing algorithm is guaranteed tosucceed on a DT, without being stuck at a local optimum[3]. We prove a similar property that greedy routing canalso find the closest node to a given point. Clustering ofnetwork nodes is an example for which this property canbe utilized. We also present protocols for broadcast andfor multicast within a radius, and prove correctness for theprotocols.


��

Definition 1 Consider a set of nodes � in a Euclideanspace. The Voronoi diagram of � is a partitioning of thespace into cells such that a node � � � is the closest nodeto all points within its Voronoi cell � ��.

That is, � �� for any � � �� where �� denotes thedistance between and . Note that a Voronoi cell in a�-dimensional space is a convex �-dimensional polytopeenclosed by �� -dimensional facets.

Definition 2 Consider a set of nodes � in a Euclideanspace. � �� and � �� are neighboring Voronoi cells,or neighbors of each other, if and only if � �� and� �� share a facet.

Definition 3 Consider a set of nodes � in a Euclideanspace. The Delaunay triangulation of � is a graph on �

where two nodes � and � in � have an edge between themif and only if � �� and � �� are neighbors of eachother.

Figure 1 shows a Voronoi diagram (dashed lines) for a setof nodes in a 2D space and a DT (solid lines) for the sameset of nodes. Note that � �� and � �� are neigh-bors of � �� but � �� is not, since � �� and� �� share only a point. Similarly, in a 3-dimensionalspace, Voronoi cells that share only an edge or a point arenot neighbors. We also say that � and � are neighbors ofeach other when � �� and � �� are neighbors ofeach other. Also note that facets of a Voronoi cell perpen-dicularly bisect edges of a DT. Therefore, a DT is the dual ofa Voronoi diagram.1 Let us denote the DT of � as �� .

Definition 4 A distributed Delaunay triangulation of aset of nodes � is specified by � �� , where�� represents the set of �’s neighbor nodes, which is locallydetermined by �.

Definition 5 A distributed Delaunay triangulation of a setof nodes � is correct if and only if both of the followingconditions hold for every pair of nodes �� : i) if thereexists an edge between � and � on the global DT of �, then� � �� and � � ��, and ii) if there does not exist an edgebetween � and � on the global DT of �, then � �� and� �� .

That is, a distributed DT is correct when for every node�, �� is the same as the neighbors of � on �� . Since� does not have global knowledge, it is not straightforwardto achieve correctness. We will identify the condition toachieve correctness for a distributed DT in section 3.

1In geometry, polyhedra are associated into pairs called duals, wherethe vertices of one correspond to the faces of the other.

u

v

w

x

Figure 1. A Voronoi diagram (dashed lines)and the corresponding DT (solid lines) in a2-dimensional space.

��

In this section we present several protocols to illustratethe usefulness of distributed DT for networking applica-tions. We assume for now that a set of nodes � form adistributed DT. Our protocols to construct and maintain adistributed DT are deferred to section 3. We also assumethat nodes are associated with their coordinates. When anode “knows” other nodes, it also knows their coordinates.That is, a node knows its own coordinates, its neighbor’s co-ordinates, and the coordinates of other nodes that it knowssuch as the destination node in routing and the source nodein broadcasting. The distance between any two nodes canbe calculated from their coordinates.

Greedy routingA well-known property of DT is that greedy routing al-

ways succeeds on a DT [3]. In greedy routing, a nodeforwards a message to the closest node to the destinationamong its neighbors. As with many greedy approaches, thegreedy routing algorithm is prone to risk of being stuck ata local optimum. That is, on an arbitrary graph, a non-destination node may be closer than any of its neighborsto the destination, thus stopping greedy routing at the node.However, on a DT, it is guaranteed that greedy routing suc-ceeds to deliver a message to the destination node. Further-more, the quality of the greedy route is often very good,since the length of an optimal route between a pair of nodeson a DT is within a constant time of the direct distance [4].

Finding the closest existing nodeSimilar to the previous application of greedy routing,

a DT may be utilized in finding the closest existing nodeto a given point. (Note that the given point may not bea node in the DT.) Finding the closest existing node is acommon operation in many Internet applications, includingserver selection, node clustering, and peer-to-peer overlaynetworks.2 We have proved [14] that greedy routing always

2If topology-aware virtual coordinates (e.g. [17]) are used for Internetapplications, application performance would be affected by accuracy of thevirtual coordinates.


succeeds to find a closest existing node to a given point aslong as it is run on a DT, using an approach similar to Boseand Morin’s proof [3] that greedy routing always succeedsto arrive at a given destination node on a DT.

Clustering of network nodesTo illustrate an application of finding the closest exist-

ing node to a given point, we present a simple clusteringprotocol of network nodes. The protocol is a distributedversion of a centralized clustering algorithm adopted from[10].3 The upper bound � of the radius of a cluster is givenas a parameter. In the centralized algorithm, nodes are con-sidered sequentially in deciding whether they should join anexisting cluster or create a new cluster. The first node con-sidered creates a new cluster and becomes the center of it,since there is no existing cluster. From the second node on,the considered node is tested to see if its distance to the cen-ter of the closest existing cluster is within � or not. If so,the considered node joins the cluster; otherwise it createsits own cluster and becomes the center of it. The algorithmstops when all nodes are considered. Note that the result ofclustering may be different depending on the order in whichnodes are considered [10].

Our clustering protocol is a distributed version of thiscentralized algorithm. The main challenge in converting itinto a distributed version is to find the closest existing clus-ter without global knowledge. We solve this problem byutilizing greedy routing on a DT. Recall that each clusterhas a center node. In our protocol, existing center nodesform a distributed DT. A non-center node does not partici-pate in the distributed DT. When a node � joins the system,it first finds the closest existing center node by using greedyrouting on the distributed DT of the center nodes. Supposethat the center node �� is found. If the distance from � to�� is within the upper bound�, � becomes a member of thecluster centered at ��; otherwise � creates its own cluster,becomes the center node of the new cluster, and joins thedistributed DT.

Other distributed approaches to clustering are found inprior work. In [11], clusters form a random graph and ajoining node may fail to find the closest existing cluster.In [12], every node maintains links to every other cluster,limiting scalability. (The scalability issue is addressed in[12] by introducing a hierarchy of clusters.) Our protocolfinds the closest cluster for a joining node and is scalable.

Broadcast using reverse pathAs was discussed earlier, the greedy routing algorithm

finds a path from a source node to a destination. Considersuch paths from all nodes in � to a node � � �. The unionof the paths is a tree rooted at �. Therefore by reversing

3There are different measures of goodness in clustering algorithms.The main objective of this algorithm is to get clusters whose radii arewithin a given upper bound. For clusterings that satisfy the upper bound,a secondary measure may be the number of clusters. This algorithm doesnot optimize the number of clusters.

s

u

v

w

x

s

u

v

w

Figure 2. An ambiguous situation due to lim-ited knowledge in GRPB.

the direction of each path, we get a broadcast tree froma source node � to every other node in �. We introducea simple broadcast protocol which utilizes the reverse-pathtree. Note that our protocol does not require knowledge ofthe global triangulation over �. Each node � is assumedonly to know its set of neighbor nodes, and determines towhich node(s) it should forward a message based on its lo-cal knowledge.

The idea of using reverse path for broadcast goes backto as early as 1978 [13]. In the context of DT, HyperCast[5] is the first system to introduce the idea. Our protocol isdifferent in that it is based on greedy routing in an arbitrarydimension while HyperCast is based on compass routing ina 2D space. The major advantage of both approaches is thata broadcast tree does not need to be explicitly maintained. Anode can immediately determine next-hop nodes based onthe coordinates of its neighbors and the destination node,without maintaining any per-session routing information.

We name our broadcast protocol as GRPB (greedy re-verse path broadcast). In GRPB, a node � maintains a localDT for � and �’s neighbors. For each neighbor �, � for-wards a message from a source node � to � if both of thefollowing two conditions hold:

C1 � is closer to � than � is;C2 in the local DT for � and �’s neighbor nodes, there

does not exist a node � �� such that: C2.1 � is closer to� than � is, and C2.2 �, � and � are included in the sametriangle (or simplex in a �-dimensional space).

Condition C1 is easy to understand. Suppose C1 is true.Then � does not forward to � if � is sure that another node,say �, is the next hop of � in the forward greedy rout-ing. The necessary and sufficient conditions for such � are:C2.1 � is closer to � than �, and C2.3 � is a neighbor of� on the global DT. However, � does not have global infor-mation and cannot check C2.3. Hence we specify conditionC2.2 which includes C2.3. C2.1 and C2.2 are necessary butnot sufficient.

Note that in case of a tie between � and � in C2.1, �must forward to � at the cost of possible duplication, since� may or may not choose � as the next hop in the forwardgreedy routing. Note also that even if node � appears to


be �’s neighbor in �’s local DT, � may not actually be �’sneighbor in the global DT. Figure 2 illustrates an example ina 2D space. The left graph shows �’s local DT, in which �and � are neighbors. However, as shown in the right graph,there may exist a node outside �’s local knowledge andthus � may not actually be a neighbor of �. Without in-cluding C2.2 in C2, � might erroneously conclude that itdoes not need to forward to �, since � appears to be theclosest node to � among �’s neighbors. C2.2 detects suchambiguous situations and requires that � forwards to � atthe cost of possible duplication. We performed experimentsto broadcast a message using GRPB on a distributed DT of200 randomly-placed nodes in various dimensions. In ourexperiments, the number of duplicate messages was from3% to 10% of the number of nodes.

The following theorem guarantees the correctness ofGRPB, namely it delivers a message to all nodes in the sys-tem. A proof of the theorem as well as the protocol pseu-docode are presented in [14].Theorem 1 Let a set of nodes � form a correct distributedDT. The GRPB protocol delivers a message from a sourcenode � � � to all the other nodes in �.

Multicast within a radiusIn a distributed virtual reality system or a multiplayer on-

line game, multicast within a given radius from a point maybe a common operation, since an event may affect nodeswithin some distance. For example, in a war simulation, anexplosion of a bomb will be seen only by soldiers withinsome distance, and will affect those within a shorter dis-tance. We observe that in the GRPB protocol the distanceof next hop from the source monotonically increases, sincethe distance to the destination monotonically decreases inthe forward greedy routing. We utilize this observation inour multicast protocol within a given radius.

In our radius greedy reverse path multicast (RadGRPM)protocol from a source node � to all the other nodes within agiven radius �, � first sends the message to all its neighborswithin �. Then for each neighbor node �, a node � forwardsa message to � if the following condition holds as well asC1 and C2 in GRPB:

C3 the distance from � to � does not exceed the radius �.For pseudocode and correctness proof of RadGRPM, we

refer interested readers to [14].

3. Protocol designOur distributed DT protocols handle join, leave, and

maintenance operations. Our join protocol ensures that ajoining node obtains enough information to identify its cor-rect neighbors and that the joining of the new node is no-tified to all existing nodes affected by the joining node, sothat the resulting distributed DT is correct after protocol ex-ecution. Similarly, our leave protocol notifies the deletionof a leaving node to all affected nodes so that the resulting

distributed DT is correct after protocol execution. Our joinand leave protocols are proved to be correct only for serialjoins and leaves.

We assume that nodes may join, leave or fail at any time.In addition to node failures, which inevitably result in an in-correct distributed DT, concurrent joins and leaves of multi-ple nodes may result in an incorrect distributed DT as well.To address such scenarios, we introduce a maintenance pro-tocol which is run periodically to detect and repair any er-rors in the system state. (When the distributed DT of a setof nodes is incorrect, for convenience, we say “the systemstate is incorrect” or “the system state has errors.”) Lastly,to simplify our protocol descriptions, we assume reliabledelivery of protocol messages. In a real implementation,additional mechanisms such as ARQ or simply TCP can beused to ensure reliable message delivery.4

��

Our approach to construct a distributed DT is as follows.We assume that each node is associated with its coordinatesin a �-dimensional Euclidean space. Each node has priorknowledge of its own coordinates, as is assumed in previouswork [5, 6, 7, 8, 9]. The mechanism to obtain coordinates isbeyond the scope of this study. Coordinates may be givenby an application, a GPS device[16], or topology-aware vir-tual coordinates[17].5 Also when we say a node � knowsanother node �, we assume that � knows �’s coordinates aswell.

Let � be a set of nodes to construct a distributed DTfrom. We will present protocols to enable each node � � �

to get to know a set of its nearby nodes including � itself,denoted as ��, to be referred to as �’s candidate set. Then� determines the set of its neighbor nodes �� based on ��.Specifically, � determines �� by calculating a local DT of��, denoted by �� . That is, � � �� if and only ifthere exists an edge between � and � on �� .

��

Recall that a distributed DT is correct when for everynode �, �� is the same as the neighbors of � on �� .Since �� is the set of �’s neighbor nodes on �� inour model, to achieve a correct distributed DT, the neigh-bors of � on �� must be the same as the neighborsof � on �� . Note that �� is local information of �while � is global knowledge. Therefore in designing ourprotocols, we need to ensure that �� is “enough” for � tocorrectly identify its global neighbors. If �� is too lim-ited, � cannot identify its global neighbors. For the extreme

4Due to the overhead of opening and closing connections, TCP maynot be a practical choice.

5Application performance on a DT may be affected by the accuracy ofvirtual coordinates.


case of �� , � can identify its neighbors on the globalDT since�� ; however, the communicationoverhead for each node to acquire global knowledge wouldbe extremely high.

Theorem 2 (Correctness Condition) Let � be a set ofnodes and for each node � � �, � � �� and �� . Let��, � � � be the set of �’s neighbor nodes on �� . Adistributed DT of � is correct if and only if, for every � � �,�� includes all the neighbor nodes of � on �� .

Theorem 2 identifies a necessary and sufficient conditionfor a distributed DT to be correct, namely: the candidate setof each node must contain all of its global neighbors. Aproof of the theorem is presented in [14]. In the followingsubsections, we use the above correctness condition as aguide to design our protocols.

��

In our join protocol,6 we assume that a joining node �is first led to the nearest existing node �, which is guaran-teed to be found using greedy routing as discussed in sec-tion 2.2. �� is initialized as ��, and � sends a NEIGH-BOR SET REQUEST messages to �. When � receivesNEIGHBOR SET REQUEST from �, � puts � into��, up-dates �� by recalculating �� , computes ��

� which isthe set of the neighbor nodes of � on �� , and replies�� to �. When � receives the reply, �� is updated to in-

clude all nodes in the reply, and � determines its neighbornodes again using the updated��. If � finds any new neigh-bor nodes, � sends NEIGHBOR SET REQUEST messagesto them. This process is repeated until � does not find anynew neighbor node. The protocol pseudocode is presentedin [14].

Theorem 3 guarantees that the join protocol, if run ona correct distributed DT, results in a correct distributed DTfor a single join. The main ideas of the proof are the fol-lowing: i) the closest existing node will be a neighbor of ajoining node, ii) all neighbor nodes of the joining node areconnected by existing neighbor relations, thus it is possibleto find them all by following the neighbor relations, and iii)the neighbor nodes of the joining node are also notified ofthe joining node’s addition in the process. A detailed proofof the theorem is presented in [14]. Note that this proof isbased on Theorem 2, which determines the condition whena distributed DT is correct.

Theorem 3 Let � be a set of existing nodes and the dis-tributed DT of � be correct. Let a node � �� join to thedistributed DT using our join protocol. Assume that thereis no other join, leave, or failure. After the join protocolfinishes, the updated distributed DT is correct.

Though the join protocol achieves a correct distributedDT after it finishes, the transient states are not correct,

6This protocol is similar to the basic generalized join algorithm in [7].

which may result in malfunction of upper-layer applica-tions. For example, a new node in an early stage of the join-ing process may not have a complete set of neighbors andmay not be able to properly forward a message for greedyrouting. To address such situations, we introduce a mecha-nism for a joining node to defer to be a part of the systemuntil it establishes its complete set of neighbor nodes. Whenan existing node receives NEIGHBOR SET REQUEST, itdoes not immediately update its neighbor set. When thejoining node � finishes its joining process, it then notifiesall its neighbors that it is safe to update their candidate setsand their neighbor sets to include �. Due to delay of no-tification message delivery, some transient states may stillbe incorrect. However, greedy routing will work well evenwith imperfect states, as to be shown in section 4.

Also note that the join protocol is proved to be correctonly for serial joins. In case of concurrent joins, the pro-tocol may not result in a correct distributed DT. Such im-perfection is addressed by the maintenance protocol to bepresented in subsection 3.5.

��

We first address the case of graceful leaves. The caseof ungraceful leaves or failures is addressed by our main-tenance protocol in subsection 3.5. A straightforward ap-proach to address graceful leave would be that a leavingnode, before it leaves, notifies all of its neighbors that itis about to leave. This simple notification is, however, notenough to maintain a correct distributed DT.

Suppose that a node � leaves and it notifies a neighbornode � that it is leaving. Then � should remove � from ��

and update ��. The problem is that in some cases � mayhave a new neighbor � that was not previously a neighborof � and may not be in �� . In such cases, the straightfor-ward approach may resulting in an incorrect distributed DT.However, we observe that such � is always a neighbor of�. Therefore it is possible for � to notify � that � is leavingand also introduce � to �, resulting in a correct distributedDT.

When a node � leaves, � calculates a local DT of itsneighbor nodes, but not including itself. Then � notifieseach of its neighbors, say �, that � is leaving as well asa list of the neighbors of � on the local DT of �. Uponreceiving such notification, � updates its candidate set andneighbor set. In addition, a DELETE message that � is leav-ing is propagated using GRPB. Note that even if � is not aneighbor node of another node , may have � in ��. TheDELETE message ensures that � is removed from such ��,if any. The protocol pseudocode is presented in [14].

The following theorem assures that the leave protocol iscorrect for serial leaves. The theorem is based on the pre-vious observation that if a node � becomes a new neighborof � after � leaves, � was a neighbor of � before � leaves.


A complete proof of the theorem is found in [14].

Theorem 4 Let � be a set of nodes and the distributed DTof � be correct. Let a node � � � leave the distributedDT using our leave protocol. Assume that there is no otherjoin, leave, or failure. After the leave protocol finishes, theupdated distributed DT is correct.

Note that the leave protocol is correct only for serialleaves. Similar to the case of concurrent joins, concurrentleaves may result in an incorrect distributed DT. Such casesare addressed by our maintenance protocol, to be discussedin subsection 3.5. In our implementation, propagation of aDELETE message is stopped when the message arrives ata node that does not have the leaving node in its candidateset. This modification greatly reduces communication cost,without affecting correctness of the leave protocol in almostall cases. A very rare case where a left node remains in acandidate set and causes incorrectness can be addressed bythe maintenance protocol.

Also, similar to the case of a join, transient incorrectstates during a leave may result in malfunction of upper-layer applications. Thus it is desirable for a leaving nodeto defer leaving after making sure that each of its neighborshas updated its neighbor set. This may be achieved by re-quiring an acknowledgement of a LEAVE message.

��

The join and leave protocols are proved correct only forserial joins and leaves, assuming that there is no other con-current join, leave, or failure. In practice, however, nodesmay join and leave concurrently, or even fail at any time,causing errors in the system state. Therefore an additionalmechanism is needed to repair errors in the system state.To address system churn and failures, we present a mainte-nance protocol, which is run periodically to detect and re-pair errors, if any, in the system state.

From Definition 5, for a distributed DT to be correct, twoconditions must be satisfied: i) Each node � must include inits neighbor set�� all of its neighbors on the global DT, andii) �� must not contain any node that is not in the system.

To satisfy the first condition, a node periodically ex-changes information with each of its neighbors. Specifi-cally, a node � informs its neighbor node � of the neighborsof � on �’s local DT. Note that the process is essentially thesame as what is done when a node joins, since the goals ofthe two protocols are same i.e. each node learns its neigh-bors on the global DT.

To satisfy the second condition, a node probes its neigh-bors by sending ping messages periodically. If a probednode does not reply, the node is considered to be not in thesystem and removed from the candidate set. This mech-anism also addresses the case of ungraceful node failures.Note that this probing can be easily integrated with the ex-change of information for the first condition.

The maintenance protocol is as follows. A node � sendsout NEIGHBOR SET REQUEST to its neighbor node �.When � receives the request, it replies with ��

� , which isthe set of neighbors of � on �� . That is, ��

� is the setof �’s neighbors in �’s local view. � also checks whether �is in its candidate set �� . If � �� , � puts � in �� . When� receives the reply ��

� , � checks whether �� . If

there exists any node in �� that is not in ��, it is added

to ��. In case � does not receive a reply from � beforeTIMEOUT, � is considered to have failed and removed from��. � also propagates the deletion of � similarly as in theleave protocol. Once �� is updated, � recalculates the lo-cal DT and determines its set of neighbor nodes ��. Ifthere are any new neighbor nodes in ��, � sends NEIGH-BOR SET REQUEST to them. The protocol pseudocode ispresented in [14].

From a large number of simulation experiments, wefound that the maintenance protocol converged to a cor-rect distributed DT in every experiment, for different di-mensionalities (2D to 6D), numbers of nodes (200 to 800),scenarios (random initial graph, severe churn with node fail-ures) as long as the system is not partitioned. Note thatit is extremely difficult to prove correctness of the main-tenance protocol for any combinations of concurrent joins,leaves and failures. Furthermore, in an environment wheresystem churn occurs continually, another join or leave mayoccur before the system converges to a correct state. Asa result, convergence to a correct system state may be im-possible during system churn. Fortunately, some applica-tions can still benefit from an imperfect DT as long as it is“accurate” enough. Therefore the accuracy of a distributedDT over time is more important in practice than eventualconvergence to a correct distributed DT for systems underchurn.

Even if our maintenance protocol converged to a cor-rect distributed DT some time after churn and failure havestopped in every one of our experiments, this does not meanthat our join and leave protocols are no longer needed. Notethat it takes time for the maintenance protocol to detect andrepair errors, resulting in a lower average accuracy. Fur-thermore, the maintenance protocol requires a much highercommunication overhead than those of the join and leaveprotocols, and thus should be run only periodically, withthe period being a design parameter to be tuned.

We define an accuracy metric of a distributed DT as fol-lows, which is used for all of our experiments. Let ��be a distributed DT of a set of in-system nodes �. We con-sider a node to be in-system from when it finishes joiningto when it starts leaving. (Note that some nodes may bein the process of joining or leaving and not included.) Let�� be the number of correct neighbor en-tries of all nodes and �� be the number ofwrong neighbor entries of all nodes on ��. A neigh-


bor entry � of a node � is correct when � is a neigh-bor of � on the global DT (namely, �� ), and wrongwhen � and � are not neighbors on the global DT. Let�� be the number of edges on �� . Note thatedges on a global DT are undirectional and thus are countedtwice to be compared with neighbor entries. The accu-racy of �� is defined as follows: accuracy(��) ��

�� A distributed DT is correctif and only if its accuracy is 1.

To demonstrate accuracy and effectiveness of the main-tenance protocol, we designed a “ring” scenario beginningwith a barely connected graph, in which each node initiallyknows only one other node. That is, node ��, � � �, has only�� in its candidate set and its neighbor set. The numberof nodes is 200. Figure 3 shows accuracy of the distributedDT versus time as the maintenance protocol runs. Note thatthe maintenance protocol achieved a correct distributed DTwithin a few rounds of protocol execution. The convergencetends to be faster in a higher dimension space, since nodeshave more neighbors and information is exchanged faster.

-0.2

0

0.2

0.4

0.6

0.8

1

0 2 4 6 8 10

Acc

urac

y

Rounds

2D3D4D5D

Figure 3. Accuracy of the maintenance proto-col in a “ring” scenario.

4. Experimental results��

The per-node communication cost of our distributed DTprotocols largely depends on the average number of neigh-bors per node. Since the average number of neighbors of anode on a DT does not increase as the number of nodes inthe system increases,7 the scalability of our distributed DTprotocols is generally very good. However, there are twominor factors that affect the per-node cost as the system sizeincreases. First, greedy routing to locate the closest exist-ing node in the join process will take ��

�� steps, where

� is the dimensionality of the space and � is the numberof nodes in the system. In addition, nodes on the bound-ary of a DT have fewer neighbors than those in the middle.

7For a DT of randomly-placed nodes, the average number of neighborsasymptotically depends on the dimensionality of the space.

0

5000

10000

15000

20000

25000

30000

200 300 400 500 600 700 800 900 1000

Num

ber

of m

essa

ges

Initial number of nodes

joinleave

maintenance

0

200

400

600

800

1000

1200

1400

1600

200 300 400 500 600 700 800 900 1000

Am

ount

of m

essa

ges(

KB

)

Initial number of nodes

joinleave

maintenance

Figure 4. The communication cost of proto-cols versus number of nodes in 3D.

When the network size is smaller, the fraction of bound-ary nodes is larger, making the average number of neigh-bors smaller. Figure 4 shows the number of messages (top)and the amount of messages in Kbytes (bottom) versus sys-tem size in 3D. The join and leave curves represent the totalcosts of 100 serial joins and 100 serial leaves respectively,and are more or less independent of system size showingvery good scalability. The maintenance curve represents theper-round cost of the maintenance protocol for all nodes,which increases linearly with system size; thus the averagecost per node is constant versus system size.

�� !�� "��

Figure 5 (top) shows accuracy of a distributed DT as afunction of time for a system under churn, more specifi-cally, when nodes are joining and leaving concurrently butnot failing. Initially, the system has 200, 400, or 800 nodeswith a correct distributed DT. Then node joins are scheduledfollowing a Poisson process with average inter-arrival timeof 1 second, until time 110 second. Graceful node leavesare also scheduled in the same way. That is, 1 node joinsand 1 node leaves once every second on the average, usingour join and leave protocols.8 Our maintenance protocolis run once every 10 seconds. In spite of the churn, accu-

8By Little’s Law, for an initial system size of 200, the average lifetimeof a node is 200 seconds. For P2P systems, this is a very high churn rate[15]. Note that accuracy in Figure 5 (top) improves as the system sizeincreases.


0.97

0.975

0.98

0.985

0.99

0.995

1

0 50 100 150 200 250

Acc

urac

y

Time (seconds)

200 nodes400 nodes800 nodes

0.995

0.9955

0.996

0.9965

0.997

0.9975

0.998

0.9985

0.999

0.9995

1

0 50 100 150 200 250

Rou

ting

succ

ess

rate

Time (seconds)


Figure 5. Accuracy (top) and greedy routingsuccess rate (bottom) of a system in 3D un-der churn versus time.

racy of the distributed DT remains very high. The smallerror is due to concurrent joins and leaves, and is repairedby our maintenance protocol periodically. Figure 5 (bot-tom) shows the success rate of a greedy routing protocolfor a system under churn while running our join, leave andmaintenance protocols. Note that the success rate is muchhigher than the accuracy value, due to careful design of ourjoin and leave protocols. In our join protocol, the neighbornodes of a joining node defer adding the joining node totheir neighbor sets until the joining node finishes its joiningprocess and is ready to function properly. Similarly, in ourleave protocol, a leaving node continues service until all ofits neighbors are notified.

�� !�� #�" ��

Figure 6 shows accuracy of distributed DT (top) andgreedy routing success rate (bottom) for a system in whichnodes join and fail concurrently, while running our join andmaintenance protocols, but not our leave protocol. Exceptfor nodes failing instead of leaving, we use the same simu-lation parameters as in subsection 4.2 to compare the impactof node failures versus graceful node leaves.

Both accuracy and greedy routing success rate are muchworse than in the previous case of system churn with grace-ful leaves instead of failures. Since any error caused by afailed node cannot be recovered until the maintenance pro-tocol detects it by a message timeout, the lower accuracy

0.84

0.86

0.88

0.9

0.92

0.94

0.96

0.98

1

0 50 100 150 200 250

Acc

urac

y

Time (seconds)


0.98

0.982

0.984

0.986

0.988

0.99

0.992

0.994

0.996

0.998

1

0 50 100 150 200 250

Rou

ting

succ

ess

rate

Time (seconds)


Figure 6. Accuracy (top) and greedy routingsuccess rate (bottom) of a system in 3D withconcurrent joins and failures versus time.

and routing success rate are to be expected considering thehigh failure rate. Lastly, note that in all simulations themaintenance protocol converged to a correct system statein a few rounds after churning stopped.

5. Related workThe first protocol to construct a distributed DT was pro-

posed by Liebeherr and Nahas [5]. The protocol utilizes thelocally equiangular property of DT in a 2D space. Nodesare assumed to have pre-assigned logical coordinates in a2D space. Each node checks whether the equiangular prop-erty holds among itself and its neighbor nodes. Whenevera violation is detected, the node flips triangles to maintaina correct DT. Their application was application-layer mul-ticast, called HyperCast. Since compass routing on a DTis guaranteed to succeed, a multicast tree can be implicitlydetermined for a given source using reverse path.

Steiner and Biersack [6] proposed a distributed approachto construct a distributed DT in a 3D space. The tetrahedronwhich includes a joining node is determined and split. Thenthe new tetrahedra are checked whether they include anynodes in their circumspheres and flipped if necessary.

Simon et al. [7] proposed two sets of distributed al-gorithms for �-dimensional spaces: basic generalized al-gorithms and improved generalized algorithms.9 Each set

9The improved generalized insertion algorithm is designed to lower


consists of an entity insertion algorithm and a deletion al-gorithm. They assume that no � � � nodes are on the samehyperplane and no �� nodes are on the same hypersphere.It is also assumed that a new node is in the interior of theconvex hull of existing nodes. Our join protocol is similarto their basic generalized insertion algorithm, but we provedcorrectness for our join and leave protocols without any oftheir assumptions. Their improved generalized algorithms,which are the focus of their paper, use a fundamentally dif-ferent approach from ours; in particular, our protocols arebased on our correctness condition and theirs are based onan already-proven centralized flip algorithm. In addition,while their work is focused on explaining correct operationof their algorithms in different scenarios for a single inser-tion or deletion,10 we rigorously proved correctness for ourjoin and leave protocols. We also performed simulation ex-periments to measure the communication cost of our proto-cols and to investigate accuracy under concurrent node joinsand leaves.

While DT has been extensively studied in computationalgeometry, most work in the field focuses on centralized al-gorithms. Ohnishi et al. [8] proposed an incremental algo-rithm to construct a distributed DT in a 2D space. Yoo etal. [9] proposed a distributed algorithm to maintain DT formoving nodes in a 2D space.

6. ConclusionsWhile DT has been known and used for a long time,

the design of protocols for constructing and maintaining aDT for a dynamic system has not received much attention.In this paper, we investigate the design of join, leave andmaintenance protocols for a set of nodes to construct andmaintain a distributed DT dynamically, as well as severalapplication-level protocols to support DT applications. Wepresent a necessary and sufficient condition for a distributedDT to be correct, which was used as a guide for our proto-col design. Experimental results show that our protocols arescalable and they achieve high accuracy for systems underchurn and with node failures.

References[1] B. Delaunay, Sur la sphere vide, Otdelenie Matem-

aticheskii i Estetvennyka Nauk, 7:793-800, 1934.[2] G. Voronoı, Nouvelles applications des parametres

continus a la theorie des formes quandratiques, Jour-nal fur die Reine and Angewandte Mathematik,133:97-178, 1908.

[3] Prosenjut Bose and Pat Morin, Online routing in tri-angulations, SIAM Journal on Computing, 33(4):937-951, 2004.

computational cost by reducing duplicated computation among nodes.10A version of their paper submitted for publication has correctness

proof for their improved entity deletion algorithm for 3D only.

[4] D.P. Dobkin, S.J. Friedman, and K.J. Supowit, De-launay graphs are almost as good as complete graphs,Discrete Computational Geometry, 5:399–407, 1990.

[5] Jorg Liebeherr, Michael Nahas, Application-LayerMulticasting With Delaunay triangulation Overlays,IEEE Journal on Selected Areas in Communications,VOL. 20, NO. 8, October 2002.

[6] Moritz Steiner, Ernst Biersack, A fully distributedpeer to peer structure based on 3D Delaunay Trian-gulation, Algotel 2005.

[7] Gwendal Simon, Moritz Steiner, Ernst Biersack,Distributed Dynamic Delaunay Triangulation in d-Dimensional Spaces, Technical report, Institut Eure-com, August 2005.

[8] Masaaki Ohnishi, Ryo Nishide, Shinichi Ueshima,Incremental Construction of Delaunay Overlaid Net-work for Virtual Collaborative Space, The third inter-national Conference on Creating, Connecting and Col-laborating through Computing, 2005.

[9] Taewon Yoo, Hyonik Lee, Jinwon Lee, Sunghee Choi,Junehwa Song, Distributed Kinetic Delaunay Triangu-lation, CS/TR-2005-240, KAIST, Korea, 2005.

[10] Min Sik Kim, Taekhyun Kim, Yong-June Shin, Si-mon S. Lam, Edward J. Powers, Scalable clusteringof Internet paths by shared congestion, IEEE Infocom2006, April 2006.

[11] Xin Yan Zhang, Qian Zhang, Zhensheng Zhang, GangSong, Wenwu Zhu, A Construction of locality-awareoverlay network: mOverlay and its performance,IEEE journal on selected areas in communications,Vol. 22, No. 1, Jan 2004.

[12] X. Brian Zhang, Simon S. Lam, Huaiyu Liu, EfficientGroup Rekeying Using Application Layer Multicast,Proceedings of 25th IEEE ICDCS, Columbus, Ohio,June 2005.

[13] Yogen K. Dalal and Robert M. Metcalfe, Reverse pathforwarding of broadcast packets, Communications ofthe ACM, Volume 21, Issue 12, Dec 1978.

[14] Dong-Young Lee and Simon S. Lam, Protocol de-sign for dynamic Delaunay triangulation, The Univ. ofTexas at Austin, Dept. of Computer Sciences, Techni-cal Report TR-06-48, Oct 2006.

[15] S. Sariou, P. K. Gummadi, and S. D. Gribble, A mea-surement study of peer-to-peer file sharing systems, InProc. of Multimedia Computing and Networking, Jan-uary 2002.

[16] Tommaso Melodia, Dario Pompili, Ian F. Akyildiz,A Communication Architecture for Mobile WirelessSensor and Actor Networks, Proceedings of IEEESECON 2006, September 2006.

[17] T. S. Eugene Ng and Hui Zhang, Predicting Inter-net Network Distance with Coordinates-Based Ap-proaches, Proceedings of INFOCOM 2002.


[ieee 27th international conference on distributed computing systems (icdcs '07) - toronto, on,...

Documents