COSI: Cloud Oriented Subgraph Identificationin Massive Social Networks

Matthias BrochelerUniversity of MarylandA.V. Williams Building

College Park, MD 20742, USAmatthias@cs.umd.edu

Andrea PuglieseUniversita della Calabria

Via P. Bucci, 41/CRende, Italy

apugliese@deis.unical.it

V.S. SubrahmanianUniversity of MarylandA.V. Williams Building

College Park, MD 20742, USAvs@cs.umd.edu

Abstract—Subgraph matching is a key operation on graphdata. Social network (SN) providers may want to find allsubgraphs within their social network that “match” certainquery graph patterns. Unfortunately, subgraph matching isNP-complete, making its application to massive SNs a majorchallenge. Past work has shown how to implement subgraphmatching on a single processor when the graph has 10-25Medges. In this paper, we show how to use cloud computingin conjunction with such existing single processor methods toefficiently match complex subgraphs on graphs as large as 778Medges. A cloud consists of one “master” compute node and k“slave” compute nodes. We first develop a probabilistic methodto estimate probabilities that a vertex will be retrieved by arandom query and that a pair of vertices will be successivelyretrieved by a random query. We use these probability estimatesto define edge weights in an SN and to compute minimal edgecuts to partition the graph amongst k slave nodes. We developalgorithms for both master and slave nodes that try to minimizecommunication overhead. The resulting COSI system can answercomplex queries over real-world SN data containing over 778Medges very efficiently.

I. INTRODUCTION

During the last decade, there has been viral growth in socialnetworks. FaceBook, Flickr, Twitter, YouTube and Blogger,all implement social networks. Both SN owners and SN usersare interested in a variety of queries that involve subgraphmatching. For example, consider the small social networkshown in Fig. 1. Users of such a network might ask queriessuch as:

(Q1) Find all vertices ?v1, ?v2, ?v3, ?p such that ?v1 worksat the University of Maryland and ?v1 is a facultymember and ?v2 is an Italian university and ?v3 isa faculty member at ?v2 who is a friend of ?v1

and ?v3 has commented on a posting (or paper) ?pby ?v1. This query corresponds to a subgraph asshown in Fig. 2 – it might be used by a UniversityPresident to find existing interactions between hisfaculty and those in Italy (e.g., just before he goesfor a meeting with the Italian embassy). When thisquery subgraph is posed against an enormous SN, wedo not have the option of matching the subgraph in anaive way against the graph – without intelligent pre-processing, the query would simply take too long.

(Q2) Consider a financial network whose nodes are ac-count holders and banks - edges capture bankingtransactions including bank transfers, lines of credit,etc. A financial crimes investigator who has deter-mined that a particular bank (say Bank1) is suspi-cious might want to ask a query of the form: “Findall vertices ?v1, ?v2 such that ?v1 wired money toBank1 and Bank1 received a wire from ?v2 and both?v1 and ?v2 have a common friend ?v3 who has beenlabeled suspicious”. The financial crimes investigatormight believe that such pairs of people (?v1, ?v2) aresuspicious and worth a further look.

Both queries above contain multiple vertices and differentrelationships between them, demonstrating the need to executecomplex queries over social networks. In addition, answeringSPARQL queries in the Semantic Web’s RDF frameworklargely involves subgraph matching. We show how to answersuch queries and more complex ones over large social net-works efficiently.

Prof Smith

Prof Dooley

Prof Roma

Prof Jones

Prof Baneri

Universita Calabria

University MD

SDU Odense

Prof Larsen

Prof Lund

Prof Olsen

Prof Calero

UMD CS

UMD Physics

Jamie Lock

department in

department in

dean

member

student of

USA in

in

ASONAM 09

collaborates

Italy UC CS

faculty

member

Social Science

Paper “ABC” author

author

collaborate

friends

attended presented

Social Science

Denmark

KPLLC 09

John Doe colleagues

Odense Physics

visited

Karl Oede

student of

visited

Paper “UVW” Paper

“HIJ” Paper “XYZ”

attended

author submitted

submitted

accepted

organized

attended comment comment

author

member

department

friends

dean

faculty

comment

faculty

faculty member

department

dean member

faculty

author

author

Fig. 1. Example social network

While all graphs can be represented trivially in relationaldatabase format, these queries both involve very large joins –thus, executing these queries on a standard relational DBMS

2010 International Conference on Advances in Social Networks Analysis and Mining

978-0-7695-4138-9/10 $26.00 © 2010 IEEE

DOI 10.1109/ASONAM.2010.80

248

2010 International Conference on Advances in Social Networks Analysis and Mining

978-0-7695-4138-9/10 $26.00 © 2010 IEEE

DOI 10.1109/ASONAM.2010.80

248

2010 International Conference on Advances in Social Networks Analysis and Mining

978-0-7695-4138-9/10 $26.00 © 2010 IEEE

DOI 10.1109/ASONAM.2010.80

248

Italy University MD

faculty

?v1 ?v3

?v2

faculty

in

friends

?p comment author

Suspicious labeled

?v1 ?v2

?v3 friends

Bank1 wired wired

Fig. 2. Example queries Q1 (left) and Q2 (right)

incurs a very high cost. For instance, on a PostGres relationaldatabase using a graph of size 16M edges, queries of similarcomplexity took an average of 168 seconds to execute. Incontrast, graph-based indexing systems such as DOGMA [1]take much less time (0.31 seconds on average) on the samehardware. The case for graph-based indexes has separatelybeen argued in several works (see, e.g., [2]).

Section II defines a social network S as a labeled directedgraph. We define queries over SNs as graphs Q – answersto queries are all subgraphs in S that “match” the querygraph Q. Our cloud oriented approach to executing suchqueries has k slave nodes and one master node. Computationis completely distributed and asynchronous and our goal isto minimize communication between nodes. To achieve this,we start with a known probability distribution over queries(which can, for example, be derived from historical data ofan SN) and we transform the SN S into a weighted graph.This transformation is achieved by identifying probabilitiesthat a vertex in S will be retrieved from a slave node inresponse to a random query and by identifying probabilitiesthat an arbitrary pair of vertices in S will be retrieved from aslave node. Section III defines these probabilities and showshow the minimal edge cuts of the resulting weighted graphcorrespond exactly to a partition of the graph that has minimalexpected cost of execution when amortized across arbitraryqueries. In Section IV, we present an algorithm to partitionan SN across k slave nodes in accordance with the abovestrategy and includes algorithms for batch insertion of newedges (and vertices) into slave nodes. Within a slave node, anystrategy to store graph data on a single disk (e.g., DOGMA [1])can be used. Section V presents two cloud oriented queryprocessing algorithms for subgraph matching queries — othertypes of queries are not handled in this paper. Section VIincludes experimental results showing that our algorithms canefficiently answer complex queries over SNs containing 778Medges. In Sections VII and VIII we discuss related work andoutline conclusions.

The approach to efficiently answering subgraph matchingqueries using a cloud of compute nodes presented in this workapplies to general graph datasets and therefore makes this workapplicable beyond the social networking realm. For instance,a significant fragment of the SPARQL query language forRDF, a popular data format used on the semantic Web, can berepresented as subgraph matching queries.

II. BASIC NOTATION

Throughout this paper, we assume the existence of anarbitrary, but fixed set V whose elements are called vertices.For example, V might consist of all strings that can form avalid userid and/or the set of all valid identifiers for commentsin a social network like Facebook (for the sake of simplicity,we are ignoring files, etc.). We also assume the existence ofa finite set P of predicate symbols.

A social network S is a triple (V,E, λ) where V ⊆ V is theset of vertices in the social network, E ⊆ V ×V is a multisetof edges from vertices to vertices and λ : E → P assigns apredicate symbol to each edge in E.

Throughout this paper, we assume an arbitrary, but fixedsocial network S = (V,E, λ) — we use the toy university SNexample in Fig. 1 for illustrative purposes.

The out neighborhood of vertex v is the set out(v) ={u | (v, u) ∈ E}; the in neighborhood of node v is the setin(v) = {u | (u, v) ∈ E}. The neighborhood of v is the setngh(v) = out(v) ∪ in(v). Each of these neighborhoods canbe restricted to a particular predicate symbol p: for example,outp(v) = {u | (v, u) ∈ E ∧ λ(v, u) = p}.

When formulating queries, we assume the existence of aset V AR of variable symbols ranging over V . Each variablesymbol starts with a ?. A query Q is a triple (VQ, EQ, λQ)where VQ ⊆ V ∪ V AR, EQ ⊆ VQ × VQ is a multiset ofedges, and λQ : EQ → P . We use V ARQ to denote the setof variable vertices in query Q.

Suppose S is an SN and Q is a query. A substitutionfor query Q is a mapping V ARQ ∩ VAR → V . If θ is asubstitution for query Q, then Qθ denotes the replacement ofall variables ?v in VQ by θ(?v). Hence, the graph structure ofQθ is exactly like that of Q except that nodes labeled withvariables are replaced by vertices in the SN S. A substitution θis an answer for query Q w.r.t. SN S iff Qθ is a subgraph of S.The answer set for query Q w.r.t. an SN S is the set {θ |Qθis a subgraph of S}. For example, the answer to the query(Q1) w.r.t. the SN in Fig. 1 is {(?v1 = Prof Dooley, ?v2 =Universita Calabria, ?v3 = Prof Calero, ?p = Paper ABC)}.

III. PARTITIONING SOCIAL NETWORKS ACROSS ACOMPUTE CLOUD

This section describes how an SN may be “split” acrossa compute cloud so that we can efficiently process subgraphmatching queries. We assume that a compute cloud consists ofk “slave” nodes and one “master” node. Slave nodes communi-cate directly without going through the master, thus preventingthe master from becoming a communication bottleneck. Themaster takes an initial query Q and directs it or parts of it toone or more slave nodes that then complete the computationof the answer with no further interaction with the master tillthe complete answer is assembled. At this stage, the completeanswer is shipped to the master which sends the result to theuser. The master is primarily an interface to the user.

We transform the SN first into a weighted graph. Intuitively,the weight of an edge (v1, v2) refers to the sum of theprobability that v2 will be retrieved immediately after v1 and

249249249

vice versa when an arbitrary query is processed. Intuitivelyif this probability is (relatively) high, then the two verticesshould be stored on the same slave node. We then use theseto partition the SN across k slave nodes so that expectedcommunication costs are minimized.

Throughout this section, we assume there is a probabilitydistribution P over the space of all queries. Intuitively, P(q)is the probability that a random query posed to a SN is q. Forany real world SN like FaceBook or Orkut, P can be easilylearned from frequency analysis of past query logs.

A. Probability of Vertex (Co-)Retrievals

A query plan qp(Q) for a query Q is a sequence of twotypes of operations: the first type retrieves the neighborhood ofvertex v (from whichever slave node it is on), and the secondtype performs some computation (e.g. check a selection con-dition or perform a join) on the results of previous operations.This definition is compatible with most existing definitions ofquery plans in the database literature.

Definition 3.1 (Query trace): Suppose x = qp(Q) is aquery plan for a query Q on an SN S. The query trace ofexecuting x on S, denoted qt(x,S), consists of (i) all the ver-tices v in S whose neighborhood is retrieved during executionof query plan x on S, and (ii) all pairs (u, v) of vertices whereimmediately after retrieving u’s neighborhood, the query planretrieves v’s neighborhood (in the next operation of x). �Traces contain consecutive retrievals of vertex neighborhoods.This allows us to store neighborhoods of both u and v on thesame slave node, avoiding unnecessary communication.

When processing a query, we make the reasonable assump-tion that index retrievals are cached so that repeated vertexneighborhood retrievals are read from memory and hence thequery trace qt(x,S) can be defined as a set rather than as amultiset. The probability distribution P on queries can be usedto infer a probability distribution P over the space of feasiblequery plans. P(x) =

∑Q∈Q:qp(Q)=x P(Q). This says that the

probability of a query plan is the sum of the probabilities ofall queries which use that query plan.1 We can now define theprobabilities of retrieval and co-retrieval as follows.Probability of retrieving vertex v. The probability, P(v),of retrieving v when executing a random query plan is∑x∈qp(Q):v∈qt(x,S) P(x). Thus, the probability of retrieving v

is the sum of the probabilities of all query plans that retrievev.Probability of retrieving v2 immediately after v1. Theprobability P(v1, v2) of retrieving v2 immediately after v1 is∑x∈qp(Q):(v1,v2)∈qt(x,S) P(x). This says that the probability of

retrieving v2 immediately after v1 is sum of the probabilitiesof all query plans that retrieve v2 immediately after v1.

B. Partitioning an SN Across a Cloud

We can associate a weighted graph WG(S) with the SNS = (V,E, λ). The weighted graph is the complete graph(V, V × V,w) where w(v1, v2) = P(v1, v2) + P(v2, v1).

1In the rest of the paper, we will abuse notation and denote both PDFs byP.

The following important theorem shows that the minimaledge cuts2 of WG(S) correspond to the partition of S acrossk slave nodes that minimizes expected cost of executing aquery.

Theorem 3.2: Assuming uniform costs (across all slavenodes) for retrieving a vertex neighborhood and sending amessage to a node, the partition of S which minimizes thetotal query execution time of a random query coincides withthe partition that minimizes the cost of a minimal edge cut ofWG(S). �To see why Theorem 3.2 is true, suppose we choose a queryQ uniformly at random. Q’s cost depends on the time toretrieve vertex neighborhoods of all vertices v ∈ qt(qp(Q),S).The total time needed to retrieve vertex neighborhoods isindependent of the partition (as we assume all slave nodesare equally fast at retrieving vertex neighborhoods). Thus weneed a constant amount of time to retrieve ngh(v) for allv ∈ qt(qp(Q),S) irrespective of the partition block containingv.

Vertex co-retrievals costs, however, are dependent on thepartition P = {P1, . . . , Pn} of S. Suppose (v1, v2) ∈qt(qp(Q),S) with v1 ∈ Pi, v2 ∈ Pj , and i 6= j. In thiscase, the neighborhood retrieval for vertex v1 is followed bythat of v2. As v2 is in a different partition block residingon another slave node, we need to send a message to thisnode at a communication cost of c which adds to the totalquery execution time and depends on the partition P . Wedefine the indicator random variable X(v1,v2) = c if (v1, v2) ∈qt(qp(Q),S), and X(v1,v2) = 0 otherwise. Using standardexpected values, the total expected cost of communication forQ is E(

∑(v1,v2)∈Ed

X(v1,v2)) =∑

(v1,v2)∈EdE(X(v1,v2)) =∑

(v1,v2)∈EdP(E(v1,v2)) × c, where Ed = {(v1, v2)|v1, v2 ∈

VS , v1 ∈ Pi, v2 ∈ Pj , i 6= j}.As the size of an edge cut is the sum of weights of all edges

that connect vertices in different partition blocks, the partitionwhich minimizes the edge cut of the graph constructed in thestatement of Theorem 3.2 is also the partition which minimizesthe total expected cost of communication and therefore thevariable part of the total expected query execution time.

IV. PARTITIONING IN PRACTICE

Even though Theorem 3.2 says we can partition the graphusing edge cuts, this method is not usable in practice becausecomputing minimal edge cuts is NP-complete [3]. Hence, weneed fast algorithms to partition the graph. In this paper, weprovide algorithms for batch insertion of edges (includingpotentially new vertices) into a social network S. We do thisvia the important notion of a vertex force vector.

Definition 4.1 (Vertex force vector): Let P ={P1, . . . , Pk} be a partition of S and consider any

2An edge cut C of a weighted graph is a partition of the vertices into“blocks”. An edge (u, v) in the graph is said to cross the edge cut C if uis in one block of the partition and v is in another. The size of an edge cutis the sum of the weights of the edges that cross the cut. C is said to be aminimum cut iff there is no other cut C’ such that the the size of C′ is lessthan the size of C.

250250250

block Pi. The vertex force vector, denoted |~v|, ofany vertex v ∈ S is a k-dimensional vector where|~v|[i] = fP(

∑x∈ngh(v)∩Pi

w((v, x))) and fP : R+ → R is afunction called the affinity measure. �A vertex force vector intuitively specifies the “affinity” be-tween a vertex and each partition block as measured by theaffinity measure fP . An affinity measure takes the connected-ness between a vertex v and the respective partition block as anargument. The vertex force vector captures the strength withwhich each partition block “pulls” on the vertex and is used asthe basis for a vertex assignment decision. If an inserted edgeintroduces a new vertex v, we first compute the vertex forcevector |~v| and then assign v to the partition block Pj wherej = argmax1≤i≤k |~v| [i]. COSI uses an affinity measure thatis a linear combination of three factors. We discuss the choiceof coefficients in the experimental section.Connectedness. Obviously, evaluating the connectedness ofa vertex v to a partition block Pi is crucial for edge cutminimization – we measure this as the number of edges thatconnect v to the vertices in Pi.Imbalance. Balanced partitions lead to even workloaddistribution, thus enhancing parallelism. Let |Pi|E =∑x∈Pi

deg(x) be the number of edges in Pi; let T be anestimate (even a bad one) of the total number of edges that agiven graph is expected to be. Then a reasonable measure ofimbalance is the standard deviation of {|Pi|E}1≤i≤k

T .Excessive size. In addition to imbalance, we regulate the sizeof partition blocks by comparing the actual size of a block toits expected one. If a block grows beyond its expected size, wewant to punish such growth more aggressively than imbalancedoes alone by reducing the affinity further according to themetric min(− |Pi|E−T

k

T , 0).

A. Batch Edge Insertion

We now show how to insert a new set of edges into anSN, given that a partition P = P1, . . . , Pk of the SN alreadyexists.3 A naive GreedyInsert insertion algorithm iteratesover all new vertices v: for each vertex v it computes thevertex force vector and assigns v to the block Pi such that|v|[i] is maximal — fortunately we can do better. This greedyalgorithm is used as a baseline in our experiments.

Our COSI Insert algorithm leverages graph modularity [4]to identify a strongly connected subgraph that is looselyconnected to the remaining graph. However, modularity cannotbe used blindly as our balance requirement must also be met.

Definition 4.2 (Modularity): The modularity of a partitionP of an undirected graph G = (V,E) with weight functionw : E → R is defined as

mod(P) =∑P∈P

(W (P, P )

2 |E|− degW (P )2

(2 |E|)2

)where degw(v) =

∑x∈V w((v, x)) is the weighted degree

of vertex v, W (X,Y ) =∑x∈X,y∈Y w((x, y)) is the sum of

3This can be used to create a partition of an SN for the first time byassuming S = ∅.

edge weights connecting two sets of vertices X,Y ⊂ V , anddegW (X) =

∑x∈X degw(x) is the weighted degree of a set

of vertices X ⊂ V . �Intuitively, blocks with high modularity are densely connectedsubgraphs which are isolated from the rest of the graph. Ouralgorithm iteratively builds high modularity blocks and thenassigns all vertices in a block to one slave node based onthe vertex force vector. Let B ⊂ V be a set of vertices. Wegeneralize the notion of a vertex force vector to sets of verticesB by defining

∣∣∣ ~B∣∣∣ [i] = fP

(∑v∈B

∑x∈ngh(v)∩Pi

w((v, x)))

.The intuition behind our partitioning algorithm is that assign-ing vertices at the aggregate level of isolated and denselyconnected blocks yields good partitions because (i) we respectthe topology of the graph, (ii) most edges are within blocksand therefore cannot be cut, and (iii) force vectors of setsof vertices combine the connectedness information of manyvertices leading to better assignment decisions.

The COSI Partition algorithm (Fig. 3) initially assignseach vertex to be its own block via the assignment function c.The assignment function from initial blocks to the partition Pis initialized to ∅ and subsequently updated to account for priorvertex assignments according to the parameter function α.The algorithm then repeatedly iterates over all new vertices uand determines whether moving u into any neighboring blockincreases modularity. If so, u is moved into the block whichyields the largest increase and the assignments are updated.We continue iterating over all new vertices until the numberof vertex moves l in the previous iteration falls below somethreshold δ. This first part of the algorithm determines modularblocks by iteratively moving individual vertices to maximizemodularity improvement (∆M(u, t) denotes the change inmodularity resulting from moving u into the block of t).

Algorithm COSI PartitionInput: Undirected graph G = (V,E), partial assignment function α : V → P ,

weight function w : E → ROutput: Total assignment function α : V → P

1 (c : V → C)← ∅; c(v) = {v}∀v ∈ V ; (b : C → P)← ∅2 for all v ∈ domain(α)3 b(c(v)) = α(v)4 repeat5 l← 06 for all u ∈ V − domain(α)

7 x← argmaxt∈ngh(u)[∆M(u, t) =2W ({u},b(t))−2W ({u},b(u)−u)

2|E| −2degw(u)[degW (b(t))−degW (b(u)−u)]

(2|E|)2]

8 if ∆M(u, x) > 09 c(x)← c(x) ∪ {u}; c(u)← c(u)− {u}; c(u)← c(x); l← l + 110 until l < δ

11 GC ← (C, EC) where E = {(x, y) | ∃u ∈ c−1(x), v ∈ c−1(y) : (u, v) ∈ E}12 wC ← EC → R where wC((x, y)) =

∑u∈c−1(x)

∑v∈c−1(y) w((u, v))

13 if |C| < θ1 or |C||V | > θ214 for all i = 1, . . . , n15 ei ← 016 for all C randomly chosen from C − domain(b)

17 bi(C)← Pm where m = argmax1≤j≤k

∣∣∣~C∣∣∣ [j]18 b← bj where j = argmin1≤i≤n

∑Cs,Ct∈C,bi(Cs) 6=bi(Ct)

w((Cs, Ct))

19 else20 b← COSI Partition(GC, b, wC)21 for all v ∈ V22 α(v)← b(c(v))23 return α

Fig. 3. COSI Partition algorithm

The algorithm constructs a new graph GC from the originalgraph G by collapsing all vertices assigned to the same block

251251251

during the modularity finding phase. If the graph GC has lessthan some threshold number of vertices θ1 or if its size isnot significantly smaller than the original graph G, we stopand assign vertices to partition blocks. Otherwise, we callCOSI Partition recursively on the collapsed graph GC tofind modular blocks comprised of modular blocks, therebybuilding multiple levels of modular graphs. If the collapsedgraph is small enough, we sequentially assign each vertex inGC to the partition block which maximizes the force vectorcomponent as we did before. We repeat this process n timesusing different random permutation of the vertices and choosethe assignment that minimizes the edge cut. Finally, we mapthe vertex assignment onto the original graph G, therebyprojecting it down one level. COSI Partition is guaranteedto respect prior vertex assignments (as specified by parameterα). Moreover, the size of the collapsed graph GC is a constantfactor smaller than the original graph and hence the size of theinput graph decreases exponentially as the function calls itselfrecursively which contributes to the speed of the algorithm.

V. QUERY ANSWERING

The COSI basic parallel algorithm, shown in Fig. 4,operates asynchronously and in parallel across all slave nodes.A user issues query Q to the master node which “prepares”the query. In particular, it selects one constant vertex c from Qand determines the slave node S that hosts c using the functioncall location(c). The prepared query is then forwarded to S.

Algorithm COSI basicOn master nodeInput: Graph query QOutput: Answer set A, i.e. set of substitutions θ s.t. Qθ is a subgraph of S

1 for all z ∈ VQ ∩ VAR2 Rz ← null /∗ no candidate substitutions for any vars in the query initially ∗/3 for all z ∈ VQ ∩ (S ∪ V)4 Rz ← {z}/∗ constant vertices only have themselves

as possible substitutions ∗/5 qid← next query ID /∗ uniquely identifies query ∗/6 c← argminv∈VQ

hopt(v)/∗ pick optimal vertex to process next ∗/7 send (Q, qid, c, {(c→ c)}, {Rz}) message to location(c) slave node

On slave nodeInput: Graph query Q, query ID qid, designated vertex c,

partial substitution θ, candidate sets {Rz}, local index D8 for all edges e = (c, v) incident on c and some v ∈ VQ ∩ VAR9 if Rv = null

10 Rv ← retrieveNeighbors(D, c, λQ(e)) /∗ use index to retrieveall nbrs of c with same label as e ∗ /

11 else12 Rv ← Rv∩ retrieveNeighbors(D, c, λQ(e)) /∗ restrict space

of possible subst. for z ∗/13 if 6 ∃v, u ∈ VQ ∩ VAR : (u, v) ∈ EQ/∗ have we found an answer? ∗/14 {v1, . . . , vl} ← {v | v ∈ VQ ∩ VAR ∧ v /∈ θ}15 for (s1, . . . , sl) ∈ Rv1 × Rv2 × . . .× Rvl16 θ′ ← θ ∪ (v1 → s1) ∪ . . . ∪ (vl → sl)17 Send (qid, θ′) to master18 else if ∃w ∈ VQ : Rw = ∅/∗ reached dead end? ∗/19 return “NO”20 else21 w ← argminv∈VQ∧|Rv|>0hopt(v)/∗ pick optimal vertex to process next ∗/22 for all m ∈ Rw

23 θ′ ← θ ∪ {w → m}24 send (Q, qid,m, θ′, {Rz}) message to location(m) slave node

Fig. 4. COSI basic algorithm

The algorithm proceeds depth first, substituting verticesfor variables in Q one at a time. We maintain a set ofresult candidates for each variable in Q. This set is eitheruninitialized or is a superset of all result substitutions for thatparticular variable. The slave node query algorithm assumes

there is an index retrieval function retrieveNeighbors(D, v, l)that retrieves nghl(v) in sorted order from the local index D(which could be implemented many ways) on the slave. Fornow, hopt arbitrarily chooses the next vertex to be substituted.A better definition of hopt will be provided in the COSI heuralgorithm.

Incoming queries come with a selected variable to be instan-tiated with a vertex ID. The algorithm updates the candidateresult sets by retrieving the neighborhood of the newly sub-stituted vertex from the index. Since the result sets are sorted,this operation takes linear time. It then checks if any resultshave been found or whether the current substitution cannotyield a valid result. All query results are sent to the masterwhich returns them to the user. If neither condition holds, thealgorithm selects the next variable v′ to be substituted andforwards the query to those slave nodes that host potentialsubstitution candidates for v′.

The COSI basic algorithm uses two optimizations to re-duce messaging costs: (i) if the source and destination ofthe message in Lines 22-23 coincide, then the algorithmrecursively calls itself with updated data structures ratherthen sending the message; (ii) the query processor groupsall messages with the same query ID targeted at the sameslave node and sends them in one message, thus reducingcommunication cost. COSI basic does not rely on centralorchestration – it uses depth first search so the branches ofthe search tree are traversed in parallel while ensuring thatno branch gets explored multiple times. After forwarding theprepared query to a slave node, the master waits for incomingresults of that query and forwards those to the user. As weexplore branches in parallel, the master cannot be notifiedwhen the search for query results has completed. Keeping trackof the current number of parallel executions for each querywould introduce significant synchronization cost. Instead, themaster keeps track of the time tlast at which the last resultof a running query has come in. If the difference between thecurrent time and tlast exceeds a threshold, the master asks allslave nodes for a list of query IDs of all currently runningqueries. The master merges these lists and closes all querieswhose IDs are not contained. To avoid the case where a queryis being forwarded to another slave node at the very momentthat the master asks for all query IDs, each slave node keepsquery IDs in their local list up to a certain grace period.

A. The COSI heur algorithm

The choice of the next variable to be instantiated hasprofound implications on the running time of COSI basic,as some substitutions yield larger branching factors in thesearch than others. COSI heur handles this by choosing thevariable vertex v′ which has the lowest cost according tofunction hopt . First, to reduce branching factor, we couldchoose the variable vertex v′ with the smallest number ofresult candidates. This heuristic only considers the branchingfactor of the immediate next iteration, but is nevertheless animportant metric to consider in the cost heuristic. Second,whenever we instantiate a vertex on a remote partition block,

252252252

we have to send a message to the appropriate slave whichis expensive. Therefore, we consider the fraction of resultcandidates which are not stored locally as a cost metric.When we have to send a query to remote slaves for furtherprocessing, we would like to distribute the workload evenlyacross all slaves. Hence, we also analyze the distribution ofresult candidates by slave via the cost metric

ds(v) =

√√√√ ∑1≤i≤k

(|Riv| −

|Rv|k

)2

where Riv is the set of result candidates for vertex v restrictedto those which reside on slave node i. Finally, we define

hopt(v) = |Rv| × (1− |Rlv|α× |Rv|

)× (1 + β × ds(v)

|Rv|)

where l is the ID of the local slave node and α and β areconstants that determine how much the model favors localityover parallelism. Our experiments study how α, β impactquery run-times.

VI. IMPLEMENTATION EXPERIMENTAL RESULTS

COSI is implemented in Java. We developed a communi-cation infrastructure for the compute nodes based on the JavaNIO libraries which is used to send the graph data during theloading and the queries during the query answering stages. Thecommunication infrastructure handles contention at individualnodes and variations in network latency. It is optimized toensure that the dispatcher’s requests for outstanding queriesare answered quickly. COSI uses a modified version of theDOGMA graph database [1] for local storage of the graphdata. DOGMA itself interfaces against BerkeleyDB 4.8 [5]for on-disk storage. However, we emphasize that COSI isimplemented independently from the underlying graph storagebackend and can utilize alternative databases. The dispatchernode handles all user requests and maintains a vertex labellookup table on disk using BerkeleyDB. To reduce storagesize as well as message size, all vertex labels are mapped ontounique IDs at the dispatcher node. When a query is issued tothe dispatcher, it first retrieves the vertex ids for the labelsbefore forwarding the query to the storage node. Conversely,it looks up the vertex labels for all ids contained in a queryanswer before forwarding it to the issuing client. Moreover,to facilitate efficient retrieval, vertex IDs include a namespacewhich uniquely identifies the storage node; thus, vertices canbe directly located without a routing table.

In our experiments, we used a cluster of 16 compute nodesout of which one served as a dispatcher and the remaining15 nodes served as storage nodes. All storage nodes had anidentical hardware configuration with two Intel Xeon QuadCore 2.3 GHz Processors, 8 GB of RAM, and 73 GB SAS 10kRPM hard drive. The dispatcher’s hardware differed slightlywith 16 GB of RAM and two 146 GB 10k RPM SAS disksin RAID1 mirror configuration.

We fixed the coefficients for the affinity measure by hand.Both, the imbalance and excessive size metric, were given

an equal weight of 1. The connectedness measure was setrelative to the number of edges we considered per batch.We experimented with different batch sizes and found bestperformance for half a million edges.

By the definition of COSI basic query answering algo-rithm in Figure 4, any two co-retrieved vertices must beconnected. Hence, we set the probability of vertex co-retrievalto 0 for all unconnected pairs of vertices. The co-retrievalprobability for connected vertices was set to 1.4 Ideally, theseprobabilities would be derived from an analysis of queryexecution logs, however, such data was not available to us.

We used the social network data set studied in [6] forexperiments. This data set contains 778M edges and describespersonal relationships and group memberships crawled fromFacebook, Orkut, Flickr, and LiveJournal.

To evaluate the performance of our proposed partitioningand query answering strategies, we designed 11 queries ofvarying size. In designing the queries, we first fixed the querygraph topology and then randomly chose edge and vertexlabels from the social network dataset while ensuring that theresulting query has a non-empty result set. The size of a querygraph is measured by the number of edges and vertices itcontains. We list some of the queries used in our experimentsin the appendix.

A. Performance of COSI Partition

Fig. 5 compares COSI Partition’s performance with thatof the GreedyInsert algorithm. To validate our experiments,we used a random partitioning scheme, which assigns verticesto slave nodes uniformly at random, as the naive baseline inour experiments and report all results in comparison to thisbaseline. COSI Partition achieves a substantial 36% improve-ment in edge cut over the naive baseline at a total runningtime of 10.5 hours for all 778M edges. GreedyInsert onlyachieves a marginal improvement in edge cut. COSI Partitionsignificantly outperforms greedy batch insertion by 33% withonly slightly higher imbalance as measured in the standarddeviation in partition block size relative to average size of ablock. We observe that COSI Partition substantially outper-forms both baselines on the important edge cut quality metric.

Fig. 5. Comparison of partitioning methods

4Note, that multiplying the probabilities of co-retrieval by a constant factordoes not affect the edge cut minimization problem.

253253253

100  

1000  

10000  

100000  

1000000  

10000000  

6E/3V   7E/4V   8E/3V   9E/3V   10E/3V   11E/4V   11E/5V   14E/5V   16E/7V   17E/5V   23E/6V  

Time  (m

s)  

Size  of  the  query  (#  edges  /  #  verDces)  

COSI_heur  1.2/0.1   COSI_heur  8.0/5.0   COSI_heur  2.0/0.5   COSI_basic  

Fig. 6. Query times by query answering algorithm on the 778M edge Facebook/LiveJournal/Orkut data set

B. Query AnsweringFig. 6 compares COSI basic against COSI heur for three

different parameter settings of the heuristic hopt: (α =1.2, β = 0.1) which strongly favors locality over parallelism,(α = 8.0, β = 5.0) which strongly favors parallelism overlocality, and (α = 2.0, β = 0.5) which balances localityand parallelism. The queries have increasing complexity asmeasured by the number of edges (E) and variables (V) inthe query graph. All query times were averaged across 6independent runs with complete system restarts after each runto empty caches. Note, that the graph is plotted in logarithmicscale to accommodate the huge differences in query times.

COSI heur drastically outperforms COSI basic by upto 4 orders of magnitude on all but two queries, and theperformance gap seems to grow exponentially with the querycomplexity. A close look at the difference in performancebetween the variants of COSI heur reveals that the thirdconfiguration outperforms the first one on 9 queries, with a tieon the remaining 2, and outperforms the second configurationon 8 queries, being slower only on 3. These results suggest,that a balanced choice of parameters leads to a better hopt.

C. Partition ImpactWe derived theoretically in Theorem 3.2 that smaller edge

cut leads to shorter expected query times. To verify thetheorem experimentally, we compare the average query an-swering times over the partition generated by COSI Partitionagainst that produced by GreedyInsert. We have shownabove that COSI Partition produces partitions with loweredge cut which should reflect in shorter query times. We usedthe COSI heur query answering with the third parameterconfiguration to compute the times for our set of queriesand report the results in Fig. 7. The results support ourhypothesis by showing that the partition produced by COSIyields significantly better query times for complex queries.

VII. RELATED WORK

A wide variety of methods for SN analysis have beenproposed and include tool kits such as UCINet [7] or Pajek [8].

However, most SN algorithms operate solely in memory,loading the entire graph from disk and then executing theanalysis (see [9] for a survey of SN analysis software). Forsocial networks of the size of Facebook, Flickr, or Orkut, suchan approach becomes infeasible. To handle social networksof such magnitude one needs to store and query networkdata efficiently on disk. More importantly, complex queriesinvolving even a few joins (as shown in the Introduction)can quickly cause such approaches to run into trouble. Ronenand Shmueli [10] introduce a social network specific querylanguage and show how such queries can be answered onmoderately sized datasets. However, their query language isgeared toward users of a social network in helping them com-municate with friends. Earlier work on database technologiesfor general graph data such as Lore [2] considered muchsmaller graphs than the social networks we study here. Graphstructured RDF [11] data has been studied in the semanticWeb community. Initial approaches to RDF storage [12], [13]stored triples in relational tables and then used a relationalquery engine to answer queries. [14] showed that storingRDF in a vertical database leads to significant query timeimprovements. [15] uses several B+tree indexes for differentpermutations of subject, predicate, and object are maintained.All these approaches only work for single machines. Inresponse to the increasing need of scalability when facingextremely large RDF datasets, two approaches have essentiallybeen proposed so far: scale up and scale out. In scaling up,existing RDF databases, such as RDF-3X [15], Sesame [12],or YARS [16], are simply run on more powerful machines. Assuch it requires no technological innovation, but is extremelycostly and limited by current hardware. In scaling out, multiplemachines are utilized to store the data but all operations onthe data are centrally executed. Parallel storage regimes, suchas YARS2 [17], are cheaper but still limited in their scalabilitydue to central execution. Our approach demonstrated efficientquery answering across multiple machines without centralorchestration.

254254254

100  

1000  

10000  

100000  

6E/3V   7E/4V   8E/3V   9E/3V   10E/3V   11E/4V   11E/5V   14E/5V   16E/7V   17E/5V   23E/6V  

Time  (m

s)  

Size  of  the  query  (#  edges  /  #  verDces)  

COSI  ParDDon   GreedyInsert  

Fig. 7. Query times by partitioning method on the 778M edge Facebook/LiveJournal/Orkut data set

VIII. CONCLUSIONS

In this paper, we study subgraph matching on very largegraph data using a cloud architecture. We develop a proba-bilistic model for retrieval of individual vertices and successiveretrieval of vertices and use this model to show how splittingan SN S across k compute nodes can be reduced to a kindof minimal edge cut problem. As finding such edge cuts isNP-complete, we propose the COSI Partition algorithm forcreating such an index based on minimizing disconnectedness,imbalance and excessive size. Our COSI basic algorithm andCOSI heur algorithms rely on a distributed communicationmechanism between slave nodes to efficiently process querieson very large datasets. We show our framework works effi-ciently, answering many complex queries over a 778M edgereal-world SN dataset derived from Flickr, LiveJournal, andOrkut in under one second.

REFERENCES

[1] M. Brocheler, A. Pugliese, and V. S. Subrahmanian, “DOGMA: A disk-oriented graph matching algorithm for RDF databases,” in ISWC, 2009,pp. 97–113.

[2] R. Goldman, J. McHugh, and J. Widom, “From semistructured data toXML: migrating the Lore data model and query language,” in WebDB,1999, pp. 25–30.

[3] G. Karypis and V. Kumar, “A fast and high quality multilevel schemefor partitioning irregular graphs,” SIAM Journal on Scientific Computing,vol. 20, pp. 359–392, 1999.

[4] V. Blondel, J. Guillaume, R. Lambiotte, and E. Lefebvre, “Fast unfoldingof communities in large networks,” Journal of Statistical Mechanics:Theory and Experiment, vol. 2008, p. P10008, 2008.

[5] BerkeleyDB, “http://www.oracle.com/technology/products/berkeley-db.”[6] A. Mislove, M. Marcon, K. P. Gummadi, P. Druschel, and B. Bhattachar-

jee, “Measurement and analysis of online social networks,” in IMC,2007, pp. 29–42.

[7] S. P. Borgatti, M. G. Everett, and L. C. Freeman, “Ucinet for windows:Software for social network analysis,” Harvard: Analytic Technologies,2002.

[8] W. Nooy, A. Mrvar, and V. Batagelj, “Exploratory social networkanalysis with pajek,” Structural analysis in the social sciences, vol. 27,2005.

[9] M. Huisman and M. A. V. Duijn, “Software for social network analysis,”Models and methods in social network analysis, p. 270316, 2005.

[10] R. Ronen and O. Shmueli, “Evaluating very large datalog queries onsocial networks,” in EDBT, 2009, pp. 577–587.

[11] O. Lassila and R. Swick, Resource Description Framework (RDF)model and syntax specification. W3C, 1998. [Online]. Available:http://www.w3.org/TR/1999/RECrdf -syntax-19990222.

[12] J. Broekstra, A. Kampman, and F. van Harmelen, “Sesame: An archi-tecture for storing and querying RDF data and schema information,” inSpinning the Semantic Web, 2003, pp. 197–222.

[13] K. Wilkinson, C. Sayers, H. A. Kuno, and D. Reynolds, “Efficient RDFstorage and retrieval in Jena2,” in SWDB, 2003, pp. 131–150.

[14] D. J. Abadi, A. Marcus, S. Madden, and K. J. Hollenbach, “Scalablesemantic Web data management using vertical partitioning,” in VLDB,2007, pp. 411–422.

[15] T. Neumann and G. Weikum, “RDF-3X: a RISC-style engine for RDF,”PVLDB, vol. 1, no. 1, pp. 647–659, 2008.

[16] A. Harth and S. Decker, “Optimized index structures for querying RDFfrom the Web,” in LA-Web, 2005, pp. 71–80.

[17] A. Harth, J. Umbrich, A. Hogan, and S. Decker, “YARS2: A federatedrepository for querying graph structured data from the web,” in ISWC,2007, pp. 211–224.

APPENDIX: SUBSET OF EVALUATION QUERIESIn the following, we list 3 of the 11 subgraph matching queries used in the

experiments. Question marks denote variables and numbers denote anonymized vertexlabels. Queries are represented as lists of directed edges:

start-vertex -- edge-type --> end-vertex

Query 1?A -- member --> group675?A -- knows --> ?B?B -- member --> group224?C -- knows --> ?B?C -- member --> group333?B -- knows --> ?A

Query 2?A -- member --> group3085?B -- knows --> ?A?B -- knows --> ?C?C -- knows --> ?A?A -- member --> group4087?C -- knows --> ?B?B -- knows --> ?D

Query 9?A -- member --> group682?A -- knows --> ?B?B -- knows --> ?A?D -- knows --> ?A?B -- knows --> user121?D -- knows --> ?B?D -- member --> group511?A -- knows --> ?D?C -- knows --> ?F?C -- knows --> user1208?F -- knows --> ?D?F -- knows --> ?A?B -- knows --> ?F?C -- member --> group1209?A -- knows --> ?F?D -- knows --> ?F?F -- member --> group311

255255255