scalable social sensing of interdependent phenomena

Scalable Social Sensing of Interdependent Phenomena

Shiguang Wang1, Lu Su2, Shen Li1, Shaohan Hu1, Tanvir Amin1, Hongwei Wang1

Shuochao Yao1, Lance Kaplan3, Tarek Abdelzaher1

1Department of Computer Science, University of Illinois at Urbana-Champaign, Urbana, IL 618012Department of Computer Science and Engineering, SUNY at Buffalo, Buffalo, NY 142603Networked Sensing and Fusion Branch, US Army Research Labs, Adelphi, MD 207843

ABSTRACTThe proliferation of mobile sensing and communication de-vices in the possession of the average individual generatedmuch recent interest in social sensing applications. Sig-nificant advances were made on the problem of uncover-ing ground truth from observations made by participantsof unknown reliability. The problem, also called fact-findingcommonly arises in applications where unvetted individualsmay opt in to report phenomena of interest. For exam-ple, reliability of individuals might be unknown when theycan join a participatory sensing campaign simply by down-loading a smartphone app. This paper extends past socialsensing literature by offering a scalable approach for ex-ploiting dependencies between observed variables to increasefact-finding accuracy. Prior work assumed that reportedfacts are independent, or incurred exponential complexitywhen dependencies were present. In contrast, this paperpresents the first scalable approach for accommodating de-pendency graphs between observed states. The approach istested using real-life data collected in the aftermath of hur-ricane Sandy on availability of gas, food, and medical sup-plies, as well as extensive simulations. Evaluation shows thatcombining expected correlation graphs (of outages) with re-ported observations of unknown reliability, results in a muchmore reliable reconstruction of ground truth from the noisysocial sensing data. We also show that correlation graphscan help test hypotheses regarding underlying causes, whendifferent hypotheses are associated with different correlationpatterns. For example, an observed outage profile can be at-tributed to a supplier outage or to excessive local demand.The two differ in expected correlations in observed outages,enabling joint identification of both the actual outages andtheir underlying causes.

Categories and Subject DescriptorsH.4 [Information Systems Applications]: Miscellaneous

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full cita-tion on the first page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re-publish, to post on servers or to redistribute to lists, requires prior specific permissionand/or a fee. Request permissions from [email protected] ’15, April 14 - 16, 2015, Seattle, WA, USA.Copyright 2015 ACM 978-1-4503-3475-4/15/04 ...$15.00.http://dx.doi.org/10.1145/2737095.2737114.

KeywordsSocial Sensing, Data Reliability, Maximum Likelihood Esti-mators, Expectation Maximization

1. INTRODUCTIONRecent advent of sensing applications, where unvetted

participants can perform measurements, generated interestin techniques for uncovering ground truth from observationsmade by sources of unknown reliability. This paper ex-tends prior work by offering scalable algorithms for exploit-ing known dependency graphs between observed variablesto improve the quality of ground truth estimation.

Consider, for example, post-disaster scenarios, where sig-nificant portions of a city’s infrastructure are disrupted. Com-munication resources are scarce, rumors abound, and meansto verify reported observations are not readily available.Survivors report to a central unit the locations of damageand outages, so that help may be sent. Some reports are ac-curate, but much misinformation exists as well. Not know-ing the individual sources in advance, it may be hard to tellwhich reports are more reliable. Simply counting the num-ber of reports that agree on the facts (called voting in priorliterature) is not always a good measure of fact correctness,as different sources may have different reliability. Hence, adifferent weight should be associated with each report (orvote), but that weight is not known in advance.

Prior work of the authors addressed the above problemwhen the reported observations are independent [22] andconsidered the case where second-hand observations werereported by other than the original sources [21]. In workthat comes closest to the present paper, an algorithm waspresented for the case, where the reported variables are cor-related [20]. Unfortunately, the computational and repre-sentational complexity of the correlation was exponential inthe number of correlated variables. Hence, in practice, itwas not feasible to consider more than a small number ofcorrelated variables at a time.

In sharp contrast to the above results, in this paper, weconsider the case where reported variables have non-trivialdependency graphs. For example, upon the occurrence ofa natural or man-made disaster, flooding, traffic conditions,outages, or structural damage in different parts of a citymay be correlated at large scale. Furthermore, the struc-ture of the correlations might be partially known. Areas ofthe same low elevation may get flooded together. Nearbyparts of the same main road may see correlated traffic con-ditions. Buildings on the same power line may suffer corre-lated power outages. Gas stations that have the same sup-

202

plier might have correlated availability of gas. Correlations(e.g., among failures) can also shed light on the root cause.For example, in a situation where a supply line simultane-ously feeds several consumers, a failure in the line will resultin correlated failures at the downstream consumers. If thetopology of the supply lines is known, so is the correlationstructure among expected consumer failures. If consumersbuild products that need multiple suppliers, knowing thepattern of the corelated consumer failures can give strongevidence as to which one of the suppliers may have failed.

Clearly, if the aforementioned correlation structure is notknown, we cannot use this approach. Scenarios where cor-relations structures are not known can be addressed usingprior work that simply views the underlying variables as un-correlated [22]. This paper offers performance and accuracyimprovements in the special (but important) case, where hy-potheses regarding possible correlations are indeed available.Exploiting such correlations reduces problem dimensional-ity, allowing us to infer the state of points of interest moreaccurately and in a more computationally efficient manner.

What complicates the problem (in social sensing scenar-ios) is that actual state of the underlying physical system isnot accurately known. All we have are reports from sourcesof unknown reliability. This paper develops the first scalablealgorithm that takes advantage of the structure of correla-tions between (large numbers of) observed variables to bet-ter reconstruct ground truth from reports of such unreliablesources. We show that our algorithm has better accuracythan prior schemes such as voting and maximum likelihoodschemes based on independent observations [22]. It also sig-nificantly outperforms, in terms of scalability, previous workthat is exponential in dependency structures [21].

The general idea behind the scalability of the new schemelies in exploiting conditional independence, when one cata-lyst independently causes each of multiple consequences tooccur. Identification of such conditional independence rela-tions significantly simplifies reasoning about the joint cor-relations between observed values, thus simplifying the ex-ploitation of such correlations in state estimation algorithms.Although previous work [20] considers correlated variablesin social sensing applications, it does not exploit conditionalindependence. The computational complexity of the previ-ous solution increases exponentially in the number of corre-lated variables, which makes it applicable only to applica-tions with a small number of such variables. By modelingthe structural correlations of variables as a Bayesian networkand exploiting conditional independence, our algorithm ismore computationally efficient. Its computational complex-ity depends on the size of the largest clique (i.e., completesub-graph) in the Bayesian network, while the complexityof the previous solution [20] depends on the total number ofnodes in the Bayesian network making the latter intractablefor applications with a large number of correlated variables.

The contributions of this paper are thus summarized asfollows:

• We extend the previous solution to a new applicationdomain in which a large number of variables are struc-turally interdependent. Our algorithm is shown to bemore accurate and computationally efficient, comparedwith previous solutions.

• The interdependent structure is formulated as a Bayesiannetwork, which enables us to exploit well-established

techniques of Bayesian network analysis. We also showthat the Bayesian network generalizes the models weproposed in previous work.

• Our algorithm is evaluated by both extensive simula-tions and a real-world data set. The evaluation resultsshow that our solution outperforms the state of theart.

The rest of the paper is organized as follows. We formulateour problem in Section 2. In Section 3, we argue that oursolution is general and can be applied to solve previous socialsensing challenges by showing that all the previous modelsare special cases of our Bayesian model. We propose oursolution in Section 4 and evaluate our algorithm in Section 5.A literature review is presented in Section 6. The paperconcludes in Section 7.

2. PROBLEM FORMULATIONSocial sensing differs from sensing paradigms that use in-

field physical sensors (e.g. Wireless Networked Sensing [15])in that it exploits sensors in social spaces. Examples in-clude sensor-rich mobile devices like smartphones, tablets,and other wearables, as well as using humans as sensors.The involvement of humans in the sensing process enablesan application to directly sense variables with higher-levelsemantics than what traditional sensors may measure. How-ever, unlike physical devices, which are usually reliable orhave the same error distribution, the reliability of humansources is more heterogeneous and may be unknown a pri-ori. This source reliability challenge in social-sensing sys-tems was recently articulated by Wang et al. [22]. Solutionsthat estimate source reliability were improved in follow-uppublications [20, 21, 23].

In recent work, the authors modeled human sources as bi-nary sensors, reporting events of interest. The rationale be-hind the binary model is that humans are much better at cat-egorizing classes of observations than at estimating precisevalues. For example, it is much easier to tell whether a roomis warm or not than to tell its exact temperature. Binaryvariables can be easily extended to multivalued ones [23],which makes the binary model versatile. In this paper, weadopt the binary model and assume that a group of humansources, denoted by S, participate in a sensing applicationto report values of binary variables, we call the event vari-ables. For example, they may report the existence or absenceof gas at a set of gas stations after a hurricane. These vari-ables are collectively denoted by C. The goal of this paperis to jointly estimate both the source reliability values andground-truth measured variable values, given only the stringof noisy reports. In contrast to prior work, we assume thatthe underlying variables are structurally correlated at scale.The question addressed in this paper is how to incorporateknowledge of these correlations into the analysis.

2.1 Modeling Interdependent Event VariablesIn previous work, event variables were either assumed to

be independent [22, 21] or were partitioned into groups ofsmall size [20, 23] with no dependencies among groups. So-lution complexity grew exponentially with the maximumgroup size. In practice, it is not uncommon that all or alarge portion of event variables are interdependent. For ex-ample, in an application that monitors traffic conditions ina city, pertinent variables might denote weather conditions

203

(e.g., snowy or rainy weather), local entertainment eventsthat impact traffic (e.g., football games or concerts), roadsurface conditions (e.g., potholes on road surfaces), and traf-fic speed, among others. These variables are correlated. Badweather results in slow traffic. So do the local entertainmentevents and bad road surfaces. Traffic congestion on one roadsegment might cause congestion on another road segment.The pervasive dependencies among variables make previouswork (e.g., Wang et al. [20]) inapplicable due to intractabil-ity, thus calling for a better model to handle them. Thispaper is the first to study the reliable social-sensing prob-lem with interdependent variables, at scale.

Our solutions are based on the insight that although in-dependence is uncommon in real applications, conditionalindependence does often arise. As stated in [12], depen-dencies usually expose some structure in reality. The de-pendency structure encodes conditional independence, thatcan be leveraged to greatly simplify the estimation of val-ues of variables. In the previous application example, giventhat the weather is snowy, the resulting traffic congestionon two road segments can be assumed to be conditionallyindependent. Both are caused by snowy weather but nei-ther is affecting the other (assuming they are sufficiently farapart). However, without knowing the state of the weather,we are not able to assume that congestion on both segmentsis independent. Measuring congestion on those segments, itwill tend to be correlated (in the presence of snow events).

In this paper, we model dependencies among variablesby a Bayesian network [13]. The underlying structure ofa Bayesian network is a directed acyclic graph (DAG) inwhich each node V corresponds to a variable, v, and each arcU → V denotes a conditional dependence such that the valueof variable v is dependent on the value of u. The Bayesiannetwork is a natural way to model causal relations betweenvariables. Since Bayesian networks are well-established toolsfor statistical inference, we can leverage prior results to solveour reliable social-sensing problem.

Of course, in some cases, the underlying dependences canform a complete graph in which any pair of variables are di-rectly interdependent. In this extreme situation, there wouldbe no efficient inference algorithm with computational com-plexity inferior to Θ(2N ), where N denotes the total numberof variables (i.e., |C|). All inferences should be made by con-sidering the joint distribution of all variables. However, asstated in [12], the complete graph structure does not oftenhappen in real applications, and so we are not interested inthis extreme case.

2.2 Categorized Source ReliabilityAlthough previous work in social sensing assumes that

sources have different reliability, for a specific source, itsreliability is assumed to be fixed (e.g., [22, 20, 21]. Thisfixed-reliability assumption does not hold in many practicalscenarios. For example, a diabetic person who is in need ofinsulin might be a better source to ask about pharmaciesthat remained open after a natural disaster, than a personwho is not in need of medication. The same diabetic personmight not be a good source to ask about gas stations thatare open, if the person does not own a car. In the abovescenario, if we assume that a single source has the same re-liability in reporting all types of variables, the performanceof estimating the ground truth of these variables might bedegraded. To make the source reliability model more practi-

cal, and thus the estimation more accurate, we assume thatsource reliability differs depending on the variable reported.Measured variables are classified into different categories.Source reliability is computed separately for each category.We call it categorized source reliability.

With the categorized-source-reliability model, the relia-bility of each source is represented by a vector (where eachelement is corresponding to the reliability for some reportedcategory of variabls, rather than a scalar as in previous work.Please note that the previous reliability model is a specialcase of our model as a single-element vector.

2.3 Problem DefinitionNext, we formally define our reliable social-sensing prob-

lem with interdependent variable at scale. We denote the j-th measured variable by Cj , and Cj is assumed to be binary.More specifically, Cj ∈ {T, F} where T represents True (e.g.,“Yes, the room is warm”), and F represents False (e.g., “No,the room is not warm”). One can think of each variableas the output of a different application-specific True/Falsepredicate about physical world state. Each variable Cj be-longs to some category `, denoted by `Cj . We use L todenote the category set.

In social-sensing, a source reports the values of variables.We call those reports, claims. We use a matrix SC to repre-sent the claims made by all sources S about all variables C.We call it the source-claim matrix. In the source-claim ma-trix, an element SCi,j = v means that the source Si claimsthat the value of variable Cj is v. It is also possible that asource does not claim any value for some variable, in whichcase the corresponding item in the source-claim matrix SCi,jis assigned value U (short for “Unknown”) meaning that thesource did not report anything about this variable. There-fore, in the source-claim matrix, SC, each item SCi,j hasthree possible values T , F and U .

We define the reliability of source Si in reporting valuesof variables of category ` as the probability that variablesbelonging to that category indeed have the values that thesource claims they do. In other words, it is the probabilitythat `Cj = v, given that SCi,j = v. In the following, weshall use the short notation Xv to denote that the variableX is of value v (i.e., X = v). Let `ti denote the reliability ofsource Si in reporting values of variables of category `. Weformally define the source reliability as follows:

`ti = Pr(`Cvj |SCvi,j

). (1)

Let `T vi denote the probability that source Si reports thevalue of variable `Cj correctly. In other words, the proba-bility that Si reports value v for variable `Cj given that itsvalue is really v. Furthermore, let `F vi denote the probabil-ity of an incorrect report by Si. In other words, it is theprobability that Si reports that `Cj has value v̄ given thatits value is v. Here x̄ is the complement of x (T̄ = F andF̄ = T ). `T vi and `F vi are formally defined below:

`T vi = Pr(SCvi,j |`Cvj

), `F vi = Pr

(SC v̄i,j |`Cvj

). (2)

Note that, `T vi + `F vi ≤ 1, since it is possible that the sourceSi does not report anything of a variable. Therefore, wehave:

1− `T vi − `F vi = Pr(SCUi,j |`Cvj

). (3)

204

Table 1: The summary of notations

Set of sources SSet of variables C

Binary variable, j CjVariable X of category ` `X

Binary value set {T, F}Source-claim matrix SC

Source reliability `ti = Pr(`Cvj |SCvi,j

)Correctness probability `T vi = Pr

(SCvi,j |`Cvj

)Error probability `F vi = Pr

(SC v̄i,j |`Cvj

)We denote the prior probability that source Si makes a pos-itive claim (i.e., claims a value T ) by sTi and denote theprior probability that source Si makes a negative claim (i.e.,claims a value F ) by sFi . We denote the prior probabilitythat variable Cj is of value v by dv. By the Bayesian theo-rem, we have:

`T vi =`ti · svi`dv

, `F vi =(1− `ti) · sv̄i

`dv. (4)

Table 1 summarizes the introduced notations.The dependencies between variables are given by a Bayesian

network. In the underlying dependency structure of theBayesian network (i.e., a DAG, denoted by G), each ver-tex Vj corresponds to a variable Cj , and each arc Vj → Vkcorresponds to a causal relation between variables Cj andCk in which the value of Ck is dependent on that of Cj .For any variable Cj , we use par(Cj) to denote the set ofvariables on whom the value of Cj directly depends (i.e.,not including transitive dependencies). Since the causal re-lation is encoded in the Bayesian network, G, there is an arcfrom each node denoting a variable in par(Cj) to the nodedenoting Cj in G. Please note that each node Vj in theBayesian network is associated with a probability functionthat takes, as input, a particular set of values of par(Cj),and gives, as output, the probability of Cj being true. Inother words, given the Bayesian network, we know the con-ditional probability Pr

(`Cj

v|par(Cj))

for any event variableCj . We assume that the Bayesian network is known fromapplication context (e.g., we might have a map that sayswhich outlet depends on which suppliers), or can be empir-ically learned from historic data by the algorithms such asthose introduced in [13]. Hence, our estimation algorithmassumes that the Bayesian network is an input.

Finally, we formulate our reliable social-sensing problemas follows: Given a source-claim matrix SC, a category la-bel ` for each reported variable, and a Bayesian network G,encoding the dependencies among variables, how to jointlyestimate both the reliability of each source and the true valueof each variable in a computationally efficient way? Here analgorithm is defined as efficient if its time complexity is sub-linear to the exponential (i.e., o(2|G|) for a Bayesian networkthat is not a complete graph, where |G| is the total numberof nodes in the Bayesian network.

3. GENERALIZATION OF PREVIOUSMODELS

Before we propose our estimation algorithm, in this sec-tion, we show that our social-sensing model is more generalthan those proposed in our previous work [22, 20, 21, 23]. In

other words, the social-sensing models proposed by the pre-vious work are all special cases of ours. Therefore, thanks tothe general model, the estimation algorithm proposed in ourpaper can be directly applied to any of the problems definedin the previous work.

Figure 1: Model connections with previous work.

First, we show that the model proposed by Wang et al. [22]is a special case of our model. In their model, both theevent variables and the sources were assumed independent.Thus, the structure of a Bayesian network for this modelis just a DAG with arcs only connecting the event variableand its corresponding claims from the sources, as shown inFigure 1(a).

In [20], the model was extended to consider physical con-straints of the sources (i.e., a variable might not be observedby some source), as well as correlated variables that fall intoa bunch of independent groups. The structure of a Bayesiannetwork for this model has disjoint cliques (complete sub-graphs) where each clique has a constant number of nodes,as shown in Figure 1(b). In this figure, there are two cliques;one has two nodes and the other has three. Furthermore,since the physical constraints of the sources are considered,there are some variable that can only be observed by a sub-set of sources. Therefore, in the corresponding Bayesian net-work, if the variable is not observed by some source, thenthere is no arc between them in the DAG (such as the right-most one that is observed only by the red source and theorange source).

Source dependencies were considered in [21], where a claimmade by a source can either be original or be re-tweeted fromsome other source. The variables are assumed independent.The corresponding Bayesian network for this model is shownas in Figure 1(c). If a source i is dependent on some othersource j, which means that the claim made by j actually af-fects that made by i as shown in Figure 1(c). In Figure 1(c),the black source is dependent on the red source, thereforethere is an arc from the red node to the black node for eachevent variable. The arc from the SCj· to SCi· is enough tomodel this dependence.

Recently, Wang et al. [23] further extended the previousmodel by considering time-varying ground truth, in whichthe value of each variable could vary over time. They provedthat given the evolving trajectory of each variable, by con-sidering a sliding window of past states, the estimation resultis greatly improved compared with estimators that only con-sider the current state. Their model can be represented bya dynamic Bayesian network with time-varying dependencystructures. Figure 1(d) gives an example of a Bayesian net-

205

work representation of their model. Here, we omit the ver-tices corresponding to claims made by sources SCi,j . Inthe figure, the variable nodes with the same color are corre-sponding to a variable in different time-slots. The evolvingtrajectory of each variable can be represented by some de-pendency structure among all its history states, as shown inFigure 1(d).

The above examples illustrate how previous models canbe special cases of our model. Therefore, once we solved theproblem with the general model, using the same algorithm,we are able to solve all the previous problems as definedin [22, 20, 21, 23]. We propose our estimation algorithm inthe following section.

4. ESTIMATING THE STATES OF INTER-DEPENDENT VARIABLES

In this section, we describe our ground truth estimation al-gorithm for social-sensing applications with the interdepen-dent variables at scale. Our algorithm follows the Expectation-Maximization (EM) framework [4] that jointly estimates (1)the reliability of each source, and (2) the ground truth valueof each reported variable. Here we assume that sources in-dependently make claims; for dependent sources, we can ap-ply the algorithm proposed in [21]. We call the proposedalgorithm EM-CAT (EM algorithm with CATegory-specificsource reliability.

4.1 Defining Estimator Parameters and Like-lihood Function

EM is a classical machine-learning algorithm to find themaximum-likelihood estimates of parameters in a statisti-cal model, when the likelihood function contains latent vari-ables [4]. To apply the EM algorithm, we first need to definethe likelihood function L(θ;x, Z), where θ is the parame-ter vector, x denotes the observed data, and Z denotes thelatent variables. The EM algorithm iteratively refines theparameters by the following formula until they converge:

θ(n+1) = arg maxθEZ|x,θ(n) [logL(θ;x, Z)] (5)

The above computation can be further partitioned into an E-step that computes the conditional expectation of the latentvariable vector Z (i.e., Q(θ) = EZ|x,θn [logL(θ;x, Z)]), andan M-step that finds the parameters θ that maximize theexpectation (i.e., θ(n+1) = arg maxθ Q(θ)). In our problem,we define the parameter vector θ as:

θ = {(`T vi , F vi )|∀i ∈ S, v ∈ Λ, ` ∈ L}

where Λ = {T, F} denotes the set of binary values and Lis the set of event categories. The data x is defined as theobservations in the source-claim matrix SC, and the latentvariable vector Z is defined as the values of the event vari-ables.

After defining θ, x and Z, the likelihood function is derivedas follows:

L(θ;x, Z) = Pr (x, Z|θ) = Pr (Z|θ) Pr (x|Z; θ)= Pr (Z1, · · · , ZN ) · Pr (x1, · · · , xN |Z1, · · · , ZN ; θ) .

(6)

Here N = |C| is the number of event variables, and xj de-notes all the claims made by the sources about the j-th vari-able. In Equation (6), Pr (Z|θ) = Pr (Z) because the jointprobability of the event variables Pr (Z) is independent fromthe parameters θ.

Next, we are going to simplify the likelihood function byproving that for any event variables j1 and j2, xj1 and xj2are conditionally independent given the latent variables Z.We useX ⊥⊥ Y to denote thatX and Y are independent, andsimilarly X ⊥⊥ Y |Z to denote that X and Y are conditionallyindependent given Z. Before proving xj1 ⊥⊥ xj2 |Z, we firstintroduce the definition of d-separation.

Definition 1 (d-separation). Let G be a Bayesian net-work, and X1 · · · Xn be a trail in G. Let Z be a subsetof the observed variables. The trail X1 · · · Xn

1 isactive given Z if

• Whenever we have a V-structure Xi−1 → Xi ← Xi+1

in the trail, then Xi or one of its descendants are inZ, and

• no other node along the trail is in Z.

If for any trail between X1 and Xn is not active, then X1

and Xn are d-separated in G by Z [10].

Here a trail between X1 and Xn is an undirected paththat is computed by simply ignoring the directions of thedirected edges in the Bayesian network G. Note that if X1

or Xn is in Z, the trail is not active. Next, we introduce aclassical lemma showing that the d-separation implies con-ditional independence.

Lemma 1. If Xi and Xj are d-separated in the Bayesiannetwork G given Z, then Xi ⊥⊥ Xj |Z [10] .

Now we are ready to prove that xj1 and xj2 are condi-tionally independent given the latent variables Z for anyj1, j2 ∈ C in Theorem 1.

Theorem 1. For any pair of event variables j1 and j2,xj1 and xj2 are conditionally independent given the latentvariables Z, i.e., ∀j1, j2 ∈ C, xj1 ⊥⊥ xj2 |Z.

Proof. To prove the theorem, we first need to define thecausal relationship between a claim SCi,j and the value Cj ofevent j. Obviously, the value Cj of the event is independentof how a source claims it, but the claim SCi,j made by asource does rely on the value Cj of event j. Therefore, it isclear this causal relationship between SCi,j and Cj should bemodeled by an arc from Zj to xi,j in the Bayesian network,as illustrated in Figure 2. Here we do not distinguish thevariable Zj and its corresponding vertex in G, and the samefor xi,j and its corresponding vertex.

Therefore, for any pair of xi1,j1 and xi2,j2 , and for what-ever event dependency graph G of the event variables, wecan find two vertices Zj1 and Zj2 in G such that all thetrails have the same structure: xi1,j1 ← Zj1 · · · Zj2 → xi2,j2 , as shown in Figure 2. Since Zj1 and Zj2 arein Z = {Z1, · · · , ZN}, and by Definition 1, we know thatxi1,j1 and xi2,j2 are d-separated by Z. Please note that thisd-separation is valid for any pair of sources i1 and i2, thusxi ⊥⊥ xj |Z by Lemma 1.

By Theorem 1 and the independent source assumption,the likelihood function in (6) can be simplified as:

L(θ;x, Z) = Pr (Z1, · · · , ZN )∏j∈C

∏i∈S Pr (xi,j |Zj ; θ) .

(7)1We use X → Y to denote the directed edge (arc) thatpoints from X to Y in G, and X Y to denote the arcthat connects X and Y whose direction, however, is not ofinterest.

206

Figure 2: An illustration of the Bayesian network.

4.2 The EM AlgorithmGiven the likelihood function, following (5), we can de-

rive the EM algorithm. We omit the detailed mathematicalderivations here since it is a standard procedure, and di-rectly show the final results of how to update the parametersθ = {`T vi , `F vi |∀i ∈ S, v ∈ Λ, ` ∈ L} in (8).

`T vi(n+1)

=

∑j∈`Cv

iPr(Zj=v|x;θ(n))∑

j∈`C Pr(Zj=v|x;θ(n)),

`F vi(n+1)

=

∑j∈`Cv̄

iPr(Zj=v|x;θ(n))∑

j∈`C Pr(Zj=v|x;θ(n)).

(8)

In (8), θ(n) denotes the parameters in the n-th iteration ofthe EM algorithm, `C denotes the set of event variables withlabel `, and `Cvi is a subset of `C with each element that thesource i claims its value being v (i.e. `Cvi = {j|SCi,j =v, L(j) = `}, where L(j) denotes the label of event variablej).

In Equation (8), the key step for refining the parameters

is to compute Pr(Zj = v|x; θ(n)

)for each j ∈ C and v ∈ Λ.

Once we have computed this value, the rest of the compu-tation for updating the parameters becomes trivial. Sincewe are using a Bayesian network to encode the dependencesbetween variables, we know the conditional probability foreach variable given a particular set of values of its parentvariables. These are given as an input of our algorithm.

Hence, we can compute Pr(Zj = v|x; θ(n)

), the marginal

probability of variable Zj given the evidence x. Similarly,

Pr(Zj = v|x; θ(n)

)can be computed. The pseudocode of

our estimation algorithm is given in Algorithm 1.

5. EVALUATIONIn this section, we study the performance of our EM-CAT

algorithm through extensive simulations as well as a real-world data set. While empirical data is always better, suchdata often constitutes isolated points in a large space of pos-sible conditions. The simulation, in contrast, can extensivelytest the performance of our algorithm under very differentconditions that are impractical to cover exhaustively in anempirical manner. The limitation of simulation is that read-world data might not follow exactly the assumed model.Therefore, we also evaluate our algorithm with a real-worlddata set. Evaluation results show that the new algorithmoffers better estimation accuracy compared to other state-of-the-art solutions.

5.1 Simulation StudyIn this simulation study, we build a social-sensing simula-

tor in Matlab R2013b. For Bayesian network inference, we

Algorithm 1 EM-CAT: Expectation-Maximization Algo-rithm with Category-specific Source Reliability

Input: The source-claim matrix SC, the Bayesian network G,and event category ` ∈ L for each event j ∈ C.Output: The estimated variable values, and the reliability vectorof each source.

1: Initialize θ(0) with random values between 0 and 0.5.2: n← 03: repeat4: n← n+ 15: for Each j ∈ C, each v ∈ Λ do6: Compute Pr

(Zj = v|x; θ(n)

)from the Bayesian net-

work G.7: end for8: for Each i ∈ S, each v ∈ Λ and each label ` ∈ L do

9: Compute `T vi(n)

and `F vi(n)

from (8)10: end for11: until θ(n) and θ(n−1) converge12: for Each j ∈ C and v ∈ Λ do13: Z(j, v)← Pr

(Zj |x; θ(n)

).

14: if Z(j, T ) > Z(j, F ) then15: Assign variable j with value T16: else17: Assign variable j with value F18: end if19: end for20: for Each i ∈ S, each category ` ∈ L do21: Compute its reliability `ti from (4)22: end for

exploit an existing Bayesian network toolbox developed byKevin Murphy [8]. Below we present our simulation setupand results.

5.1.1 MethodologyWe simulated 100 interdependent binary variables. The

underlying dependency graph is a random DAG that changesin each simulation run. The Bayesian network is createdwith the dependency structure defined by the DAG and pa-rameters randomly generated using the toolbox. The ex-pected ground truth for all variables is set to 0.5 (i.e., withprobability 0.5, the variable will be True). The actual (marginal)probability distribution for each variable is defined by theBayesian network.

The ground truth values of variables are generated basedon the Bayesian network in a topologically-sorted order.That is, we wait to generate the value of variable v untilthe values of all of its parents, par(v), have been generated.Therefore, the ground truth value distribution of our vari-ables follows the Bayesian network. Each event variable isalso assigned a label ` randomly from a label set L to sim-ulate the event category.

The simulator randomly assigns a reliability vector foreach source. We randomly select a set of the sources tobe “experts” at some category. Hence, for each, we choosea category, `, and give the source a high reliability valuein reporting variables of that category. The other valuesin the reliability vector are assigned lower values, makingthe average reliability of each source roughly the same. Weuse ti to denote the average reliability of source Si. In thesimulation, ti is in the range (0.5, 1). We also simulate the“talkativeness” of the sources, which denotes the probabilitythat a source would make a claim, denoted by si.

The source-claim matrix SC is then generated accordingto the reliability vector of each source, ti, and the talkative-

207

ness of each source, si. For each source Si and each eventvariable Cj , we first decide whether the source will make aclaim about the variable by flipping a biased coin with prob-ability si that the source will claim something. If it does notclaim, then SCi,j = Unknown, otherwise we generate thevalue of SCi,j based on T vi and F vi which can be computedfrom the reliability vector of the source.

The source-claim matrix SC is the evidence x in the Bayesiannetwork. To include the claim nodes in the Bayesian net-work, we extend the DAG such that for each vertex Vj inthe DAG G we create one vertex V ′j and add an arc (Vj , V

′j )

to G. We tried to add one vertex Vi,j for each xi,j anddirectly set the parameters of Vi,j to the corresponding T viand F vi in the parameter vector θ(n). This implementationis straightforward. However, it adds too many extra (thatis |S| × |C|) vertices to the Bayesian network, which greatlyslows down the inference computation. Therefore, in ourimplementation, we just add one vertex V ′j for each variableVj in G, and set the evidence (the observed value) of V ′j to

False (which means Pr(V ′j = False|Vj

)= p(xj |Zj ; θ(n)) =∏

i∈S p(xi,j |Zj ; θ(n))). In this implementation, we only dou-

ble the size of the DAG, which makes the inference compu-tation of the Bayesian network much more efficient.

The default values of the simulation parameters are asfollows: the number of sources is 40, the expected source(average) reliability ti is 0.6, the talkativeness of the sourcesi is 0.6. The number of event variables is set to 100, and werandomly generate the Bayesian network parameters suchthat, in expectation, the probability that each variable isTrue, is set to 0.5. The number of edges in the Bayesiannetwork is 100. There are 2 categories by default.

We compare our algorithm to the algorithm proposed in [22]and two intermediate extensions towards the current solu-tion. We also include a simple baseline. We use EM-CATto denote our algorithm, and EM-REG to denote the algo-rithm proposed in [22]. Note that, EM-REG assumes thatvariables are independent, and all the variables share thesame category (i.e., it assumes that there is only one cate-gory). The first extension of EM-REG is to add the Bayesiandependency structure to the event variables. We call thisextension EM-T (EM algorithm with sTructed variables).The second extension of EM-REG is to consider event cate-gories, that is called EM-C. The simple baseline algorithmis just voting, and is denoted by VOTING. VOTING esti-mates each variable to be equal to the majority vote. Eachsimulation runs 100 times and each result is averaged overthe 100 executions.

5.1.2 Evaluation ResultsFigure 3 shows the performance of our EM-CAT algo-

rithm as the number of sources varies from 20 to 80. In Fig-ure 3(a), we observe that our EM-CAT algorithm has thelowest estimation error, and EM-T and EM-C work betterthan the regular EM algorithm which is better than sim-ple baseline voting. The reason is that when the underlyingevent variables follow some dependency structure, exploit-ing this piece of information will result a better estimator.EM-CAT also considers the category-specific reliability ofeach source. For each event category, EM-CAT will alwaysselect the sources with higher reliability for the category.Therefore, it achieves higher accuracy. Please note that asthe number of sources increases, the accuracy of all the es-timators increases. More data sources will result in more

data. Therefore, the accuracy of the learning algorithm willbe improved.

Figure 3(b) shows the error in estimating source reliability.Both the EM-CAT and EM-C are better in estimating sourcereliability than the other two algorithms. The reason is thatthe other algorithms ignore event categories. Thus, the in-formation regarding differences in source reliability acrossdifferent categories of observations is not exploited.

Figure 4 shows the performance of the estimators as afunction of source reliability. From the figure, we observethat with more reliable sources, the accuracy of the esti-mators is greatly improved. Even the voting can result invery reliable estimates when source reliability is 0.9 or above(i.e., 90% of their reports are true). Among all the estima-tors, our new EM-CAT is the best at both estimating theground truth values of reported variables and the reliabilityof sources.

Figure 5 explores the effect of “talkativeness,” si, of thesources on estimation accuracy. As mentioned earlier, thetalkativeness of a source denotes the probability that thesource will make a claim regarding some variable. In theexperiment, talkativeness is varied from 0.4 to 0.8. Withhigher talkativeness, we have more data. This is the reasonwhy accuracy of the estimators improves as si increases.Again, our EM-CAT algorithm is the best among all theestimators.

In Figure 6, we study the performance of the estimatorswhen the number of edges in the dependency structure (aDAG) varies. A larger number of edges in the DAG meansmore dependencies among variables. Figure 6 shows thatperformance of the state estimators does not change muchwith the number of dependencies. The reason could be thatthe parameters of the Bayesian network are generated uni-formly at random in (0, 1). Therefore, in expectation, thebias of the ground truth is around 0.5. However, if the biasof the ground truth is skewed, we will observe a differenceamong different dependency structures, since the value ofa variable will be affected more by its depended variables.Our algorithm is the best among all the algorithms sincewe exploit the dependency structure of the variables. Al-though the accuracy of estimators does not vary much asthe structure changes, exploiting this information leads to amore accurate state estimator.

We study the performance of the estimators as the numberof category labels varies from 2 to 5 in Figure 7. From Fig-ure 7(a), we observe that as the number of labels increases,the performance of EM-CAT becomes worse. This is be-cause we end up with fewer and fewer data in each category.With fewer data, the parameters of the estimator cannotbe learned accurately. Therefore the performance of estima-tion degrades. This figure suggests that when the data sizeis small, it is better to ignore category, but with a large datasize, it would be better to exploit it.

Figure 8, we study the performance of the estimators asthe number of variables varies from 80 to 110. From thefigure, we observe that the accuracy of the estimators im-proves as the number of variables increases. The reason isthat we have more data to learn the estimation parametersmore accurately. Actually, the voting algorithm does notvary much, since in voting each source has the same weightas the others. Therefore, even when the number of variablesincreases, the weight of the sources does not change, leav-ing performance the same. Again, our algorithm EM-CAT

208

(a) Variable State Estimation

(b) Source Reliability Estimation

Figure 3: Performance as the num-ber of sources varies.



Figure 4: Performance as the sourcereliability varies.



Figure 5: Performance as the sourcetalkativeness varies.

is the best among all the algorithms in both the estimationof ground truth values of variables and estimation of sourcereliability.

Next, we study the scalability of our EM-CAT algorithm.We mainly compare our algorithm with the algorithm pro-posed in [20]. Their algorithm considers the full joint distri-bution of all the correlated variables.

We compare three inference algorithms: 1. the junctiontree algorithm (JTree), 2. the variable eliminationalgorithm (VarElim), and 3. the method used in [20],i.e., inference with the full joint distribution (Total).Please note that all the three algorithms compute the exactinference probility of the Bayesian network, there are alsoalgorithms that compute the approximate inference proba-bility [10] which compromises the inference accuracy but hasa better computational complexity.

In Figure 9, we fixed the expected node degrees to be 2,and varied the number of nodes in the Bayesian networkfrom 10 to 25. Note that, the y-axis is in log-scale. FromFigure 9, we clearly observe that the computation time ofthe Total algorithm increases exponentially as the numberof nodes increase linearly. The computation time of boththe JTree algorithm and the VarElim algorithm grows in amuch less rapid way. The time complexity of both the al-gorithms actually depends on the size of the largest clique(the complete sub-graph in the Bayesian network); since theexpected node degree is 2, it is possible that the resultinggraph has a clique of size n, where n is less than the totalnumber of nodes N in the Bayesian network. As the totalnumber of nodes N increases, the gap between n and Nwill also increase as shown in the Figure 9. The JTree algo-rithm is more efficient than the VarElim algorithm, since itmaintains a data structure which can simultaneously update

the potential of local cliques but the the VarElim algorithmeliminates the variables sequentially. Furthermore, the timecomplexity of the VarElim algorithm also depends on theorder of the variables to eliminate. It is NP-hard to find theoptimal order based on which the VarElim algorithm elim-inates the variables. When the total number of nodes N is25, we can observe that the Total algorithm needs 20 secondsin average, while VarElim needs 2 seconds and JTree needsonly 0.2 seconds in average. We can use JTree algorithm inour EM-VTC algorithm for the Bayesian inference compu-tation, compared with the previous solution that does notexploit the dependence structure of the variables [20] (i.e.using the Total algorithm), it is not hard to observe howscalable our algorithm is as the number of total variablesvaries.

Figure 10 shows the CPU time of the three inference al-gorithm when the number of nodes is fixed at 24 but theexpected degree of each node varies from 1 to 3. From thefigure, we can observe that the CPU time of the VarElimalgorithm increases as the number of node degree increases.This is because that with a larger node degree, the chancethat the VarElim algorithm selects a bad variable eliminat-ing order become larger. Thus, the time complexity of VarE-lim increases. However, the time complexity of the JTree al-gorithm only depends on the size of each local clique, whenthe node degree is small such as less than 3 the complexityof JTree actually changes very little which does not show inFigure 10. Figure 10 shows the scalability of our algorithmcompared with previous solution [20] as the expected nodedegree varies in the Bayesian network.

Next, we evaluate our algorithm using data from a realdisaster scenario. In addition to estimation accuracy, weevaluate its efficacy at hypothesis testing.

209



Figure 6: Performance as the num-ber of edges in the Bayesian networkvaries.



Figure 7: Performance as the num-ber of event categories varies.



Figure 8: Performance as the num-ber event variables varies.

Figure 9: Computation time comparison with fixednode degree.

5.2 Performance with Real DataHere, we test the estimation accuracy of our algorithm

with a real-world data set about the availability of groceries,pharmacies, and gas stations during Hurricane Sandy inNovember 20122. Sandy was reported as the second-costliesthurricane in the history of the United States (surpassed onlyby hurricane Katrina). It caused widespread shortage ofgas, food, and medical supplies, as gas stations, grocery re-tail shops and pharmacies were forced to close. Some wereclosed for as long as a month. The data set was documented

2The data set is available at:http://www.ahcusa.org/hurricane-Sandy-assistance.htm

Figure 10: Computation time comparison with fixednumber of nodes.

by the All Harzard Consortium (AHC)[2], a state-sanctionednon-profit organization fucused on homeland security, emer-gency management, and business continuity issues in themid-Atlantic and northeast regions of the US. The data cov-ered states including WV, VA, PA, NY, NJ, MD, and DC,and the information was updated daily. Figure 11 shows aportion of the location distribution of the gas stations in thedata set.

5.2.1 MethodologyIn this evaluation, the reported variables denote whether

a given gas station has gas, whether a given grocery storeis open, and whether a given pharmacy is open on a given

210

Figure 11: The location distribution of the gas sta-tion data.

day following the hurricane. Hence, the total number ofvariables is equal to the total number of gas stations, gro-cery shops and pharmacies, combined, multiplied by the pe-riod of observation in days. All variables are binary. Theground truth for these variables is given in the AHC dataset. The experiment has two goals. The first goal is to testthe accuracy of our algorithm at reconstructing the groundtruth values of these variables from observations reported byunreliable sources. The second goal is to determine whichof several hypothesized dependency structures among thevariables is borne out by data reported by these unreliablesources (which is an instance of hypothesis testing). In par-ticular, it is interesting to see whether a hypothesis thatcorresponds to the ground-truth dependency structure willactually be picked out despite observation noise. Since wehad ground truth data only, we added noise artificially bysimulating 100 unreliable sources. The average reliabilityof a source was set to 0.7. The expected talkativeness of asource was set 0.8. Obsevations reported by these simulatedsources were then used.

The hypothesis testing experiment exemplifies the typeof analysis a decision-maker might perform to understandthe cause of failures in a large system. For example, whenpower is lost or restored, when flooding occurs or is drained,and when available resources are depleted in a neighborhooddue to local demand, correlated changes in the state of ouraforementioned variables occur. The dependency structureamong these variables depends on the cause of outage. Thisobservation allows us to compare hypotheses regarding thecause of failure. Each hypothesis would correspond to a dif-ferent dependency graph between failures. A dependencygraph that results in a higher value for the likelihood func-tion when the EM algorithm converges would imply thatthe corresponding hypothesis is better supported by data.Hence, by comparing the converged values of the likelihoodfunctions when running EM with different hypothesized de-pendency graphs, we could determine which hypothesis ismore likely to be the case. For illustration, we propose threehypotheses regarding the dependency structures among theobserved variables in the hurricane Sandy scenario:

1. Independent hypothesis: This hypothesis trivially statesthat the variables are independent.

2. Supply line hypothesis: This hypothesis states that allvariables located in the same state are connected by adirected path; the supply line.3

3. Exact hypothesis: Prior work on the same data set [5]did identify the exact dependency between our vari-ables by observing which variables tend to change to-gether. This real dependency structure was computedbased on ground truth data. We include it here to testour EM algorithm. The algorithm, if correct and ro-bust to the noise introduced by less reliable sources,should generate a higher final value for the likelihoodfunction when this hypothesis is used for the depen-dency structure.

These three hypotheses are named Indep, SupLine, andCoJump, respectively. We test which one is the most prob-able. Our hypotheses lead to different DAGs from which webuild the corresponding Bayesian networks. In our evalua-tion, we randomly select four days in November, 2012, andtest our hypotheses based on data from these four days.

In addition to choosing a winning hypothesis, we studythe performance of our EM-CAT algorithm in terms of esti-mation errors in the state of gas stations, pharmacies, andgroceries, and error in estimating the reliability of the simu-lated sources. The evaluation is averaged over 20 executionsto smooth out the noise.

5.2.2 Evaluation ResultsFigure 12 shows the evaluation results on the pharmacy

data. In this figure, we observe that the Indep hypothesisand the SupLine hypothesis lead to an inferior estimationaccuracy compared to the CoJump hypothesis. This is con-sistent with the fact that the CoJump hypothesis did coin-cide with the real dependency structure seen in the data.

The estimation errors of the three hypotheses with ourEM-CAT algorithm for the data of grocery retail stores areshown in Figure 13. We can observe that the CoJump hy-pothesis is the best of the three. SupLine works better thanIndep on the data generally.

The evaluation results for the gas station data is shownin Figure 14. In general, the CoJump hypothesis is betterthan the other two.

The above results show that the “right” hypothesis re-garding the dependency structure among variables does infact result in better estimation accuracy, but how would auser know which of multiple hypotheses is the right hypothe-sis? The answer, as mentioned earlier, lies in computing theconverged value of the likelihood function (when EM termi-nates) when dependencies among variables are given by eachof the compared hypotheses. The hypothesis correspond-ing to the highest value of likelihood is the best hypothesis.These values are summarized in Table 2 for the hypothesesdescribed above. From the table, we observe that the Co-Jump hypothesis has the highest likelihood according to thedata set. This is what we hoped to see. It shows that indeed

3Ideally, we should have considered the real topology of sup-ply lines or supply routes connecting the retailers under con-sideration. However, we did not have access to this informa-tion, so we just assumed, for the sake of an example, that weknew the supply line topology and that it was as we defined.

211

Figure 12: Estimation error withthe pharmacy data.

Figure 13: Estimation error withthe grocery data.

Figure 14: Estimation error withthe gas station data.

comparing the converged values of the likelihood functioncomputed by the EM algorithm identifies the right hypoth-esis regarding dependencies among variables, even in thepresence of noise from unreliable observations. The evalua-tion illustrates use of the EM-CAT algorithm for hypothesistesting.

Table 2: Likelihood of the hypotheses for each dataset.

Pharmacy Grocery Gas StationCoJump 0.83 0.74 0.79SupLine 0.66 0.64 0.74Indep 0.70 0.58 0.73

6. RELATED WORKInferring the structure of Bayesian Network is, in general,

an NP-complete problem [3]. In order to learn a BayesianNetwork in a tractable way, various algorithms are proposed.There are mainly two categories of approaches, score-basedand constraint-based [19]. The former one tries to searchfor the optimum structure based on goodness-of-fit. Thelatter one utilizes conditional independence to build the net-work. Depending on the different data types and relation-ships, various hypothesis tests are available. For continuousdata, if the relationship among variables is believed to belinear, tests based on Pearson’s correlation are widely used.Asymptotic χ2 tests can also be used to test independencebetween two continuous variables [6]. In cases of categori-cal variables, one of the most classical tests is Pearson’s χ2

test [1]. It works on the contingency table and tests if pairedobservations from two categorical variables are independent.In addition, likelihood-ratio statistic (or G2) can also beused on either categorical or continuous variables; Jonck-heere’s trend test provides an independence test on ordinalvariables [7]. Although in some cases separate test statis-tics can be used by different tests, they usually provide thesame conclusions. If two variables are conditionally depen-dent, an edge between these two variables should be drawnin the Bayesian Network. Different from traditional work,where the inference is employed on the data provided by asingle source, our proposed mechanism is able to conducthypothesis testing upon crowdsourced data by accountingfor a variety of sources of different and unknown reliability.

The problem studied in this paper bears some resemblanceto the fact-finding problem that has been studied extensively

in recent years. The goal of fact-finding, generally speaking,is to ascertain correctness of data from sources of unknownreliability . As one of the earliest efforts in this domain, Hubsand Authorities [9] presented a basic fact-finder, where thebelief in a claim and the truthfulness of a source are com-puted in a simple iterative fashion. Later on, Yin et al.introduced TruthFinder as an unsupervised fact-finder fortrust analysis on a providers-facts network [24]. Pasternacket al. extended the fact-finder framework by incorporat-ing prior knowledge into the analysis and proposed severalextended algorithms: Average.Log, Investment, and PooledInvestment [14]. Su et al. proposed semi-supervised learningframeworks to improve the quality of aggregated decisionsin distributed sensing systems [16, 17]. Towards a joint es-timation on source reliability and claim correctness, Wanget al. [22, 21] and Li et al. [11, 18] proposed expectationmaximization and coordinate descent methods to deal withdeterministic and probabilistic claims, respectively. Thoughyielding good performance in many cases, none of these ap-proaches considers situations where the variables in questionhave wide-spread dependencies. To address this problem,Wang et al. [20, 23] further extended their framework tohandle limited dependencies. However, their algorithm hasexponential computational complexity in the number of cor-related variables, and thus can only be applied in scenarioswhere the number of dependencies is small. In contrast totheir work, we consider a model for more general scenarioswhere a considerable number of dependencies exists amongthe variables reported by the unreliable sources.

7. CONCLUSIONIn this paper, we addressed the reliable crowd-sensing

problem with interdependent variables. Crowd-sensing is anovel sensing paradigm in which human sources are treatedas sensors. The challenge is that the reliability of sources isunknown in advance. Recently, several efforts tried to ad-dress this reliability challenge by formulating the problemgiven different source and event models. However, they didnot address the problem when the reported variables areinterdependent at large scale. In this paper, dependenciesbetween reported variables were formulated as a Bayesiannetwork. We demonstrated that our formulation is moregeneral than previous work; previous models being specialcases of ours. Evaluation results showed that our EM-CATalgorithm outperforms the state-of-the-art solutions. We

212

also showed that the algorithm can be used for hypothesistesting on the dependency structure among variables.

AcknowledgmentsWe sincerely thank Jorge Ortiz for shepherding the finalversion of this paper, and the anonymous reviewers for theirinvaluable comments. Research reported in this paper wassponsored by the Army Research Laboratory and was ac-complished under Cooperative Agreement W911NF-09-2-0053,DTRA grant HDTRA1-10-10120, and NSF grants CNS 09-05014 and CNS 10-35736. The views and conclusions con-tained in this document are those of the authors and shouldnot be interpreted as representing the official policies, eitherexpressed or implied, of the Army Research Laboratory orthe U.S. Government. The U.S. Government is authorized toreproduce and distribute reprints for Government purposesnotwithstanding any copyright notation here on.

8. REFERENCES[1] A. Agresti. An introduction to categorical data

analysis, volume 135. Wiley New York, 1996.

[2] All hazards consortium. http://www.ahcusa.org/.

[3] D. M. Chickering. Learning bayesian networks isnp-complete. In Learning from data, pages 121–130.Springer, 1996.

[4] A. P. Dempster, N. M. Laird, D. B. Rubin, et al.Maximum likelihood from incomplete data via the emalgorithm. Journal of the Royal statistical Society,39(1):1–38, 1977.

[5] S. Gu, C. Pan, H. Liu, S. Li, S. Hu, L. Su, S. Wang,D. Wang, T. Amin, R. Govindan, G. Aggarwal,R. Ganti, M. Srivatsa, A. Barnoy, P. Terlecky, andT. Abdelzaher. Data extrapolation in social sensingfor disaster response. In Proceedings of the 10th IEEEInternational Conference on Distributed Computing inSensor Systems. IEEE Press, 2014.

[6] R. I. Jennrich. An asymptotic χ2 test for the equalityof two correlation matrices. Journal of the AmericanStatistical Association, 65(330):904–912, 1970.

[7] A. R. Jonckheere. A distribution-free k-sample testagainst ordered alternatives. Biometrika, pages133–145, 1954.

[8] Kevin Murphy. Bayes Net Toolbox for Matlab.https://code.google.com/p/bnt/.

[9] J. M. Kleinberg. Authoritative sources in ahyperlinked environment. Journal of the ACM(JACM), 46(5):604–632, 1999.

[10] D. Koller and N. Friedman. Probabilistic graphicalmodels: principles and techniques. MIT press, 2009.

[11] Q. Li, Y. Li, J. Gao, B. Zhao, W. Fan, and J. Han.Resolving conflicts in heterogeneous data by truth

discovery and source reliability estimation. InProceedings of the 2014 ACM SIGMOD internationalconference on Management of data, pages 1187–1198.ACM, 2014.

[12] Mark Paskin. A short course on graphical models.http://ai.stanford.edu/ paskin/gm-short-course/.

[13] T. D. Nielsen and F. V. Jensen. Bayesian networksand decision graphs. Springer, 2009.

[14] J. Pasternack and D. Roth. Knowing what to believe(when you already know something). In COLING,2010.

[15] C. S. Raghavendra, K. M. Sivalingam, and T. Znati.Wireless sensor networks. Springer, 2004.

[16] L. Su, J. Gao, Y. Yang, T. F. Abdelzaher, B. Ding,and J. Han. Hierarchical aggregate classification withlimited supervision for data reduction in wirelesssensor networks. In Proceedings of the 9th ACMConference on Embedded Networked Sensor Systems,pages 40–53. ACM, 2011.

[17] L. Su, S. Hu, S. Li, F. Liang, J. Gao, T. F.Abdelzaher, and J. Han. Quality of information baseddata selection and transmission in wireless sensornetworks. In RTSS, pages 327–338, 2012.

[18] L. Su, Q. Li, S. Hu, S. Wang, J. Gao, H. Liu,T. Abdelzaher, J. Han, X. Liu, Y. Gao, andL. Kaplan. Generalized decision aggregation indistributed sensing systems. In Real-Time SystemsSymposium (RTSS), 2014.

[19] I. Tsamardinos, L. E. Brown, and C. F. Aliferis. Themax-min hill-climbing bayesian network structurelearning algorithm. Machine learning, 65(1):31–78,2006.

[20] D. Wang, T. Abdelzaher, L. Kaplan, R. Ganti, S. Hu,and H. Liu. Exploitation of physical constraints forreliable social sensing. In RTSS, 2013.

[21] D. Wang, T. Amin, S. Li, T. A. L. Kaplan, S. G. C.Pan, H. Liu, C. Aggrawal, R. Ganti, X. Wang,P. Mohapatra, B. Szymanski, and H. Le. Humans assensors: An estimation theoretic perspective. In IPSN,2014.

[22] D. Wang, L. Kaplan, H. Le, and T. Abdelzaher. Ontruth discovery in social sensing: a maximumlikelihood estimation approach. In IPSN, 2012.

[23] S. Wang, D. Wang, L. Su, L. Kaplan, andT. Abdelzaher. Towards cyber-physical systems insocial spaces: The data reliability challenge. InReal-Time Systems Symposium (RTSS), 2014.

[24] X. Yin, J. Han, and P. S. Yu. Truth discovery withmultiple conflicting information providers on the web.Knowledge and Data Engineering, IEEE Transactionson, 20(6):796–808, 2008.

213

scalable social sensing of interdependent phenomena

Documents