Machine Vision and ApplicationsDOI 10.1007/s00138-013-0535-8

SPECIAL ISSUE PAPER

Context-based person identification framework for smart videosurveillance

Liyan Zhang · Dmitri V. Kalashnikov ·Sharad Mehrotra · Ronen Vaisenberg

Received: 1 February 2013 / Revised: 4 June 2013 / Accepted: 11 July 2013© Springer-Verlag Berlin Heidelberg 2013

Abstract Smart video surveillance (SVS) applicationsenhance situational awareness by allowing domain analyststo focus on the events of higher priority. SVS approachesoperate by trying to extract and interpret higher “seman-tic” level events that occur in video. One of the key chal-lenges of SVS is that of person identification where the taskis for each subject that occurs in a video shot to identifythe person it corresponds to. The problem of person iden-tification is especially challenging in resource-constrainedenvironments where transmission delay, bandwidth restric-tion, and packet loss may prevent the capture of high-qualitydata. Conventional person identification approaches whichprimarily are based on analyzing facial features are oftennot sufficient to deal with poor-quality data. To address thischallenge, we propose a framework that leverages heteroge-neous contextual information together with facial features tohandle the problem of person identification for low-qualitydata. We first investigate the appropriate methods to utilizeheterogeneous context features including clothing, activity,human attributes, gait, people co-occurrence, and so on. Wethen propose a unified approach for person identificationthat builds on top of our generic entity resolution frame-work called RelDC, which can integrate all these contextfeatures to improve the quality of person identification. Thiswork thus links one well-known problem of person identi-fication from the computer vision research area (that deals

This work was supported in part by NSF grants CNS-1118114,CNS-1059436, CNS-1063596. It is part of NSF supported projectSherlock @ UCI (http://sherlock.ics.uci.edu): a UC Irvine project onData Quality and Entity Resolution [1].

L. Zhang (B) · D. V. Kalashnikov · S. Mehrotra · R. VaisenbergDepartment of Computer Science, University of California,Irvine, CA, USAe-mail: zhangliyan.uci@gmail.com

with video/images) with another well-recognized challengeknown as entity resolution from the database and AI/MLareas (that deals with textual data). We apply the proposedsolution to a real-world dataset consisting of several weeksof surveillance videos. The results demonstrate the effective-ness and efficiency of our approach even on low-quality videodata.

Keywords Person identification · Context information ·Entity resolution · Smart video surveillance

1 Introduction

Advances in sensing, networking, and computational tech-nologies have allowed the possibility of creating sentientpervasive spaces wherein sensors embedded in physical envi-ronments are used to monitor its evolving state to improvethe quality of our lives. There are numerous physical worlddomains in which sensors are used to enable new func-tionalities and/or bring new efficiencies including intelli-gent transportation systems, reconnaissance, surveillancesystems, smart buildings, smart grid, and so on.

In this paper, we focus on smart video surveillance (SVS)systems wherein video cameras are installed within buildingsto monitor human activities [17,18,33]. Surveillance systemcould support variety of tasks: from building security to newapplications such as locating/tracking people, inventory, ortasks like analysis of human activity in shared spaces (such asoffices) to bring improvements on how the building is used.One of the key challenges in building smart surveillance sys-tems is that of automatically extracting semantic informationfrom the video streams [31,32,35,36]. This semantic infor-mation may correspond to human activities, events of inter-est, and so on that can then be used to create a representation

123

L. Zhang et al.

Fig. 1 Example of surveillance video frames

of the state of the physical world, e.g., a building. This seman-tic representation, when stored inside a sufficiently power-ful spatio-temporal database, can be used to build variety ofmonitoring and/or analysis applications. Most of the currentwork in this direction focuses on computer vision techniques.Automatic detection of events from surveillance videos is adifficult challenge and the performance of current techniquesoften leaves a room for improvement. While event detectionconsists of multiple challenges, (e.g., activity detection, loca-tion determination, and so on), in this paper we focus on aparticularly challenging task of person identification [3,4].

The challenge of person identification (PI) consists ofassociating each subject that occurs in the video with a real-world person it corresponds to. In the domain of computervision, the most direct way to identify a person is to performface detection followed by face recognition, the accuracy ofwhich is limited even when video data are of high quality,due to the large variation of illumination, pose, expression,and occlusion, etc. Thus, in the resource-constrained envi-ronments, where transmission delay, bandwidth restriction,and packet loss may prevent the capture of high quality data,face detection and recognition becomes more complex. Wehave experimented with Picasa’s face detector on our videodataset, 1 and found that it can detect faces in only 7 % of thecases and then among them it can recognize only 4 % of faces.

Figure 1 illustrates the example of frames in our videodataset, where only one face is successfully detected (solid-line rectangle) utilizing the current face detection techniques.Several reasons account for the low detection rate: (1) facescannot be captured if people walk with their back to thecameras; (2) faces are too small to be detected when peo-ple are far away from cameras; (3) the large variation ofpeople’s pose and expression brings more challenges to face

1 704 × 480 resolution per frame.

detection. Thus the traditional face detection and recogni-tion techniques are not sufficient to handle the poor-qualitysurveillance video data.

To deal with the poor-quality video data and overcome thelimitation of current face detection techniques, we shift ourresearch focus to context-based approaches. Contextual datasuch as time, space, clothing, people co-occurrence, gait, andactivities, are able to provide the additional cues for personidentification. Consider the frames illustrated in Fig. 1 forexample. Although only one face is detected and no recogni-tion results are provided, the identities of all the subjects canbe estimated by analyzing the contextual information. First,time and foreground color continuities split the eight framesinto two sequences or shots. The first four frames constructthe first shot, and the following four frames form the secondshot, where subjects within each shot describe the same enti-ties. Furthermore, some other contextual features reveal thehigh possibility that the two subjects in these shots are thesame entity. For instance, they both share the similar clothing(red T-shirt and gray pants), they perform similar activities(walking in front of the same camera, though in oppositedirections), and they have the same gaits (walking speed).Thus the context features help to reveal that the subjects inthe eight frames very likely refer to the same entity. To iden-tify this person, face recognition process is usually inevitable.However, the activity information can also provide extra cuesto recognize people’s identity. In the above example, supposethat the first shot in Fig. 1 is the first shot of that day where aperson enters the corner office which belongs to “Bob”. Thenmost probably this person is “Bob” because in most cases,the first person entering the office should have the key. There-fore, by analyzing contextual information even without facerecognition results, we can predict that the very likely iden-tity of subject in all the eight frames is “Bob”. The exampledemonstrates the essential role that contextual data play in

123

Context-based person identification framework

the person identification issue for the low-quality video data.Another significant advantage of context information is itsweaker sensitivity to video data quality as compared with thatof face recognition. That makes context-driven approachesmore robust and reliable when dealing with poor quality data.

In this paper, we extend our previous work [39] to explorea novel approach to leverage contextual information, includ-ing time, space, clothing, people co-occurrence, gait, andactivities to improve the performance of person identifica-tion. To exploit contextual information, we connect the prob-lem of person identification with a well-studied problem ofentity resolution [5,21,26], which typically deals with tex-tual data. Entity resolution is a very active research areawhere many powerful and generic approaches have been pro-posed, some of which could potentially be applied to theperson identification problem. In this paper, we first investi-gate methods for extracting and processing several differenttypes of context features. We then demonstrate how to applya relationship-based approach for entity resolution, calledRelDC [23], to the person identification problem. RelDCis an algorithmic framework for analyzing object featuresas well as inter-object relationships, to improve the qualityof entity resolution. In this paper we will demonstrate howRelDC framework for entity resolution could be leveraged tosolve a person identification problem that arises when ana-lyzing video streams produced by cameras installed in the CSDepartment at UC Irvine. Our empirical evaluation demon-strates the advantage of the context-based solution over thetraditional techniques, as well as its effectiveness and robust-ness. The proposed approach shows clear improvements overapproaches that only exploit facial features. The improve-ment is even more pronounced for low-quality data, as itrelies on contextual features that are less sensitive to deteri-oration of data quality.

The rest of this paper is organized as follows: We start byintroducing the related work in Sect. 2. Then in Sect. 3, wepresent the proposed approach for context based person iden-tification. Section 4 demonstrates experiments and results.Finally, we conclude in Sect. 5 by highlighting key points ofour work.

2 Related work

2.1 Video-based person identification

The conventional approaches for person identification areto first use face detection followed by face recognition.Figure 2 illustrates the basic schema for person identification.Given a face frame, after locating faces via a face detector,the extracted faces are passed to a matcher which leveragesthe face recognition techniques to measure the similaritiesbetween the extracted faces and “gallery faces” (where true

Fig. 2 Example of basic person identification process

identities of people are known) to determine the identities ofthe extracted faces.

In general, face detection is the first and essential com-ponent used in person identification. However, in our testdataset, only a small proportion of faces (7 %) could bedetected using the current face detection techniques, and outof them very few (4 %) could be recognized, due to the poorquality of video data in our surveillance setting. The failureof detection for most faces makes it impossible to apply thesubsequent face recognition process in 93 % of the cases.Hence, the task of achieving high-quality person identifica-tion becomes a challenge for video of poor quality.

Face recognition is another active topic of research thathas attracted significant attention in the past two decades.Most of the research efforts have focused on techniquesfor still images, especially face representation methods.Recently, descriptor-based face representation approacheshave been proposed and proven to be effective. Theyinclude Local Binary Pattern (LBP) [2] describing the micro-structure of faces, SIFT and Histogram of Oriented Gradients(HOG) [10], and so on. These face recognition techniquesare able to achieve good performance in controlled situa-tions, but tend to suffer when dealing with uncontrolled con-ditions where faces are captured with a large variation in pose,illumination, expression, scale, motion blur, occlusion, etc.These nuisance factors may cause the differences in appear-ance between distinct shots of the same person to be greaterthan those between two people viewed under similar condi-tions. Thus leveraging context features could bring signifi-cant improvement on top of techniques that rely on low-level

123

L. Zhang et al.

visual features only, especially in the context of surveillancevideos.

Compared with still images, videos often have more use-ful features and additional context information that can aidin face recognition. For example, a video sequence wouldoften contain several images of the same entity, which poten-tially shows the entity’s appearance under different condi-tions. Surveillance videos usually have temporal and spatialinformation available, which still images do not always have.In addition, video frames are capable of storing the objectsin different angles, which contain 3-D geometric informa-tion. To better leverage these properties, some face recogni-tion algorithms have been proposed to operate on video data.They include using temporal voting to improve the identi-fication rates, extracting 2-D or 3-D face structures fromvideo sequences [12–14]. However, these methods do notfully exploit the context information and very few of themaddress the problem of integration of heterogeneous contextfeatures.

In this paper, we propose to leverage heterogeneous con-textual information to improve the performance of video-based face recognition. To integrate the heterogeneous con-textual features together, we connect the problem of personidentification with the well-studied entity resolution prob-lem and apply our entity resolution RelDC framework toconstruct a relationship graph to resolve the correspondingperson identification problem.

2.2 Entity resolution

High quality of data is a fundamental requirement for effec-tive data analysis which is used by many scientific anddecision-support applications to learn about the real-worldand its phenomena [15,16,24]. However, many InformationQuality (IQ) problems such as errors, duplicates, incomplete-ness, etc., exist in most real-world datasets. Among theseIQ problems, Entity Resolution (also known as deduplica-tion or record linkage) is among the most challenging andwell-studied problem. It arises especially when dealing withraw textual data, or integrating multiple data sources to cre-ate a single unified database. The essence of ER problem isthat the same real-world entities are usually referred to indifferent ways in multiple data sources, leading to ambigu-ity. For instances, the real-world person name ‘John Smith’might be represented as ‘J. Smith’, or misspelled as ‘JohnSmitx’. Besides, two distinct individuals may be referred asthe same representation, e.g., both ‘John Smith’ and ‘JaneSmith’ referred to as ‘J. Smith’. Therefore, the goal of ER isto resolve these entities by identifying the records represent-ing the same entity.

There are two main instances for ER problem: Lookup[5,21] and Grouping [5,26]. Lookup is a classificationproblem, with the goal of identifying the object that each

reference refers to. Grouping is a clustering problem, whosegoal is to correctly group the representations that refer to thesame object. We primarily will be interested in an instanceof the lookup problem. Our research group at the Universityof California, Irvine, has also contributed significantly to thearea of ER in the context of Project Sherlock@UCI, e.g.,[8,19,20,28–30,38]. The most related work of our group issummarized next.

2.2.1 Relationship-based data cleaning (RelDC)

To address the entity resolution problem, we have developeda powerful disambiguation engine called the Relationship-based Data Cleaning (RelDC) [6,7,21–23,27]. RelDC isbased on the observation that many real-world datasets arerelational2 in nature, as they contain information not onlyabout entities and their attributes, but also relationshipsamong them as well as attributes associated with the rela-tionships. RelDC provides a principled domain-independentmethodology to exploit these relationships for disambigua-tion, significantly improving data quality.

Relationship-based data cleaning (RelDC) works by rep-resenting and analyzing each dataset in the form of entity-relationship graph. In this graph, entities are represented asnodes and edges correspond to relationships among enti-ties. The graph is augmented further to represent ambiguityin data. The augmented graph is then analyzed to discoverinterconnections, including indirect and long connections,between entities which are then used to make disambiguationdecisions to distinguish between same/similar representa-tions of different entities as well as to learn different rep-resentations of the same entity. RelDC is based on a simpleprinciple that entities tend to cluster and form multiple rela-tionships among themselves.

After the construction of entity-relationship graphs, thealgorithm computes the connection strengths between eachuncertain reference and each of the reference’s potential“options” that entities it could refer to. For instance, reference‘J. Smith’ might have two options: ‘John Smith’ and ‘JaneSmith’. The reference will be resolved to the option that hasthe strongest combination of the connection strength and thetraditional feature-based similarity. Logically, the computa-tion of the connection strength can be divided into two parts:first finding the connections which correspond to paths inthe graph and then measuring the strength in the discoveredconnections. In general, many connections between a pairof nodes may exist. For efficiency, only the important pathsare considered, e.g., L-short simple paths. The strength ofthe discovered connections is measured by employing one ofthe connection strength models [21]. For instance, one model

2 We use the standard definition of relational datasets as used in thedatabase literature.

123

Context-based person identification framework

computes the connection strength of a path as the probabilityof following the path in the graph via a random walk.

After the connection strength is computed, this problemis transformed into an optimization problem of determiningthe weights between each reference and each of reference’soption nodes. Once the weights are computed by solvingthe optimization problem, RelDC resolves the ambiguousreference to the option with the largest weight. Finally, theoutcome of the disambiguation is used to create a regular(cleaned) database.

3 Context-based framework for person identification

3.1 Problem definition

Let D be the surveillance video dataset. The dataset containsK video frames F = { f1, f2, . . . , fK } wherein motion hasbeen detected. Let ti denote the time stamp of each frame fi .When a frame fi contains just one subject, we will refer tothe subject as xi , or as x fi . Let P = {p1, p2, . . . , p|P|} be theset of (known) people of interest that appear in our dataset.Then the goal of person identification is for each subject xi tocompute wi j which denotes the probability that xi is personp j , and correctly identify the person pk ∈ P that subject xi

corresponds to. If the subject is not in P , then the algorithmshould output xi = other. Table 1 summarizes some of thenotations throughout this paper.

Figure 3 illustrates an example of the person identificationproblem, where the goal is to determine whom the subject ineach video frame refers to: “Bob” or “Alice”. We can observethat the entity resolution problem has a very similar goal, thatis, to associate each uncertain reference to an object in thedatabase with the real-world object. Hence in this paper we

demonstrate how to apply one entity-resolution frameworkcalled RelDC to the problem of person identification. Theframework will exploit the relationships between contextualfeatures of subjects in the video surveillance to improve thequality of the person identification task.

3.2 General framework

Figure 4 illustrates the general framework of context-basedperson identification for surveillance videos. Given the storedvideos from surveillance cameras, the framework first seg-ments the frames with motion into shots based on temporalinformation. To facilitate person identification based on per-son faces the framework performs several preliminary stepssuch as face detection, extraction, facial representation. andrecognition. It then extracts the contextual features includ-ing people’s clothing, attributes, gait, activities, etc. Afterthe extraction of face and contextual features, the frameworkconstructs the entity-relationship graph and then applies theentity resolution algorithm RelDC on the graph to performthe corresponding person identification task.

In the following we discuss how to extract contextualfeatures from surveillance videos and then leverage RelDCframework to integrate these features together to resolve theperson identification problem.

3.3 Contextual feature extraction

Contextual features can provide additional cues to facilitatevideo-based person identification, especially for poor-qualityvideo data. In the following, we describe how to extractand leverage contextual features, such as people’s clothing,attribute, gait, activities, co-occurrence, and so on, to improvethe performance of person identification.

Table 1 Notation anddescription Notation Meaning

D The surveillance video dataset being processed

F = { f1, f2, . . . , fK } The set of detected motioned video frames

S = {s1, s2, . . . , s|S|} The set of shots after video segmentation

X = {x1, x2, . . . , x|X |} The set of subjects appearing in the video

P = {p1, p2, . . . , p|P|} The set of real-world people of interest

wi j The probability that subject xi is person p j

SC (xi , x j ) The cloth similarity between subjects xi and x j

Sacti j (actm

i , actnj ) The similarity between activity actm

i and actnj

P(xi = p j |acti , tk) The probability that subject xi is person p j based on activity and time

Ai The attribute vector for subject xi

F R(xi , p j ) The probability that subject xi is person p j based on facial features

G = (V, E) The entity relationship graph

cs(xi , p j ) The connection strength measure between subject xi and person p j

123

L. Zhang et al.

Fig. 3 Example of personidentification for surveillancevideos

Fig. 4 General framework forcontext-based approach

3.3.1 Temporal segmentation

We first describe temporal segmentation which is an essentialpart in video processing. We segment videos into shots. Intu-itively, subjects appearing in consecutive frames are likelyto be the same person. Hence, we initially group frames intoshots just based on the time continuity. But time continu-ity alone cannot guarantee person continuity. If the subjects’color histograms of two consecutive frames are significantlydifferent indicating potentially different people, the shot issplit further at such break points.

Suppose that we obtain a set of shots S = {s1, s2, . . . , s|S|}after the video segmentation. Most of the time the framesthat belong to the same shot describe the same entities. Thusthe person identification task reduces from identifying thesubjects in an image to identifying the subjects in a shot.We next describe how to extract contextual features for ashot.

3.3.2 Clothing

People’s clothing can be a good discriminative feature for dis-tinguishing among people [11,37]. Although people changetheir clothes across different days, they do not change it toooften within shorter period of time, and hence the same cloth-ing in such cases is often strong evidence that two imagescontain the same person. To accurately capture the clothinginformation of an individual in an image, we separate theperson from the background by applying a background sub-traction algorithm [4]. After color extraction processing, theforeground area is represented by a 64-dimensional vector,which consists of a 32-bin hue histogram, a 16-bin satura-tion histogram, and a 16-bin brightness histogram. Figure 5shows an example of the extracted foreground image andcorresponding color histogram.

The extracted clothing features can be used to computethe clothing-based similarity among subjects. For each pair of

123

Context-based person identification framework

Fig. 5 Example of foregroundextraction

subjects xi and x j , let Ci and C j be their clothing histogramsand ti and t j by the timestamps when xi and x j have beencaptured in video. We can choose an appropriate similaritymeasure to compute the similarities between them, such asthe cosine similarity. For instance, if we assume that peoplekeep the same clothing during the same day, we can define

SC (xi , x j ) ={

Ci ·C j|Ci ||C j | if day(ti ) = day(t j )

0 otherwise

To compute the similarity of subjects from two shots, thealgorithm selects a subject from a certain frame in a shot torepresent the shot. Usually the algorithm chooses a frametowards the middle, which tends to capture the profile of theperson better.

3.3.3 Activity

Activities and events associated with subjects prove to bevery relevant to the problem of person identification [9,40].The trajectory and walking direction can serve as a cue indi-cating the identity of the individual. For example, the activityof entering an office can provide strong evidence about theidentity of the subject entering the office: it is likely to beeither (one of the) person(s) who works in this office, or theircollaborators and friends. Furthermore, considering the timeof the activity in addition to the activity itself can often pro-vide even better disambiguation power. For example, on anygiven weekday, the person who enters an office first on that

day is likely to be the owner of the office. In addition, byanalyzing past video data the behavior routines for differentpeople can be extracted, which later can provide clues aboutthe identify of subjects in video. For instance, if we discoverthat “Bob” is accustomed to entering the coffee room to drinkhis coffee at about 10 a.m. each weekday, then the subjectwho enters the coffee room at around 10 a.m. is possibly“Bob”. Therefore, subject activities can often provide addi-tional evidence to recognize people. We now discuss how toextract and analyze certain people’s activities.

Bounding Box and Centroid Extraction. To track the trajec-tory of a subject and obtain his activity information, we needto extract bounding box and centroid of the subject. To do thatwe consider three consecutive frames with the same object.We first compute the differences of the first two frames bysubtraction and then compute the differences of the last twoframes. By combining the two different parts, we get thelocation of objects. After obtaining the bounding box, wedetermine the centroid of subjects by averaging the points ofx-axes and y-axes.

Walking Direction. The most common activity in surveil-lance dataset is walking. The walking direction (towards oraway from the camera) is an important factor to predict thesubsequent behavior of a person. The walking direction canbe obtained automatically by analyzing the changes of thecentroid between two consecutive frames in a shot. For exam-ple, as illustrated in Fig. 6, by determining that the centroid

123

L. Zhang et al.

Fig. 6 Example of walking direction

Fig. 7 Example for location clustering

of the subject is moving from the bottom to the top in thecamera view, we can determine that this person is walkingaway from the camera.

Activity Detection. We focus on detecting simple regulartype of behavior of people, including entering and exitinga room, walking through the corridor, standing still, and soon. These types of behavior can be determined by analyz-ing the bounding box of a person. For instance, for walkingthe algorithm focuses on the first and last frame in a shot,which we are called entrance and exit frames. By analyzingthe bounding box (BB) of a subject in the entrance frame, wecould predict where the subject has come from. Similarly,the exit frame could tell us where this person is headed to.

If we consider all the BBs in entrance and exit frames,we can find several locations in the camera view, wherepeople are most likely to appear or disappear. These loca-tions, denoted as L = {l1, l2, . . . , l|L|}, can be automati-cally computed in an unsupervised way by clustering thecentroid of entrance/exit BBs. Based on this analysis, weautomatically obtain the entrance and exit point in an image.Figure 7 demonstrates an example of the clustering result ofthe entrance and exit locations.

After computing the set of entrance and exit locations L ={l1, l2, . . . , l|L|}, we compute the distance between them anddetermine the entrance and exit points in each shot. Suppose

that in a shot sm the subject xi walks from location l p to lq .Then we can denote the activity as actm

i : {l p → lq}.

Activity Similarity. For each shot the algorithm extractsthe activity information by performing the aforementionedprocess. We assume that two subjects with similar activitieshave a certain possibility to describe the same person. Thusbased on this assumption, we connect the potentially samesubjects through the similar activities. Suppose that for twosubject xi and x j from shot sm and shot sn , respectively, thealgorithm extract activity information actm

i : {la → lb} andactn

j : {lc → ld}. We can define the activity similarity asfollows:

Sacti j

(actm

i , actnj

)=

{1 if la = lc(ld) and lb = ld(lc)0.5 if la = lc(ld) or lb = ld(lc)

In this equation, activities with the exact oppositeentrance/exit points are defined to be equal, for example,the subject xi with activity acti : {la → lb} and the subjectx j with activity act j : {lb → la} are considered to sharethe same activity. Thus the activity similarities can be lever-aged to connect the subjects which share the same/similaractivities.

Person Estimation Based on Activity. The intuition is thatthe identity of a person can be estimated by analyzing hisactivities. In general, given labeled past data we can com-pute priors such as P(xi = pm |acti ), which correspond tothe probability that the observed subject xi is the real-worldperson pm , given that the subject participates in activity acti ,such as entering/exiting a certain location. Similarly, we cancompute P(xi = pm |acti , tk) which also considers time.

3.3.4 Person gait

Gait is also a good feature to identify a particular person,because different people’s gaits are often different. For exam-ple, somebody might walk very fast or slow, somebody mightwalk with swinging arms or head. Thus by analyzing the char-acteristics of people’s gaits, we might be able to better predictthe identity of one subject or the sameness of two subjects.

123

Context-based person identification framework

For example, if the walking speed of two subjects differssignificantly, then they might not refer to the same entity.

3.3.5 Face-derived human attributes

Face-derived human attributes that could be estimated byanalyzing people faces, such as gender, age, ethnicity, facialtraits, and so on, are important evidence to identify a per-son. By considering these attributes, many uncertainties anderrors for person identification can be avoided, such as con-fusing a “men” with a “women”, an “adult” with a “child”,and so on. To obtain attribute values from a given face, weuse the attribute system [25]. It contains 73 types of attributesclassifiers, such as “black hair”, “big nose”, or “wearing eye-glasses”. Thus for each subject xi , the algorithm computes73-D attribute vector, denoted as Ai . The attribute similar-ity of two subjects xi and x j can be measured as the cosinesimilarity between Ai and A j . In addition, if the extractedattribute for xi is significantly different from that of the real-world person pm , then xi is not likely to be pm .

However, the extraction of reliable attribute valuesdepends on the quality of video data. This limitation usuallyleads to the failure of attribute extraction on lower qualitydata.

3.3.6 People co-occurrence

To recognize the identity of a person, people that frequentlyco-occur/present with that person in the same frames canprovide vital evidence. For example, suppose that “Bob” and“Alice” are good friends and usually walk together, then theidentity of one person might imply that of the other. Thusgiven the labeled past video data, we can statistically ana-lyze the people co-occurrence information, and compute theprior probability of one person in the presence of the other.Furthermore, from the co-occurrence/presence of two peoplein one frame we can derive that the two subjects are differentpeople. This observation can help to differentiate subjects.

3.4 Face detection and recognition

Face detection and recognition is a direct way to identifya person. However, it does not perform well in our datasetdue to several reasons. First, the surveillance cameras usedare of low quality and also the resolution of each frame isnot very high: 704 × 480. Second, people may actually walkaway from cameras, in which case the cameras only cap-ture their backs and not faces. Because of that, the best facedetection algorithms we have tried could only detect facesin about 7 % of frames and recognize 1 or 2 faces for a fre-quently appearing person out of all of his/her images in thedataset. Although the result is not ideal, we could still lever-age it for further processing. We define a function F R(xi , p j )

which reflects the result obtained by the face recognition.If xi and p j are the same according to face recognition, weset F R(xi , p j ) = 1, and otherwise F R(xi , p j ) = 0.

3.5 Solving the person identification problem with RelDC

In the previous sections we have described how to extractcontextual features including the people’s clothing, face-derived attributes, gait, activities, co-occurrence, etc, andobtain the face recognition results. In this section we showhow to represent the person identification problem as anentity resolution problem to be solved by our graph-basedRelDC entity resolution framework.

Relationship-based data cleaning (RelDC) performs entityresolution by analyzing object features as well as inter-objectrelationships to improve the data quality. To analyze relation-ships, RelDC leverages the entity-relationship graph of thedataset. The proposed framework will utilize inherent andcontextual features, as well as the relationships, to improvethe quality of person identification.

Figure 8 shows an example of the person identificationprocess that employs both the inherent and contextual fea-tures. The simple person identification task in the exampleis to discover whether the subject in the given frames is“Bob” or someone else. The example shows that, by usingface recognition, only one face (marked in the red rectan-gle) can be detected and recognized to be “Bob”, whereasthe remaining subjects cannot be identified. On the otherhand, by leveraging the context information, the identityof all the subjects can be recognized. Context informationsuch as activity, clothing, gait, face-derived attributes can beextracted from the both the probe frames (the ones to be dis-ambiguated) and gallery frames (the references frames wherethe labels/indentities are known). First, based on the timecontinuity, the frames are segmented into two shots, wherein each shot the frames describe the same person. Thus, thefour subjects in Shot 1 all refer to “Bob”. For Shot 2, althoughno face-based features can is computed (since the person iswalking with his back towards the camera), the subjects inShot 2 can also be connected to “Bob” through contextualfeatures. One such connection is the similar contextual fea-tures between Shot 2 and Shot 1 that we now know refers to“Bob”. Another connection is the special activity of Shot 2which illustrates that the subject is the first person enter-ing “Bob” offices on that day. Therefore, by constructing anentity-relationship graph which considers both inherent andcontextual feature, the identity of subjects in all the probeframes can be resolved.

3.5.1 Entity-relationship graph

In order to apply RelDC, the algorithm first constructs anentity-relationship graph G = (V, E) to represent the given

123

L. Zhang et al.

Fig. 8 Example ofcontext-based personidentification

person identification task, where V is the set of nodes and E isthe set of edges. Each node corresponds to an entity and eachedge to a relationship. The graph will contain several differ-ent types of nodes: shot, subject, person, clothing, attribute,gait, and activity. The edges linking these nodes correspondto the relationships. For instance, the edge between a shotnode and a subject node corresponds to the “appears in” rela-tionship.

In graph G, edges have weights where a weight is a realnumber in [0,1] that reflects the degree of confidence in therelationship. For example, if there is an edge with weight 0.8between a subject node and a person node, this implies thealgorithm has 80 % confidence that this subject and personare the same. The edge weight between two color histogramnodes denotes their similarity.

Figure 9 illustrates an example of an entity-relationshipgraph. It shows a case where the set of people of interest con-sists of just two persons: Alice and Bob. It considers threeshots s1, s2, s3, where s1 captures two subjects x11 and x12,shot s2 captures x2, and s3 has x3. The graph only shows theclothing and activity contextual features; the other contex-tual features are not shown for clarity. The goal is to matchpeople with shots.

Subject x11, x12, x2, x3 in the graph are connected withtheir corresponding clothing color histograms C11, C12, C2,

C3. An edge between two color histogram nodes representsthe similarity between them. For instance, the similarity ofC2 and C3 is 0.8. In addition, subjects are connected to thecorresponding activities, which could be indicative of whothese subjects are. For example, if the past labeled data areavailable, from the fact that subject s3 is connected to activityact3, we can get the prior probability of 0.7 that s3 is Bob. Thegraph also shows that according to face recognition subjectx2 in shot s2 is Bob.

Fig. 9 Example of entity-relationship graph

The main goal is to analyze the relationships betweenthe subject nodes and person nodes and compute the weightwi j that each subject xi associating with person p j . Notice,weights wi j are the only variables in the graph, whereas allother edge-weights are fixed constants. After constructing thegraph, RelDC will compute the value of those wi j weightsbased on the notion connection strength discussed next. Aftercomputing the weights, RelDC will use them to resolve eachsubject to the person that has the highest weight.

3.5.2 Connection strength computation

The constructed entity-relationship graph G illustrates theconnections and linkages between subjects appearing in thevideo shots and real-world people. Intuitively, the more pathsexist between two entities, the stronger the two entitiesare related. Thus we introduce the definition of connection

123

Context-based person identification framework

strength cs(xl , p j ) between each subject node xl and personnode p j , to reflect how strongly subject xl and person p j

are related. The value of cs(xl , p j ) can be computed accord-ing to some connection strength model. The computationprocess logically consists of two parts: finding the connec-tions (paths) between the two nodes and then measuring thestrength in of the discovered connections.

Generally, many different paths can exist between twonodes and considering very long paths could be inefficient.Therefore, in our approach, only important connection pathsare taken into account, for instance, L-short simple paths(e.g., L ≤ 4). For example, in Fig. 9 one 4-short simple pathbetween subject x2 and person “Bob” is “x2-C2-C3-x3-Bob”.We will use PL(xl , p j ) to denote the set of all the L-shortsimple paths between subject node xl and person node p j .

To measure the strength of the discovered connections,some connection strength models [21] can be leveraged. Forinstance, we can compute the connection strength of a pathpa as the probability of following path pa in graph G viarandom walks. The connection strength cs(xl , p j ) can becomputed as the sum of the connection strengths of paths inPL(xl , p j ).

cs(xl , p j ) =∑

pa∈PL (xl ,p j )

c(pa). (1)

3.5.3 Weight computation

After computing the connection strength measures cs(xl , p j )

for each unresolved subject xl and real-world person p j , thenext task is to determine the desired weight wl j which shouldrepresent the confidence that subject xl matches person p j .RelDC computes these weights based on the Context Attrac-tion Principle (CAP) [21] that states that if cr� ≥ cr j thenwr� ≥ wr j , where cr� = c(xr , p�) and cr j = c(xr , p j ).In other words, the higher weight should be assigned to thebetter connected person. Therefore, the weights are com-puted based on the connection strength. In particular, RelDCsets the weight proportional to the corresponding connec-tion strengths: wr j cr� = wr�cr j . Using this strategy andgiven that

∑Nj=1 wr j = 1 (if each possible “option node”,

that is, each possible person, are listed), the weight wr j , forj = 1, 2, . . . , N , can be computed as follows:

wr j =⎧⎨⎩

cr j∑Nj=1 cr j

if∑N

j=1 cr j > 0;1N if

∑Nj=1 cr j = 0.

(2)

Thus, since some paths can go through edges labeled withwi j weight, the desired weight wr j can be defined as a func-tion of other option weights w: wr j = fr j (w).{

wr j = fr j (w) (for all r, j)

0 ≤ wr j ≤ 1 (for all r, j)(3)

The goal is to solve System (3). System (3) might nothave a solution as it can be over-constrained. Thus, a slackis added to it by transforming each equation wr j = fr j (w)

into fr j (w) − ξr j ≤ wr j ≤ fr j (w) + ξr j . Here, ξr j is a slackvariable that can take on any real nonnegative value. Theproblem transforms into solving the optimization problem,where the objective is to minimize the sum of all ξr j :⎧⎪⎪⎪⎪⎪⎪⎨⎪⎪⎪⎪⎪⎪⎩

Constraints:fr j (w) − ξr j ≤ wr j ≤ fr j (w) + ξr j (for all r, j)0 ≤ wr j ≤ 1 (for all r, j)0 ≤ ξr j (for all r, j)

Objective: Minimize∑

r, j ξr j

(4)

System (4) always has a solution and it can be solved bya solver or iteratively. In our scenario, we solve this systemin an iterative way [21]. The solution of this system are thevalues for all wr j weights.

3.5.4 Interpretation procedure

The computed weight wr j reflects the algorithm’s confidencethat subject xr is person p j . The next task is to decide whichperson to assign to xr given the weights. The original RelDCchooses the person p j who has the largest weight wr j amongwr1, wr2, . . . , wr |P|, when resolving the references of sub-ject xr .

The original strategy is meant for the case where eachpossible person p j that xr can refer to is known beforehand.However, in the setting of the person identification problem,this is not the case, as the algorithm is trying to decide ifxr refers to one of the known people p j of interest or tosome “other” person. To handle this new “other” category,we modify the original RelDC algorithm to also check ifall of the computed weights are above a certain predefinedthreshold t . If they are below the threshold, this means thealgorithm does not have enough evidence to resolve subjectxr , in which case it assigns xr to “other”. Otherwise, it willpick the person with the largest weight—the same way as theoriginal algorithm.

4 Experiments and results

4.1 Experimental datasets

Our experimental dataset consists of two weeks’ surveillancevideos from two adjacent cameras located in the second floorof CS Department building at UC Irvine [34]. These camerasare distributed in the corners of a corridor, near the officesof the Information System Group (ISG) members. Activitiesof graduate students and faculty, such as entering and exit-ing offices, hallway conversations, walking, and so on, are

123

L. Zhang et al.

captured by these cameras. Frames are collected continu-ously when motion is detected with the frame rate of 1 framea second for each camera. The resulting video shots are rela-tively simple, with one (or, rarely, a few) person(s) perform-ing simple activities. The task is to map the unknown subjectsinto known people.

To test the performance of the proposed algorithm, wemanually labeled four people from the video dataset to assignthe ground truth labels. The video collected over 2 weekscontains several (over 50) individuals of which we manuallylabeled 4. We then have divided the dataset into 2 parts. Thefirst week has been used as training data and the second weekas test data. From the training data, we get the faces of thechosen 4 people and train a face recognizer. We also extractactivities of people and compute priors based on activities.

4.2 Evaluation metrics

We have applied RelDC (in a limited form with a simplifiedconnection strength model) to identify the four people fromthe testing dataset. After obtaining the weight wr j for eachsubject xr to person p j , we decide which person that eachsubject should be assigned to using our strategy. The subjectxr can be assigned to p j only if two requirements are sat-isfied: (1) wr j is the largest among wr1, wr2, . . . , wr |P|, (2)wr j ≥ threshold. If the weights of a subject for each optionalperson are almost equal, and none of them is larger than thethreshold, then this subject will be considered as “others”. Bysetting different thresholds, we can get different recognitionresults.

To evaluate the performance of the proposed method, wechoose precision and recall as the evaluation metrics. Byselecting a particular threshold value, each subject xr canbe assigned a label denoted as L(xr ). The ground truth ofidentity for each subject xr is referred as T (xr ). Then as toeach person p j in the person set P , we can compute the cor-responding precision and recall based on L(xr ) and T (xr ).Thus the total precision and recall can be obtained by aver-aging the precision and recall for each targeted person p j .

Precision = 1

|P||P|∑j=1

|{xr |L(xr ) = p j ∧ T (xr ) = p j }||{xr |L(xr ) = p j }| (5)

Recall = 1

|P||P|∑j=1

|{xr |L(xr ) = p j ∧ T (xr ) = p j }||{xr |T (xr ) = p j }| (6)

4.3 Results

Figure 10 illustrates the precision-recall curve achievedby selecting different threshold values. We compare ourapproach with two conventional approaches.

0 0.2 0.4 0.6 0.8 10.2

0.4

0.6

0.8

1

Recall

Pre

cisi

on

Precision-Recall Curve

Our ApproachFacial FeautresKNN

Fig. 10 Precision-recall curve

– Facial features based method. As shown in Fig. 10, ifmerely leveraging facial visual features, the performanceof person identification is very poor. The recall is prettylow because most faces in the dataset cannot be detecteddue to the low quality of data, and thus the followingrecognition process is not able to be performed.

– K nearest neighbors method (K N N ). To perform K N N ,we just simply aggregate all the heterogeneous contextfeatures to obtain the overall subject similarities andthen label the K nearest neighbors of the resolved sub-ject with the same identity. By introducing context fea-tures, this method can achieve better performance thanthe facial feature based method. However, in this method,the underlying relationships between different contextfeatures are not considered.

The comparison with the above two approaches demon-strates the superiority of our approach. The advantages of ourapproach lie in that we not only leverage heterogeneous con-text features, but also explore the underlying relationships tointegrate heterogeneous context features together to improvethe recognition performance.

To test the robustness of our approach, we degrade theresolution and sampling rate of frames in our dataset respec-tively, and run a series of experiments on such dataset. Ouralgorithm mainly relies on context features such as activ-ities, which are less sensitive to the deterioration of videoquality. Figure 11 indicates that the decrease of frame reso-lution does not affect the performance of activity detectionsince the contextual information (such as time and location)is less sensitive to the frame resolution. But the performanceof activity detection (suppose the performance with the orig-inal resolution and sampling rate is 100 %) drops when sam-pling rate reduces from 1 frame/sec to 1/2 and 1/3 frame/sec,because many important frames are lost with the decreaseof sampling rate. Figure 12 illustrates that person identifica-tion result drops with the reduction of resolution and sam-pling rate, due to the loss of activity and color information.However, person identification result of our algorithm even

123

Context-based person identification framework

Fig. 11 Activity detection with decreasing of resolution and samplingrate

Fig. 12 PI result with decreasing of resolution and sampling rate

with the lowest resolution and sampling rate is much betterthan the baseline results of Naive Approach (which predictsresults just based on the occurrence probability in the trainingdataset). Consequently, Fig. 12 demonstrates the robustnessof our approach with low-quality video data, because ourapproach leverages contextual data rather than merely rely-ing on the quality of video data.

5 Conclusion

In this paper we considered the task of person identifica-tion in the context of Smart Video Surveillance. We havedemonstrated how an instance of indoor person identifica-tion problem (for video data) can be converted into the prob-lem of entity resolution (which typically deals with textualdata). The area of entity resolution has become very activeas of recently, with many research groups proposing pow-erful generic algorithms and frameworks. Thus, establishinga connection between the two problems has the potential tobenefit the person identification problem, which could beviewed as a specific instance of ER problem. Our experi-ments of using a simplified version of RelDC framework forentity resolution have demonstrated the effectiveness of ourapproach. This paper is, however, only a first step in exploit-ing ER techniques for video data cleaning tasks. Our current

approach has numerous assumptions and limitations: (1) Theapproach assumes that color of clothing is a strong identifierfor a person on a given day; if several people wear similarcolor clothes and have similar activities, it is hard to distin-guish them using the current approach. (2) If several peopleappear together, it is sometimes hard for the algorithm to cor-rectly separate these subjects, and this negatively affects theresult. Our future work will explore how additional featuresderived from video, as well as additional semantics in theform of context and metadata (e.g., knowledge of buildinglayout, offices, meeting times, etc.) can be used to furtherimprove person identification.

References

1. Project sherlock @ uci. http://sherlock.ics.uci.edu2. Ahonen, T., Hadid, A., Pietik, M.: Face description with local

binary patterns: Application to face recognition. IEEE Trans. Pat-tern Anal. (2006)

3. An, L., Kafai, M., Bhanu, B.: Dynamic bayesian network for uncon-strained face recognition in surveillance camera networks. IEEE J.Emerg. Sel. Top. Circuits Syst. 3(2), 155–164 (2013)

4. Balcan, M., Blum, A., Choi, P.P., Lafferty, J., Pantano, B., Rwe-bangira, M.R., Zhu, X.: Person identification in webcam images:An application of semi-supervised learning. In: ICML Workshopon Learning from Partially Classified Training Data (2005)

5. Chaudhuri, S., Ganjam, K., Ganti, V., Kapoor, R., Narasayya, V.,Vassilakis, T.: Data cleaning in Microsoft SQL Server 2005. In:ACM SIGMOD Conference (2005)

6. Chen, S., Kalashnikov, D.V., Mehrotra, S.: Adaptive graphicalapproach to entity resolution. In: Proceedings of ACM IEEE JointConference on Digital Libraries (JCDL 2007), Vancouver, BritishColumbia, Canada, June 17–23 (2007)

7. Chen, Z., Kalashnikov, D.V., Mehrotra, S.: Exploiting relationshipsfor object consolidation. In: Proceedings of International ACMSIGMOD Workshop on Information Quality in Information Sys-tems (ACM IQIS 2005), Baltimore, MD, USA, June 17 (2005)

8. Chen, Z.S., Kalashnikov, D.V., Mehrotra, S.: Exploiting contextanalysis for combining multiple entity resolution systems. In: Pro-ceedings of ACM SIGMOD International Conference on Manage-ment of Data (ACM SIGMOD 2009), Providence, RI, USA, June29–July 2 (2009)

9. Cristani, M., Bicego, M., Murino, V.: Audio-visual event recog-nition in surveillance video sequences. IEEE Trans. Multimed.(2007)

10. Dalal, N., Triggs, B.: Histograms of oriented gradients for humandetection. In: CVPR (2005)

11. Gallagher, A., Chen, T.: Clothing cosegmentation for recognizingpeople. In: IEEE CVPR (2008)

12. Gao, Y., Tang, J., Hong, R., Yan, S., Dai, Q., Zhang, N., Chua,T.: Camera constraint-free view-based 3D object retrieval. IEEETrans. Image Process. 21(4), 2269–2281 (2012)

13. Gao, Y., Wang, M., Tao, D., Ji, R., Dai, Q.: 3D object retrieval andrecognition with hypergraph analysis. IEEE Trans. Image Process.21(9), 4290–4303 (2012)

14. Gao, Y., Wang, M., Zha, Z., Shen, J., Li, X., Wu, X.: Visual-textualjoint relevance learning for tag-based social image search. IEEETrans. Image Process. 22(1), 363–376 (2013)

15. Grossman, R.L., Kamath, C., Kegelmeyer, P., Kumar, V., Namburu,R.R.: Data Mining for Scientific and Engineering Applications.Kluwer, Dordrecht (2001)

123

L. Zhang et al.

16. Han, J., Altman, R.B., Kumar, V., Mannila, H., Pregibon, D.:Emerging scientific applications in data mining. Commun. ACM45(8), 54–58 (2002)

17. Hong, R., Tang, J., Tan, H.-K., Ngo, C.-W., Yan, S., Chua, T.-S.: Beyond search: event-driven summarization for web videos.TOMCCAP 7(4), 35 (2011)

18. Hong, R., Wang, M., Li, G., Nie, L., Zha, Z.-J., Chua, T.-S.: Multi-media question answering. IEEE MultiMed. 19(4), 72–78 (2012)

19. Kalashnikov, D.V.: Super-EGO: fast multi-dimensional similarityjoin. Int. J. Very Large Data Bases 4(2), 561–585 (2013)

20. Kalashnikov, D.V., Chen, Z., Mehrotra, S., Nuray, R.: Web peo-ple search via connection analysis. IEEE Trans. Knowl. Data Eng.20(11) (2008)

21. Kalashnikov, D.V., Mehrotra, S.: Domain-independent data clean-ing via analysis of entity-relationship graph. ACM Trans DatabaseSyst. 31(2), 716–767 (2006)

22. Kalashnikov, D.V., Mehrotra, S., Chen, S., Nuray, R., Ashish, N.:Disambiguation algorithm for people search on the web. In: Pro-ceedings of the IEEE 23rd International Conference on Data Engi-neering (IEEE ICDE 2007), Istanbul, Turkey, April 16–20 2007(short publication)

23. Kalashnikov, D.V., Mehrotra, S., Chen, Z.: Exploiting relation-ships for domain-independent data cleaning. In: SIAM Interna-tional Conference on Data Mining (SDM 2005), Newport Beach,CA, USA, April 21–23 (2005)

24. Kantardzic, M., Zurada, J.: Next Generation of Data-Mining Appli-cations. Wiley, London (2005)

25. Kumar, N., Berg, A.C., Belhumeur, P.N., Nayar, S.K.: Describablevisual attributes for face verification and image search. In: IEEETPAMI (2011)

26. McCallum, A., Wellner, B.: Object consolidation by graph par-titioning with a conditionally-trained distance metric. In: KDDWorkshop on Data Cleaning, Record Linkage and Object Con-solidation (2003)

27. Nuray-Turan, R., Kalashnikov, D.V., Mehrotra, S.: Self-tuningin graph-based reference disambiguation. In: Proceedings of the12th International Conference on Database Systems for AdvancedApplications (DASFAA 2007), Springer LNCS, Bangkok, Thai-land, April 9–12 (2007)

28. Nuray-Turan, R., Kalashnikov, D.V., Mehrotra, S.: Exploiting webquerying for web people search. ACM Trans. Database Syst. 37(1)(2012)

29. Nuray-Turan, R., Kalashnikov, D.V., Mehrotra, S.: Adaptive con-nection strength models for relationship-based entity resolution.ACM J. Data Inf. Qual. 4(2) (2013)

30. Nuray-Turan, R., Kalashnikov, D.V., Mehrotra, S., Yu, Y.: Attributeand object selection queries on objects with probabilistic attributes.ACM Trans. Database Syst. 37(1) (2012)

31. Tang, J., Hong, R., Yan, S., Chua, T.-S., Qi, G.-J., Jain, R.: Imageannotation by knn-sparse graph-based label propagation over nois-ily tagged web images. ACM TIST 2(2), 14 (2011)

32. Tang, J., Yan, S., Hong, R., Qi, G.-J., Chua, T.-S.: Inferring seman-tic concepts from community-contributed images and noisy tags.ACM Multimed., pp. 223–232 (2009)

33. Tian, Y., Brown, L.M.G., Hampapur, A., Senior, M.L., Shu, C.:Ibm smart surveillance system (s3): event based video surveil-lance system with an open and extensible framework. Mach. Vis.Appl.(2008)

34. Vaisenberg, R., Mehrotra, S., Ramanan, D.: Semartcam scheduler:emantics driven eal- ime data collection from indoor amera net-works to maximize event detection. J. Real Time Image Process.5(4) (2010)

35. Wang, M., Hua, X.-S., Hong, R., Tang, J., Qi, G.-J., Song, Y.: Uni-fied video annotation via multigraph learning. IEEE Trans. CircuitsSyst. Video Technol. 19(5), 733–746 (2009)

36. Wang, M., Li, H., Tao, D., Lu, K., Wu, X.: Multimodal graph-based reranking for web image search. IEEE Trans. Image Process.21(11), 4649–4661 (2012)

37. Yang, M., Yu, K.: Real-time clothing recognition in surveillancevideos. In: ICIP (2011)

38. Zhang, L., Kalashnikov, D.V., Mehrotra, S.: A unified frameworkfor context assisted face clustering. In: ACM International Confer-ence on Multimedia Retrieval (ACM ICMR 2013), Dallas, Texas,USA, April 16–19 (2013)

39. Zhang, L., Vaisenberg, R., Mehrotra, S., Kalashnikov, D.V.: Videoentity resolution: applying er techniques for smart video surveil-lance. In: IQ2S Workshop in Conjunction with IEEE PERCOM2011 (2011)

40. Zhong, H., Shi, J., Visontai, M.: Detecting unusual activity in video.In: CVPR (2004)

Author Biographies

Liyan Zhang is currently aPh.D candidate in Computer Sci-ence Department of the Uni-versity of California, Irvine.Her research interests includeinformation retrieval, multime-dia search, computer vision andcyber physical systems. Shereceived the BSc degree in com-puter science from Hebei Uni-versity of Science and Technol-ogy, China, in 2006 and theMSc degree in software engi-neering from Tsinghua Univer-sity, China, in 2009.

Dmitri V. Kalashnikov is anAssociate Adjunct Professor ofComputer Science at the Uni-versity of California, Irvine. Hereceived his Ph.D degree in Com-puter Science from Purdue Uni-versity in 2003. He received hisdiploma in Applied Mathemat-ics and Computer Science fromMoscow State University, Russiain 1999, graduating summa cumlaude. His general research inter-ests include databases and datamining. Currently, he specializesin the areas of entity resolutionand data quality and real-time sit-

uational awareness. In the past, he has also contributed to the areas ofspatial, moving-object, and probabilistic databases. He has receivedseveral scholarships, awards, and honors, including an Intel Fellowshipand Intel Scholarship. His work is supported by the NSF, DH&S, andDARPA.

123

Context-based person identification framework

Sharad Mehrotra is a Professorin the School of Information andComputer Science at Universityof California, Irvine and found-ing Director of the Center forEmergency Response Technolo-gies (CERT) at UCI. Mehrotra’sresearch interests include vari-ous aspects of data management,multimedia, and distributed sys-tems. Mehrotra is the recipient ofthe SIGMOD test of time awardin 2012, DASFAA test of timeaward in 2013, and numerousbest paper awards including SIG-MOD best paper award in 2004

and ICMR best paper award in 2013. Mehrotra’s recent research focuseson data quality, data privacy and sensor driven situational awarenesssystems.

Ronen Vaisenberg received hisB.A degree with distinction,president list of excellence, fromthe Open University of Israelin 2001. He received his M.Scdegree in Information SystemsEngineering from Ben-GurionUniversity of the Negev in 2005and his M.Sc in computer sci-ences in 2008 from the Univer-sity of California, Irvine. He hasgraduated with a Ph.D degreefrom the University of Califor-nia, Irvine. Under the mentor-ship of Professor Sharad Mehro-tra, Vaisenberg’s Ph.D disserta-

tion deals with the issues related to the data management support forsentient systems, motivated by real-world emergency-response appli-cation needs. His work has been supported by an IBM Ph.D fellowship,NSF’s ITR-Rescue (RESponding to Crisis and Unexpected Events) andDHS’s Safire (Situational Awareness for Firefighters). He was a visit-ing researcher with the Event processing group at IBM-Haifa researchlab, Fusion Tables team at Google Research and the Ads backend teamat Google. He is currently working on building Google’s Knowledgegraph—a large-scale semantic representation of the world’s entities.Ronen is excited about making the work a better place using informa-tion technology.

123