detecting moving shadows: formulation, moving shadows: formulation, algorithms and evaluation 1...

Click here to load reader

Post on 16-Apr-2018




7 download

Embed Size (px)



    Detecting Moving Shadows:Algorithms and Evaluation

    Andrea Prati 1,2, Ivana Mikic2, Mohan M. Trivedi2, Rita Cucchiara1

    AbstractMoving shadows need careful consideration inthe development of robust dynamic scene analysis systems.Moving shadow detection is critical for accurate object de-tection in video streams, since shadow points are often mis-classified as object points causing errors in segmentation andtracking. Many algorithms have been proposed in the litera-ture that deal with shadows. However, a comparative evalu-ation of the existing approaches is still lacking. In this paper,we present a comprehensive survey of moving shadow de-tection approaches. We organize contributions reported inthe literature in four classes, two of them are statistical andtwo are deterministic. We also present a comparative em-pirical evaluation of representative algorithms selected fromthese four classes. Novel quantitative (detection and discrim-ination rate) and qualitative metrics (scene and object inde-pendence, flexibility to shadow situations and robustness tonoise) are proposed to evaluate these classes of algorithms ona benchmark suite of indoor and outdoor video sequences.These video sequences and associated ground-truth dataare made available at to al-low for others in the community to experiment with new al-gorithms and metrics.

    Keywords Shadow detection, performance evaluation,object detection, tracking, segmentation, traffic scene anal-ysis, visual surveillance


    DETECTION and tracking of moving objects is atthe core of many applications dealing with im-age sequences. One of the main challenges in theseapplications is identifying shadows which objectscast and which move along with them in the scene.Shadows cause serious problems while segmentingand extracting moving objects, due to the misclas-sification of shadow points as foreground. Shadowscan cause object merging, object shape distortion andeven object losses (due to the shadow cast over an-

    Corresponding author.1 Dipartimento di Ingegneria dellInformazione, Universita di

    Modena e Reggio Emilia, Via Vignolese, 905 - Modena - Italy- phone: +39-059-2056136 - fax: +39-059-2056126 - e-mail:{prati.andrea/cucchiara.rita}

    2 Computer Vision and Robotics Research Laboratory, Depart-ment of Electrical and Computer Engineering, University of Cali-fornia, San Diego, 9500 Gilman Drive - La Jolla, CA 92037-0407- USA - phone: (858) 822-0075 - fax: (858) 534-0415 - e-mail:{ivana/trivedi}

    other object). The difficulties associated with shadowdetection arise since shadows and objects share twoimportant visual features. First, shadow points are de-tectable as foreground points since they typically dif-fer significantly from the background; second, shad-ows have the same motion as the objects castingthem. For this reason, the shadow identification iscritical both for still images and for image sequences(video) and has become an active research area espe-cially in the recent past. It should be noted that whilethe main concepts utilized for shadow analysis in stilland video images are similar, typically the purposebehind shadow extraction is somewhat different. Inthe case of still images, shadows are often analyzedand exploited to infer geometric properties of the ob-jects causing the shadow (shape from shadow ap-proaches) as well as to enhance object localizationand measurements. Examples of this can be found inaerial image analysis for recognizing buildings [1][2],for obtaining 3-D reconstruction of the scene [3] oreven for detecting clouds and their shadows [4]. An-other important application domain for shadow detec-tion in still images is for the 3-D analysis of objectsto extract surface orientations [5] and light source di-rection [6].

    Shadow analysis, considered in the context ofvideo data, is typically performed for enhancingthe quality of segmentation results instead of de-ducing some imaging or object parameters. Inthe literature shadow detection algorithms are nor-mally associated with techniques for moving ob-ject segmentation. In this paper we present a com-prehensive survey of moving shadow detection ap-proaches. We organize contributions reported inthe literature in four classes and present a com-parative empirical evaluation of representative algo-rithms selected from these four classes. This com-parison takes into account both the advantages andthe drawbacks of each proposal and provides a quan-titative and qualitative evaluation of them. Novelquantitative (detection and discrimination rate) and


    qualitative metrics (scene and object independence,flexibility to shadow situations and robustness tonoise) are proposed to evaluate these classes of al-gorithms on a benchmark suite of indoor and out-door video sequences. These video sequences andassociated ground-truth data are made availableat toallow for others in the community to experiment withnew algorithms and metrics. This availability fol-lows the idea of data-sharing embodied in Call forComparison, like the project of European COST 211Group (see at forfurther details).

    In the next Section we develop a two layer tax-onomy for surveying various algorithms presented inthe literature. Each approach class is detailed and dis-cussed to emphasize its strengths and its limitations.In Section III, we develop a set of evaluation metricsto compare the shadow detection algorithms. This isfollowed by Section IV where we present a results ofempirical evaluation of four selected algorithms on aset of five video sequences. The final Section presentsconcluding remarks.


    Most of the proposed approaches take into accountthe shadow model described in [7] . To account fortheir differences, we have organized the various al-gorithms in a two-layer taxonomy. The first layerclassification considers whether the decision processintroduces and exploits uncertainty. Deterministicapproaches use an on/off decision process, whereasstatistical approaches use probabilistic functions todescribe the class membership. Introducing uncer-tainty to the class membership assignment can re-duce noise sensitivity. In the statistical methods (as[8][9][10][11][12]) the parameter selection is a criti-cal issue. Thus, we further divide the statistical ap-proaches in parametric and non-parametric methods.The study reported in [8] is an example of the para-metric approach, whereas [10][11] are examples ofthe non-parametric approach. The deterministic class(see [6][7][13][14]) can be further subdivided. Sub-classification can be based on whether the on/off de-cision can be supported by model based knowledge ornot. Choosing a model based approach achieves un-doubtedly the best results, but is, most of the times,

    too complex and time consuming compared to thenon-model based. Moreover, the number and thecomplexity of the models increase rapidly if the aimis to deal with complex and cluttered environmentswith different lighting conditions, object classes andperspective views.

    It is also important to recognize the types of fea-tures utilized for shadow detection. Basically, thesefeatures are extracted from three domains: spectral,spatial and temporal. Approaches can exploit differ-ently spectral features, i.e. using gray level or colorinformation. Some approaches improve results by us-ing spatial information working at a region level or ata frame level, instead of pixel level. This is a classifi-cation similar to that used in [15] for the backgroundmaintenance algorithms. Finally, some methods ex-ploit temporal redundancy to integrate and improveresults.

    In Table I we have classified 21 papers dealing withshadow detection in four classes. We highlight spec-tral, spatial and temporal features used by these algo-rithms. In this paper, we focus our attention on fouralgorithms (reported in bold in Table I) representativeof three of the above-mentioned classes. For the sta-tistical parametric class we choose the algorithm pro-posed in [8] since this utilizes features from all threedomains. The approach reported in [11] can be con-sidered to be a very good representative of the statisti-cal non-parametric class and is also cited and used in[17]. Within the deterministic non-model based classwe choose to compare the algorithm described in [13]because is the only one that uses HSV color spacefor shadow detection. Finally, algorithm reported in[7] has been selected for its unique capability to copewith penumbra. The deterministic model-based classhas not been considered due to its complexity and dueto its reliance on very specific task domain assump-tions. For instance, the approach used in [14] modelsshadows using a simple illumination model: assum-ing parallel incoming light, they compute the projec-tion of the 3D object model onto the ground, exploit-ing two parameters for the illumination direction setoff-line and assumed to be constant during the entiresequence. However, as stated in the previous Sec-tion, in outdoor scene the projection of the shadow isunlikely to be perspective, since the light source cannot be assumed to be a point light source. Therefore,the need for object models and illumination positionsmanual setting make this approach difficult to be im-


    Statistical parametric Statistical non-parametricPaper Spectral Spatial Temporal Paper Spectral Spatial TemporalFriedman and Russell 1997 [12] C L D Horprasert et al. 1999 [11] C L SMikic et al. 2000 [8][9] C R D Tao et al.4 2000 [16] C F D

    McKenna et al. 2000 [17] C L SDeterministic model based Deterministic non