1

Real Time, Online Detection of Abandoned Objects in Public Areas

Proceedings of the 2006 IEEE International Conference on Robotics and Automation

Authors : Nathaniel Bird, Stefan Atev, Nicolas Caramelli, Robert Martin, Osama Masoud, Nikolaos Papanikolopoulos

2

Outline

Introduction Method Description

A. Low-Level Processing B. Short-Term Logic C. Long-Term Logic D. Image Similarity

Results Conclusions and Future Work

3

I. Introduction

This paper addresses this issue by presenting an algorithm for automated detection of abandoned objects.

Abandoned object : a stationary object has not been touching a person for some time threshold.

The method must Online in real time Stay active around the clock Not detect still people as abandoned objects Detect abandoned objects even if they are

occluded by moving crowds of people for periods of time

4

II. Method DescriptionA. Low-Level Processing

Background segmentation is performed using [6].

Processing only every tenth frame is still a high enough rate considering the temporal scale at which the events of interest occur.

[6] C. Stauffer and W. E. L. Grimson, “Adaptive background mixture models for real-time tracking,” Proceedings of the IEEE Computer Vision and Pattern Recognition, vol. 2, pp. 2246-2252, June 1999

5

II. Method DescriptionA. Low-Level Processing

The background segmentation method is restricted to user-specified regions of interest in the image. block out areas of the image where any background

changes detected can be considered as noise (walls) Remove areas that are too far from the camera for

accurate abandoned object identification

6

II. Method DescriptionA. Low-Level Processing

Binary foreground mask Blob extraction is then performed

on the binary foreground mask. Correlating the blobs detected in

the last frame with the blobs detected in the current frame.

7

II. Method DescriptionB. Short-Term Logic

Blob types: Abandoned Object (A) Person (P) Still Person (SP) Unknown (U)

Blob behaviors: Creation Splits Merges Blob centroid velocity

8

II. Method DescriptionB. Short-Term Logic

Person Group (PG) A PG is created containing the new blobs that

split from a P or an SP. All blobs contained within a PG are classified as U until one of them becomes a P, at which all other blobs within it can be classified normally.

This is to stop a sitting person from being incorrectly classified as an abandoned object if they place a bag beside them that splits from their blob.

: threshold velocity above which an abandoned object, still person, or unknown becomes a person.

9

II. Method DescriptionB. Short-Term Logic

10

II. Method DescriptionB. Short-Term Logic

11

II. Method DescriptionC. Long-Term Logic

The long term logic maintains a set of potential abandoned objects and a set of still people. Contour in the image plane Timestamp of when they were first detected.

When an A type blob is found, it first checked if it does not overlap with any item in the potential abandoned object or still person sets. Yes => copied into potential abandoned object

set No => ignore

So did SP type blob.

12

II. Method DescriptionC. Long-Term Logic

All items in the potential abandoned object set and the still person set are checked every time a frame is processed. If their corresponding area in the binary

foreground mask is filled less than some percentage, p, then the item is dropped.

P were empirically found to be between 75% and 80%

13

II. Method DescriptionC. Long-Term Logic

time threshold t If during a check it is discovered that t

time has elapsed since an item was added to the potential abandoned object set, an alarm is triggered for that object.

After the alarm is triggered, the long-term logic adjusts a mask used by the background segmentation module so that it will not improperly learn the abandoned object into the background.

14

II. Method DescriptionD. Image Similarity

When the potential abandoned object is first detected, a copy of the image at its location is saved.

At every time step, the area surrounding the object ( a “halo” excluding the object) is checked for significant foreground activity.

If there is no noticeable foreground activity in the halo, the current image is compared pixel-by-pixel to the stored image for the potential abandoned object, and an average per-pixel difference is calculated.

15

II. Method DescriptionD. Image Similarity

An exponential running average of this difference is then updated.

If the value of the exponential running average exceeds an empirically determined threshold, the potential abandoned object is deemed to be moving too much to be a stationary object. => reclassified to a still person.

sumavgicountssumavg )1(][

16

III. ResultsA. Alarm Description

The following information is what is recorded for every alarm : 1. Identification Number 2. Start Time 3. Trigger Time 4. End Time 5. Image-Plane Location

17

III. ResultsB. Ground Truth

The ground truth for a given video sequence is determined manually for every sequence by a human operator.

18

III. ResultsC. PED/PAT Score Description

A candidate match is declared if there is sufficient spatial proximity and/or overlap between the two alarms as well as a temporal distance below a specific tolerance.

The candidate matches will usually result in a many-to-many relationship Bipartite graph

We therefore give each edge a weight equal to the timestamp difference between the two alarms and then find the minimum-weighted maximum-cardinality matching.

19

III. ResultsD. Overall Score Description

We define an overall score as follows:

x is relative importance we wish to give to the PED score over the PAT score. x=0 => PAT plot X=0.5 => weight PED and PAT equally X=1 => PED plot

We use a value of x=0.75 because we consider finding true alarms more important than some false positives.

20

III. ResultsE. Test Sequences

21

III. ResultsF. Results

22

III. ResultsF. Results

23

IV. Conclusions and Future Work

We have presented a method to detect abandoned objects that works online in real time, uses color data, can adapt to scene changes around the clock, does not detect still people as abandoned objects and detects abandoned objects even if they are occluded by moving crowds of people for periods of time.

The results for densely populated scenes are not as good, indicating that future research should look into defining a short-term logic that characterizes the behavior of blobs corresponding to crowds.