WALLFLOWER PRINCIPLES AND PRACTICE OF BACKGROUND

MAINTENANCE

Costache Theodor “Hermann Oberth” Faculty of Engineering – Advanced Computing Systems Master

OVERVIEW

Introduction

The Wallflower Algorithm

Experiments

Analysis and Principles

Conclusions

Video surveillance systems seek to automatically identify people, objects, or events of interest in different kinds of environments.

A common element of such surveillance systems is a module that performs background subtraction for differentiating background pixels from foreground pixels.

The difficult part of background subtraction is not the differencing itself, but the maintenance of a background model.

INTRODUCTION

An ideal background maintenance system would be able to avoid the following problems:

INTRODUCTION

Moved objects

Time of day

Light switch

Waving trees

Camouflage

Bootstrapping

Foreground aperture

Sleeping person

Waking person Shadows

Wallflower was intended to solve as many of the canonical problems as possible.

To handle problems that occur at various spatial scales, it processes images at the pixel, region, and frame levels.

The pixel- level processing makes the prel iminary classifications of foreground versus background and also handles adaptation to changing backgrounds. The pixel- level avoids many of the common problems immediately: moved objects, t ime of day, waving trees, camouflage, and bootstrapping.

The region-level considers inter-pixel relationships that might help to refine the raw classification of the pixel level; doing so avoids the foreground aperture problem.

The frame level addresses the l ight switch problem: it watches for sudden changes in large parts of the image and swaps in alternate background models that explain as much of the new background as possible

THE WALLFLOWER ALGORITHM

 

THE PIXEL LEVEL

The idea is to segment whole objects, rather than isolated pixels, as foreground.

When an object is homogeneously colored, the problem is an instance of the classic vision problem where a moving homogeneous region exhibits no perceivable motion.

Moving homogeneous regions are necessarily bracketed by pixels which occur on the frontier edges of object movement, and which exhibit identical properties as those inside the region

So, we cast new foreground regions that have been discovered by the pixel level as seed regions that are grown by backprojecting the pixel values that occur in the seeds.

THE REGION LEVEL

As e ac h n e w p a i r o f r a w a n d f o re g ro u n d - m a r ked i m a g e s , a n d , a r r i v es ,

C om p u t e i m a g e d i ff e re n c e s ( a a n d b ) :

C om p u t e t h e s u b s e t o f p i xe l s w h i c h oc c u r a t t h e i n t e r s e c t i o n o f ad j a c e n t p a i r s o f d i ff e re n c e d i m ag es a n d t h e p re v i o u s f o re g rou n d i m a g e ( c ) :

Fi n d 4 - c o n n ec t e d re g i o n s , , i n , d i s c a rd i n g re g i o n s c on s i s t i n g o f l e s s t h an p i xe l s

C om p u t e , t h e n o rm a l i z ed h i s t o g r a m o f e ac h , a s p ro j e c t e d o n t o t h e i m a g e ( s i s a p i xe l v a l u e ) :

Ba c k p ro j e c t h i s t o g r a m s i n : Fo r e ac h , c om p u t e , an d f ro m e a c h p o i n t i n t h e i n t e r s e c t i o n , g ro w , t h e 4 - c o n n e c t e d re g i o n s i n t h e i m a g e ,

w h e re w e u s e = 1 6 , = 8 , ε = 0 . 1 .

THE REGION LEVEL

THE FRAME LEVEL

EXPERIMENTS

EXPERIMENTS

Background maintenance seems simple at first. But it turns out to be a problem rich with hard cases and subtle tradeoffs. Therefore, a set of principles to which background maintenance modules should adhere, need to be established.

ANALYSIS AND PRINCIPLES

•No SemanticsPrinciple 1

•Proper Initial Segmentation

Principle 2

•Stationarity CriteriaPrinciple 3

•AdaptationPrinciple 4

• Multiple Spatial Levels

Principle 5

The Wal lflower a lgor i thm outputs fa lse posit ives in the waving trees sequence, where part of the sky, considered background by the p ixel - level , becomes foreground after region- level processing. The region- level a lgor i thm is therefore an unsound heur ist ic , the use of which is not just ified in general because i t is an attempt to extract object semantics f rom low- level vis ion.

Semantic differentiation of objects should not be handled by the background maintenance module.

A background maintenance module handles the default model for everything in a scene that is not modeled expl ic i t ly by other processing modules.

The module performing background maintenance should not attempt to extract the semant ics of foreground objects on i ts own.

While background maintenance might be usefu l in determining gross traffic stat ist ics of objects such as people and cars, which can al ternately be moving or motionless, attempts to use i t a lone as a preprocess ing step for cont inuous, accurate tracking are bound to fa i l .

PRINCIPLE 1 – NO SEMANTICS

As long as there is to be a background maintenance module, we must differentiate the task of finding foreground objects from the task of understanding

Background subtraction should segment objects of interest when they first appear (or reappear) in a scene .

In environments where adaptation is necessary, maintaining foreground objects as foreground is not a reasonable task for background modelers, since such accurate maintenance requires semantic understanding of foreground.

Object recognition and tracking can be computationally expensive tasks – good background subtraction eliminates the need to perform these tasks for each frame, on every subregion.

PRINCIPLE 2 – PROPER INITIAL SEGMENTATION

The principle of proper init ial segmentat ion depends on the notion of object sal ience. Objects are sal ient when they deviate from some invariant property of the background. The key quest ion, then, is how this invar iance is modeled and what it means to deviate from it.

Backgrounds, for instance, are not necessari ly defined by absence of motion. Consider a fluttering leaf on a tree. As the leaf moves on and off a pixel, that pixel ’s value wi l l change radical ly. No unimodal distr ibut ion of pixel values can adequately capture such a background, because these models implic it ly assume that the background, apart from some minimal amount of noise, is stat ic. Indeed, a l l unimodal models fa i led to capture the complexity in the background required to handle the waving trees and camouflage experiments.

An appropriate pixel-level stationarity criterion should be defined. Pixels that satisfy this criterion are declared background and ignored.

We define stat ionarity as that qual ity of the background that a part icular model assumes to be approximately constant. Carelessness in defining the stat ionarity cr i ter ion can lead to the waving trees and camouflage problems.

PRINCIPLE 3 – STATIONARITY CRITERIA

Backgrounds often change, even with liberal definitions of stationarity.

The background model must adapt to both sudden and gradual changes in the background.

The line between gradual and non-gradual changes should be chosen to maximize the distinction between events that cause them.

PRINCIPLE 4 – ADAPTATION

Sudden l ight changes were best handled by normalized block correlation, eigenbackground, and Wallflower. On the other hand, neither eigenbackgrounds nor block correlation deals with the moved object problem or the bootstrapping problem, because they lack adaptive pixel level models. So, our final principle is the following:

Background models should take into account changes at differing spatial scales.

Most background maintenance algorithms maintain either pixel-wise models or whole-frame models, but not both. Pixel- level models are necessary to solve many of the most common background maintenance problems, while the l ight switch and t ime-of-day problems suggests that frame-wide models are useful, as well.

A good background maintenance system is l ikely to explicit ly model changes that happen at different spatial scales. Much of Wallflower’s success is attributable to its separate models for pixels and frames.

PRINCIPLE 5 – MULTIPLE SPATIAL LEVELS

Background maintenance, though frequently used for video surveillance applications, is often implemented ad hoc with little thought given to the formulation of realistic, yet useful goals. We presented Wallflower, a system that attempts to solve many of the common problems with background maintenance. Comparison of Wallflower with other algorithms establishes a case for five principles that we proposed based on analysis of the experiments

CONCLUSION