Background Subtraction for Temporally Irregular

Dynamic TexturesGerald Dalley, Joshua Migdal,

and W. Eric L. Grimson

Workshop on Applications of Computer Vision (WACV) 2008

8 Jan. 2008 2

Key Prior Approaches

l MoG – learn modes for all pixels & track changesl Works as long as dynamic textures are regular

l Per-pixel kernels (Mittal & Paragios)l Can pay attention to higher-level features like optical flowl How many kernels to keep?

l Spatial kernels (Sheikh & Shah)l Handles temporal irregularity in dynamic texturesl How many kernels to keep?

8 Jan. 2008 3

Results with Mittal & Paragios Traffic Clip

Key:l bboxes from

ground truthl True

positivel False

positivel False

negative

l No ground truth available

8 Jan. 2008 4

Our Goals

l Compactnessl Speedl Complexity growth

l Grow with scene complexity, not with the amount of time to see the complexity§ If a leaf blows in front of a pixel once a minute, we don’t

want to have to keep 1800 kernels

l Usable in MoG frameworksl MoG variants are very widely used (original

Stauffer & Grimson paper has 1000+ citations)

8 Jan. 2008 5

Pixels Affected by a GivenMixture ComponentOur Model

l Mixture of Gaussians (MoG)

Standard MoG

Ours

p(ci|Φ) ∝∑

j∈NiwjN (ci;µj ,Σj)

ci the observed color at pixel i

Φ the model {wj , µj ,Σj}j

Ni neighborhood of pixel i

p(ci|Φ) ∝∑

j∈NiwjN (ci;µj ,Σj)

ci the observed color at pixel i

Φ the model {wj , µj ,Σj}j

Ni neighborhood of pixel i

8 Jan. 2008 6

Foreground/Background Classification

l Find best matching background Gaussian, jl Use neighborhood

l Squared Mahalanobis Distance

-

=

input

d2ij

mixture model

Φ1 Φ2 Φ3

dij = (ci − µj)TΣ−1

j (ci − µj)dij = (ci − µj)TΣ−1

j (ci − µj)

8 Jan. 2008 7

hard Stauffer-Grimson

soft Stauffer-Grimson

best match only

update all matches

Model Update OptionsHard UpdatesSoft Updates

Loca

lR

egio

nal

8 Jan. 2008 8

Results on a Classic Dataset: Wallflower WavingTrees

Ground

Truth

Toyam

aet al.

Best

MoG

Ours

Final ResultAfter MRFMahalanobisDistances

8 Jan. 2008 9

ROC (Wallflower)

8 Jan. 2008 10

Traffic #410G

round T

ruthM

ittal &

Paragios

Best

MoG

Ours

Final ResultAfter MRFMahalanobisDistances

8 Jan. 2008 11

Traffic #150G

round T

ruthM

ittal &

Paragios

Best

MoG

Ours

Final ResultAfter MRFMahalanobisDistances

8 Jan. 2008 12

Pushing the Limits:Ducks

Ground

Truth

Best

MoG

Ours

Final ResultAfter MRFMahalanobisDistances

8 Jan. 2008 13

Pushing the Limits:Jug

Ground

Truth

Zhong

&

Sclaroff

Best

MoG

Ours

Final ResultAfter MRFMahalanobisDistances

8 Jan. 2008 14

Parameter Sensitivity

0 1000 2000 3000 4000 5000 6000 70000

0.2

0.4

0.6

0.8

1

min number of mistakes

freq

uenc

y

ws=7ws=5ws=3ws=1

traffic

0 500 1000 1500 20000

0.05

0.1

0.15

min number of mistakes

freq

uenc

y

ws=7ws=5ws=3ws=1

wallflower

max max

8 Jan. 2008 15

Summary

l Model representationl Same as MoG

l FG/BG Classifier Inputl Add local neighborhood loop

l Updatel Several options, including standard MoG updates

l Resultsl Reduces foreground false positivesl Reduces need for exotic input features (e.g. optical flow)l Cost window size

8 Jan. 2008 16

Also in the Paper…

l Analysisl Timing experimentsl More ROC analysis

l Implementation Detailsl MRF choicel Update formulas

8 Jan. 2008 17

Thank You

Questions...

8 Jan. 2008 18

Backup Slides

8 Jan. 2008 19

PureSoft SoftLocal PureHard HardLocal0

10

20

30

40

50

60R

elat

ive

Tim

e C

ost

Model Update Computational Costs

W=12

W=32

W=52

W=72

8 Jan. 2008 208 Jan. 2008 20

PureSoft SoftLocal PureHard HardLocal0

20

40

60

80

100R

elat

ive

Tim

e C

ost

Mahalanobis Map Computational Costs

W=12

W=32

W=52

W=72

8 Jan. 2008 21

1

2

3

4

5

6

7

8

1 2 3 4 1 2 3 4 1 2 3 41 2 3 4

Background

Model

Observed

Image

9x9

Matching

3x3

Matching

Small Windows are Still Useful

8 Jan. 2008 22

c(t−1)ic(t−1)ic(t−1)i c

(t)ic(t)ic(t)i

φ(t)jφ(t)j

z(t−1)iz(t−1)i

l(t−1)jl(t−1)jl(t−1)j l

(t)jl(t)jl(t)j

JJ

II

z(t)iz(t)i

φ(t−1)jφ(t−1)j

Generative Model

Particle locations

Particle appearances

Pixel-to-particle assignments

Pixel appearance

8 Jan. 2008 23

Independent Assignments

l(t−1)1

l(t−1)2

l(t−1)3

z(t)i = 2

l(t)1

l(t)2

l(t)3

l(t−1)1l(t−1)1l(t−1)1

l(t−1)2l(t−1)2l(t−1)2

l(t−1)3l(t−1)3l(t−1)3

z(t)i = 2z(t)i = 2z(t)i = 2

l(t)1l(t)1l(t)1

l(t)2l(t)2l(t)2

l(t)3l(t)3l(t)3