video segmentation

Video SegmentationLabeling independently moving image regions

2

Motion SegmentationForeground (object) and Background (noise)

Result could be a Binary image, containing foreground onlyProbability image, containing the likelihood of each

pixel being foreground

ApproachesMotion-based (optical flow)Color-basedTexture-based

Motion (Change) Detection

Static camera : changed and unchanged regions.Moving camera : global and local motion regions.Limitations associated with motion estimationAperature problem : pixels in a flat image region

may appear stationary even if they are moving as a result of an aperture problem (hence the need for hierarchical methods)

Occlusion Problem : erroneous labels may be assigned to pixels in covered or uncovered image regions as a result of an occlusion problem.

Motion Detection: using two frames : Image Differencing

5

Subtract ImageCompute pixel-wiseSubtract previous image from input image:

Usually the absolute distance is applied

),(),(),( yxByxIyxF

),( yxF

1. Save image in last frame2. Capture camera image3. Subtract image4. Threshold5. Delete noise

1

2

3

4

5

6

7

8

1

2

5

4

3

6

7

8

9

10

7

ThresholdDecide, when a pixel is supposed to be

considered as a background pixel, or when it is to be considered as a foreground pixel:

Pixel is foreground pixel, if

Pixel is background pixel, if

Problem: What TH?!?

),( yx

THyxF ),(),( yx

THyxF ),(

1. Save image in last frame2. Capture camera image3. Subtract image4. Threshold5. Delete noise

1

2

3

4

5

6

7

8

125 4 3678

Frame no. 1 2 3 4 5 6 7 8 9

x 154 160 160 160 157 130 112 79 0

y 13 19 19 19 16 19 19 19 0

w s s w w w w w

Dominant Motion : no optical flowSpatio temporal intensity gradient.Dominant motion segmentation

fitting a single parametric motion modelpartition the frame in two pixel groupsRepeat step 1 only to well represented pixels

group.

ThresholdingNoise Removal Median Filter

Image subtraction

Bounding Box Centroid

Multiple Motion SegmentationMultiple motion models compete against each

other at each decision site. They consist estimating:Motion within each region (motion model)spatial support of each regionnumber of regions.

The problem : associate each pixel to the right motion model, while simultaneously estimating motions and supports. Clustering (K-means, hough transform),Maximum Likelihood (ML) (pixel based) and Maximum A Posteriori probability (MAP).Region based label assignment

Optical flow estimationmotion vectors at each frame

Thresholding Morphological closing on the motion

vectors

Object tracking using kalman filters (prediction)

15

Motion ModelPredicted position at time t:

Brownian Motion: According to a Gaussian model

0’th order:

1’th order:Similar for y

2’th order Similar for y

),( tt yx

),(),( 11 tttt yxyx

1 at time in velocity :1

11

t-xx

xtxx

t

ttt

1 at time in on accelerati :2

1

1

112

1

t-xx

xtxtxx

t

tttt

16

Color Based segmentation : Background Subtraction

Use Neighborhood relation!! Compare pixel with its neighbors!! Weight them!!

Learn the background and its variations!!E.g. Gaussian models (mean,var) for each pixel!!!E.g. a Histogram for each Pixel The more images you train on the better!! Algorithm:

Consider each pixel (x,y) in the input image and check, how much it varies with respect to the mean and variance of the learned Gaussian models?

1. Calculate mean and variance for each pixel

2. Capture camera image3. Subtract image (= motion)4. Weight the distances (new)5. Threshold according to variance6. Delete noise

Color Segmentation with Histograms

18

Color Segmentation with Histogramsbrightness

19

Color Segmentation with Gaussian Distribution

N(m, C)

Thank you