Shadow Detection In Video

Submitted by: Hisham Abu saleh

Why detecting shadows?

Moving objects detection:Shadows can cause object merging, object

shape distortion or object loses. Shadow points are detected as

foregroundThey differ from background.They move the same as the object .

Why detecting shadows? cont.

Shadow Detection Algorithms

Algorithms are divided into: Deterministic approaches

Model based Best results. Too Complex.

Non-Model based

Statistical approaches (use probabilistic functions). Parametric methods. Nonparametric methods. Selecting parameters is critical issue.

Features presented by algorithms

Spectral Gray level Color (next lecture)

Spatial Pixel level Region level Frame level

Temporal Static Dynamic

Algorithm Taxonomy

Algorithm Comparison

We will compare between four algorithms: Statistical Nonparametric Statistical Parametric Deterministic Nonmodel-based (DNM1) Deterministic Nonmodel-based (DNM2)

The deterministic model-based class has not been considered due to its complexity and due to its reliance on very specific task domain assumptions.

Statistical Nonparametric Approach

HorprasertColor, Pixel level, Static.Shadows have similar chromaticity, but

lower brightness than the background model.

A statistical learning procedure is used to automatically determine the appropriate thresholds.

Statistical Parametric Approach

Mikie’ Color, Region level, Dynamic. The probability of a pixel belonging to shadow is

computed by assuming the color of the pixel not shadowed, and by using an approximated linear transformation to estimate the color of the point covered by a shadow.

Manual segmentation of a certain number of frames has to be done to collect statistics and to compute the values of matrix of the transformation.

Deterministic Nonmodel-based Approach (DNM1)

Cucchiara, Color, Pixel level, Static.This algorithm works in the HSV color space

(closer to the human perception of color and it has more accuracy in distinguishing shadows).

A shadow cast on a background does not change its hue significantly.

Deterministic Nonmodel-based Approach (DNM2)

StauderGray level, Frame level, Dynamic.The only algorithm that deals with

penumbra.Reference frame

Previous frame Has limitations: Object speed, Noise sensetive.

Background frame

Deterministic Nonmodel-based Approach (DNM2) cont.

),(),(),(sk yxpyxEyx kk The image luminance:

A

pA

pA

c

LyxNcyxkc

LyxNcc

yx )),(cos(),(

)),(cos(

),(Ek

If illuminated

If penumbra

If umbra1),(0 yxk Describes the transition inside the penumbra

p(x,y) – The reflectance of the object surface.

E(x,y) – irradiance.

Cp – Intensity of light source.

CA – ambient light.

Deterministic Nonmodel-based Approach (DNM2) cont.

Assumptions: 1. light source intensity cp is high.

The frame difference at a pixel changed by a moving cast shadow will be large.

If pixel is inside the umbra of a cast shadow at time instant k and outside the shadow at time instant k + 1. The reflectance of a static background does not change with time, thus the frame difference will be then:

0)),(cos(),(),(s-y)(x,s k1k LyxNcyxpyx pk

Deterministic Nonmodel-based Approach (DNM2) cont.

2. Camera and background are static.3. Background is plane, light source

position is distant from background.4. Light source size and distance

between moving object and background are not negligible compared to the distance between light source and object.

Deterministic Nonmodel-based Approach (DNM2) cont.

A. Detection of Static Background Edgesedges in the current

and in the previous image are detected and classified into moving and static edges.

Deterministic Nonmodel-based Approach (DNM2) cont.

B. Detection of Uniform Changes of Shading Frame Ratio:

If the frame ratio is locally spatially constant, a moving cast shadow is assumed at position x,y.

),(

),(),( 1

yxE

yxEyxFR

k

k

Deterministic Nonmodel-based Approach (DNM2) cont.

A shadow is decided to be in a region if it there is “darker” uniform region.

Static edges hint at a static background and can be exploited to detect nonmoving regions inside the frame difference.

Deterministic Nonmodel-based Approach (DNM2) cont.

C. Penumbra DetectionAccording to assumption 4 the

cast shadow has penumbra.The penumbra causes a soft

luminance step at the contour of a shadow that is characterized by its step height, step width and its gradient.

Deterministic Nonmodel-based Approach (DNM2) cont.

According to a table we put a candidate as a penumbra or not.

Algorithm comparison

Two important quality measures: good detection

Low probability of misclassifying a shadow point.Minimizing the shadow points classified as

background/foreground.good discrimination

the probability of classifying nonshadow points as shadow should be low.

Algorithm comparison

We define two metrics:

ss

s

FNTP

TP

FF

F

FNTP

TP

TP – True Positive

FN – False Negative

S – Shadow

F – Foreground (object)

Algorithm comparison

The Results:

SNP – Very effective, sensitive to dark objects and noise.

SP – Not very effective since the assumption that the transformation matrix is constant on the entire image.

DNM1 – Good accuracy, stable performance.

DNM2 – fails when the background is not planar.

Example

Physics-Based Shadow Detection

This step detects moving pixels (object, shadow, and some erroneous pixels).

Physics-Based Shadow Detection cont.

pixels on a detected surface cannot be shadow if they have higher intensity than the actual background.

This test does not reduce the shadow areas, but may successfully reduce object areas.

Physics-Based Shadow Detection cont.

Physics-Based Shadow Detection cont.

Shadow pixels falling on neutral surfaces, such as asphalt roads, tend to be more blue.

This test is not applied to all the pixels, but to neutral or gray surfaces that have low saturation (< 0.3)

Physics-Based Shadow Detection cont.

Physics-Based Shadow Detection cont.

This step performs surface segmentation. The connectivity C of two neighboring pixels p1 and p2 with

intensities u and v:

Otherwise

TvuHifppC

0

|),(|1)2,1(

Physics-Based Shadow Detection cont.

H(u,v) is defined as follows:

21

21),(,2,1

1

1

1

1

RR

RRvuH

vv

vvR

uu

uuR

tt

tt

tt

tt

If two neighboring pixels belong to the same surface they will have temporal ratios that are close together.

Physics-Based Shadow Detection cont.

Physics-Based Shadow Detection cont.

This step removes the effect of sky illumination.

We subtract the foreground pixel values from the background over the masked area.

Physics-Based Shadow Detection cont.

The user selects a patch of shadow region from a frame during the training.

The materials colors are stored to the verification step.

Physics-Based Shadow Detection cont.

This step performs matching of body color of various surfaces with the stored body color of materials that we expect to see in the scene.

The algorithm has training and testing phases.

Physics-Based Shadow Detection cont.

References

A. Prati, I. Mikic, M. Trivedi, R. Cucchiara, “Detecting Moving Shadows: Algorithms and Evaluation”, IEEE Transactions on pattern analysis and machine intelligence, vol. 25, no. 7, July 2003.

S. Nadimi, B. Bhanu, “Physical Models for Moving Shadow and Object Detection in Video”, IEEE Transactions on pattern analysis and machine intelligence, vol. 26, no. 8, August 2004.

J. Stauder, R. Mech, J. Ostermann, “Detection of Moving Cast Shadows for Object Segmentation”, IEEE Transactions on multi media, vol. 1, no. 1, March 1999.

References cont.

A. Prati, I. Mikic, C. Grana, M. Trivedi, “Shadow Detection Algorithms for Traffic Flow Analysis: a Comparative Study”.