1 video processing lecture on the image part (8+9) automatic perception 12+15 volker krüger aalborg...
TRANSCRIPT
![Page 1: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/1.jpg)
1
Video Processing Lecture on the image part (8+9)
Automatic Perception 12+15
Volker KrügerAalborg Media Lab
Aalborg University [email protected]
![Page 2: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/2.jpg)
2
Agenda
• Segmentation in video (finding the object(s))– MED3 applications
• Small and gentle intro to statistics
Probably next time:
• Tracking: follow the object(s) over time
• What to remember
![Page 3: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/3.jpg)
3
Segmentation in Video
![Page 4: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/4.jpg)
4
Videos
• Videos are Image Sequences over Time
y
x
t
•An image is a function over x and y:•A video is a function of images over time t:
• At each time step t we have an image • Framerate = the number of images per second, e.g., 25 Images/s
),(),,( yxftyxf t
),( yxf
),(),,( yxftyxf t
![Page 5: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/5.jpg)
5
Segmentation in Video• General Segmentation is application-dependent!
Knowledge base!!• One application: Finding the object(s)
– MED3 applications– Preprocessing, Segmentation
• Tracking = follow the object(s) over time:– being able at each time t to give, e.g, position, color, orientation, etc.– Representation, Description
Knowledge base
Problemdomain Image
acquisition
Preprocessing
SegmentationRepresentationand description
Recognitionand Interpretation
Result
![Page 6: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/6.jpg)
6
SegmentationLots of applications!! One is:• Separation of Foreground (object) and Background
(everything else = noise)• Result could be a
– Binary image, containing foreground only– Probability image, containing the likelihood of each pixel being
foreground
• Useful for further processing, such as using silhouettes, etc.• Approaches
– Motion-based– Color-based– Some approaches can learn!
(demos!)
![Page 7: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/7.jpg)
7
Foreground-Background Segmentation using Motion and
Color
![Page 8: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/8.jpg)
8
Foreground-Background Segmentation using Motion and Color
• Motion-based– Model-free
– No learning
– Image differencing
• Color-based– Background subtraction
• Background used as a model
• No learning
– Advanced background subtraction• Background is learned
– Very advanced background subtraction• Background is learned
![Page 9: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/9.jpg)
9
Image Differencing
![Page 10: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/10.jpg)
10
Image Differencing
• The motion in an image can be found by subtracting the current image from the previous image
• Algorithm1. Save image in last frame
2. Capture current camera image
3. Subtract image (= difference = motion)
4. Threshold
5. Delete noise
![Page 11: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/11.jpg)
11
3. Subtract Image• Compute pixel-wise
• Subtract previous image from input image:
• Usually the absolute distance is applied
),(),(),( yxByxIyxF
),( yxF
1. Save image in last frame2. Capture camera image3. Subtract image4. Threshold5. Delete noise
![Page 12: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/12.jpg)
12
4. Threshold
• Decide, when a pixel is supposed to be considered as a background pixel, or when it is to be considered as a foreground pixel:
• Pixel is foreground pixel, if
• Pixel is background pixel, if
• Problem: What TH?!?
),( yx
THyxF ),(),( yx
THyxF ),(
1. Save image in last frame2. Capture camera image3. Subtract image4. Threshold5. Delete noise
![Page 13: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/13.jpg)
13
5. Deleting Noise
• Singular pixels are likely to appear: – Pixel-noise!!
• Apply Median filter:– Depending on filter size, bigger spots can be erased
• Alternative: Morphologic
1. Save image in last frame2. Capture camera image3. Subtract image4. Threshold5. Delete noise
(show: patch: diff)
![Page 14: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/14.jpg)
14
Background Subtraction
![Page 15: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/15.jpg)
15
Background Subtraction
• Foreground is moving, background is stable
• Algorithm1. Capture image containing background2. Capture camera image3. Subtract image (= difference = motion)4. Threshold5. Delete noise
(show: patch: bg_1)
![Page 16: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/16.jpg)
16
Advanced Background Subtraction
![Page 17: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/17.jpg)
17
Advanced Background Subtraction• What if we have small motion in the
background?– Bushed, leaves, etc. and noise in the camera/lighting– (show histo patch)
• Learn(!) the background
• Capture N images and calculate the average background image (no object present)
1. Calculate average background image 2. Capture camera image3. Subtract image (= motion)4. Threshold5. Delete noise
![Page 18: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/18.jpg)
18
Very Advanced Background Subtraction
![Page 19: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/19.jpg)
19
Very Advanced Background Subtraction
• Use Neighborhood relation!!– Compare pixel with its neighbors!!– Weight them!!
• Learn the background and its variations!!– E.g. Gaussian models (mean,var) for each pixel!!!– E.g. a Histogram for each Pixel Both will be revisited, later!!– The more images you train on the better!!– Idea:
• Some pixel may vary more than other pixels– Algorithm:
• Consider each pixel (x,y) in the input image and check, how much it varies with respect to the mean and variance of the learned Gaussian models?
1. Calculate mean and variance for each pixel 2. Capture camera image3. Subtract image (= motion)4. Weight the distances (new)5. Threshold according to variance6. Delete noise
![Page 20: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/20.jpg)
20
Weight the Distances, Correlation between Pixel values
• If one pixel is considered to be a foreground pixel– Its neighbor is also likely to be a foreground pixel– If its neighbor is not considered to be a foreground
pixel, one of the two might be wrong– Neighboring pixels are highly correlated (similar)
1. Calculate mean and variance for each pixel 2. Capture camera image3. Subtract image (= motion)4. Weight the distances (new)5. Threshold according to variance6. Delete noise
![Page 21: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/21.jpg)
21
Weight the Distances
• What does a pixel say about a foreground pixel, that is further away?– Pixels with an increasing distance to each other
are saying less about each other– The correlation between pixels degreases with
distance1. Calculate mean and variance for each pixel 2. Capture camera image3. Subtract image (= motion)4. Weight the distances (new)5. Threshold according to variance6. Delete noise
![Page 22: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/22.jpg)
22
Weight the Distances• Use a Gaussian for Weighting• To test a pixel I(x,y)
– Center a Gaussian on this pixel and weight the neighboring pixels accordingly => Convolution!
• 1D example:
60 48 2 3 1 222 100Signal:
0.25 0.5 0.25
39.5 13.75 2.25 56.75 136.3
Gaussian filter:
Output:
![Page 23: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/23.jpg)
23
Weight the Distances
• Likelihood image or Probability image, containing the likelihood for each pixel of being a background/foreground pixel
1. Calculate mean and variance for each pixel 2. Capture camera image3. Subtract image (= motion)4. Weight the distances (new)5. Threshold according to variance6. Delete noise
![Page 24: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/24.jpg)
24
A little detour to Statistics
![Page 25: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/25.jpg)
25
Statistics
• Mean– Center of gravity of the object
• Variance– The Variance measures the variations of the object pixel
positions around the center of gravity
N
iimean x
Nx
1
1
N
iimean y
Ny
1
1
N
imeanivar xx
Nx
1
2)(1
N
imeanivar yy
Ny
1
2)(1
xvar is big. yvar is small
N is the numberof object pixels
![Page 26: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/26.jpg)
26
Statistics
• Standard deviation: sigma ()• Normal distribution = Gaussian distribution
varsigma xx
99.99 %
99.73 %
95.44 %
68.26 %
SamplesRange234
(bb)
![Page 27: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/27.jpg)
27
Statistics
• How to use it– ”Automatic” thresholding based on statistics
• Example: the color of the hand– Training =>mean color
– Algorithm: hand pixel if: THmin < pixel < Thmax
– How do we define THmin and Thmax ?
– Use statistics: THmin = mean-2 and THmax = mean+2
![Page 28: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/28.jpg)
28
Threshold According to Variance• Threshold can be chosen depending on the variance
– A local threshold
• Standard Deviation• For example:
– If Thmin < dist < Thmax => object pixel– Thmin = mean – – Thmax = mean +
),var(),( yxyx
99.99 %
99.73 %
95.44 %
68.26 %
SamplesRange234
1. Calculate mean and variance for each pixel 2. Capture camera image3. Subtract image (= motion)4. Weight the distances (new)5. Threshold according to variance6. Delete noise
![Page 29: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/29.jpg)
29
Segmentation using Color
![Page 30: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/30.jpg)
30
Segmentation using Color
• Assumption: the object has a different color than the background– As seen in Chroma keying
• Two approaches– Color histogram– Gaussian distributions
![Page 31: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/31.jpg)
31
Segmentation using Color Histogram
![Page 32: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/32.jpg)
32
Segmentation using Color Histogram
• Given an object, defined by its color Histogram• Algorithm:
– Segment each pixel in the input image according to the histogram
• The result: binary or probability image. – White pixels: pixel in input image had color as defined
by the histogram
– Black pixel: pixel in input image did not have color defined by the histogram
![Page 33: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/33.jpg)
33
Learning the Color Histogram
• Recall a grayvalue histogram• The color histogram summarizes the
color distribution in an image (region)• There is a histogram bin for each
possible color!• The columns of the histogram are high
for those colors appearing often in the image (region) and low for those appearing seldom in the image.
• The more images you train on the better!!
bins
![Page 34: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/34.jpg)
34
Color Segmentation with Histograms
• Algorithm:– For each pixel in the input I(x,y)
• Go to the bin having the same color as the pixel H( I(x,y) )
• Assign the probability value at the bin to the output image: O(x,y)
– O(x,y) = H( I(x,y) )
• Using the above leads to a probability image
• Run a threshold on the output image to get a binary image
![Page 35: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/35.jpg)
35
Segmentation using Gaussian Distributions
![Page 36: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/36.jpg)
36
Segmentation with Gaussian Distribution
• Given an object, calculate the mean and standard deviation of its color– Represent each color component (e.g., R,G,B) by a Gaussian
model– That is, a Gaussian distribution: (mean,sigma)
• The same principle as in “very advanced background subtraction” !!
• Algorithm: – Given an input image, segment each pixel by comparing it to the
Gaussian models of the object color• For example:
– If Thmin < dist < Thmax => object pixel– Thmin = mean – – Thmax = mean +
![Page 37: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/37.jpg)
37
Comparing the Methods• When to use a color histogram and when
to use Gaussian models?• Plot the training data and look at it!• Gaussian models assume data is
distributed equally within a rectangle• Histograms do not assume anything• Gaussian models require less training• Histograms require a decision regarding
the resolution of the bins and the threshold.
• Thresholds in Gaussian models have a physical interpretation
r
g
r
g
![Page 38: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/38.jpg)
38
Tracking
![Page 39: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/39.jpg)
39
Tracking
• Follow the object(s) over time– Finding the trajectory (curve connecting the
positions over time)
• Simple tracking:
• Advanced tracking: – Cluttered background and multiple objects
![Page 40: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/40.jpg)
40
Tracking
• For MED3 projects– Use tracking to predict where the object will be
in the next image – This allow us to specify the ROI (region of
interest)– Focus the algorithm
• Save computational resources
• Less likely that we will find an incorrect object
(show: patch: color_track)
![Page 41: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/41.jpg)
41
Prediction
• Given the position of the object in previous images, where do we think (predict) the object will be in the current image?
• We need a motion model• The size of the search region depends on:
– The uncertain of the prediction– The framerate– How fast can the object max. move?
?
Search region
![Page 42: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/42.jpg)
42
Motion Model• Predicted position at time t:
• Brownian Motion: According to a Gaussian model
• 0’th order:
• 1’th order:– Similar for y
• 2’th order – Similar for y
• Many other types exist: look at your application!
),( tt yx
),(),( 11 tttt yxyx
1 at time in velocity :1
11
t-xx
xtxx
t
ttt
1 at time in on accelerati :2
1
1
112
1
t-xx
xtxtxx
t
tttt
![Page 43: 1 Video Processing Lecture on the image part (8+9) Automatic Perception 12+15 Volker Krüger Aalborg Media Lab Aalborg University Copenhagen vok@media.aau.dk](https://reader030.vdocuments.mx/reader030/viewer/2022032722/56649cf35503460f949c1178/html5/thumbnails/43.jpg)
43
What to remember
• Statistics: mean and variance• Motion segmentation
– Image differencing (two images)– Background subtraction (one bg. image)– Advanced background subtraction (many bg. image)– Very advanced background subtraction (learn each pixel)
• Color segmentation– Using a histogram – Gaussian models
• Tracking– Prediction– Motion model