multi video camera calibration and synchronization

43
Multi video camera calibration and synchronization

Upload: alpha

Post on 25-Feb-2016

93 views

Category:

Documents


1 download

DESCRIPTION

Multi video camera calibration and synchronization. Motivation. Multi camera applications become common. Example: Stereo, Surveillance … . Using multi camera we can over come problems like hidden objects. In general more cameras equal more information on the scene. How does it look. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Multi video camera calibration and synchronization

Multi video camera calibration and synchronization

Page 2: Multi video camera calibration and synchronization

Motivation

• Multi camera applications become common.Example: Stereo, Surveillance….

• Using multi camera we can over come problems like hidden objects.

• In general more cameras equal more information on the scene.

Page 3: Multi video camera calibration and synchronization

How does it look

• Multi camera setup

Page 4: Multi video camera calibration and synchronization

The scene

• The filmed scene 1/3

Page 5: Multi video camera calibration and synchronization

The scene

• The filmed scene 2/3

Page 6: Multi video camera calibration and synchronization

The scene

• The filmed scene 3/3

Page 7: Multi video camera calibration and synchronization

Perspective projection

Perspective projection:

Projection plane

P(x, y, z)

P'(xp, yp, zp)

d

Zv

Xv

Yv

P(x, y, z)

P(x, y, z)

Xp

Xv

Yv

Zv

Zv

yp

Im * objectage

object

Xx d

Z

Im * objectage

object

Yy d

Z

Page 8: Multi video camera calibration and synchronization

The projection matrix

• Object point • Image point• Using the model:

• And the projection matrix (so far) is:

*Homogenous coordinates

( , , )TP X Y Z( , )Tp x y

( , , ) ( , ) ( * , * )object objectT T T

object object

X YX Y Z x y d d

Z Z

0 0 00 0 0

1 0 0 1 01

Xx d

Yy d

Z

Page 9: Multi video camera calibration and synchronization

Internal matrix

• The internal matrix represent the inner camera settings

• Focal length (d)• Principle point location usually

(0,0)• Scaling factor

( , )x yo o

( , )x ys s

Page 10: Multi video camera calibration and synchronization

External matrix

• Includes all the orientation properties of the camera

• Rotation• Translation

( )camera worldp R p T

Page 11: Multi video camera calibration and synchronization

Projection matrix sum up

• Internal parameters

• External parameters

• The result p=MP

int

0 00 00 0 1 0

x x

ernal y y

a oM a o

int *ranal externalM M M

11 12 13 1

21 22 23 2

31 32 33 3

0 0 0 1

extenal

R R R RTR R R RT

MR R R RT

Page 12: Multi video camera calibration and synchronization

Calibration

• Camera calibration is used to coordinate between cameras.

• Given a 3D point in the real word finding the projected point in the camera.

• The goal is to fined the projection matrix M.• Using known 3D points and there

corresponding image points p=MP can be solved.

Page 13: Multi video camera calibration and synchronization

When a full calibration is not necessary Homography

• Mapping between a point on a ground plane as seen from one camera, to the same point on the ground plane as seen from a second camera.

Page 14: Multi video camera calibration and synchronization

When an Homography can be used

• When the images are of the same planeCamera 1 Camera 2 Result

Page 15: Multi video camera calibration and synchronization

When an Homography can be used

• When images taking using same camera by only rotating it

Page 16: Multi video camera calibration and synchronization

Homography computation

• Using the Homography matrix H we can map point from one image to second image

• So we have : p’=Hp• P and p’ are given in homogeneous

coordinates11 12 13

21 22 23

31 32 331 1

x h h h xy h h h y

h h h

Page 17: Multi video camera calibration and synchronization

Homography computation

• H is 3x3• That is 8 D.O.F• To find H we need 4 corassponding points

11 12 13 31 32' ' ' 0h x h y h h xx h yx x

21 22 23 31 32' ' ' 0h x h y h h xy h yy y

Page 18: Multi video camera calibration and synchronization

Finding corresponding points

• Manually, detecting by hand similar features.

• Not accurate• Good for 2 cameras, what about 9 or more.

Page 19: Multi video camera calibration and synchronization

Known solution

• Automatic Detection of known features.

• Large working volume need large objects.• very hard to detect from far distance.

Page 20: Multi video camera calibration and synchronization

Features detection in wide base line

• Noise • Hidden parts• Assuming detection is possible finding the

corresponding is hard.

Page 21: Multi video camera calibration and synchronization

Example of feature detection problems

Page 22: Multi video camera calibration and synchronization

Goals of the calibration object

• 360 degrees view.• Robust to noise.• Accurate regardless the distance (or zoom).• Easy to find corresponding points.• Automated as possible.

Page 23: Multi video camera calibration and synchronization

Solution

• Use easy to detect features (Active features).

• Use the benefits of time dimension video.• This will create a easy to detect

corresponding point list.• Find Homography using the list of points.

Page 24: Multi video camera calibration and synchronization

Calibration object

• Ultra bright LEDs.

• Very bright, easy to detect.

Page 25: Multi video camera calibration and synchronization

Use flickering as identifier

• features flicker in constant rate • Each feature has a different rate• The cameras filming in constant rate• The LED flicker can be found• The result a list of points in an increasing

frequency rate for each camera

Page 26: Multi video camera calibration and synchronization

Detection method first stage

• Filter unnecessary noise• Use the red channel only as filter.• What about acceptable red channel filters in

RGB such as:R = ((R-B)+(R-G)).• Remove white pixels (All channels have

high intensities ).• Not good for a case a LED caused high

saturation (appears as white).

Page 27: Multi video camera calibration and synchronization

Filter Example

• Red channel only ((R-B)+(R-G))

Page 28: Multi video camera calibration and synchronization

Detection method second stage

• Take advantage of video camera time line• The LED is going from on to off state• Subtracting following frames (similar to

background subtraction).• Detect features pixels candidates using a

threshold.• Save detection frame number to detect

flickering rate.

Page 29: Multi video camera calibration and synchronization

Detection method third stage

• So far we have points candidate and there frequencies.

• Yet some of the candidates are noise.• Use a frequency as a second filter • Most of the noises have a very short and not

consist frequency.

Page 30: Multi video camera calibration and synchronization

Noise and feature frequencies

• Noise

• Feature

Page 31: Multi video camera calibration and synchronization

Frequency filter

• Before

Page 32: Multi video camera calibration and synchronization

Frequency filter

• After

Page 33: Multi video camera calibration and synchronization

Detection method fourth stage

• Once we have the LED pixels detected we need to detect a pixel to represent it

• Local maximum, the pixel with the highest intensity level.

• Solution to different distances of camera from the features and different zoom rates.

Page 34: Multi video camera calibration and synchronization

Local maximum example

• Before

Page 35: Multi video camera calibration and synchronization

Local maximum example

• After

Page 36: Multi video camera calibration and synchronization

Full tool example

Page 37: Multi video camera calibration and synchronization

Synchronization

• Given the frame number k in the first camera find the corresponding frame in the second camera.

• Not all the cameras starts to film in the same time.

• Known solution using temporal features

Page 38: Multi video camera calibration and synchronization

Temporal features

• Hard to find, not suitable for 9 cameras or more

Page 39: Multi video camera calibration and synchronization

Automatic synchronization

• Each feature has a different rate• The signature is based on the gap between the

pools vibrate.• Given an index we search for the first time after

this index the pool with the lowest frequency vibrate and so on.

• Given that the polls turned on in t0,t1,t2,t3,t4,t5 the resulting signature is

• (t1-t0,t2-t1,t3-t2,t4-t3,t5-t4)

Page 40: Multi video camera calibration and synchronization

Synchronization graph 1/2

Page 41: Multi video camera calibration and synchronization

Synchronization graph 2/2

Page 42: Multi video camera calibration and synchronization

Tool synchronization example

Page 43: Multi video camera calibration and synchronization

The end

• Thank you!!!