3d tracking : chapter2-1 mathematical tools

Monocular Model-Based 3D Tracking of Rigid Ob-jects: A Survey

2008. 12. 04.백운혁

Chapter 2. Mathematical Tools

Agenda

Monocular Model-Based 3D Tracking of Rigid Objects : A Survey

Chapter 2. Mathematical Tools 2.1 Camera Representation 2.2 Camera Pose Parameterization 2.3 Estimating the External Parameters

Matrix 2.4 Least-Squares Minimization Techniques 2.5 Robust Estimation 2.6 Bayesian Tracking

the standard pinhole camera model

2.1 Camera Representation

2.1.1 The Perspective Projection Model

1

Z

Y

X

MWorld Coordinates

1

v

u

mImage Coordinates(in the image)

]|[ tRKP Projection Matrix

]|[ tR

K

Image coordinate system

2.1.2 The Camera Calibration Matrix internal parameters

100

0 0

0

v

us

K v

u

f focal length

fkuu

fkvv uk the number of pixels per unit distance

in the uvk the number of pixels per unit distance

in the v

0

0

v

uc principal point

s skew parameter

2.1.2 The Camera Calibration Matrix projection

Z

X

f

mx

f focal length

Z

Xfmx

Image Plane

2.1.2 The Camera Calibration Matrix projection to image

uk the number of pixels per unit distance in the u

vk the number of pixels per unit distance in the v

0

0

v

uc principal point (center of image

plane)

2.1.2 The Camera Calibration Matrix skew

field of view

s referred as the skew, usually

0

fkuu fkvv

image plane size and field of view are as-sumed to be fixed,but not fixed focal length

2.1.3 The External Parameters Matrix world coordinate to camera coordinate

]|[ tRThe 3x4 external pa-rameters

R rotation ma-trix

t translation vector

tRMM wc

wM in the world coordinate system

cM in the camera coordinate system

2.1.3 The External Parameters Matrix

2.1.4 Estimating the Camera Calibration Matrix

internal parameters are assumed to be fixed make use of a calibration pattern of known size

inside the field of view correspondence

between the 3D points and the 2D image points

2.1.5 Handling Lens Distortion

gentialradial duduuu tan

gentialradial dvdvvv tan

urkrkduradial )1( 42

21 radial distor-

tion

uvpvrp

urpuvpdu gential

222

1

2221

tan2)2(

)2(2tangential dis-tortion

(usually ig-nored)

22 vur

can be avoided by locally re-pametrizing the ro-tation

2.2 Camera Pose Parameteriza-tion

2.2.1 Euler Angles

cossin0

sincos0

001

cos0sin

010

sin0cos

100

0cossin

0sincos

R

α,β,γ to be rotation angles around the Z, Y, and X axis respec-tively yields

one rotation has no effect gimbal lock problem

2.2.2 Quaternions

2

1sin,

2

1cos wq

A rotation about the unit vector by an angle

w

a scalar plus a 3-vector),( va

2.2.3 Exponential Map

32

!3

1

!2

1)exp( I

A rotation about the unit vector by an angle

w

w

Let be a 3D vector

Tzyx wwww ,,

2.2.3 Exponential Map

is the skew-symmetric matrix

)( AAT

0

0

0

zy

xx

yz

ww

ww

ww

Rodrigues’ formula2ˆ)cos1(ˆ)(sin)exp()( IR

2)cos1

()sin()exp()(

IR

the exponential map represents a rotation as a 3-vector that gives its axis and magnitude.

iZ

2.2.4 Linearization of Small Rota-tions

RMM

MM

MI

)(

estimated camera positions

(when the internal parameters are known)

2.3 Estimating the External parameters Ma-trix

2.3.1 How many Correspondences are nec-essary?

n=3 known correspondencesproduce 4 possible solution(P3P Problem)

n>=4 known correspondencesproduce 2 possible solution

n>=4 known correspondences(points are coplanar)produce unique solution

n>=6 known correspondences produce unique solution

2.3.2 The Direct Linear Transformation (DLT)

to estimate the whole matrix P by solving a linear systemeven when the internal parameters are

not known Each correspondence

gives rise to two linearly independent equations

iiii

iii uPZPYPXP

PZPYPXP

34333231

14131211

iiii

iii vPZPYPXP

PZPYPXP

34333231

24232221

ii mM

iiii

iii uPZPYPXP

PZPYPXP

34333231

14131211

iiii

iii vPZPYPXP

PZPYPXP

34333231

24232221

03433323114131211 iiiiiiiiii uPuZPuYPuXPPZPYPXP

03433323114131211 iiiiiiiiii vPvZPvYPvXPPZPYPXP

010000

00001

34

11

P

P

vvZvYvXZYX

uuZuYuXZYX

iiiiiiiiii

iiiiiiiiii

0

34

11

P

P

B Stacking all the equation into B yields the lin-ear system :

2.3.2 The Direct Linear Transformation (DLT)

2.3.2 The Direct Linear Transformation (DLT)

6 correspondences must be known

for 3D tracking , using a calibrated cameraand estimating only its orientation and

position

0

34

11

P

P

B is the eigen vector of B corresponding to the smallest eigenvalue of B

]|[ tR

PKtR 1]|[

2.3.3 The Perspective-n-Point (PnP) Prob-lem

2.3.4 Pose estimation from a 3D Plane

The relation between a 3D plane and its image

projection can be represented by a homogeneous 3x3

matrix(homography matrix)

Let us consider the plane

0Z

MPm~~

1

1

1

0

21

321

Y

X

H

Y

X

tRRK

Y

X

tRRRK

2.3.4 Pose estimation from a 3D Plane

The matrix H can be estimatedfrom four correspondencesusing a DLT algorithm

the translation vec-tor

last column is given by the cross-product since the columns of R must be orthonormal

ii mM

tRRKH 21 t

3R 21 RR

2.3.5 non-Linear Reprojection Error

i

ii

tR

mMPdisttR ,~

]|[ 2

]|[

minarg

finding the pose that minimizes a sum of residual errors

2.4 Least-Squares Minimiza-tion Techniques ir

2.4.1 Linear Least-Squares

the function is linear

the camera pose parameters

the unknowns of a set of linear equationsin matrix form as

can be estimated as

f

p

TT AAAA 1)( pseudo-inverse of A

p bAp

bAp

2.4.2 Newton-Based Minimization Algo-rithms

the function is not linear

algorithms start from an initial estimate of the minimum and update it iteratively

is chosen to minimize the residual at itera-tion

and estimated by approximating to the first order

f

iii pp 1

0p

iiii Jpfpf )()(

i 1if

2.4.2 Newton-Based Minimization Algo-rithms

Jacobian matrixthe partial derivatives of all these functions

n

mm

n

x

y

x

y

x

y

x

y

J

1

1

1

1

mn RRf :

bpf ii

)(minarg

i

i

i

J

J

bJpf

minarg

)(minarg

iTT

i JJJ 1)(

iTT

i JIJJ 1)(

I stabilizes the begav-ior

inliers

data whose distribution can be explainedby some set of model parameters

outliers

which are data that do not fit the model

the data can be subject to noise

M-estimatorsgood at finding accurate solutionsrequire an initial estimate to converge correctly

RANSACdoes not require such an initial estimatedoes not take into account all the available datalacks precision

2.5 Robust Estimation

2.5.1 M-Estimators

least-squares estimation the assumption that the observations are in-

dependent and have a Gaussian distribution

Instead of minimizingi

ir2

ir are residual er-rors is an M-estimator that reduce the influence

of outliers

i

ir )(

2.5.1 M-Estimators

otherwisec

xc

cxifx

xHub

2

2)(

2

otherwisec

cxifc

xc

xTuk

6

116)(

2

322

Huber estimator

Tukey estimator

Huber estimator : linear to reduce the influence of large residual errors

Tukey estimator : flat so that large residual errors have no influ-ence at all

2.5.1 M-Estimators

2.5.2 RANSAC

samples of data points are randomly selected estimate model parameters find the subset of points (consistent with

the estimate) the largest is retained

and refined by least-squares minimization

N n

ip

SSi

iS

Sa set of mea-surements nthe model parameters require a

minimum of

2.5.2 RANSAC linear least-square esti-mation

2.5.2 RANSAC random sampling

2.5.2 RANSAC random sampling

2.5.2 RANSAC random sampling

2.5.2 RANSAC random sampling

estimating the density of successive states

in the space of possible camera poses.

2.6 Bayesian Trackingts

Thank you for your attention