affine structure from motion

ComputerVision

Affine structure from motion

Marc PollefeysCOMP 256

Some slides and illustrations from J. Ponce, A. Zisserman, R. Hartley, Luc Van Gool, …

ComputerVision

2

Last time: Optical Flow

0 tyx IvIuI uIx

It

Ixu

Ixu=It

Aperture problem:

two solutions:- regularize (smoothness prior)- constant over window (i.e. Lucas-Kanade)

Coarse-to-fine, parametric models, etc…

ComputerVision

3

Jan 16/18 - Introduction

Jan 23/25 Cameras Radiometry

Jan 30/Feb1 Sources & Shadows Color

Feb 6/8 Linear filters & edges Texture

Feb 13/15 Multi-View Geometry Stereo

Feb 20/22 Optical flow Project proposals

Feb27/Mar1 Affine SfM Projective SfM

Mar 6/8 Camera Calibration Silhouettes and Photoconsistency

Mar 13/15 Springbreak Springbreak

Mar 20/22 Segmentation Fitting

Mar 27/29 Prob. Segmentation Project Update

Apr 3/5 Tracking Tracking

Apr 10/12 Object Recognition Object Recognition

Apr 17/19 Range data Range data

Apr 24/26 Final project Final project

Tentative class schedule

ComputerVision

4

AFFINE STRUCTURE FROM MOTION

Reading: Chapter 12.

• The Affine Structure from Motion Problem• Elements of Affine Geometry• Affine Structure from Motion from two Views

• A Geometric Approach• Affine Epipolar Geometry• An Algebraic Approach

• Affine Structure from Motion from Multiple Views• From Affine to Euclidean Images• Structure from motion of multiple and deforming object

ComputerVision

5

Affine Structure from Motion

Reprinted with permission from “Affine Structure from Motion,” by J.J. (Koenderink and A.J.Van Doorn, Journal of the Optical Society of America A, 8:377-385 (1990). 1990 Optical Society of America.

Given m pictures of n points, can we recover• the three-dimensional configuration of these points?• the camera configurations?

(structure)(motion)

ComputerVision

6

Orthographic Projection

Parallel Projection

ComputerVision

7

Weak-Perspective Projection

Paraperspective Projection

ComputerVision

8

The Affine Structure-from-Motion Problem

Given m images of n fixed points P we can write

Problem: estimate the m 2x4 matrices M andthe n positions P from the mn correspondences p .

i

j ij

2mn equations in 8m+3n unknowns

Overconstrained problem, that can be solvedusing (non-linear) least squares!

j

ComputerVision

9

The Affine Ambiguity of Affine SFM

If M and P are solutions, i j

So are M’ and P’ wherei j

and

Q is an affinetransformation.

When the intrinsic and extrinsic parameters are unknown

ComputerVision

10

Affine Spaces: (Semi-Formal) Definition

ComputerVision

11

Example: R as an Affine Space2

ComputerVision

12

In General

The notation

is justified by the fact that choosing some origin O in Xallows us to identify the point P with the vector OP.

Warning: P+u and Q-P are defined independently of O!!

ComputerVision

13

Barycentric Combinations

• Can we add points? R=P+Q NO!

• But, when we can define

• Note:

ComputerVision

14

Affine Subspaces

ComputerVision

15

Affine Coordinates

• Coordinate system for U:

• Coordinate system for Y=O+U:

• Coordinate system for Y:

• Affine coordinates:

• Barycentric coordinates:

ComputerVision

16

When do m+1 points define a p-dimensional subspace Y of an n-dimensional affine space X equipped with some coordinate frame basis?

Writing that all minors of size (p+2)x(p+2) of D are equal to zero gives the equations of Y.

Rank ( D ) = p+1, where

ComputerVision

17

Affine Transformations

Bijections from X to Y that:• map m-dimensional subspaces of X onto m-dimensional subspaces of Y;• map parallel subspaces onto parallel subspaces; and• preserve affine (or barycentric) coordinates.

In E they are combinations of rigid transformations, non-uniform scalings and shears.

Bijections from X to Y that:• map lines of X onto lines of Y; and• preserve the ratios of signed lengths of line segments.

3

ComputerVision

18

Affine Transformations II

• Given two affine spaces X and Y of dimension m, and two coordinate frames (A) and (B) for these spaces, there exists a unique affine transformation mapping (A) onto (B).

• Given an affine transformation from X to Y, one can always write:

• When coordinate frames have been chosen for X and Y,this translates into:

ComputerVision

19

Affine projections induce affine transformations from planesonto their images.

ComputerVision

20

Affine Shape

Two point sets S and S’ in some affine space X are affinely equivalent when there exists an affine transformation : X X such that X’ = ( X ).

Affine structure from motion = affine shape recovery.

= recovery of the corresponding motion equivalence classes.

ComputerVision

21

Geometric affine scene reconstruction from two images(Koenderink and Van Doorn, 1991).

ComputerVision

22

Affine Structure from Motion

(Koenderink and Van Doorn, 1991)

Reprinted with permission from “Affine Structure from Motion,” by J.J. (Koenderink and A.J.Van Doorn, Journal of the Optical Society of America A, 8:377-385 (1990). 1990 Optical Society of America.

ComputerVision

23

The Affine Epipolar Constraint

Note: the epipolar lines are parallel.

ComputerVision

24

Affine Epipolar Geometry

ComputerVision

25

The Affine Fundamental Matrix

where

ComputerVision

26

An Affine Trick.. Algebraic Scene Reconstruction

ComputerVision

27

The Affine Structure of Affine Images

Suppose we observe a scene with m fixed cameras..

The set of all images of a fixed scene is a 3D affine space!

ComputerVision

28

has rank 4!

ComputerVision

29

From Affine to Vectorial Structure

Idea: pick one of the points (or their center of mass)as the origin.

ComputerVision

30

What if we could factorize D? (Tomasi and Kanade, 1992)

Affine SFM is solved!

Singular Value Decomposition

We can take

ComputerVision

31

From uncalibrated to calibrated cameras

Weak-perspective camera:

Calibrated camera:

Problem: what is Q ?

Note: Absolute scale cannot be recovered. The Euclidean shape(defined up to an arbitrary similitude) is recovered.

ComputerVision

32

Reconstruction Results (Tomasi and Kanade, 1992)

Reprinted from “Factoring Image Sequences into Shape and Motion,” by C. Tomasi andT. Kanade, Proc. IEEE Workshop on Visual Motion (1991). 1991 IEEE.

ComputerVision

33

More examples

Tomasi Kanade’92,Poelman & Kanade’94

ComputerVision

34

More examples

Tomasi Kanade’92,Poelman & Kanade’94

ComputerVision

35

More examples

Tomasi Kanade’92,Poelman & Kanade’94

ComputerVision

36

Further Factorization work

Factorization with uncertainty

Factorization for indep. moving objects (now)

Factorization for articulated objects (now)

Factorization for dynamic objects (now)

Perspective factorization (next week)

Factorization with outliers and missing pts.

(Irani & Anandan, IJCV’02)

(Costeira and Kanade ‘94)

(Bregler et al. 2000, Brand 2001)

(Jacobs ‘97 (affine), Martinek & Pajdla‘01 Aanaes’02 (perspective))

(Sturm & Triggs 1996, …)

(Yan and Pollefeys ‘05)

ComputerVision

37

Structure from motion of multiple moving objects

ComputerVision

38

Structure from motion of multiple moving objects

ComputerVision

39

Shape interaction matrixShape interaction matrix for articulated

objects looses block diagonal structure

Costeira and Kanade’s approach is not usable for articulated bodies(assumes independent motions)

ComputerVision

40

Articulated motion subspaces

Joint (1D intersection)

Hinge (2D intersection)

(joint=origin)

(hinge=z-axis)

Motion subspaces for articulated bodies intersect

(rank=8-1)

(rank=8-2)

(Yan and Pollefeys, CVPR’05)(Tresadern and Reid, CVPR’05)

Exploit rank constraint to obtain better estimate

(Yan & Pollefeys, 06?)Also for non-rigid parts if

ComputerVision

41

Student Segmentation

Intersection

Results

Toy truckSegmentation

Intersection

ComputerVision

42

Articulated shape and motion factorization

Automated kinematic chain building for articulated & non-rigid obj.– Estimate principal angles between subspaces – Compute affinities based on principal angles– Compute minimum spanning tree

(Yan and Pollefeys, 2006?)

ComputerVision

43

Structure from motion of deforming objects

Extend factorization approaches to deal with dynamic shapes

(Bregler et al ’00; Brand ‘01)

ComputerVision

44

Representing dynamic shapes

represent dynamic shape as varying linear combination of basis shapes

k

kk (t)ScS(t)

(fig. M.Brand)

ComputerVision

45

Projecting dynamic shapes

PtScR k

kk

(figs. M.Brand)Rewrite:

ComputerVision

46

Dynamic image sequences

One image:

Multiple images(figs. M.Brand)

ComputerVision

47

Dynamic SfM factorization?

Problem: find J so that M has proper structure

ComputerVision

48

Dynamic SfM factorization

(Bregler et al ’00)

Assumption: SVD preserves order and orientation of basis shape components

ComputerVision

49

Results(Bregler et al ’00)

ComputerVision

50

Dynamic SfM factorization(Brand ’01)

constraints to be satisfied for M

constraints to be satisfied for M, use to compute J

hard!

(different methods are possible, not so simple and also not optimal)

ComputerVision

51

Non-rigid 3D subspace flow

• Same is also possible using optical flow in stead of features, also takes uncertainty into account

(Brand ’01)

ComputerVision

52

Results (Brand ’01)

ComputerVision

53

(Brand ’01)

Results

ComputerVision

54

Results(Bregler et al ’01)

ComputerVision

55

Next class: Projective structure from motion

Reading: Chapter 13