structure from motion€¦ · orthogonal structure from motion •let’s recap: – write down...
TRANSCRIPT
![Page 1: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/1.jpg)
Structure from Motion
![Page 2: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/2.jpg)
Structure from Motion
• For now, static scene and moving camera – Equivalently, rigidly moving scene and
static camera
• Limiting case of stereo with many cameras
• Limiting case of multiview camera calibration with unknown target
• Given n points and N camera positions, have 2nN equations and 3n+6N unknowns
![Page 3: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/3.jpg)
Approaches
• Obtaining point correspondences – Optical flow
– Stereo methods: correlation, feature matching
• Solving for points and camera motion – Nonlinear minimization (bundle adjustment)
– Various approximations…
![Page 4: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/4.jpg)
Orthographic Approximation
• Simplest SFM case: camera approximated by orthographic projection
Perspective Orthographic
![Page 5: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/5.jpg)
Weak Perspective
• An orthographic assumption is sometimes well approximated by a telephoto lens
Weak Perspective
![Page 6: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/6.jpg)
Consequences of Orthographic Projection
• Translation perpendicular to image plane cannot be recovered
• Scene can be recovered up to scale (if weak perspective)
![Page 7: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/7.jpg)
Orthographic Structure from Motion
• Method due to Tomasi & Kanade, 1992
• Assume n points in 3D space p1 .. pn
• Observed at N points in time at image coordinates (xij, yij), i = 1..N, j=1..n – Feature tracking, optical flow, etc.
– All points visible in all frames
![Page 8: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/8.jpg)
Orthographic Structure from Motion
• Write down matrix of data
=
NnN
n
NnN
n
yy
yyxx
xx
1
111
1
111
D
Points → Fram
es →
Frames →
![Page 9: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/9.jpg)
Orthographic Structure from Motion
• Step 1: find translation
• Translation perpendicular to viewing direction cannot be obtained
• Translation parallel to viewing direction equals motion of average position of all points
![Page 10: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/10.jpg)
Orthographic Structure from Motion
• After finding translation, subtract it out (i.e., subtract average of each row)
−−
−−−−
−−
=
NNnNN
n
NNnNN
n
yyyy
yyyyxxxx
xxxx
1
11111
1
11111
~D
![Page 11: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/11.jpg)
Orthographic Structure from Motion
• Step 2: try to find rotation
• Rotation at each frame defines local coordinate axes , , and
• Then
i j k
jiijjiij yx pjpi ~ˆ~,~ˆ~ ⋅=⋅=
![Page 12: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/12.jpg)
Orthographic Structure from Motion
• So, can write where R is a “rotation” matrix and S is a “shape” matrix
RSD =~
[ ]n
N
N ppS
j
ji
i
R ~~
ˆ
ˆˆ
ˆ
1
T
T1
T
T1
=
=
−−
−−−−
−−
=
NNnNN
n
NNnNN
n
yyyy
yyyyxxxx
xxxx
1
11111
1
11111
~D
![Page 13: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/13.jpg)
Orthographic Structure from Motion
• Goal is to factor
• Before we do, observe that rank( ) should be 3 (in ideal case with no noise)
• Proof: – Rank of R is 3 unless no rotation
– Rank of S is 3 iff have noncoplanar points
– Product of 2 matrices of rank 3 has rank 3
• With noise, rank( ) might be > 3
D~
D~
D~
![Page 14: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/14.jpg)
SVD
• Goal is to factor into R and S
• Apply SVD:
• But should have rank 3 ⇒ all but 3 of the wi should be 0
• Extract the top 3 wi, together with the corresponding columns of U and V
D~
T~ UWVD =
D~
![Page 15: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/15.jpg)
Factoring for Orthographic Structure from Motion
• After extracting columns, U3 has dimensions 2N×3 (just what we wanted for R)
• W3V3T has dimensions 3×n (just what we
wanted for S)
• So, let R*=U3, S*=W3V3T
![Page 16: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/16.jpg)
Affine Structure from Motion
• The i and j entries of R* are not, in general, unit length and perpendicular
• We have found motion (and therefore shape) up to an affine transformation
• This is the best we could do if we didn’t assume orthographic camera
![Page 17: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/17.jpg)
Ensuring Orthogonality
• Since can be factored as R* S*, it can also be factored as (R*Q)(Q-1S*), for any Q
• So, search for Q such that R = R* Q has the properties we want
D~
![Page 18: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/18.jpg)
Ensuring Orthogonality
• Want or
• Let T = QQT
• Equations for elements of T – solve by least squares
• Ambiguity – add constraints
( ) ( ) 1ˆˆ T*T* =⋅ QiQi ii
0ˆˆ1ˆˆ1ˆˆ
*TT*
*TT*
*TT*
=
=
=
ii
ii
ii
jQQi
jQQj
iQQi
=
=
010
ˆ,001
ˆ *1
T*1
T jQiQ
![Page 19: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/19.jpg)
Ensuring Orthogonality
• Have found T = QQT
• Find Q by taking “square root” of T – Cholesky decomposition if T is positive definite
– General algorithms (e.g. sqrtm in Matlab)
![Page 20: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/20.jpg)
Orthogonal Structure from Motion
• Let’s recap: – Write down matrix of observations
– Find translation from avg. position
– Subtract translation
– Factor matrix using SVD
– Write down equations for orthogonalization
– Solve using least squares, square root
• At end, get matrix R = R* Q of camera positions and matrix S = Q-1S* of 3D points
![Page 21: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/21.jpg)
Results
• Image sequence
[Tomasi & Kanade]
![Page 22: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/22.jpg)
Results
• Tracked features
[Tomasi & Kanade]
![Page 23: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/23.jpg)
Results
• Reconstructed shape
[Tomasi & Kanade]
Front view Top view
![Page 24: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/24.jpg)
Orthographic → Perspective
• With orthographic or “weak perspective” can’t recover all information
• With full perspective, can recover more information (translation along optical axis)
• Result: can recover geometry and full motion up to global scale factor
![Page 25: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/25.jpg)
Perspective SFM Methods
• Bundle adjustment (full nonlinear minimization)
• Methods based on factorization
• Methods based on fundamental matrices
• Methods based on vanishing points
![Page 26: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/26.jpg)
Motion Field for Camera Motion
• Translation:
• Motion field lines converge (possibly at ∞)
![Page 27: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/27.jpg)
Motion Field for Camera Motion
• Rotation:
• Motion field lines do not converge
![Page 28: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/28.jpg)
Motion Field for Camera Motion
• Combined rotation and translation: motion field lines have component that converges, and component that does not
• Algorithms can look for vanishing point, then determine component of motion around this point
• “Focus of expansion / contraction”
• “Instantaneous epipole”
![Page 29: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/29.jpg)
Finding Instantaneous Epipole
• Observation: motion field due to translation depends on depth of points
• Motion field due to rotation does not
• Idea: compute difference between motion of a point, motion of neighbors
• Differences point towards instantaneous epipole
![Page 30: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/30.jpg)
SVD (Again!)
• Want to fit direction to all ∆v (differences in optical flow) within some neighborhood
• PCA on matrix of ∆v
• Equivalently, take eigenvector of A = Σ(∆v)(∆v)T
corresponding to largest eigenvalue
• Gives direction of parallax li in that patch, together with estimate of reliability
![Page 31: Structure from Motion€¦ · Orthogonal Structure from Motion •Let’s recap: – Write down matrix of observations – Find translation from avg. position – Subtract translation](https://reader033.vdocuments.mx/reader033/viewer/2022042807/5f82ce8ffef6633fe2055271/html5/thumbnails/31.jpg)
SFM Algorithm
• Compute optical flow
• Find vanishing point (least squares solution)
• Find direction of translation from epipole
• Find perpendicular component of motion
• Find velocity, axis of rotation
• Find depths of points (up to global scale)