![Page 1: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/1.jpg)
Last Two Lectures
Panoramic Image Stitching
Feature Detection and Matching
![Page 2: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/2.jpg)
Today
More on Mosaic
Projective Geometry
Single View Modeling
Vermeer’s Music LessonReconstructions by Criminisi et al.
![Page 3: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/3.jpg)
Image Alignment
Feature Detection and Matching
Cylinder:
Translation
2 DoF
Plane:
Homography
8 DoF
![Page 4: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/4.jpg)
Plane perspective mosaics
– 8-parameter generalization of affine motion
• works for pure rotation or planar surfaces
– Limitations:
• local minima
• slow convergence
![Page 5: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/5.jpg)
Revisit Homography
x
(Xc,Yc,Zc)
xcfZ
Y
X
yf
xf
y
x
c
c
100
0
0
~
1
1
1
Z
Y
X
yf
xf
y
x
c
c
R
100
0
0
~
1
2
2
21
1 ~ xxKRK
![Page 6: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/6.jpg)
Estimate f from H?
x
(Xc,Yc,Zc)
xcfZ
Y
X
f
f
yy
xx
c
c
100
00
00
~
1
1
1
1
1
Z
Y
X
f
f
yy
xx
c
c
R
100
00
00
~
1
2
2
2
2
21
1
12 ~)( xxRKK
H
{
1
222
1
1
1
1
2
***
/
/
~
f
fjfifh
fged
fcba
HKKR
??, 21 ff
![Page 7: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/7.jpg)
The drifting problem
• Error accumulation
– small errors accumulate over time
![Page 8: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/8.jpg)
Bundle Adjustment
Associate each image i withiK iR
Each image i has features ijp
Trying to minimize total matching residuals
),(
211~) and all(
mi j
mjmmiiijiifE pKRRKpR
![Page 9: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/9.jpg)
Rotations
• How do we represent rotation matrices?
1. Axis / angle (n,θ)
R = I + sinθ [n] + (1- cosθ) [n] 2
(Rodriguez Formula), with
[n] be the cross product matrix.
![Page 10: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/10.jpg)
Incremental rotation update
1. Small angle approximation
ΔR = I + sinθ [n] + (1- cosθ) [n] 2
≈ I +θ [n] = I+[ω]
linear in ω= θn
2. Update original R matrix
R ← R ΔR
![Page 11: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/11.jpg)
Recognizing Panoramas
[Brown & Lowe, ICCV’03]
![Page 12: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/12.jpg)
Finding the panoramas
![Page 13: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/13.jpg)
Finding the panoramas
![Page 14: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/14.jpg)
Algorithm overview
![Page 15: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/15.jpg)
Algorithm overview
![Page 16: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/16.jpg)
Algorithm overview
![Page 17: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/17.jpg)
Algorithm overview
![Page 18: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/18.jpg)
Algorithm overview
![Page 19: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/19.jpg)
Algorithm overview
![Page 20: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/20.jpg)
Finding the panoramas
![Page 21: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/21.jpg)
Finding the panoramas
![Page 22: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/22.jpg)
Algorithm overview
![Page 23: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/23.jpg)
Algorithm overview
![Page 24: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/24.jpg)
Algorithm overview
![Page 25: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/25.jpg)
Get you own copy!
[Brown & Lowe, ICCV 2003]
[Brown, Szeliski, Winder, CVPR’05]
![Page 26: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/26.jpg)
How well does this work?
Test on 100s of examples…
![Page 27: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/27.jpg)
How well does this work?
Test on 100s of examples…
…still too many failures (5-10%)
for consumer application
![Page 28: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/28.jpg)
Matching Mistakes: False Positive
![Page 29: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/29.jpg)
Matching Mistakes: False Positive
![Page 30: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/30.jpg)
Matching Mistakes: False Negative
• Moving objects: large areas of disagreement
![Page 31: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/31.jpg)
Matching Mistakes
• Accidental alignment
– repeated / similar regions
• Failed alignments
– moving objects / parallax
– low overlap
– “feature-less” regions
• No 100% reliable
algorithm?
![Page 32: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/32.jpg)
How can we fix these?
• Tune the feature detector
• Tune the feature matcher (cost metric)
• Tune the RANSAC stage (motion model)
• Tune the verification stage
• Use “higher-level” knowledge
– e.g., typical camera motions
• → Sounds like a big “learning” problem
– Need a large training/test data set (panoramas)
![Page 33: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/33.jpg)
Enough of images!
We want more
from the image
We want real 3D
scene
walk-throughs:
Camera rotation
Camera
translation
on to 3D…
![Page 34: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/34.jpg)
So, what can we do here?
• Model the scene
as a set of
planes!
![Page 35: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/35.jpg)
(0,0,0)
The projective plane
• Why do we need homogeneous coordinates?
– represent points at infinity, homographies, perspective projection, multi-view relationships
• What is the geometric intuition?
– a point in the image is a ray in projective space
(sx,sy,s)
• Each point (x,y) on the plane is represented by a ray (sx,sy,s)
– all points on the ray are equivalent: (x, y, 1) (sx, sy, s)
image plane
(x,y,1)
y
xz
![Page 36: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/36.jpg)
Projective lines
• What does a line in the image correspond to in
projective space?
• A line is a plane of rays through origin
– all rays (x,y,z) satisfying: ax + by + cz = 0
z
y
x
cba0 :notationvectorin
• A line is also represented as a homogeneous 3-vector l
l p
![Page 37: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/37.jpg)
l
Point and line duality
– A line l is a homogeneous 3-vector
– It is to every point (ray) p on the line: l p=0
p1p2
What is the intersection of two lines l1 and l2 ?
• p is to l1 and l2 p = l1 l2
Points and lines are dual in projective space
• can switch the meanings of points and lines to get another
formula
l1
l2
p
What is the line l spanned by rays p1 and p2 ?
• l is to p1 and p2 l = p1 p2
• l is the plane normal
![Page 38: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/38.jpg)
Ideal points and lines
• Ideal point (“point at infinity”)
– p (x, y, 0) – parallel to image plane
– It has infinite image coordinates
(sx,sy,0)y
x
z image plane
Ideal line
• l (a, b, 0) – parallel to image plane
(a,b,0)
y
x
z image plane
• Corresponds to a line in the image (finite coordinates)
![Page 39: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/39.jpg)
Homographies of points and lines
• Computed by 3x3 matrix multiplication– To transform a point: p’ = Hp
– To transform a line: lp=0 l’p’=0– 0 = lp = lH-1Hp = lH-1p’ l’ = lH-1
– lines are transformed by postmultiplication of H-1
![Page 40: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/40.jpg)
3D projective geometry
• These concepts generalize naturally to
3D
– Homogeneous coordinates
• Projective 3D points have four coords: P =
(X,Y,Z,W)
– Duality
• A plane N is also represented by a 4-vector
• Points and planes are dual in 4D: N P=0
– Projective transformations
• Represented by 4x4 matrices T: P’ = TP, N’
= N T-1
![Page 41: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/41.jpg)
3D to 2D: “perspective” projection
• Matrix Projection: ΠPp1
************
ZYX
w
wywx
What is not preserved under perspective projection?
What IS preserved?
![Page 42: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/42.jpg)
Vanishing points
• Vanishing point
– projection of a point at infinity
image plane
cameracenter
ground plane
vanishing point
![Page 43: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/43.jpg)
Vanishing points (2D)
image plane
cameracenter
line on ground plane
vanishing point
![Page 44: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/44.jpg)
Vanishing points
• Properties
– Any two parallel lines have the same vanishing point v
– The ray from C through v is parallel to the lines
– An image may have more than one vanishing point• in fact every pixel is a potential vanishing point
image plane
cameracenter
C
line on ground plane
vanishing point V
line on ground plane
![Page 45: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/45.jpg)
Vanishing lines
• Multiple Vanishing Points
– Any set of parallel lines on the plane define a vanishing point
– The union of all of these vanishing points is the horizon line• also called vanishing line
– Note that different planes define different vanishing lines
v1 v2
![Page 46: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/46.jpg)
Vanishing lines
• Multiple Vanishing Points
– Any set of parallel lines on the plane define a vanishing point
– The union of all of these vanishing points is the horizon line• also called vanishing line
– Note that different planes define different vanishing lines
![Page 47: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/47.jpg)
Computing vanishing points
• Properties
– P is a point at infinity, v is its projection
– They depend only on line direction
– Parallel lines P0 + tD, P1 + tD intersect at P
V
DPP t0
0/1
/
/
/
1
Z
Y
X
ZZ
YY
XX
ZZ
YY
XX
tD
D
D
t
t
DtP
DtP
DtP
tDP
tDP
tDP
PP
ΠPv
P0
D
![Page 48: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/48.jpg)
Computing vanishing lines
• Properties– l is intersection of horizontal plane through C with image plane
– Compute l from two sets of parallel lines on ground plane
– All points at same height as C project to l
• points higher than C project above l
– Provides way of comparing height of objects in the scene
ground plane
lC
![Page 49: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/49.jpg)
![Page 50: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/50.jpg)
Fun with vanishing points
![Page 51: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/51.jpg)
Perspective cues
![Page 52: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/52.jpg)
Perspective cues
![Page 53: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/53.jpg)
Perspective cues
![Page 54: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/54.jpg)
Comparing heights
Vanishing
Point
![Page 55: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/55.jpg)
Measuring height
1
2
3
4
55.4
2.8
3.3
Camera height
![Page 56: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/56.jpg)
q1
Computing vanishing points (from lines)
• Intersect p1q1 with p2q2
v
p1
p2
q2
Least squares version• Better to use more than two lines and compute the “closest” point of
intersection
• See notes by Bob Collins for one good way of doing this:
– http://www-2.cs.cmu.edu/~ph/869/www/notes/vanishing.txt
![Page 57: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/57.jpg)
C
Measuring height without a ruler
ground plane
Compute Z from image measurements
• Need more than vanishing points to do this
Z
![Page 58: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/58.jpg)
The cross ratio
• A Projective Invariant
– Something that does not change under projective transformations
(including perspective projection)
P1
P2
P3
P4
1423
2413
PPPP
PPPP
The cross-ratio of 4 collinear points
Can permute the point ordering
• 4! = 24 different orders (but only 6 distinct values)
This is the fundamental invariant of projective geometry
1
i
i
i
iZ
Y
X
P
3421
2431
PPPP
PPPP
![Page 59: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/59.jpg)
vZ
r
t
b
tvbr
rvbt
Z
Z
image cross ratio
Measuring height
B (bottom of object)
T (top of object)
R (reference point)
ground plane
HC
TBR
RBT
scene cross ratio
1
Z
Y
X
P
1
y
x
pscene points represented as image points as
R
H
R
H
R
![Page 60: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/60.jpg)
Measuring height
RH
vz
r
b
t
R
H
Z
Z
tvbr
rvbt
image cross ratio
H
b0
t0
vvx vy
vanishing line (horizon)
![Page 61: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/61.jpg)
Measuring height vz
r
b
t0
vx vy
vanishing line (horizon)
v
t0
m0
What if the point on the ground plane b0 is not known?
• Here the guy is standing on the box, height of box is known
• Use one side of the box to help find b0 as shown above
b0
t1
b1
![Page 62: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/62.jpg)
Computing (X,Y,Z) coordinates
• Okay, we know how to compute height (Z
coords)
– how can we compute X, Y?
![Page 63: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/63.jpg)
Camera calibration
• Goal: estimate the camera parameters
– Version 1: solve for projection matrix
ΠXx1
************
ZYX
w
wywx
• Version 2: solve for camera parameters separately
– intrinsics (focal length, principle point, pixel size)
– extrinsics (rotation angles, translation)
– radial distortion
![Page 64: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/64.jpg)
Vanishing points and projection matrix
************
Π4321 ππππ
1π 2π 3π 4π
T00011 Ππ = vx (X vanishing point)
Z3Y2 ,similarly, vπvπ
origin worldof projection10004
TΠπ
ovvvΠ ZYX
Not So Fast! We only know v’s up to a scale factor
ovvvΠ ZYX cba
• Can fully specify by providing 3 reference points
![Page 65: Panoramic Image Stitching - University of Wisconsin–Madison](https://reader031.vdocuments.mx/reader031/viewer/2022021307/620753ef49d709492c30566b/html5/thumbnails/65.jpg)
3D Modeling from a photograph
https://research.microsoft.com/vision/cambridge/3d/3dart.htm