image registration and 3d reconstruction · image registration and 3d reconstruction in computer...
TRANSCRIPT
![Page 1: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/1.jpg)
Image Registration and 3D Reconstruction in Computer Vision
Adrien Bartoli et al.Clermont Université, France
![Page 2: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/2.jpg)
![Page 3: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/3.jpg)
![Page 4: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/4.jpg)
![Page 5: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/5.jpg)
![Page 6: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/6.jpg)
Image Understanding
Object detection
Rendering
Augmented realityView morphing
Object recognition
Bill Murray
Scarlett Johansson
Special effects, motion capture, …
Photogrammetry
![Page 7: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/7.jpg)
Real world
ImagesVisual process
Prior knowledge
Scene description
“Vision is the act of knowing what is where by looking.”- Aristotle
![Page 8: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/8.jpg)
Computer Vision
Does not study biological vision
Handles various types of camera
From [Baker et al., CVPR’01]
Uses simple models
![Page 9: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/9.jpg)
3D Computer Vision: Shape-from-X
Shading Focus Shadows
Texture
Occlusions, silhouettes, …
![Page 10: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/10.jpg)
Techniques with Multiple Views
Photometric stereo
Structure-from-Motion - SfM
From [Higo et al. ICCV’09]
Image formation
![Page 11: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/11.jpg)
SfM – Structure-from-Motion
![Page 12: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/12.jpg)
SfM has Two Main Steps
Image registration 3D reconstruction
![Page 13: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/13.jpg)
Rigid vs Deformable SfM
Image registration 3D reconstruction
Image registration 3D reconstruction
![Page 14: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/14.jpg)
Why is Registration Difficult?
![Page 15: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/15.jpg)
Rigid SfM: A Mature Paradigm
Inputs: a set of images
1 – registration
2 – projective reconstruction
3 – camera self-calibration
4 – surface reconstruction
Outputs: camera parameters and scene structure
![Page 16: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/16.jpg)
Keypoint Detection and Matching
• Feature-based methods: abstract the images by features
• Keypoints lie at local maxima in intensity variations
• They should be stable under change of viewpoint
• They are matched by comparing invariant descriptors
Detected keypoints(with scale)
Matched keypoints(using winner-takes-all)
![Page 17: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/17.jpg)
Robust Estimation Methods• Objective: estimate the model parameters and classify
each datum as valid or erroneous
• This is a chicken-and-egg problem
• RANSAC (Random Sample Consensus)
– Handles more than 50% outliers in the data
– Basic idea: sample minimal data sets and maximize the support
Homography based panorama
Inliers Outliers
![Page 18: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/18.jpg)
Rigid SfM: A Mature Paradigm
Inputs: a set of images
1 – registration
2 – projective reconstruction
3 – camera self-calibration
4 – surface reconstruction
Outputs: camera parameters and scene structure
![Page 19: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/19.jpg)
The Projective Camera
=Imagepoint
Intrinsics Extrinsics
3D point
3D pointRetina point
Image pointCentre of projection
Linear in homogeneous coordinates
=1
Rational in euclidean coordinates
![Page 20: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/20.jpg)
Projective Geometry
![Page 21: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/21.jpg)
3D Reconstruction
• The basic constraint: intersection of the viewing rays for matching points
A camera
The scene
surface
![Page 22: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/22.jpg)
![Page 23: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/23.jpg)
Projective Reconstruction
• Nonlinear optimization problem
• Initialization– Split the image set
– Reconstruct from 2 – 4 views
– Merge the reconstructions
• Refinement: bundle adjustment
=
![Page 24: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/24.jpg)
Camera Self-Calibration
• Uses constraints on the intrinsics, such as
– Constant intrinsics for all cameras
– Known intrinsics, except the (varying) focal length
• Linear or nonlinear least squares
• Why using camera self-calibration?
– Flexibility: no prior knowledge on structure and extrinsics
– Simplicity: projective is easier than metric
• Bundle adjustment
=
![Page 25: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/25.jpg)
Bundle Adjustment
• Minimization of the reprojection error• Thousands of unknowns• Exploits the problem structure
![Page 26: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/26.jpg)
Rigid SfM: A Mature Paradigm
Inputs: a set of images
1 – registration
2 – projective reconstruction
3 – camera self-calibration
4 – surface reconstruction
Outputs: camera parameters and scene structure
![Page 27: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/27.jpg)
Dense 3D Reconstruction• Based on geometric primitives (planes, cubes, etc)
Keypoint detectionand matching 3D reconstruction
Piecewise planarsegmentation
![Page 28: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/28.jpg)
Dense 3D Reconstruction
• Stereopsis, voxel colouring
• Based on matching colours between the images
One of two images Disparity map
One of many images 3D model
Results from[Boykov et al, PAMI’04]
Results from[Furukawa et al, CVPR’07]
![Page 29: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/29.jpg)
Augmentation
![Page 30: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/30.jpg)
Deformable Structure-from-Motion
Time
• The viewing rays for matching points do not generally meet
• The problem is generally ill-posed
![Page 31: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/31.jpg)
Deformable Structure-from-Motion
• Priors on the structure– Single object
– Empirical: smoothness, etc
– Physical: inextensibility, developability, mechanical models, etc
– Statistical: face, body, etc
Image registration 3D reconstruction
![Page 32: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/32.jpg)
Deformable Surface Registration
• Find transformations that make the input images similar
• Geometric transformation: changes pixel locations
• Photometric transformation: changes pixel colours
• Sequential registration
Template
![Page 33: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/33.jpg)
Deformable Surface RegistrationCurrent image
Template
Unknown transformation
Transformedimage
Differenceimage
2
→ minimization over the transformation parameters
![Page 34: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/34.jpg)
Explicit Photometric Transformation
No photometric model Affine photometric model
Template Current image
Videos of the difference image
![Page 35: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/35.jpg)
Other Registration Results and Methods
Curve-basedregistration
Retexturing
![Page 36: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/36.jpg)
Other Registration Results and Methods
Fluid flow registration
Sea breeze tracking
Computation of mechanical constants
![Page 37: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/37.jpg)
Retargetting
![Page 38: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/38.jpg)
Deformable Surface 3D Reconstruction
• Highly dependent on the surface (cloth, paper, skin, etc)
• Example algorithm for isometric surfaces d
Template Current image
d
![Page 39: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/39.jpg)
Isometric Deformable 3D Reconstruction
focal plane
1. Compute the upper bounds for each pair of points
2. Keep the tightest upper bounds
3. Recompute the bounds and loop to 2
4. Interpolate the upper bounds to get a surface
Side view
![Page 40: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/40.jpg)
![Page 41: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/41.jpg)
![Page 42: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/42.jpg)
Summary for Rigid Structure-from-Motion
• Mature for textured environments
• Camera and ‘sparse’ structure recovery, self-calibration
• Companies: 2d3 (University of Oxford then Vicon), RealViz (INRIA then Autodesk), etc
• Textbooks: [Hartley et al, 03 ; Faugeras et al, 01 ; Forsyth et al, 03]
Image registration 3D reconstruction
SfM
![Page 43: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/43.jpg)
Summary for Deformable Structure-from-Motion
• Some results in the ‘surface’ case for registration and 3D reconstruction
• Image registration for fluid flow
• The general case is completely open
• Prior knowledge on the environment is required
Image registration 3D reconstruction
![Page 44: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/44.jpg)
Bibliography• Keypoint detection, description and matching:
[Lowe, IJCV’04 ; Mikolajczyk et al, IJCV’05]• Robust estimation (RANSAC): [Fischler et al, CVGIP’81]• Projective reconstruction: [Nister, PhD-Thesis’00]• Bundle adjustment: [Triggs et al, VA’00]• Self-calibration: [Gurdjos et al, ICCV’09]• (Rigid) dense stereo:
[Boykov et al, PAMI’04 ; Strecha et al, CVPR’08]• Deformable surface registration:
[Gay-Bellile et al, PAMI’10; Pilet et al, IJCV’07]• Deformable surface reconstruction:
[Perriollat et al, IJCV’10; Salzmann et al, ICCV’09]
![Page 45: Image Registration and 3D Reconstruction · Image Registration and 3D Reconstruction in Computer Vision Adrien Bartoli et al. Clermont Université, France ... abstract the images](https://reader036.vdocuments.mx/reader036/viewer/2022071404/60f81cd15284c870f819c05e/html5/thumbnails/45.jpg)
Image Registration and 3D Reconstruction in Computer Vision
Adrien Bartoli et al.Clermont Université, France