photographing long scenes with multi-viewpoint panoramaslazebnik/research/fall08/xiaowei_li.pdf ·...
TRANSCRIPT
![Page 1: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/1.jpg)
Photographing Long Scenes with Multi-Viewpoint Panoramas
Agarwala, M. Agrawala, M. Cohen, D. Salesin, R. Szeliski
SIGGRAPH 2006
Presented by Xiaowei Li
![Page 2: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/2.jpg)
Keywords in the Title
•Multi-ViewpointSingle-Viewpoint
•Panoramas•Long Scenes
![Page 3: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/3.jpg)
Single-Viewpoint
One “camera”, one shot;Unique perspective rule on one picture.
![Page 4: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/4.jpg)
Single-Viewpoint
Ancient artists knew this.
![Page 5: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/5.jpg)
Multi-Viewpoint
• Just many Single Viewpoint ...
• Inside one picture, different portion has different perspective rules.
• In this paper, many single-viewpoint photos rendered in one picture naturally.
![Page 6: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/6.jpg)
Panoramas
• Strip Panoramas
• Single-Viewpoint Panoramas
• Multi-Viewpoint Panoramas
![Page 7: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/7.jpg)
Panoramas• Strip Panoramas
- also known as “Slit Scan”. - pushbroom cameras/ 1D camera- satellite images.
- also can obtained by sampling normal 2D image sequences [Zheng2003, Levin2005]. Vertical pixel strips from each image in the sequence.
![Page 8: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/8.jpg)
Panoramas• Strip Panoramas by sampling image
sequences [Zheng 2003]
![Page 9: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/9.jpg)
Panoramas• Strip Panoramas by sampling image
sequencesOrthographic projection along horizontal axis;
Perspective projection along vertical axis.
Main ProblemDifferent aspect ratio at different depth->closer squashed; further stretched.
![Page 10: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/10.jpg)
Panoramas• Strip Panoramas by sampling image
sequencesMain Problem
Different aspect ratio at different depth->closer squashed; further stretched.
many adaptive or interactive method to choose different width for pixel strip for objects at different depth. however, still open problem.
![Page 11: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/11.jpg)
Panoramas•Strip Panoramas by sampling image sequences
Main ProblemDifferent aspect ratio at different depth->closer squashed; further stretched.
![Page 12: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/12.jpg)
Panoramas•Strip Panoramas by sampling image sequences
by different slit method [Roman 04, thesis 06]
![Page 13: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/13.jpg)
Panoramas• Strip Panoramas by sampling image
sequencesOther problems- Lose local perspective effects horizontally.- video cameras on cars: in general, lower resolution; shake, blurring; motion restricted as camera moving on a flat plane (ground).
![Page 14: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/14.jpg)
Panoramas• Single-Viewpoint Panoramas
- most normal panoramas: wide angle cameras. - using images from pure rotating cameras [Szeliski 97]- using approximate rotating cameras [Lowe’s autostitch]
Hard for long scene ...
![Page 15: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/15.jpg)
Panoramas
• Multi-Viewpoint Panoramas- by using multiple images- single cross-slits- multiple cross-slits
• This paper, totally different scheme.
![Page 16: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/16.jpg)
Panoramas
• Multi-Viewpoint Panoramas Why?- A photograph with a wider field of view would cause distortion towards the edges of the image. - Far enough away from the scene we will lose the depth cues of the scene.- Such panoramas can be used to visually convey directions through a city,- or to visualize how proposed architecture would appear within the context of an existing street.
![Page 17: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/17.jpg)
Long Scene Panorama Applications
• Street View - really long [Roman 04, thesis 06]...
![Page 18: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/18.jpg)
Long Scene Panorama Applications
• Ancient Street View by royal artist Zhang Zeduan800-900 years ago
![Page 19: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/19.jpg)
Long Scene Panorama Applications
• Virtual Earth/Google Earth- really long in 2 directions
![Page 20: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/20.jpg)
This work• Multi-viewpoint panorama long roughly
planar scenes (facades of the buildings along a city street).
• Significantly different from previous strip panoramas.
• After a small user interaction, the system will automatically compute a panorama with a MRF optimization.
• Users may exert additional control over the appearance of the result.
![Page 21: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/21.jpg)
Quick Example
![Page 22: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/22.jpg)
Quick Example
![Page 23: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/23.jpg)
Quick Example
![Page 24: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/24.jpg)
Quick Example
![Page 25: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/25.jpg)
![Page 26: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/26.jpg)
Previous Strip Panoramas
• Assumptions• Orthographic projection along the horizontal axis
• Perspective projection along the vertical axis
• Shortcomings• Only objects at a certain depth from the camera
plane can be shown with a correct aspect ratio.
• Further objects appear horizontally stretched.
• Closer objects appear squashed.
![Page 27: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/27.jpg)
What is a GoodMulti-Viewpoint Panorama
• Each object in the scene is rendered from a viewpoint roughly in front of it.
• The panoramas are composed of large regions of linear perspective.
• Local perspective effects are evident.
• The seams between these perspective regions do not draw attention.
![Page 28: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/28.jpg)
System Overview
Text
![Page 29: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/29.jpg)
A Key Observation• Images projected onto the picture
surface from their original 3D viewpoints will agree in areas depicting scene geometry lying on the dominant plane.
![Page 30: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/30.jpg)
Data Capture
• Handheld camera- walk along street and take picture every meter. - manually control exposure.
• Fisheye lens for some scenes. - Cover more scene content in one picture to avoid frequent “viewpoint transition”.
![Page 31: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/31.jpg)
Preprocessing
• Image Correction- For photos from fisheye lens, use PtLens to remove the radial distortion. Then treat them as normal images.
![Page 32: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/32.jpg)
Preprocessing• Recovering of the projection matrices of
each camera using the structure-from-motion system - The one in Photo Tourism, which is now open source.- Pair-wise matching on SIFT features enforces strong constraints for optimization.
Camera 3D position
![Page 33: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/33.jpg)
Preprocessing• SfM result
- Each image’s projection matrix and 3D point cloud for the scene strucure.
Picture not from this project!!
![Page 34: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/34.jpg)
Preprocessing
• Compensation of exposure variation• Brightness scale factor kii for each image Ii
• For pixels that depict the same geometry, asserting that ki*Ii = kj*Ij for Ii , Ij
• Each SIFT point match gives us three linear constraints of these form.
![Page 35: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/35.jpg)
• Picture surface?A virtual 3D surface upon which the panorama will be formed. - It should be roughly aligned to the dominant plane of the scene.
why? Good property 1.
Picture Surface Selection
![Page 36: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/36.jpg)
Blue curves
Picture Surface Selection
X- axis
Z- axis
![Page 37: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/37.jpg)
• Two steps: 1 Find world coordinate.2 Draw curve in xz plane.
• The system offers an automatic and an interactive approaches for choosing the coordinate system: • automatic approach -- PCA; largest variation is
x-axis, least is y-axis. Because it’s facade scene.
• interactive approach -- The cross product from two selected vectors along the y and x-axis forms the z-axis; the cross product of z and y forms the new x-axis.
Picture Surface Selection
![Page 38: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/38.jpg)
Picture Surface Selection
• The user is asked to draw a polyline into the plan view (xz slice).
• The system sweeps the polyline up and down the y−axis.• Directly draw manually
• Interactively draw -- select clusters of scene points; remove outliers; fit a third-degree polynomial z (x ) as a function of their x−coordinates; swept up this surface and down the y-axis.
![Page 39: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/39.jpg)
Red: recovered camera trajectory.Blue: user drawn polyline
Data Videos
directly
interactively
![Page 40: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/40.jpg)
Sampling the Picture Surface• The system samples the picture surface to form a
regular 2D grid. This 2D grid will map to make the final panorama image space.
• S (i , j ) refers the 3D location of the (i,j) sample.
![Page 41: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/41.jpg)
Sampling the Picture Surface
i
j...
...
This surface will form the final panorama after projected to 2D
it’s a 3D surface.so (i,j) --> S(i, j)
![Page 42: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/42.jpg)
• Project all S(i, j) with each image’s projection matrix.
• Each sample S (i , j ) forms one pixel, if its projection is located inside that image. - so there will be black holes and edges highly distorted. (white board)
Sampling the Picture Surface
![Page 43: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/43.jpg)
Sampling the Picture Surface
One Source Image
Its Sampled Image(after a circular crop)
![Page 44: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/44.jpg)
Sampling the Picture Surface
• They also call this process as projecting source images onto picture surface.
![Page 45: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/45.jpg)
• Project all source images to picture surface by “sampling picture surface”.
• Produce an average image with all these projected images.
Average Image
...average
![Page 46: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/46.jpg)
Average Image• After all source images get projected to
3D surface and sampled.
Average
Average with un-warping and cropping
Street not straight, due to Sfm drifting
Corrected by un-warping
![Page 47: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/47.jpg)
Average Image
Recall: image areas on dominant plane will be consistent after reprojected to picture surface.
![Page 48: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/48.jpg)
Viewpoint Selection
• We have now a series of n images Ii of equivalent dimension.
• Image Ii represents the i’th viewpoint.
• It’s necessary to choose one source image Ii for each pixel p = (px , py)
![Page 49: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/49.jpg)
Again, pixel labelling problem
![Page 50: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/50.jpg)
Objective Function
• The MRF optimization computes a labeling L(p), where L(p) = i if pixel p of the panorama is assigned color Ii (p).
• The objective function for choosing the viewpoint has three terms.
![Page 51: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/51.jpg)
First Term• The first term reflects the property that an object in the scene should be imaged from a viewpoint roughly in front of it.
• Assuming the cameras have roughly the same distance from the picture surface.
• It’s possible to find pixel pi whose corresponding 3D Sample S (pi) is closest to camera position Ci .
• So, if pixel p chooses its color from Ii, formulate this heuristic as:
![Page 52: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/52.jpg)
Second Term
• The second term encourages transitions between different regions of linear perspective to be natural and seamless.
For all neighboring pixels.
![Page 53: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/53.jpg)
Third Term• The third term encourages the panorama to resemble the average image in areas where the scene geometry intersects the picture surface. - Recall: the key observation
Also, it’s used for many other variance, like motion blur, occlusion
![Page 54: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/54.jpg)
• A vector median filter is used to compute across the three color channels for a robust mean value.
• The median absolute deviation (MAD) is calculated as median L2 distance from the median color.
Third Term
![Page 55: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/55.jpg)
Third Term
• Assuming that image color channels vary from 0 to 255, it’s possible to define the cost function:
![Page 56: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/56.jpg)
Complete Cost Function• Pixels in Image Ii to which the i’th camera does not project are set as null --> the black holes.
• L(p) = i is not possible if Ii (p) = null.
• Higher values for α encourage pixels from more straight-on views at the expense of more noticeable seams.
• Lower values of both α and β are more likely to remove objects off of the dominant plane.
• 100 β 0 25 i t ll
![Page 57: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/57.jpg)
Solve it as a MRF Optimization
• The panorama is computed at a lower resolution so that the MRF optimization can be computed in reasonable time.
• Using the hierarchical approach a higher-resolution is recreated.
• The final panorama is composited in the gradient domain to smooth the seams.
![Page 58: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/58.jpg)
Interactive Refinement
• View selection - A certain viewpoint should be used for a certain region of the composite.
• Seam suppression - The user can indicate objects in a scene across which seams should never placed.
• Inpainting (Similar to image completion)The user can draw strokes to indicate areas that should be filled with zero gradients during gradient-domain composition (to remove for example power lines)
![Page 59: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/59.jpg)
Seam Suppression• The MRF Optimization try to route seams
around objects that lie off the dominant plane.
• However, such objects don’t always exist.
Shortened car in the automatic panorama
User strokes in
one sourceimage
![Page 60: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/60.jpg)
Seam Suppression• Propagate the stroked pixels
- assume the strokes are drawn on 3D planes.- transfer the pixels to other source images by a homography (2D/2D relationship)
![Page 61: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/61.jpg)
Seam Suppression
• Constraints on these stroked pixels (on all images):for each pixel q adjacent to p that L(p) = i, if and only if L(q) = i.
--> keep that whole region as much as possible.
![Page 62: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/62.jpg)
Full Example 1
![Page 63: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/63.jpg)
Data
![Page 64: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/64.jpg)
Projected Source
![Page 65: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/65.jpg)
Average Image
![Page 66: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/66.jpg)
Seams Found
Red lines are where the labeling changes-->Seams
![Page 67: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/67.jpg)
Result
Automatically computed
![Page 68: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/68.jpg)
Full Example 2
![Page 69: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/69.jpg)
Data
![Page 70: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/70.jpg)
Projected Source
![Page 71: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/71.jpg)
Initial Result
Automatically computed
![Page 72: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/72.jpg)
User Strokes
User wants to maintain perspective in these areas.
![Page 73: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/73.jpg)
Seams
No seams pass the area covered by strokes.
![Page 74: Photographing Long Scenes with Multi-Viewpoint Panoramaslazebnik/research/fall08/xiaowei_li.pdf · Photographing Long Scenes with Multi-Viewpoint Panoramas Agarwala, M. Agrawala,](https://reader034.vdocuments.mx/reader034/viewer/2022042515/5fa667b7fdba5434d377934b/html5/thumbnails/74.jpg)
Final Result
Before After