image-based modeling - vis center quan_imageimage... · image-based modeling long quan ... the...
TRANSCRIPT
![Page 1: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/1.jpg)
ImageImage--based Modelingbased Modeling
Long [email protected]
with contributions from T. Fang, M. Lhuillier, P. Tan, Y. Wei, J. Xiao, G. Zeng and P. Zhao
The department of Computer Science & Engineering, HKUST
May 18, 2011, Hangzhou, 3DIMPVT-2011
![Page 2: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/2.jpg)
OverviewIntroductionFrom pixels to 3D points
Structure from motionA quasi-dense approach
From 3D points to objectsSmall-scale objects
Smooth surfaces, Hairs, TreesLarge-scale buildings
Façade, Buildings, Cities
Conclusions
![Page 3: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/3.jpg)
IntroductionComputer vision
3D reconstruction from imagesOne of the fundamental vision problems
Computer graphicsModeling: creating 3D digital models
Interactive methods (using softwares like Maya)Use sanners
Use photographs image-based modelingFor existing objectsPhotorealistic modelsCan be automated
![Page 4: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/4.jpg)
TV/movie post-production, special effects, Virtual realities, 3D game creationPhotogrammetryRobotics navigation…Modeling the world:
Google Earth, Virtual Earth!
Modeling/3D reconstruction applications
![Page 5: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/5.jpg)
One example of small-scale object
Rendered by Maya3D mesh modeltake images(1 of 35)
![Page 6: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/6.jpg)
One example of large-scale object
![Page 7: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/7.jpg)
How to get 3D points and camera positions from 2D pixels? (geometry computation or structure-from-motion (SFM))
How to get 3D objects from unstructured 3D points? (object modeling)
Objects are of different natures:Smooth objects (surface reconstruction)Curvilinear objects (hair modeling)Complex fine details (tree modeling)Large-scale objects (city modeling)
Two challenging components:
Smooth surface
Curves
Open surface patchTexturesRegular objects
![Page 8: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/8.jpg)
Structure from MotionStructure from Motion
![Page 9: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/9.jpg)
Standard Sparse SFMSparse feature detection (points of interest, or SIFT)Heuristic Initial matching of feature points by local image similaritiesCorrespondence using RANSAC with minimum algebraic solutionAuto-calibrationOptimization of cameras and points (bundle adjustment)
There are textbooks (Faugeras,Hartley/Zisserman) and commercial softwares for sparse SFM.
captured 51 images
Sparse 3D points and camera positions
……
![Page 10: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/10.jpg)
A Quasi-dense approach
Dense (per pixel) reconstruction is impossible Standard Sparse (feature) reconstruction is insufficient
Quasi-dense: re-sampled (un-interpolated) disparity maps
captured 26 images
3D points and camera positions
… …
A Quasi-Dense Approach to Surface Reconstruction from Uncalibrated ImagesLhuillier and Quan, IEEE PAMI 2005
![Page 11: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/11.jpg)
Advantages of Quasi-dense:
Hand-held camera (uncalibrated)Fully automatic computationRobust and accurate (the example is impossible with other existing methods)Pave the way for further object modeling (with sufficient density of points cloud, often impossible with sparse)
![Page 12: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/12.jpg)
Object ModelingObject Modeling
![Page 13: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/13.jpg)
Modeling
SFM
Structuring points into objects!
![Page 14: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/14.jpg)
Surface reconstruction
Model a wide variety of relatively smooth objectsPiror is ‘smoothness’Mathematically well studied:
Implicit surfaceVariational principle Level-set implementation
![Page 15: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/15.jpg)
Examples
![Page 16: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/16.jpg)
Hair Modeling
Difficult due to insufficient image resolution
Typically 500*500 pixels for 50,000 hair fibers
Only Hair strands, group of hair fibers, are visible
![Page 17: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/17.jpg)
Hair modeling approachPrior model: a curve approximated by a polylineConstraint: Polyline Growth by following edge orientation
![Page 18: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/18.jpg)
Modeling Hair from Images,Wei, Ofek, Quan, Shum, Siggraph 2004.
Examples, fiber by fiber, 300K fibers
![Page 19: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/19.jpg)
Tree modeling
Trees are ubiquitous but difficult to modelComplex geometry and topologyFine texture details
relative leaf size
Tree spectrum
![Page 20: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/20.jpg)
But, trees have self-similar structure: a natural generative model for both leaves and branchesAll leaves are similar: a generic model for all leaves of a type of plants
3D 2D
Structurefrom Motion
Leaf Segmentation Leaf Reconstruction Plant Model
![Page 21: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/21.jpg)
one of the 35 input images recovered model
Image-based Plant Modeling,Quan, Tan, Zeng, Yuan, Wang, and Kang, Siggraph 2006.
Example
Each leaf has individual geometry and texture.
![Page 22: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/22.jpg)
Structurefrom Motion
Synthesis of occluded branches
Plant ModelReconstruction ofvisible branches
by self-similarity!
But, trees have self-similar structure: a natural generative model for both leaves and branchesAll leaves are similar: a generic model for all leaves of a type of plantsThe branches are self-similar, a kind of fractal!
![Page 23: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/23.jpg)
One of the 18 input images
“This is probably the first image-based tree modeling method that produces a quality that could beadequate for movies. Earlier work produced results that were suitable for virtual reality applications, but not really anything that could be confused with a real image. Here, that seems within reach!”from one siggraph reviewer.
Recovered tree model
Image-based Tree Modeling, Tan, Zeng, Wang, Kang, and Quan, SIGGRAPH 2007.
Example
![Page 24: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/24.jpg)
Provisional summary:
generative rulesgeneric modelsData(image)-driven
![Page 25: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/25.jpg)
LargeLarge--Scale Object ModelingScale Object Modeling
![Page 26: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/26.jpg)
A traditional street in Canton:
Challenging ‘large-scale’ buildings vs ‘small-scale’ objectsSpatial constraints for capturingScale and automation
City buildings façades
![Page 27: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/27.jpg)
Generic shape: rectangle (approximation at a given scale, the ground level street-view)
Generative rules: rectangular boxes rectangles nested rectangles with different offsets
Prior knowledge on buildings and façades
![Page 28: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/28.jpg)
Basic ideas: facade by facade
Initialization as a flat rectangle in spaceDecomposition:
Dividing recursively the facade into smallrectangular patches (building blocks)Regularize the depth (quasi-dense 3D point)
Augmentation by quasi-denseCompletion (texture refetching)
Image-based Façade Modeling, Xiao, Fang, Tan, Zhao, Ofek, and Quan, SIGGRAPH ASIA 2008.
![Page 29: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/29.jpg)
![Page 30: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/30.jpg)
![Page 31: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/31.jpg)
Example for Hennepin avenue:
![Page 32: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/32.jpg)
Comparison with Virtual Earth models
![Page 33: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/33.jpg)
614 images by a handheld camera
Example of Canton street
![Page 34: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/34.jpg)
Large-scale full automation
Interactive automaticSegmentation is the keyA fully automatic approach to building modeling
![Page 35: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/35.jpg)
Classification and segmentation
Over-segmentation Segmentation (multi-view)
Classification (single view)
Input image
Five classes: buildings, sky, ground, vegetation, the others
![Page 36: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/36.jpg)
![Page 37: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/37.jpg)
Pipeline overviewPipeline overview
• (scene) classification and segmentation• (building block) decomposition scheme• (façade) structure analysis and modeling
![Page 38: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/38.jpg)
Building partition: using vertical linesBuilding partition: using vertical lines
• Detection of lines• Match only horizontal and vertical lines
using vanishing points• Reconstruct only vertical lines
for the lateral motion of the camera
![Page 39: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/39.jpg)
Building partition façadesBuilding partition façades
Decompose the buildings into small modelable blocks or facades
![Page 40: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/40.jpg)
Façade modeling: Patch reconstruction and ortho composition Façade modeling: Patch reconstruction and ortho composition
• Over-segmentation [Felzenszwalb and Huttenlocher 2004]. Estimate centroidsEstimate normal directions
• Ortho-image space (object space)
![Page 41: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/41.jpg)
Patch-reconstruction and Ortho-compositionPatch-reconstruction and Ortho-composition
From single image
From multi image
Texture mapDepth map
Input image Ortho image
![Page 42: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/42.jpg)
Façade modeling: Structure analysisFaçade modeling: Structure analysis
• Joint segmentation• Shape regularization
Segmentation Regularization
![Page 43: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/43.jpg)
Façade modeling: final resultsFaçade modeling: final results
Image-based Street-side City Modeling, Xiao, Fang, Zhao, Lhuillier, and Quan, SIGGRAPH ASIA 2009.
2.5D Façade 3D Building Texture-mapped buidling
![Page 44: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/44.jpg)
![Page 45: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/45.jpg)
Façades buildings citiesFaçades buildings cities
![Page 46: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/46.jpg)
![Page 47: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/47.jpg)
Conclusion
An image-based approach to 3DAt the intersection of vision and graphicsScale (small and large) and automation
Methodology: Generative rulesGeneric modelsData(image)-driven
![Page 48: Image-based Modeling - Vis Center QUAN_ImageImage... · Image-based Modeling Long QUAN ... The branches are self-similar, a kind of fractal! One of the 18 input images ... Conclusion](https://reader033.vdocuments.mx/reader033/viewer/2022052917/5aa96f347f8b9a86188cc385/html5/thumbnails/48.jpg)
New bookImage-based Modeling, Long QUAN,Springer-Verlag,2010.
Part I Geometry: fundamentals of multi-view geometryPart II Computation: from pixels to 3D pointsPart III Modeling: from 3D points to objects