how to fit a surface to a point cloud? or optimization of surface functionals in computer vision
DESCRIPTION
TRICS seminar Computer Science Department. How to fit a surface to a point cloud? or optimization of surface functionals in computer vision Yuri Boykov. Optimization of surface functionals in computer vision. Computer vs. human vision Model fitting in computer vision - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/1.jpg)
The University of
Ontario
How to fit a surface to a point cloud?or
optimization of surface functionals in computer vision
Yuri Boykov
TRICS seminarComputer Science Department
![Page 2: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/2.jpg)
The University of
Ontario
Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision
• templates, pictorial structures, trees, deformable models, contours/snakes, meshes, surfaces, complexes, graphs, weak-membrane model, Mumford-Shah, Potts model,……
Optimization in computer vision• dynamic programming, gradient descent, PDEs, shortest paths, min. spanning
trees, linear and quadratic programming, primal-dual schema, network flow algorithms, QPBO, ...
Applications• segmentation, stereo, multi-view reconstruction, optical
flows• surface fitting
![Page 4: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/4.jpg)
The University of
Ontario+Shading
M.C. EscherDrawing hands
3D shapeunderstanding
![Page 5: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/5.jpg)
The University of
Ontario+Color
Da VinciMadonna Litta
Recognition
![Page 6: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/6.jpg)
The University of
Ontario+Texture
MagritteSouvenir de Voyage
recognizingmaterial
![Page 7: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/7.jpg)
The University of
Ontario+Texture
The New Yorker Album of Drawings, The Viking Press, NY, 1975
recognizing 3D perspective
![Page 8: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/8.jpg)
The University of
OntarioWhat do humans get by ‘looking’?
J. VermeerThe Guitar Player
Contours Shading Color Texture …
basic image cues:
![Page 9: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/9.jpg)
The University of
OntarioWhat do humans get by ‘looking’?
Contours Shading Color Texture …
basic image cues: Segmentation Motion 3D shape perception 3D scene geometry Detection/Recognition …
higher-level perception:
![Page 10: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/10.jpg)
The University of
OntarioWhat do computers get by ‘looking’?
x
y
3D plot of image intensity I(x,y) x
y
I(x,y)
x
y
![Page 11: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/11.jpg)
The University of
OntarioWhat do computers get by ‘looking’?
P. PicassoThe Guitar Player
Intensity discontinuities (contours)
Intensity gradients (shading)
Multi-valued intensities (color)
Filtering (e.g. texture)
…
basic image cues: higher-levelgrouping?
![Page 12: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/12.jpg)
The University of
Ontario
model
Bayesian approach
Prior + Data Low-level cues
(local info)high-level knowledge
(global picture)
Fit some prior model into data
![Page 13: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/13.jpg)
The University of
OntarioRigid Template Matching
• In matching we estimate “position” of a rigid template in the image
• “Position” includes global location parameters of a rigid template:
- translation, rotation, scale,…
Face templateimage
image
translation,rotation,scaling
![Page 14: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/14.jpg)
The University of
OntarioNon-rigid (parametric) matching
1. Pick one image (red)2. Warp the other images to match it (homographic transform)3. Blend
panorama mosaicing
![Page 16: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/16.jpg)
The University of
Ontarioe.g…. using flexible templates
• In flexible template matching we estimate “position” of each rigid component of a template
• For tree-structured models, efficient global optimization is possible via DP (Felzenswalb&Huttenlocher 2002)
![Page 17: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/17.jpg)
The University of
Ontariotracking parameters => activity recognition
Bottom-up tracker
![Page 18: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/18.jpg)
The University of
Ontario
5-18
deformable contours (“snakes”)
2D curve which matches to image data Initialized near target, iteratively refined Can restore missing data
initial intermediate final
Optimization gets harder when a loop is introduced. DP does not apply.
One solution: gradient descent
Kass, Witkin, Terzopoulos 1987
![Page 19: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/19.jpg)
The University of
Ontario
6-19Cremers, Tischhäuser, Weickert, Schnörr, “Diffusion Snakes”, IJCV '02
local minima, fixed contour topology
![Page 20: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/20.jpg)
The University of
Ontario
6-20
A contour may be approximated from
u(x,y) with near sub-pixel accuracy
C
-0.8 0.2
0.5
0.70.30.6-0.2
-1.7
-0.6
-0.8
-0.4 -0.5
),( ppp yxuu
• Level set function u(x,y) is normally discretized/stored over image pixels• Values of u(p) can be interpreted as distances or heights of image pixels
Implicit representation of contoursOsher&Sethian 1989
![Page 21: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/21.jpg)
The University of
Ontario
6-21
[Visualization is courtesy of O. Juan] Simple evolution
Morphological Operation:Erosion
1p
ppp NCd
|| ppp udu
![Page 22: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/22.jpg)
The University of
Ontario
6-22
Visualization is courtesy of O. Juan
Example of gradient descent evolution 2
2
2
2)(yu
xu
p dtu
Gradient descent w.r.t.
Euclidean length
![Page 23: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/23.jpg)
The University of
Ontario
6-23
Example of gradient descent evolution
Laplacian
Osher&Sethian 1989
Gradient descent w.r.t.
Euclidean length
2
2
2
2)(yu
xu
p dtu
![Page 24: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/24.jpg)
The University of
Ontario
6-24
[example from Goldenberg, Kimmel, Rivlin, Rudzsky, IEEE TIP ’01] Geodesic Active Contours via Level-sets
yug
xug
pyxdtu
)()()(
C
dsgCE )()(
![Page 25: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/25.jpg)
The University of
Ontario
6-25
Other geometric energy functionals besides length [courtesy of Ron Kimmel]
weighted length C
dsgCE )()(
Functional E( C ) gradient descent evolution
,~ Ngg
weighted area
dafCE )( ~ f
alignment(flux)
C
dsNCE ,)(v )div(~ v
Geometric measures commonly used in segmentation
NdC
![Page 26: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/26.jpg)
The University of
Ontario
in 3D…deformable meshes, level-sets, …
Sphere.mpg
Estimation of positionfor mesh points
Many loops.optimization - gradient
descent
GOALS: global optima (?) “right” functional (?)
Typical problems:- local minima (clutter,
outliers)-over-smoothing
![Page 27: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/27.jpg)
The University of
Ontario
Global Optimization andSurface Functionals
![Page 28: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/28.jpg)
The University of
Ontario
More generally...
Estimate labels for graph nodes
I
p
L
along one scan line in the image
observed noisy image I image labeling L(restored intensities)
NOTE:similar to robust regression modelestimation
![Page 29: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/29.jpg)
The University of
Ontario
(simple example)
Piece-wise smooth restoration
Markov Random Fields (MRF) approach
weak membrane model (Geman&Geman’84, Blake&Zisserman’83,87)
pL qL
Nqp
qpp
pp LLVILLE),(
2 ),()()(
discontinuity preserving prior
optimizing E(L) is NP hard!
,V
T T
(Continuous analogue: Mamford-Shah functional, 1989)
![Page 30: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/30.jpg)
The University of
Ontario
I
p
L
observed noisy image I image labeling L(restored intensities)
(simple example)
Piece-wise constant restoration
along one scan line in the image
![Page 31: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/31.jpg)
The University of
Ontario
(simple example)
Piece-wise constant restoration
Potts model Boykov Veksler Zabih ’01Greig et al.’89 (for 2 labels)
Nqp
qpp
pi
i
LLILLEi ),(
2 )()()(
global optimization is still NP hard, but there are fast provably good
combinatorial approximation algorithms, linear and quadratic programming, QPBO, primal-dual
schema
pL qL
qp LL
0}2L:p{ p2
“perceptual grouping”
}1L:p{ p1
}0L:p{ p0
![Page 32: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/32.jpg)
The University of
OntarioPerceptual grouping from stereo
(Birchfield &Tomasi’99) constant label = plane
![Page 33: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/33.jpg)
The University of
Ontario
Binary labeling(binary image restoration)
)|Pr()()( 2pppppp LIILLD
}1,0{pL
original binary image I optimal binary labeling L
Nqp
qpp
pp LLLDLE),(
)()()( Greig Porteous Seheult ’89
Globally optimal solution is possible using combinatorial graph cut algorithms• pseudo-boolean optimization Hammer’65, Picard&Ratlif’75
![Page 34: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/34.jpg)
The University of
Ontario
Binary labeling(object extraction)
1pL
0pL
object segmentation
left ventricle of heart
}1,0{pL
![Page 35: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/35.jpg)
The University of
Ontario
Binary labeling(object extraction)
10
C
Globally optimal solution is possible using graph cut algorithms• pseudo-boolean optimization (Hammer’65, Picard&Ratlif’75)
surface extraction
Nqp
qppqp
pp LLwLDLE),(
)()()(
Boykov&Jolly’01
left ventricle of heart
}1,0{pL
![Page 36: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/36.jpg)
The University of
OntarioImplicit surface representation via graph-cuts• Any contour (or surface in 3D) satisfying labeling of
exterior/interior points (pixel centers) is acceptable if some explicit surface has to be output.
0 1
1
1 1 1 0
0
0
0
0 0
![Page 37: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/37.jpg)
The University of
Ontario
)(
)(,)()(CC C
x dpxfdsdsgCE vN
)int(
)(,)()(CC
xC
dpxfdsdsgCE vN
Geometric length any convex,
symmetric metric (e.g. Riemannian)
Flux any vector field v
Regional bias any scalar function f
“edge alignment”
Tight characterization for geometric functionals of contour C that can be globally optimized by graph cut algorithms (Kolmogorov&Boykov’05)
disclaimer: for pairwise interactions only
Global optimization of geometric surface functionals
N)q,p(
qpp
pp )L,L(V)L(D)L(E
![Page 38: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/38.jpg)
The University of
OntarioGlobally optimal surfaces in 3D
Volumetric segmentation(BJ01,BK’03,KB’05)
![Page 39: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/39.jpg)
The University of
Ontario
Binary labeling(object extraction)
Blake et al.’04, Rother et al.’04
}1,0{pL
iteratively re-estimate color models e.g. using mixture of Gaussians
![Page 42: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/42.jpg)
The University of
OntarioOptimal surfaces in 3D
3D reconstructionVogiatzis, Torr, Cippola’05
Local cues: voxel’s photoconsistency
Prior:smoothness,
projective geometry constraints
![Page 43: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/43.jpg)
The University of
OntarioGlobally optimal surfaces in 3D
Lempitsky&Boykov, 2006
from a cheap digital camera
![Page 44: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/44.jpg)
The University of
Ontario
3D model
multi-view reconstruction set up
Furukawa&Ponce 2006
Globally optimal surfaces in 3D
(texture mapped)
![Page 45: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/45.jpg)
The University of
OntarioSurface fitting to point clouda cloud of 3D points (e.g. from a laser scanner)
3D model:
surface fitting:
![Page 47: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/47.jpg)
The University of
OntarioSurface fitting functional
CC
dsdsCE 1,)( vN Fitting a surface into a cloud of oriented points (Lempitsky&Boykov, 2007)
ip
in
S
data fit prior
![Page 48: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/48.jpg)
The University of
OntarioOptimal surfaces in 3DFitting a surface into a cloud of oriented points (Lempitsky&Boykov, 2007)
From 10 views
No initialization is needed
CC
dsdsCE 1,)( vN
![Page 49: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/49.jpg)
The University of
OntarioGlobal vs. local optimization
regional potentials )(vdivf
CC
dsdpdivCE 1)()int(
vFitting a surface into a cloud of oriented points (Lempitsky&Boykov, 2007)
initial solution local minima global minima
![Page 53: How to fit a surface to a point cloud? or optimization of surface functionals in computer vision](https://reader036.vdocuments.mx/reader036/viewer/2022062301/5681613b550346895dd09c4a/html5/thumbnails/53.jpg)
The University of
OntarioSummary
Global optimization- Your solution is as good as your functional
- No need to worry about initial guess or convergence issues- Polynomial algorithms, but many practical issues (efficient data structures, memory limitations, parallelization, dynamic applications,…)- Many useful functionals are NP hard (lots of approximation methods are developed)- New approaches allowing global optimization are introduced (including new version of level-sets)