taubin ciarp 2012
DESCRIPTION
Keynote lecture given by Gabriel Taubin at CIARP 2012 (http://www.ciarp.org) on Wednesday, September 5, 2012, in Buenos Aires, ArgentinaTRANSCRIPT
Smooth Signed Distance Surface Reconstruc4on and Applica4ons
Gabriel Taubin Brown University
CIARP 05 Sep 2012
Reconstruc4on Method
Surface Representa4on
Oriented Points
Posi4ons & Normals Implicit Surface Polygon Mesh
Typical Surface Reconstruc3on Pipeline
• Medical Imaging – Visualiza4on – Segmenta4on
Numerous Applica3ons • Industry
– Reverse engineering – Fast metrology – Physical simula4ons
• Entertainment – Anima4ng digital clays for movies or games
• Archeology and Art – Digi4za4on of cultural heritage and ar4s4c works
• …
Oriented Points
Posi4ons, Normals, Colors
Methods to Capture Oriented Points
Laser Scanning
Mul4-‐View Stereo
Structured Ligh4ng
Laser range scanning Mul4-‐View Stereo
Data Acquisi3on
Structured Ligh4ng
Gray Code Structured Ligh3ng Results
hBp://mesh.brown.edu/byo3d/
hBp://mesh.brown.edu/byo3d
Surface Reconstruc3on from Mul3-‐View Data
SSD: Smooth Signed Distance Surface Reconstruc3on F. Calakli, G. Taubin, Computer Graphics Forum, 2011.
• Con4nuous mathema4cal formula4on • Par4cular discre4za4on and algorithm • Reconstructs water4ght surface as polygon mesh • From a sta4c oriented point cloud
hBp://mesh.brown.edu/ssd
Related Papers & Projects
• Vector Field Isosurface-‐Based Reconstruc4on From Oriented Points, by Sibley & Taubin, Siggraph 2005 (Sketch).
• Smooth Signed Distance Surface Reconstruc4on, by Calakli & Taubin, PG 2011 & Computer Graphics Forum 2011.
• Smooth Signed Distance Colored Surface Reconstruc4on, by Calakli & Taubin, chapter in State-‐of-‐the-‐Art Volume on Computer Graphics, Visualiza4on, Visual Analy4cs, VR and HCI, 2012.
• Accurate 3D Footwear Impression Recovery from Photographs, by Andalo, Calakli, Taubin, and Goldenstein, Proceedings of the 4th. Interna4onal Conference on Imaging for Crime Detec4on and Preven4on (ICDP-‐2011).
• High Resolu4on Surface Reconstruc4on from Mul4-‐view Aerial Imagery, Calakli, Ulusoy, Restrepo, Mundy & Taubin, 3DIMPVT 2012
• REVEAL Digital Archaeology Project • Cuneiform Automa4c Transla4on Project
Par3cularly Good at Extrapola3ng Missing Data
< 0
> 0
0 )( fZ
Oriented Points, D (samples from unknown surface S)
Find a scalar valued func4on , whose zero level set Z(f) = S’ is the es4mate for true surface S
Computed Implicit Surface, S’
f
ℜ→Df :
Implicit func3on framework
• Input: oriented point set: D = { ( pi, ni ) i=1,…,N}
contained in a bounding volume V • Output: implicit surface
S = { x | f (x) = 0 } with the func4on defined on V, such that
f (pi) = 0 and ∇f (pi) = ni ∀(pi,ni) ∈ D • A family of implicit func4ons with a finite number of parameters has to be chosen
• Parameters must be es4mated so that the condi4ons stated above are sa4sfied, if not exactly, then in the least-‐squares sense
Implicit Curve and Surface Reconstruc3on
Challenges
Uniform sampling
Non-‐uniform sampling
Noisy data
Misaligned scans
• Interpola4ng polygon meshes Boissonnat [1984], Edelsbrunner [1984] Amenta et al. [1998], Bernardini et al. [1999] Dey et al. [2003][2007], …
• Implicit func4on fijng Taubin [1991], Hoppe et al. [1992], Curless et al. [1996] Whitaker [1998], Carr et al.[2001], Davis et al. [2002], Ohtake et al. [2004], Turk et al. [2004], Shen et al. [2004] Sibley-‐Taubin [2005]
General Approaches
Poisson Surface Reconstruc3on
Kazhdan et al. [2006] Manson et al. [2008]
Poisson Surface Reconstruc3on
1. Extend oriented points to con4nuous vector field defined on the whole volume, so that
2. Integrate vector field, by minimizing
3. Determine isolevel, by minimizing
||∇f (p)− v(p) ||2 dpV∫
( f (pi )− f0 )2
i=1
N
∑
v(pi ) ≈ ni
v(p)
Main problem with this approach
Main problem with this approach
Main problem with this approach
Main problem with this approach
What kind of implicit func3on?
Indicator Func4on Smooth Signed Distance Func4on
1
0
0
Vectorfield Isosurface-‐Based Reconstruc3on From Oriented Points P. Sibley and G. Taubin [Siggraph 2005 Sketch]
• Surface reconstruc4on from cloud of oriented points • Implicit representa4on can deal with missing data • Rather than fijng analy4c func4on (RBFs, etc), and then
extract isosurface for visualiza4on, fit isosurface directly to data
E ( f ) =P
i f ( pi ) 2+P
i kr f ( pi ) ¡ n i k2+RV kH f ( x ) k2dx
Vectorfield Isosurface-‐Based Reconstruc3on From Oriented Points P. Sibley and G. Taubin [Siggraph 2005 Sketch]
• Surface reconstruc4on from cloud of oriented points • Implicit representa4on can deal with missing data • Rather than fijng analy4c func4on (RBFs, etc), and then
extract isosurface for visualiza4on, fit isosurface directly to data • Ini4al approach: 3 steps
1. Fit con4nuous vector field to data normal vectors 2. Integrate vector field on regular grid 3. Determine Isolevel
E ( f ) =P
i f ( pi ) 2+P
i kr f ( pi ) ¡ n i k2+RV kH f ( x ) k2dx
VFIso: Representa3on
• Regular voxel grid • Implicit func4on defined by values at ver4ces • And tri-‐linear interpola4on • The output is:
∑= α ααϕ )()( pfpf
αf )(pαϕ
}{ αf
VFIso: Varia3onal Approach
• Extend data to vector field on the volume minimizing
tionregularizadata
DvvnpvvE ∑∑ −+−=
),()()(
βαβαλ22
∑=α
ααϕ )()( :where pvpv
• Then integrate vector field ?
VFIso: Details
• Problem: Vector field v(p) not integrable! • Instead: Fit scalar field directly by minimizing
∑∑ ∇−∇+−∇=),(
22)(})({βα
βαα λ ffnpffED
}{ αf
• Where ∑ ∇=∇α ααϕ )()( pfpf
αf∇
∑ −=D
fpffE200 )()(
• And is discre4zed using central differences • This is a standard (Laplacian) Least Squares problem
• Minimize to find isolevel
Squirrel(9K) Angel(24K) Bunny(35K) Ram(678K)
Some Methods to Capture 3D Point Clouds
Shadow
Turntable
Backdrop
8 Megapixel Camera
Multi-Flash Attachment
Multi-Flash Camera
Beyond SilhoueBes: Surface Reconstruc3on using Mul3-‐Flash Photography
D. Crispell, D. Lanman, P. Sibley, Y. Zhao and G. Taubin [3DPVT 2006]
33
Shadow
Turntable
Backdrop
8 Megapixel Camera
Mul3-‐Flash ABachment
Mul3-‐Flash Camera
Multi-Flash 3D Photography:
Capturing the Shape and Appearance of 3D Objects
Turntable Rota3on
A new approach for reconstructing 3D objects using shadows cast by depth discontinuities, as detected by a multi-flash camera. Unlike existing stereo vision algorithms, this method works even with plain surfaces, including unpainted ceramics and architecture.
Estimated Shape: 3D Point Cloud
Recovered Appearance: Phong BRDF Model
Multi-Flash Turntable Sequence: Input Image
Data Capture: A turntable and a digital camera are used to acquire data from 670 viewpoints. For each viewpoint, we capture a set of images using illumination from four different flashes. Future embodiments will include a small, inexpensive handheld multi-flash camera.
Recovering a Smooth Surface
The reconstructed point cloud can possess errors, including gaps and noise. To minimize these effects, we find an implicit surface which interpolates the 3D points. This method can be applied to any 3D point cloud, including those generated by laser scanners.
VFIso Results [2006 110x110x110 grid]
• Oriented point set: D = { ( pi, ni ) } sampled from a surface S
• Implicit surface: S = { x | f (x) = 0 } such that f (pi) = 0 and ∇f (pi) = ni ∀(pi,ni) ∈ D
• Least squares energy:
SSD Con3nuous Formula3on
E( f ) = f (pi )2
i=1
N
∑ +λ1 ∇f (pi )−ni2
i=1
N
∑ ∫∑∑ +−∇+===
V
N
i
N
idffffE xxnpp iii
22
1
21
1
2 ||)(H||)()()( λλ
What does the regulariza3on term do ?
• Near data points: since the data terms dominate, the func4on approximates the signed distance
• Away from data points: the regulariza4on term dominates and forces the gradient to be smooth and close to constant
∫∑∑ +−+===
V
N
i
N
idMvfMvfE xxnpp iii
22
1
21
1
2 ||)(||)()(),,( λλ
Role of each energy term
Quadra4c energy in f, v, and M
If f, v, and M are linear func3ons of the same parameters, then the minimiza4on reduces to a least squares problem
Linear families of func3ons
• Popular Smooth Basis Func4ons – Monomials [Taubin’91] – Radial basis func4ons [Carr et al., ‘01], – Compactly supported basis func4ons [Othake et al. ‘04], – Trigonometric polynomials [Kazhdan et al. ‘05], – B-‐splines [Kazhdan et al., 06], – Wavelets [Manson et al. ‘08],
cFbAFFFE tt +−= 2)(Non-‐homogenous,
Quadra4c energy
bAF =Global minimum
We can use Independent Discre3za3ons
• Hybrid FE/FD discre4za4on – Trilinear interpolant for the func4on f(x) – Primal finite differences for the gradient ∇f(x) – Dual finite differences for the Hessian Hf(x)
• As long as f(x), ∇f(x), and Hf(x) are wrixen as a linear combina4ons of the same parameter vector F
cFbAFFFE tt +−= 2)(Non-‐homogenous, Quadra4c energy
bAF =Global minimum
Implementa3on
• Primal-‐Dual octree data structure • Cascading mul4-‐grid itera4ve solver (conjugate gradient): Solve the problem on a much coarser level – Use the solu4on at that level to ini4alize the solu4on at the next level
– Refine with the itera4ve solver • Iso-‐surface extrac4on (crack-‐free)
– Dual marching cubes [Schaefer 2005]
Marching Cubes on Octrees • Non-‐conforming hexahedral mesh • Results in crack problem. • Problem solved by Dual Marching Cubes
MPU [Othake ‘03 ]
Poisson [Kazhdan ‘06 ]
D4 Wavelets [Manson ‘08 ]
SSD [Calakli ’11]
Input point cloud
SSD Surface Reconstruc3on • Theore4cal contribu4ons:
– Oriented point samples regarded as samples of Euclidean signed distance func4on
– Reconstruc4on as global minimiza4on problem – Yet sparse system of linear equa4ons
• Empirical advantages: – Robust to noise and uneven sampling density
• Future work: – Streaming out-‐of-‐core implementa4ons – Parallel/Mul4-‐core/GPU implementa4ons – Dynamic shapes
REVEAL Digital Archaeology Project Cooper, Kimia, Taubin, Galor, Sanders, Willis
• The main goal is to automate the tedious processes of data collec4on and documenta4on at the excava4on site, as well as to provide visualiza4on tools to explore the data collected in the database
• Also to solve specific problems in Archaeology using computer vision techniques.
• We first used a network of cameras to capture the ac4vity at the excava4on site, to reconstruct the shape of the environment as it is being excavated, to reconstruct layers, and to locate finds in 3D
• Then we switched to mul4-‐view stereo with photos captured with hand-‐held cameras
Semi-‐automa3cally Assemble Fragments Into Ar3facts
Assisted Data Acquisition, Algorithmic Reconstruction, Integrated multi-format analysis
REVEAL Archaeological Data Acquisition
Objects: Ar4facts
Excava4ons Areas Sites
Data Acquisi3on
Data: Text Photos Video 3D Models
Automa3cally Convert Photos to 3D Models
Site Plans
Import photos, videos, and laser scans and connect them to database objects
Improve speed and accuracy with computer assisted data entry
Advanced Algorithms
Import External Data
Laser Scanned Models
Rule-‐based Reconstruc3ons
REVEAL Database
hBp://sourceforge.net/projects/revealanalyze
Typical Activity Sequence
Data integrated and synchronized in tabular, plan drawing, 3D spa3al, image, and video formats REVEAL Archaeological Analysis
Select Ar3facts on Site Plan
Ar4facts Excava4ons
Areas Sites
Display Photos of Selected Ar3facts
Examine Rela3onship of Ar3facts in-‐situ in auto-‐generated 3D Excava3on Model
Export FormaBed Ar3fact Data for inclusion in Site Publica3on
[Furukawa and Ponce 2008]
[Snavely et. al. 2006]
MVS soaware
Patch-‐based Mul4-‐View Stereo (PMVS) hxp://grail.cs.washington.edu/soyware/pmvs/
hxp://phototour.cs.washington.edu/bundler/
Accurate 3D Footwear Impression Recovery From Photographs, F. A. Andalo, F. Calakli, G. Taubin, and S. Goldenstein,
Interna4onal Conference on Imaging for Crime Detec4on and Preven4on (ICDP-‐2011).
Comparable to 3D Laser Scanner
Handheld Interac4ve, Incremental 3D Object Scanning Kim, Honors Thesis, Brown University, May 2012
• Based on MS Kinect sensor
• Con4nuous coarse pose es4ma4on from color camera using PTAM
• 3D snapshots captured from different points of view using depth camera
• Alignment improved with Itera4ve Closest Points (ICP) algorithm
• SSD is run on aligned 3D snapshots
Screen shots of our real-‐3me stereo system working on the field
Real-‐Time High-‐Defini3on Stereo on GPGPU using Progressive Mul3-‐Resolu3on Adap3ve Windows
Y. Zhao, and G. Taubin, Image and Vision Compu4ng 2011.
¼ Resolution ½ Resolution Full Resolution¼ Resolution ½ Resolution Full Resolution
Coarse-‐to-‐fine matching on mul;ple resolu;ons
Stereo Frame Grabbing
Stereo Rec3fica3on
Mul3-‐Resolu3on Pyramid Genera3ng
Background Modeling With Shadow Removal Foreground Detec3on with Dila3on & Erosion
Stereo Matching Using Adap3ve Window with Cross-‐Checking
Disparity Refinement Re
solu3o
n Scan
from
Low
to
High
Processing Pipeline
Real-‐Time High-‐Defini3on Stereo on GPGPU using Progressive Mul3-‐Resolu3on Adap3ve Windows
Y. Zhao, and G. Taubin, Image and Vision Compu4ng 2011.
High Resolu3on Surface Reconstruc3on from Mul3-‐view Aerial Imagery by Calakli, Ulusoy, Restrepo, Mundy & Taubin, 3DIMPVT 2012
Microscopic 3D Shape Capture Liberman & Taubin (work in progress)
~ 4 mm
Ques4ons?
This material is based upon work supported by the Na4onal Science Founda4on under Grants CCF-‐0729126, IIS-‐0808718, CCF-‐0915661, and
IIP-‐1215308.