taubin ciarp 2012

Smooth  Signed  Distance  Surface  Reconstruc4on  and  Applica4ons  

Gabriel  Taubin  Brown  University  

CIARP  05  Sep  2012  

Reconstruc4on  Method  

Surface  Representa4on  

Oriented  Points    

Posi4ons  &  Normals   Implicit  Surface                              Polygon  Mesh      

Typical  Surface  Reconstruc3on  Pipeline  

•  Medical  Imaging  –  Visualiza4on  –  Segmenta4on  

Numerous  Applica3ons  •  Industry  

–  Reverse  engineering  –  Fast  metrology  –  Physical  simula4ons  

•  Entertainment    –  Anima4ng  digital  clays  for  movies  or  games  

•  Archeology  and  Art  –  Digi4za4on  of  cultural    heritage  and  ar4s4c  works  

•  …  

Oriented  Points    

Posi4ons,  Normals,  Colors  

Methods  to  Capture  Oriented  Points  

Laser  Scanning    

Mul4-­‐View  Stereo    

Structured  Ligh4ng  

Laser  range  scanning   Mul4-­‐View  Stereo  

Data  Acquisi3on  

Structured  Ligh4ng  

Gray  Code  Structured  Ligh3ng  Results  

hBp://mesh.brown.edu/byo3d/  

hBp://mesh.brown.edu/byo3d  

Surface  Reconstruc3on  from  Mul3-­‐View  Data  

SSD:  Smooth  Signed  Distance  Surface  Reconstruc3on  F.  Calakli,  G.  Taubin,  Computer  Graphics  Forum,  2011.    

•  Con4nuous  mathema4cal  formula4on  •  Par4cular  discre4za4on  and  algorithm  •  Reconstructs  water4ght  surface  as  polygon  mesh  •  From  a  sta4c  oriented  point  cloud  

hBp://mesh.brown.edu/ssd  

Related  Papers  &  Projects  

•  Vector  Field  Isosurface-­‐Based  Reconstruc4on  From  Oriented  Points,  by  Sibley  &  Taubin,  Siggraph  2005  (Sketch).  

•  Smooth  Signed  Distance  Surface  Reconstruc4on,  by  Calakli  &  Taubin,  PG  2011  &  Computer  Graphics  Forum  2011.  

•  Smooth  Signed  Distance  Colored  Surface  Reconstruc4on,  by  Calakli  &  Taubin,  chapter  in  State-­‐of-­‐the-­‐Art  Volume  on  Computer  Graphics,  Visualiza4on,  Visual  Analy4cs,  VR  and  HCI,  2012.  

•  Accurate  3D  Footwear  Impression  Recovery  from  Photographs,  by  Andalo,  Calakli,  Taubin,  and  Goldenstein,  Proceedings  of  the  4th.  Interna4onal  Conference  on  Imaging  for  Crime  Detec4on  and  Preven4on  (ICDP-­‐2011).  

•  High  Resolu4on  Surface  Reconstruc4on  from  Mul4-­‐view  Aerial  Imagery,  Calakli,  Ulusoy,  Restrepo,  Mundy  &  Taubin,  3DIMPVT  2012  

•  REVEAL  Digital  Archaeology  Project    •  Cuneiform    Automa4c  Transla4on  Project  

Par3cularly  Good  at  Extrapola3ng  Missing  Data  

<  0  

>  0  

0  )( fZ

Oriented  Points,  D    (samples  from  unknown  surface  S)  

Find    a  scalar  valued  func4on                                      ,  whose  zero    level  set  Z(f)  =  S’  is  the  es4mate  for  true  surface  S  

Computed  Implicit  Surface,  S’  

f

ℜ→Df :

Implicit  func3on  framework  

•  Input:  oriented  point  set:        D  =  {  (  pi,  ni  )  i=1,…,N}  

       contained  in  a  bounding  volume  V  •  Output:  implicit  surface    

 S  =  {  x  |  f  (x)  =  0  }          with  the  func4on  defined  on  V,  such  that    

         f  (pi)  =  0      and      ∇f  (pi)  =  ni    ∀(pi,ni)  ∈  D  •  A  family  of  implicit  func4ons  with  a  finite  number  of  parameters  has  to  be  chosen  

•  Parameters  must  be  es4mated  so  that  the  condi4ons  stated  above  are  sa4sfied,  if  not  exactly,  then  in  the  least-­‐squares  sense  

Implicit  Curve  and  Surface  Reconstruc3on  

Challenges  

Uniform  sampling  

Non-­‐uniform  sampling  

Noisy  data  

Misaligned  scans  

•  Interpola4ng  polygon  meshes  Boissonnat  [1984],  Edelsbrunner  [1984]  Amenta  et  al.  [1998],  Bernardini  et  al.  [1999]  Dey  et  al.  [2003][2007],  …  

•  Implicit  func4on  fijng  Taubin  [1991],  Hoppe  et  al.  [1992],  Curless  et  al.  [1996]  Whitaker  [1998],  Carr  et  al.[2001],  Davis  et  al.  [2002],    Ohtake  et  al.  [2004],  Turk  et  al.  [2004],  Shen  et  al.  [2004]  Sibley-­‐Taubin  [2005]  

General  Approaches  

Poisson  Surface  Reconstruc3on  

   Kazhdan  et  al.  [2006]                          Manson  et  al.  [2008]    

Poisson  Surface  Reconstruc3on  

1.  Extend  oriented  points  to  con4nuous  vector  field                      defined  on  the  whole  volume,  so  that  

2.  Integrate  vector  field,  by  minimizing  

3.  Determine  isolevel,  by  minimizing  

||∇f (p)− v(p) ||2 dpV∫

( f (pi )− f0 )2

i=1

N

∑

v(pi ) ≈ ni

v(p)

Main  problem  with  this  approach  

Main  problem  with  this  approach  

Main  problem  with  this  approach  

Main  problem  with  this  approach  

What  kind  of  implicit  func3on?  

Indicator  Func4on   Smooth  Signed  Distance  Func4on  

1  

0  

0  

Vectorfield  Isosurface-­‐Based  Reconstruc3on  From  Oriented  Points  P.  Sibley  and  G.  Taubin  [Siggraph  2005  Sketch]  

•  Surface  reconstruc4on  from  cloud  of  oriented  points  •  Implicit  representa4on  can  deal  with  missing  data  •  Rather  than  fijng  analy4c  func4on  (RBFs,  etc),  and  then  

extract  isosurface  for  visualiza4on,  fit  isosurface  directly  to  data  

E ( f ) =P

i f ( pi ) 2+P

i kr f ( pi ) ¡ n i k2+RV kH f ( x ) k2dx

Vectorfield  Isosurface-­‐Based  Reconstruc3on  From  Oriented  Points  P.  Sibley  and  G.  Taubin  [Siggraph  2005  Sketch]  

•  Surface  reconstruc4on  from  cloud  of  oriented  points  •  Implicit  representa4on  can  deal  with  missing  data  •  Rather  than  fijng  analy4c  func4on  (RBFs,  etc),  and  then  

extract  isosurface  for  visualiza4on,  fit  isosurface  directly  to  data  •  Ini4al  approach:  3  steps  

1.  Fit  con4nuous  vector  field  to  data  normal  vectors  2.  Integrate  vector  field  on  regular  grid  3.  Determine  Isolevel  

E ( f ) =P

i f ( pi ) 2+P

i kr f ( pi ) ¡ n i k2+RV kH f ( x ) k2dx

VFIso:  Representa3on  

•  Regular  voxel  grid    •  Implicit  func4on  defined  by  values  at  ver4ces  •  And  tri-­‐linear  interpola4on  •  The  output    is:  

∑= α ααϕ )()( pfpf

αf )(pαϕ

}{ αf

VFIso:  Varia3onal  Approach  

•  Extend  data  to  vector  field  on  the  volume  minimizing  

tionregularizadata

DvvnpvvE ∑∑ −+−=

),()()(

βαβαλ22

∑=α

ααϕ )()( :where pvpv

•  Then  integrate  vector  field  ?    

VFIso:  Details  

•  Problem:  Vector  field  v(p)  not  integrable!  •  Instead:  Fit  scalar  field                        directly  by  minimizing      

∑∑ ∇−∇+−∇=),(

22)(})({βα

βαα λ ffnpffED

}{ αf

• Where   ∑ ∇=∇α ααϕ )()( pfpf

αf∇

∑ −=D

fpffE200 )()(

•  And                      is  discre4zed  using  central  differences  •  This  is  a  standard  (Laplacian)  Least  Squares  problem  

•  Minimize  to  find  isolevel  

         Squirrel(9K)                Angel(24K)                  Bunny(35K)            Ram(678K)  

Some  Methods  to  Capture  3D  Point  Clouds  

Shadow

Turntable

Backdrop

8 Megapixel Camera

Multi-Flash Attachment

Multi-Flash Camera

Beyond  SilhoueBes:  Surface  Reconstruc3on  using  Mul3-­‐Flash  Photography  

D.  Crispell,  D.  Lanman,  P.  Sibley,  Y.  Zhao  and  G.  Taubin  [3DPVT  2006]  

33  

Shadow  

Turntable  

Backdrop  

8  Megapixel  Camera  

Mul3-­‐Flash  ABachment  

Mul3-­‐Flash  Camera  

Multi-Flash 3D Photography:

Capturing the Shape and Appearance of 3D Objects

Turntable  Rota3on  

A new approach for reconstructing 3D objects using shadows cast by depth discontinuities, as detected by a multi-flash camera. Unlike existing stereo vision algorithms, this method works even with plain surfaces, including unpainted ceramics and architecture.

Estimated Shape: 3D Point Cloud

Recovered Appearance: Phong BRDF Model

Multi-Flash Turntable Sequence: Input Image

Data Capture: A turntable and a digital camera are used to acquire data from 670 viewpoints. For each viewpoint, we capture a set of images using illumination from four different flashes. Future embodiments will include a small, inexpensive handheld multi-flash camera.

Recovering a Smooth Surface

The reconstructed point cloud can possess errors, including gaps and noise. To minimize these effects, we find an implicit surface which interpolates the 3D points. This method can be applied to any 3D point cloud, including those generated by laser scanners.

VFIso  Results  [2006  110x110x110  grid]  

•  Oriented  point  set:        D  =  {  (  pi,  ni  )  }    sampled  from  a  surface  S  

•  Implicit  surface:      S  =  {  x  |  f  (x)  =  0  }  such  that              f  (pi)  =  0      and      ∇f  (pi)  =  ni    ∀(pi,ni)  ∈  D  

•  Least  squares  energy:    

SSD  Con3nuous  Formula3on  

E( f ) = f (pi )2

i=1

N

∑ +λ1 ∇f (pi )−ni2

i=1

N

∑ ∫∑∑ +−∇+===

V

N

i

N

idffffE xxnpp iii

22

1

21

1

2 ||)(H||)()()( λλ

What  does  the  regulariza3on  term  do  ?  

•  Near  data  points:  since  the  data  terms  dominate,  the  func4on  approximates  the  signed  distance  

•  Away  from  data  points:  the  regulariza4on  term  dominates  and  forces  the  gradient  to  be  smooth  and  close  to  constant  

∫∑∑ +−+===

V

N

i

N

idMvfMvfE xxnpp iii

22

1

21

1

2 ||)(||)()(),,( λλ

Role  of  each  energy  term  

Quadra4c  energy  in  f,  v,  and  M  

If  f,  v,  and  M  are  linear  func3ons  of  the  same  parameters,  then  the  minimiza4on  reduces  to  a  least  squares  problem    

Linear  families  of  func3ons  

•  Popular  Smooth  Basis  Func4ons  – Monomials  [Taubin’91]  –  Radial  basis  func4ons  [Carr  et  al.,  ‘01],    –  Compactly  supported  basis  func4ons  [Othake  et  al.  ‘04],  –  Trigonometric  polynomials  [Kazhdan  et  al.  ‘05],  –  B-­‐splines  [Kazhdan  et  al.,  06],    – Wavelets  [Manson  et  al.  ‘08],    

 cFbAFFFE tt +−= 2)(Non-­‐homogenous,    

Quadra4c  energy  

bAF =Global  minimum  

We  can  use  Independent  Discre3za3ons  

•  Hybrid  FE/FD  discre4za4on    – Trilinear  interpolant  for  the  func4on  f(x)  – Primal  finite  differences  for  the  gradient  ∇f(x)  – Dual  finite  differences  for  the  Hessian  Hf(x)  

•  As  long  as  f(x),  ∇f(x),  and  Hf(x)  are  wrixen  as  a  linear  combina4ons  of  the  same  parameter  vector  F  

cFbAFFFE tt +−= 2)(Non-­‐homogenous,    Quadra4c  energy  

bAF =Global  minimum  

Implementa3on  

•  Primal-­‐Dual  octree  data  structure  •  Cascading  mul4-­‐grid  itera4ve  solver  (conjugate  gradient):  Solve  the  problem  on  a  much  coarser  level  – Use  the  solu4on  at  that  level  to  ini4alize  the  solu4on  at  the  next  level    

– Refine  with  the  itera4ve  solver  •  Iso-­‐surface  extrac4on  (crack-­‐free)  

– Dual  marching  cubes  [Schaefer  2005]  

Marching  Cubes  on  Octrees  •  Non-­‐conforming  hexahedral  mesh    •  Results  in  crack  problem.  •  Problem  solved  by  Dual  Marching  Cubes  

MPU    [Othake  ‘03  ]  

Poisson    [Kazhdan  ‘06  ]  

D4  Wavelets    [Manson  ‘08  ]  

SSD  [Calakli  ’11]  

Input  point  cloud  

SSD  Surface  Reconstruc3on  •  Theore4cal  contribu4ons:  

– Oriented  point  samples  regarded  as  samples  of  Euclidean  signed  distance  func4on  

– Reconstruc4on  as  global  minimiza4on  problem  – Yet  sparse  system  of  linear  equa4ons  

•  Empirical  advantages:  – Robust  to  noise  and  uneven  sampling  density  

•  Future  work:  – Streaming  out-­‐of-­‐core  implementa4ons  – Parallel/Mul4-­‐core/GPU  implementa4ons  – Dynamic  shapes  

REVEAL  Digital  Archaeology  Project  Cooper,  Kimia,  Taubin,  Galor,  Sanders,  Willis  

•  The  main  goal  is  to  automate  the  tedious  processes  of  data  collec4on  and  documenta4on  at  the  excava4on  site,  as  well  as  to  provide  visualiza4on  tools  to  explore  the  data  collected  in  the  database  

•   Also  to  solve  specific  problems  in  Archaeology  using  computer  vision  techniques.  

•  We  first  used  a  network  of  cameras  to  capture  the  ac4vity  at  the  excava4on  site,  to  reconstruct  the  shape  of  the  environment  as  it  is  being  excavated,  to  reconstruct  layers,  and  to  locate  finds  in  3D  

•  Then  we  switched  to  mul4-­‐view  stereo  with  photos  captured  with  hand-­‐held  cameras  

Semi-­‐automa3cally      Assemble      Fragments    Into  Ar3facts  

Assisted Data Acquisition, Algorithmic Reconstruction, Integrated multi-format analysis

REVEAL Archaeological Data Acquisition

Objects:  Ar4facts  

Excava4ons  Areas  Sites  

Data  Acquisi3on  

Data:  Text  Photos  Video  3D  Models  

Automa3cally  Convert  Photos  to  3D  Models  

Site  Plans  

Import  photos,  videos,  and  laser  scans  and  connect  them  to    database  objects  

Improve  speed  and  accuracy  with  computer  assisted  data  entry  

Advanced  Algorithms  

Import  External  Data  

Laser  Scanned  Models  

Rule-­‐based    Reconstruc3ons  

REVEAL  Database  

hBp://sourceforge.net/projects/revealanalyze  

Typical Activity Sequence

Data  integrated  and  synchronized  in  tabular,  plan  drawing,  3D  spa3al,  image,  and  video  formats  REVEAL Archaeological Analysis

Select  Ar3facts  on  Site  Plan  

Ar4facts  Excava4ons  

Areas  Sites  

Display  Photos  of  Selected  Ar3facts  

Examine  Rela3onship  of  Ar3facts  in-­‐situ  in  auto-­‐generated  3D  Excava3on  Model    

Export  FormaBed  Ar3fact  Data  for  inclusion  in  Site  Publica3on  

[Furukawa  and  Ponce  2008]  

[Snavely    et.  al.  2006]  

MVS    soaware  

Patch-­‐based  Mul4-­‐View  Stereo  (PMVS)  hxp://grail.cs.washington.edu/soyware/pmvs/  

hxp://phototour.cs.washington.edu/bundler/  

Accurate  3D  Footwear  Impression  Recovery  From  Photographs,  F.  A.  Andalo,  F.  Calakli,  G.  Taubin,  and  S.  Goldenstein,  

Interna4onal  Conference  on  Imaging  for  Crime  Detec4on  and  Preven4on  (ICDP-­‐2011).    

Comparable  to  3D  Laser  Scanner  

Handheld  Interac4ve,  Incremental  3D  Object  Scanning  Kim,  Honors  Thesis,  Brown  University,  May  2012  

•  Based  on  MS  Kinect  sensor  

•  Con4nuous  coarse  pose  es4ma4on  from  color  camera  using  PTAM  

•  3D  snapshots  captured  from  different  points  of  view  using  depth  camera  

•  Alignment  improved  with  Itera4ve  Closest  Points  (ICP)  algorithm  

•  SSD  is  run  on  aligned  3D  snapshots  

Screen  shots  of  our  real-­‐3me  stereo  system  working  on  the  field  

Real-­‐Time  High-­‐Defini3on  Stereo  on  GPGPU  using  Progressive  Mul3-­‐Resolu3on  Adap3ve  Windows  

Y.  Zhao,  and  G.  Taubin,  Image  and  Vision  Compu4ng  2011.    

¼ Resolution ½ Resolution Full Resolution¼ Resolution ½ Resolution Full Resolution

Coarse-­‐to-­‐fine  matching  on  mul;ple  resolu;ons  

Stereo  Frame  Grabbing  

Stereo  Rec3fica3on  

Mul3-­‐Resolu3on  Pyramid  Genera3ng  

Background  Modeling  With  Shadow  Removal    Foreground  Detec3on  with  Dila3on  &  Erosion    

Stereo  Matching  Using  Adap3ve  Window  with  Cross-­‐Checking  

Disparity  Refinement  Re

solu3o

n  Scan

 from

 Low

 to  

High  

Processing  Pipeline  

Real-­‐Time  High-­‐Defini3on  Stereo  on  GPGPU  using  Progressive  Mul3-­‐Resolu3on  Adap3ve  Windows  

Y.  Zhao,  and  G.  Taubin,  Image  and  Vision  Compu4ng  2011.    

High  Resolu3on  Surface  Reconstruc3on  from  Mul3-­‐view  Aerial  Imagery  by  Calakli,  Ulusoy,  Restrepo,  Mundy  &  Taubin,  3DIMPVT  2012  

Microscopic  3D  Shape  Capture  Liberman  &  Taubin  (work  in  progress)  

~  4  mm  

Ques4ons?  

This  material  is  based  upon  work  supported  by  the  Na4onal  Science  Founda4on  under  Grants  CCF-­‐0729126,  IIS-­‐0808718,  CCF-­‐0915661,  and  

IIP-­‐1215308.