learning to generate 3d shapes - umass amherstsmaji/presentations/3dshape... · 2019-12-21 ·...
TRANSCRIPT
![Page 1: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/1.jpg)
Learning to generate 3D shapesSubhransu MajiCollege of Information and Computer SciencesUniversity of Massachusetts, Amhersthttp://people.cs.umass.edu/smaji
August 10, 2018 @ Caltech
![Page 2: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/2.jpg)
ImagefromAutodesk3DMaya
Creating 3D shapes is not easy
![Page 3: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/3.jpg)
3
Inferring 3D shapes from images
What shapes are puffins?
What shapes are pumpkinseed fish?
![Page 4: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/4.jpg)
• Manytechniquesforrecognizing3Ddata,butrelativelyfewtechniquesforgeneratingthem
• Representationsforgeneration?• Voxels• Multiview• Geometryimages• Shapebasis• Set-based(points,triangles,etc.)• Procedural,e.g.,constructivesolidgeometry
Creating 3D shapes is not easy
![Page 5: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/5.jpg)
Talk overview
• Generativemodelsfor3Dshapesandapplications• Multiview[3DV’17]• Multiresolutiontreenetworks[ECCV’18]• Constructivesolidgeometry[CVPR’18]
• Learning3Dshapeswithweaksupervision[3DV’17]
![Page 6: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/6.jpg)
3D Shape Reconstruction from Sketches
via Multi-view Convolutional Networks
ZhaoliangLun MatheusGadelha
EvangelosKalogerakis SubhransuMaji
RuiWang
3DV 2017
Part 1
![Page 7: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/7.jpg)
3D Shape Reconstruction from Sketches
via Multi-view Convolutional Networks
ZhaoliangLun MatheusGadelha
EvangelosKalogerakis SubhransuMaji
RuiWang
3DV 2017
Part 1
![Page 8: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/8.jpg)
Why line drawings? Simple & intuitive
medium to convey shape!
ImagefromSuggestiveContourGallery,DeCarloetal.2003
![Page 9: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/9.jpg)
Goal: 2D line drawings in, 3D shapes out!
ShapeMVDfrontview
sideview 3Dshape
![Page 10: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/10.jpg)
Deep net architecture: U-net structure
Featurerepresentationsinthedecoderdependonpreviouslayer&encoder’scorrespondinglayer
U-net:Ronnebergeretal.2015,Isolaetal.2016
frontview
sideview
outputview1
outputview12
![Page 11: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/11.jpg)
Penalizeper-pixeldepthreconstructionerror:&per-pixelnormalreconstructionerror:&“unreal”outputs:
Training: full loss
frontview
sideview
GeneratorNetwork
outputview1
outputview12
…frontview
sideview
outputview1
outputview12
log ( )P real−
DiscriminatorNetwork
Real?Fake?
Real?Fake?
DiscriminatorNetwork
cGAN:Isolaetal.2016
| |pred gtpixels
d d−∑(1 )pred gt
pixels
n n− ⋅∑
![Page 12: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/12.jpg)
Training data
Character10Kmodels
Chair10Kmodels
Airplane3Kmodels
Modelsfrom“TheModelsResource”&3DWarehouse
![Page 13: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/13.jpg)
…
Syntheticlinedrawings
Trainingdepthandnormalmaps
12views
Training data
![Page 14: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/14.jpg)
Predict multi-view depth and normal maps!
outputview1
outputview12
frontview
sideview
Test time
![Page 15: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/15.jpg)
• Depth derivatives should be
consistent with normals
• Corresponding depths and
normals across different
views should agree
Optimization problem
Multi-viewdepth&normalmaps
Consolidated pointcloud
outputview1
outputview12
Multi-view depth & normal map fusion
![Page 16: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/16.jpg)
Multi-viewdepth&normalmaps
Consolidated pointcloud
Surfacereconstruction[Kazhdanetal.2013]
outputview1
outputview12
Surface reconstruction
![Page 17: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/17.jpg)
Multi-viewdepth&normalmaps
Consolidated pointcloud
Surface“fine-tuning”
[Nealenetal.2005]
outputview1
outputview12Surface
reconstruction[Kazhdanetal.2013]
Surface deformation
![Page 18: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/18.jpg)
Multi-viewdepth&normalmaps
Consolidated pointcloud
Surface“fine-tuning”[Nealenetal.2005]
outputview1
outputview12Surface
reconstruction[Kazhdanetal.2013]
Surface deformation
![Page 19: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/19.jpg)
Experiments
![Page 20: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/20.jpg)
Qualitative Results
reference shape
reference shape
nearestretrieval
nearestretrieval
our result
our result
volumetricnet
volumetricnet
![Page 21: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/21.jpg)
reference shape
reference shape
nearestretrieval
nearestretrieval
our result
our result
volumetricnet
volumetricnet
Qualitative Results
![Page 22: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/22.jpg)
Quantitative Results
Metric Ourmethod Volumetric NN
Hausdorffdistance 0.120 0.638 0.242Chamferdistance 0.023 0.052 0.045normaldistance 34.27 56.97 47.94
volumetricdistance 0.309 0.497 0.550
Character(humandrawing)
Hausdorffdistance 0.171 0.211 0.228Chamferdistance 0.028 0.032 0.038normaldistance 34.19 48.81 43.75
volumetricdistance 0.439 0.530 0.560
Man-made(humandrawing)
![Page 23: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/23.jpg)
Single vs two input line drawings
Singlesketch
Twosketches
Resultingshape
Resultingshape
![Page 24: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/24.jpg)
More results
![Page 25: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/25.jpg)
Multiresolution Tree Networks for 3D
Point Cloud Processing
MatheusGadelhaSubhransuMaji
RuiWang
ECCV 18
Part 2
![Page 26: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/26.jpg)
Multiresolution Tree Networks for 3D
Point Cloud Processing
MatheusGadelhaSubhransuMaji
RuiWang
ECCV 18
Part 2
![Page 27: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/27.jpg)
Point-cloud decoders
• Globalshapebasisorfully-connecteddecoders
Morphable models[Figure from Booth et al., 16]
Requires perfect correspondence
![Page 28: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/28.jpg)
How important is correspondence?
Gadelha et al., BMVC 17
Related work: Fan et al., CVPR 2017
![Page 29: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/29.jpg)
How important is correspondence?
![Page 30: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/30.jpg)
Multiresolution tree networks
• Addressesthelackof• convolutionalstructure,and• coarsetofinereasoning
• Basicidea:linearize3Dpointsanduse1Dconvolutions
Points colored with kdtree sort index
![Page 31: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/31.jpg)
Multiresolution tree networks
• Addressesthelackof• convolutionalstructure,and• coarsetofinereasoning
• Basicidea:linearize3Dpointsanduse1Dconvolutions
Implicit mulitresolution structure
![Page 32: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/32.jpg)
Multiresolution tree networks
Architecture for encoding and decoding
Multiresolution convolution block
![Page 33: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/33.jpg)
Does multiresolution analysis help?
single-resolutionfully-connected multi-resolution
Color indicates the position in the list
![Page 34: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/34.jpg)
Other shape tasks with MRTNet
![Page 35: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/35.jpg)
Single image shape reconstruction
![Page 36: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/36.jpg)
Quantitative evaluation: ShapeNet dataset
voxel-based fully-conn. multiview
Chamfer distance: pred g GT / GT g pred
Lin et al. AAAI 2018
![Page 37: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/37.jpg)
Quantitative evaluation: ShapeNet dataset
voxel-based fully-conn. multiview
Chamfer distance: pred g GT / GT g pred
![Page 38: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/38.jpg)
MRTNet summary
• Agenericarchitecturefor• Pointcloudclassification(91.7%onModelNet40)• Semanticsegmentation(seeresultsinthepaper)• Generation
• Projectpage:http://mgadelha.me/mrt/index.html
![Page 39: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/39.jpg)
CSGNet: Neural Shape Parser for
Constructive Solid Geometry
GopalSharmaRishabhGoyal
DifanLiuEvangelosKalogerakis
SubhransuMaji
interpretable and editable
Part 3
CVPR 18
![Page 40: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/40.jpg)
Constructive 2D geometry
![Page 41: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/41.jpg)
Constructive solid geometry
![Page 42: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/42.jpg)
Constructive solid geometry
Circle1Triangle1-Circle2-Triangle2-Triangle3-[STOP]
Postfix notation
CNN RNN Program
![Page 43: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/43.jpg)
Learning
• Supervisedsetting:learntopredictprogramsdirectly
• Unsupervisedsetting:Noground-truthprograms.• Learnparameterstominimizeareconstructionerrorthroughpolicygradients[REINFORCE,Willams1992]
CNN RNN Program
reconstruction loss
![Page 44: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/44.jpg)
CSGNet: 2D/3D a programs
![Page 45: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/45.jpg)
CSGNet: 2D/3D a programs
Train on synthetic data and adapt to new domains using policy gradients
Logos
Synthetic CAD shapes
![Page 46: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/46.jpg)
CSGNet: 2D/3D a programs
How well does the nearest neighbor perform? Chamfer distance 1.88 (NN), 1.36 (CSGNet) with 675K training examples
![Page 47: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/47.jpg)
CSGNet: 2D/3D a programs
How well does the nearest neighbor perform? Chamfer distance 1.88 (NN), 1.36 (CSGNet) with 675K training examples
![Page 48: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/48.jpg)
CSGNet: 2D/3D a programs
CAD shapes dataset: Chamfer distance 1.94 (NN), 0.51 (CSGNet)
![Page 49: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/49.jpg)
CSGNet: 2D/3D a programs
Input
• Moreresultsinthepaper:rewardshaping,comparisontoFasterR-CNNforprimitivedetection,resultson3D,etc.
• Preprintavailable:https://arxiv.org/abs/1712.08290
![Page 50: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/50.jpg)
Learning 3D Shape Representations
with Weak Supervision
MatheusGadelhaSubhransuMaji
RuiWang
3DV 2017
Part 4
![Page 51: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/51.jpg)
Related Work
• 3Dshapefromcollectionofimages• Visualhull—sameinstance,knownviewpoints
• Photometricstereo—sameinstance,knownlighting,simplereflectance
• Structurefrommotion—sameinstance(or3D)
• Non-rigidstructurefrommotion—knownshapefamily(e.g.,faces)
• Ourwork—unknownshapefamily,unknownviewpoints
• 3Dshapefromsingleimage• Optimization-basedapproaches;
• Recognition-basedapproaches;
![Page 52: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/52.jpg)
A motivating example
Small cubes are reddishBig cubes are bluish
Hypothesis: It is easier to generate these images by reasoning in 3D
![Page 53: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/53.jpg)
Approach• Our goal is to learn a 3D shape generator whose
projections match the provided set of the views
Generator Projection
![Page 54: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/54.jpg)
Approach• Our goal is to learn a 3D shape generator whose
projections match the provided set of the views
Generator Projection
How do we match distributions?
![Page 55: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/55.jpg)
Approach• Our goal is to learn a 3D shape generator whose
projections match the provided set of the views
Generator Projection
How do we match distributions?
min
GDKL(G||D) = min
z⇠GElog
G(z)
D(z)
�generated true
![Page 56: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/56.jpg)
Approach• Our goal is to learn a 3D shape generator whose
projections match the provided set of the views
Generator Projection
How do we match distributions?
min
GDKL(G||D) = min
z⇠GElog
G(z)
D(z)
�generated trueestimate using
logistic regression
![Page 57: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/57.jpg)
Approach• Our goal is to learn a 3D shape generator whose
projections match the provided set of the views
Generator Projection
How do we match distributions?
min
GDKL(G||D) = min
z⇠GElog
G(z)
D(z)
�generated true
min
G
max
d
Ex⇠D
[log d(x)] + Ez⇠G
[log(1� d(z))]
Generative adversarial networks [Goodfellow et al.]
estimate using logistic regression
![Page 58: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/58.jpg)
PrGAN
Projection using line integration along the view direction
I(x) = 1� exp
✓�Z 1
0V (x+ r)dr
◆
Generator maps z to a voxel occupancy grid and a viewpoint
![Page 59: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/59.jpg)
Dataset generation
![Page 60: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/60.jpg)
Airplanes
input generated
![Page 61: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/61.jpg)
Airplanes
input generated
![Page 62: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/62.jpg)
Vases
input generated
![Page 63: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/63.jpg)
Vases
input generated
![Page 64: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/64.jpg)
Mixed categories
input generated
![Page 65: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/65.jpg)
MMD metric: 2D-GAN 90.1, PrGAN 88.3
![Page 66: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/66.jpg)
MMD metric 3D-GAN 347.5PrGAN 442.9
Trained with 3D supervision
![Page 67: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/67.jpg)
Projection GAN
• Themodelisabletorecoverthecoarse3Dstructure
• Butshouldusesideinformationwhenavailable• Viewpoint• Landmarks/partlabels• Poseestimates
• Iterative:bootstrap3Dtoestimatepose&viewpoint
![Page 68: Learning to generate 3D shapes - UMass Amherstsmaji/presentations/3dShape... · 2019-12-21 · Learning to generate 3D shapes Subhransu Maji ... CSGNet: 2D/3D a programs Train on](https://reader034.vdocuments.mx/reader034/viewer/2022042220/5ec6050d5638540e6d6ee5d5/html5/thumbnails/68.jpg)
Thank you!
• Collaborators:MatheusGadelha,ZhaoliangLun,GopalSharma,RuiWang,EvangelosKalogerakis
• FundingfromNSF,NVIDIA,Facebook• https://people.cs.umass.edu/smaji/projects.html