acquiring 3d indoor environments with variability and repetition young min kim stanford university...
TRANSCRIPT
![Page 1: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/1.jpg)
1
Acquiring 3D Indoor Environments with Variability and Repetition
Young Min Kim Stanford University
Niloy J. MitraUCL/ KAUST
Dong-Ming YanKAUST
Leonidas GuibasStanford University
![Page 2: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/2.jpg)
2
Data Acquisition via Microsoft Kinect
Raw data: Noisy point clouds Unsegmented Occlusion issues
Our tool: Microsoft Kinect
Real-time Provides depth and color Small and inexpensive
![Page 3: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/3.jpg)
3
Dealing with Pointcloud Data
• Object-level reconstruction
• Scene-level reconstruction[Chang and Zwicker 2011]
[Xiao et. al. 2012]
![Page 4: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/4.jpg)
4
Mapping Indoor Environments
• Mapping outdoor environments– Roads to drive vehicles– Flat surfaces
• General indoor environments contain both objects and flat surfaces– Diversity of objects of interest– Objects are often cluttered– Objects deform and move
Solution: Utilize semantic information
![Page 5: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/5.jpg)
5
Nature of Indoor Environments
• Man-made objects can often be well-approximated by simple building blocks– Geometric primitives– Low DOF joints
• Many repeating elements – Chairs, desks, tables, etc.
• Relations between objects give good recognition cues
![Page 6: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/6.jpg)
6
Indoor Scene Understanding with Pointcloud Data
• Patch-based approach
• Object-level understanding [Silberman et. al. 2012]
[Koppula et. al. 2011]
[Shao et. al. 2012] [Nan et. al. 2012]
![Page 7: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/7.jpg)
7
Comparisons
[1] An Interactive Approach to Semantic Modeling of Indoor Scenes with an RGBD Camera[2] A Search-Classify Approach for Cluttered Indoor Scene Understanding
[1] [2] oursPrior model 3D database 3D database Learned
Deformation Scaling Part-based scaling Learned
Matching Classifier Classifier GeometricSegmentation User-assisted Iteration Iteration
Data Microsoft Kinect Mantis Vision Microsoft Kinect
![Page 8: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/8.jpg)
8
Contributions
• Novel approach based on learning stage– Learning stage builds the model that is specific to
the environment• Build an abstract model composed of simple
parts and relationship between parts– Uniquely explain possible low DOF deformation
• Recognition stage can quickly acquire large-scale environments– About 200ms per object
![Page 9: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/9.jpg)
9
Approach
• Learning: Build a high-level model of the repeating elements
• Recognition: Use the model and relationship to recognize the objects
translational
rotational
![Page 10: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/10.jpg)
10
Approach
• Learning– Build a high-level model of the repeating elements
![Page 11: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/11.jpg)
11
Output Model: Simple, Light-Weighted Abstraction
• Primitives– Observable faces
• Connectivity– Rigid– Rotational– Translational– Attachment
• Relationship– Placement information
3m3m
2m2m
1m1m
ggcontact
translational
rotational3
1l
Mlmmm },,,,{ 31321
![Page 12: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/12.jpg)
12
Joint Matching and Fitting
• Individual segmentation– Group by similar normals
• Initial matching– Focus on large parts– Use size, height, relative positions– Keep consistent match
• Joint primitive fitting– Add joints if necessary– Incrementally complete the model
![Page 13: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/13.jpg)
13
Approach
• Learning– Build a high-level model of the repeating elements
![Page 14: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/14.jpg)
14
Approach
• Learning– Build a high-level model of the repeating elements
• Recognition– Use the model and relationship to recognize the
objects
![Page 15: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/15.jpg)
15
Hierarchy
• Ground plane and desk• Objects– Isolated clusters
• Parts– Group by normals
• The segmentation is approximate and to be corrected later
S},,{ 21 oo
iopp },,{ 21
![Page 16: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/16.jpg)
16
Bottom-Up Approach
• Initial assignment for parts vs. primitives– Simple comparison of height, normal, size– Robust to deformation– Low false-negatives
• Refined assignment for objects vs. models– Iteratively solve for position, deformation and
segmentation– Low false-positives
parts
![Page 17: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/17.jpg)
17
Bottom-Up Approach
• Initial assignment for parts vs. primitive nodes• Refined assignment for objects vs. models
Input points
Initial objects
Models matched
Refined objectsobjects parts matched
![Page 18: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/18.jpg)
18
Results
Data available:http://www0.cs.ucl.ac.uk/staff/n.mitra/research/acquire_indoor/paper_docs/data_learning.ziphttp://www0.cs.ucl.ac.uk/staff/n.mitra/research/acquire_indoor/paper_docs/data_recognition.zip
![Page 19: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/19.jpg)
19
Synthetic Scene
Recognition speed: about 200ms per object
![Page 20: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/20.jpg)
20
Synthetic Scene
![Page 21: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/21.jpg)
21
Synthetic Scene
![Page 22: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/22.jpg)
22
0 0.2 0.4 0.6 0.8 1 1.20
0.2
0.4
0.6
0.8
1
1.2Data type
Gaussian 0.004 Gaussian 0.004Gaussian 0.3 Gaussian 0.3Gaussian 1.0 Gaussian 1.0
Precision
Reca
ll
Different pair Similar pair
![Page 23: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/23.jpg)
23
0 0.2 0.4 0.6 0.8 1 1.20
0.2
0.4
0.6
0.8
1
1.2Data type
Gaussian 0.004 Gaussian 0.004Gaussian 0.3 Gaussian 0.3Gaussian 1.0 Gaussian 1.0
Precision
Reca
ll
Different pair Similar pair
![Page 24: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/24.jpg)
24
0 0.2 0.4 0.6 0.8 1 1.20
0.2
0.4
0.6
0.8
1
1.2Noise
Gaussian 0.004 Gaussian 0.008Gaussian 0.3 Gaussian 0.5Gaussian 1.0 Gaussian 2.0
Precision
Reca
ll
0 0.2 0.4 0.6 0.8 1 1.20
0.2
0.4
0.6
0.8
1
1.2Density
density 0.4 density 0.5density 0.6 density 0.7density 0.8
Precision
Reca
ll
![Page 25: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/25.jpg)
25
Office 1
trash bin
4 chairs2 monitors
2 whiteboards
![Page 26: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/26.jpg)
26
Office 2
![Page 27: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/27.jpg)
27
Office 3
![Page 28: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/28.jpg)
28
Deformations
drawer deformations
monitorlaptopmissed monitor
chair
![Page 29: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/29.jpg)
29
Auditorium 1Open table
![Page 30: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/30.jpg)
30
Auditorium 2
Open table
Open chairs
![Page 31: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/31.jpg)
31
Seminar Room 1
missed chairs
![Page 32: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/32.jpg)
32
Seminar Room 2
missed chairs
![Page 33: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/33.jpg)
33
Limitations
• Missing data– Occlusion, material, …
• Error in initial segmentation– Cluttered objects are merged as a single segment– View-point sometimes separate single object into
pieces
![Page 34: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/34.jpg)
34
Conclusion
• We present a system that can recognize repeating objects in cluttered 3D indoor environments.
• We used purely geometric approach based on learned attributes and deformation modes.
• The recognized objects provide high-level scene understanding and can be replaced with high-quality CAD models for visualization (as shown in the previous talks!)
![Page 35: Acquiring 3D Indoor Environments with Variability and Repetition Young Min Kim Stanford University Niloy J. Mitra UCL/ KAUST Dong-Ming Yan KAUST Leonidas](https://reader036.vdocuments.mx/reader036/viewer/2022062518/56649f3d5503460f94c5c7bf/html5/thumbnails/35.jpg)
35
Thank You
• Qualcomm Corporation• Max Planck Center for Visual Computing and Communications• NSF grants 0914833 and 1011228• a KAUST AEA grant• Marie Curie Career Integration Grant 303541• Stanford Bio-X travel Subsidy