![Page 1: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/1.jpg)
Structured light
15-463, 15-663, 15-862Computational Photography
Fall 2017, Lecture 27http://graphics.cs.cmu.edu/courses/15-463
![Page 2: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/2.jpg)
Course announcements
• Homework 5 has been graded.- Mean: 129.- Median: 150.
• Any questions about homework 6?
• Extra office hours this Wednesday, 3-5pm.
• Make sure to take the faculty course evaluation and the end-of-semester class survey.
![Page 3: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/3.jpg)
Overview of today’s lecture
• Stereo and triangulation.
• Structured light.
• Binary coding.
• Using shadows.
• The 3D scanning pipeline.
• Dealing with global illumination.
![Page 4: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/4.jpg)
Slide credits
Most of these slides were directly adapted from:
• Srinivasa Narasimhan (CMU).• Mohit Gupta (Wisconsin).
![Page 5: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/5.jpg)
Stereo and triangulation
![Page 6: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/6.jpg)
Stereo Triangulation
I J
![Page 7: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/7.jpg)
Stereo Triangulation
I J
Correspondence is hard!
![Page 8: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/8.jpg)
Structured light
![Page 9: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/9.jpg)
I J
Structured Light Triangulation
Correspondence becomes easier!
![Page 10: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/10.jpg)
Structured Light
• Any spatio-temporal pattern of light projected on a surface (or volume).
• Cleverly illuminate the scene to extract scene properties (eg., 3D).
• Avoids problems of 3D estimation in scenes with complex texture/BRDFs.
• Very popular in vision and successful in industrial applications (parts assembly, inspection, etc).
![Page 11: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/11.jpg)
3D Scanning using structured light
![Page 12: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/12.jpg)
Light Stripe Scanning – Single Stripe
Camera
Source
Surface
Light plane
• Optical triangulation– Project a single stripe of laser light
– Scan it across the surface of the object
– This is a very precise version of structured light scanning
– Good for high resolution 3D, but needs many images and takes time
![Page 13: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/13.jpg)
Triangulation
• Project laser stripe onto object
Object
Laser
Camera
Light Plane
0 DCzByAx
![Page 14: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/14.jpg)
Camera
Triangulation
• Depth from ray-plane triangulation:• Intersect camera ray with light plane
Laser
Object
Light Plane
0 DCzByAx
)','( yx
Image Point
fzyy
fzxx
/'
/'
CfByAx
Dfz
''
![Page 15: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/15.jpg)
Example: Laser scanner
Cyberware® face and head scanner
+ very accurate < 0.01 mm − more than 10sec per scan
![Page 16: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/16.jpg)
Digital Michelangelo Projecthttp://graphics.stanford.edu/projects/mich/
Example: Laser scanner
![Page 17: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/17.jpg)
Portable 3D laser scanner (this one by Minolta)
![Page 18: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/18.jpg)
Binary coding
![Page 19: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/19.jpg)
Faster Acquisition?
![Page 20: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/20.jpg)
Faster Acquisition?
•Project multiple stripes simultaneously
•Correspondence problem: which stripe is which?
•Common types of patterns:
• Binary coded light striping
• Gray/color coded light striping
![Page 21: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/21.jpg)
Binary Coding
Pattern 1
Pattern 2
Pattern 3
Projected over time
Example:
3 binary-encoded patterns which allows the measuring surface to be divided in 8 sub-regions
Faster:
stripes in images.12 n
n
![Page 22: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/22.jpg)
Binary Coding
• Assign each stripe a unique illumination codeover time [Posdamer 82]
Space
Time
![Page 23: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/23.jpg)
Binary Coding
Pattern 1
Pattern 2
Pattern 3
Projected over time
Example: 7 binary patterns proposed
by Posdamer & Altschuler
…
Codeword of this píxel: 1010010 identifies the corresponding pattern stripe
![Page 24: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/24.jpg)
More complex patterns
Works despite complex appearances
Works in real-time and on dynamic scenes
• Need very few images (one or two).• But needs a more complex correspondence algorithm
![Page 25: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/25.jpg)
Continuum of Triangulation Methods
Slow, robust Fast, fragile
Multi-stripeMulti-frame
Single-frameSingle-stripe
![Page 26: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/26.jpg)
Using shadows
![Page 27: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/27.jpg)
![Page 28: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/28.jpg)
![Page 29: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/29.jpg)
![Page 30: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/30.jpg)
![Page 31: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/31.jpg)
![Page 32: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/32.jpg)
![Page 33: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/33.jpg)
The 3D scanning pipeline
![Page 34: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/34.jpg)
3D Model Acquisition Pipeline
3D Scanner
![Page 35: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/35.jpg)
3D Model Acquisition Pipeline
3D Scanner
View Planning
![Page 36: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/36.jpg)
3D Model Acquisition Pipeline
3D Scanner
AlignmentView Planning
![Page 37: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/37.jpg)
3D Model Acquisition Pipeline
3D Scanner
Alignment
Merging
View Planning
![Page 38: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/38.jpg)
3D Model Acquisition Pipeline
3D Scanner
Alignment
MergingDone?
View Planning
![Page 39: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/39.jpg)
3D Model Acquisition Pipeline
3D Scanner
Alignment
MergingDone?
View Planning
Display
![Page 40: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/40.jpg)
Real-Time 3D Model Acquisition
![Page 41: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/41.jpg)
Dealing with global illumination
![Page 42: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/42.jpg)
Light Transport
Light Source
Scene
Inter-reflections
Sub-surface
scattering
Volumetric
Scattering
Direct Illumination
42
Slide adapted from Nayar et al
![Page 43: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/43.jpg)
43
Why has global light transport been ignored?
Scattering
Inter-reflections
b
a
o dyyLyxKxxLxL )(),()()()(
Analytical models of light transport are complex
dsLsssLsLs st )ˆ,r()ˆ,ˆ()ˆ,r()ˆ,r(.̂4
Form-factor
Phase
Function
Absorption
coefficient
Scattering
coefficient
Albedo
![Page 44: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/44.jpg)
Bowl on a Marble Slab
![Page 45: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/45.jpg)
45
Captured images under conventional Gray codes
Pattern 1 Pattern 10 Pattern 7 Pattern 4
Lowest Frequency
Illumination
Highest Frequency
Illumination
![Page 46: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/46.jpg)
Issues due to global illumination effects
Strong Inter-reflections
Low-frequency pattern
Blurring due to
Sub-surface Scattering
High-frequency pattern
![Page 47: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/47.jpg)
3D Visualizations: State of the Art
Conventional Gray
(11 images)
Modulated Phase-Shifting
(162 images)
Errors due to
interreflections
Errors due to
sub-surface scattering
![Page 48: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/48.jpg)
V-Groove SceneInter-reflections
![Page 49: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/49.jpg)
Conventional Gray codes
Low frequency pattern
Captured Image
Inverse Pattern
Captured Image
Pattern
Edge I = 0.16 I = 0.25
![Page 50: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/50.jpg)
Binarization error
Incorrect Binarization
One (illuminated) Zero (not-illuminated)
Errors due to inter-reflections
Ground-truth Binarization
![Page 51: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/51.jpg)
Why is the Decoding Incorrect for Low-frequencies?
Captured Image Captured Image
Pattern
Edge I = 0.16 I = 0.25
I = Direct + a . Global I = (1 – a) . Global
a ~ 0, Direct < Global > I < I
![Page 52: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/52.jpg)
surface
camera
source
i
Point Light Source Illuminating the Scene
Direct Component
Global Component
![Page 53: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/53.jpg)
surface
camera
source
i
Global Illumination is Locally Constant
![Page 54: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/54.jpg)
surface
camera
source
i
Under High-Frequency Illumination…
![Page 55: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/55.jpg)
surface
camera
source
i
…Global Illumination Contribution is Constant
![Page 56: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/56.jpg)
3D Visualizations: Our Result
Our Technique
(41 images)
![Page 57: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/57.jpg)
Binarization for high-frequency pattern
Pattern
Captured Image
Inverse Pattern
Captured Image
I = 0.25 I = 0.16
I = Direct + 0.5 Global I = 0.5 Global >
![Page 58: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/58.jpg)
Binary DecodingCaptured Image
High-frequency Patterns are Decoded Correctly
![Page 59: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/59.jpg)
59
Logical Coding and Decoding
Preventing errors due to long-range effects
![Page 60: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/60.jpg)
Logical Coding and Decoding
XOR
=
Correct BinarizationIncorrect Binarization
Binarization
Binarization Binarization
![Page 61: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/61.jpg)
Depth Map Comparison
Our XOR-04 Codes (11 images)Conventional Gray Codes (11 images)
0 200 400 600 800600
700
800
900
1000
1100
1200
Pixels
Dep
th (
mm
)
Gray Codes
Our XOR-04 Codes
Ground Truth
Errors due to Inter-reflections
![Page 62: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/62.jpg)
XOR of Second-last Pattern with Patterns 1-8 XOR-04 Codes
(10 patterns)
Making the Logical XOR Codes
Conventional Gray Codes (10 patterns)
XOR of Last Pattern with Patterns 1-9 XOR-02 Codes (10 patterns)
Base PlaneBase Plane
![Page 63: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/63.jpg)
Inter-reflections vs. Sub-surface scattering
XOR Codes
Conventional Gray Codes
Long-Range
Inter-reflections
XOR Codes
Conventional Gray Codes
Short-Range
Sub-surface Scattering
![Page 64: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/64.jpg)
Gray Codes with Low Spatial Frequencies
Conventional Gray Codes
Max min-stripe-width Gray Codes
![Page 65: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/65.jpg)
Ensemble of Codes for General Scenes
Conventional Gray (10 images) Max min-SW Gray (10 images)
XOR-04 (10 images) XOR-02 (10 images)
![Page 66: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/66.jpg)
Reconstructing General Scenes
![Page 67: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/67.jpg)
Ensemble of Codes for General Scenes
Conventional Gray (10 images) Max min-SW Gray (10 images)
XOR-04 (10 images) XOR-02 (10 images)
Return the
consistent value
Ensemble of Codes (41 images)
![Page 68: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/68.jpg)
Conventional Gray
(11 images)
Our Technique
(41 images)
Modulated Phase-Shifting
(162 images)
Shape Comparison
![Page 69: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/69.jpg)
Qualitative Light Transport Analysis
Inter-reflections
(Both XOR codes agree)
Sub-surface Scattering
(Both Gray codes agree)
All four codes agree
Error-Detection
(all four codes different)
![Page 70: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/70.jpg)
Translucent Wax Candle
Modulated Phase-
Shifting (162 images)
Our Ensemble Codes
(41 images)Scene
Errors due to strong
sub-surface scattering
![Page 71: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/71.jpg)
Modulated Phase-
Shifting (162 images)
Our Ensemble Codes
(41 images)Scene
Translucent Wax Object
Errors due to strong
sub-surface scattering
![Page 72: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/72.jpg)
Ikea Lamp
Diffusion +
Inter-reflections
![Page 73: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/73.jpg)
Depth-Map Comparison
Regular Gray Codes (11 images) Our Ensemble Codes (41 images)
![Page 74: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/74.jpg)
3D Visualization using our ensemble codes
![Page 75: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/75.jpg)
Shower Curtain
Diffusion +
Inter-reflections
Goal is to reconstruct the shape of the shower-curtain. Shape of the curtain is planar
because it was taped to the rod to avoid movement while capture.
![Page 76: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/76.jpg)
Shape Comparisons
Phase-Shifting (18 images)Regular Gray Codes (11 images)
Our XOR Codes (11 images)
![Page 77: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/77.jpg)
Fruit Basket: Multiple Effects
Sub-surface Scattering Inter-reflections
![Page 78: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/78.jpg)
Regular Gray (11 images) Phase-Shifting (18 images)
Depth-maps with previous state of the art
![Page 79: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/79.jpg)
Regular Gray (11 images) Modulated Phase-Shifting (162 images)
Depth-maps with previous state of the art
![Page 80: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/80.jpg)
Depth-maps with our Ensemble Codes
Our Ensemble Codes (41 images)
![Page 81: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/81.jpg)
3D Visualizations with our ensemble codes
![Page 82: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/82.jpg)
3D Visualization with our ensemble codes
![Page 83: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/83.jpg)
Bowls and Milk: Multiple Effects
Interreflections
Subsurface Scattering
![Page 84: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/84.jpg)
Phase-Shifting (18 images) Modulated Phase-Shifting (162 images)
Our XOR Codes (11 images)Regular Gray Codes (11 images)
![Page 85: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/85.jpg)
3D Visualizations with our ensemble codes
![Page 86: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/86.jpg)
Flower-Vase
Diffusion
Sub-surface Scattering
![Page 87: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/87.jpg)
Comparison
Phase-Shifting (18 images) Regular Gray Code (11 images)
Modulated Phase-Shifting (162 images) Our Ensemble Codes (41 images)
![Page 88: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/88.jpg)
Comparison
Phase-Shifting (18 images) Regular Gray Code (11 images)
Modulated Phase-Shifting (162 images) Our Ensemble Codes (41 images)
![Page 89: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/89.jpg)
Shape Using Ensemble Codes Wax Bowl
Multiple Global Illumination Effects
![Page 90: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/90.jpg)
Shape Using Ensemble Codes Deep Wax Container
Multiple Global Illumination Effects
![Page 91: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/91.jpg)
Lamp made of shiny brushed metal
Strong and high-frequency inter-reflections
![Page 92: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/92.jpg)
Regular Gray (11 images) Our Ensemble Codes (41 images)
Depth Map Comparison
![Page 93: See Through Walls with Wi-Figraphics.cs.cmu.edu/courses/15-463/2017_fall/lectures/lecture27.pdf · • Mohit Gupta (Wisconsin). Stereo and triangulation. Stereo Triangulation I J](https://reader033.vdocuments.mx/reader033/viewer/2022051912/6002f59a0e314a03180ae5fe/html5/thumbnails/93.jpg)
References
Basic reading:• Szeliski textbook, Sections 7.1, 11.1, 12.1.• Lanman and Taubin, “Build Your Own 3D Scanner: Optical Triangulation for Beginners,”
SIGGRAPH course 2009.this very comprehensive course has everything you need to know about 3D scanning using structured light, including details on how to build your own.
• Gupta et al., “A Practical Approach to 3D Scanning in the Presence of Interreflections, Subsurface Scattering and Defocus,” IJCV 2013.
this paper has a very detailed treatment of problems to structured-light-based 3D scanning caused because of global illumination, and proposes the robust XOR patterns we discussed.
Additional reading:• Gupta et al., “A Combined Theory of Defocused Illumination and Global Light Transport,” IJCV
2012.an earlier paper discussing global illumination and 3D scanning with structured light.
• O’Toole et al., “Homogeneous codes for energy-efficient illumination and imaging,” SIGGRAPH 2015.
the epipolar imaging paper we covered in a previous class also includes a discussion of how epipolar imaging helps when performing stereo-based 3D scanning in the presence of global illumination.