polarized 3d
TRANSCRIPT
MIT Media Lab 2
Polarized 3DAchuta Kadambi, MIT Media Lab
Joint work with Vahe Taamazyan, Boxin Shi, and Ramesh Raskar
MIT Media Lab 3
MIT Media Lab 4
Microsoft Kinect v2
MIT Media Lab 5
Microsoft Kinect v2
MIT Media Lab 6
Microsoft Kinect v2
MIT Media Lab 7
MIT Media Lab 8
Multistripe Laser Scan
MIT Media Lab 9
Multistripe Laser Scan
MIT Media Lab 10
Multistripe Laser Scan
NextEngine 3D$3000 USDRaster
MIT Media Lab 11
Multistripe Laser Scan
NextEngine 3D$3000 USDRaster
MIT Media Lab 12
MIT Media Lab 13
Polarized 3D
3D Photo w. $30 Pol. Filter
MIT Media Lab 14
Polarized 3D
3D Photo w.$30 Pol. Filter
MIT Media Lab 15
Polarized 3D
3D Photo w.$30 Pol. Filter
MIT Media Lab 16
Plenoptic Light Transport
Adelson and Bergen “The Plenoptic Function…” MIT Press 1991
MIT Media Lab 17
Plenoptic Light Transport
Viewpoint Diversity (Light Field Cam)
Adelson and Bergen “The Plenoptic Function…” MIT Press 1991
MIT Media Lab 18
Plenoptic Light Transport
Viewpoint Diversity (Light Field Cam)
Wavelength Diversity (Hyperspectral Cam)
Adelson and Bergen “The Plenoptic Function…” MIT Press 1991
MIT Media Lab 19
Plenoptic Light Transport
Viewpoint Diversity (Light Field Cam)
Wavelength Diversity (Hyperspectral Cam)
Polarization Diversity(Photos, Shape, Scatter)
Adelson and Bergen “The Plenoptic Function…” MIT Press 1991
MIT Media Lab 20
Plenoptic Light Transport
Viewpoint Diversity (Light Field Cam)
Wavelength Diversity (Hyperspectral Cam)
Polarization Diversity(Photos, Shape, Scatter)
Time of Flight(3D, Scattering)
Adelson and Bergen “The Plenoptic Function…” MIT Press 1991
MIT Media Lab 21
Plenoptic Light Transport
Viewpoint Diversity (Light Field Cam)
Wavelength Diversity (Hyperspectral Cam)
Polarization Diversity(Photos, Shape, Scatter)
Time of Flight(3D, Scattering)
Bounce Index(Scattering)
Adelson and Bergen “The Plenoptic Function…” MIT Press 1991
MIT Media Lab 22
Plenoptic Light Transport
Viewpoint Diversity (Light Field Cam)
Wavelength Diversity (Hyperspectral Cam)
Polarization Diversity(Photos, Shape, Scatter)
Time of Flight(3D, Scattering)
Bounce Index(Scattering)
Adelson and Bergen “The Plenoptic Function…” MIT Press 1991
MIT Media Lab 23
Plenoptic Light Transport
Viewpoint Diversity (Light Field Cam)
Wavelength Diversity (Hyperspectral Cam)
Polarization Diversity(Photos, Shape, Scatter)
Time of Flight(3D, Scattering)
Bounce Index(Scattering)
Adelson and Bergen “The Plenoptic Function…” MIT Press 1991
MIT Media Lab 24
Plenoptic Light Transport
Viewpoint Diversity (Light Field Cam)
Wavelength Diversity (Hyperspectral Cam)
Polarization Diversity(Photos, Shape, Scatter)
Time of Flight(3D, Scattering)
Bounce Index(Scattering)
Adelson and Bergen “The Plenoptic Function…” MIT Press 1991
MIT Media Lab 25
Plenoptic Light Transport
Viewpoint Diversity (Light Field Cam)
Wavelength Diversity (Hyperspectral Cam)
Polarization Diversity(Photos, Shape, Scatter)
Time of Flight(3D, Scattering)
Bounce Index(Scattering)
Adelson and Bergen “The Plenoptic Function…” MIT Press 1991
MIT Media Lab 26
Plenoptic Light Transport
Viewpoint Diversity (Light Field Cam)
Wavelength Diversity (Hyperspectral Cam)
Polarization Diversity(Photos, Shape, Scatter)
Time of Flight(3D, Scattering)
Bounce Index(Scattering)
Adelson and Bergen “The Plenoptic Function…” MIT Press 1991
MIT Media Lab 27
Plenoptic Light Transport
Viewpoint Diversity (Light Field Cam)
Wavelength Diversity (Hyperspectral Cam)
Polarization Diversity(Photos, Shape, Scatter)
Time of Flight(3D, Scattering)
Bounce Index(Scattering)
Adelson and Bergen “The Plenoptic Function…” MIT Press 1991
MIT Media Lab 28
Plenoptic Light Transport
Viewpoint Diversity (Light Field Cam)
Wavelength Diversity (Hyperspectral Cam)
Polarization Diversity(Photos, Shape, Scatter)
Time of Flight(3D, Scattering)
Bounce Index(Scattering)
Adelson and Bergen “The Plenoptic Function…” MIT Press 1991
MIT Media Lab 29
Polarization of Light
MIT Media Lab 30
Polarization of Light
MIT Media Lab 31
Polarization of Light
Plane of Polarization
MIT Media Lab 32
Polarization of Light
Plane of Polarization Plane of Polarization
MIT Media Lab 33
Brewster’s Angle
MIT Media Lab 34
Cool 2D Photos
Photo Credit: Bob Atkins
MIT Media Lab 35
Cool 2D Photos
Photo Credit: Bob Atkins
MIT Media Lab 36
Polarization used in 2D photography…
… But what about Polarizers for 3D Cams?
MIT Media Lab 37
Shape from Polarization
MIT Media Lab 38
Shape from Polarization
MIT Media Lab 39
Shape from Polarization
MIT Media Lab 40
Shape from Polarization
MIT Media Lab 41
Shape from Polarization
MIT Media Lab 42
Shape from Polarization
Old Principle [Fresnel 1819]
cos cosc oso cs
i
i t
tnrn
θ θθ θ⊥ +−
=
coscos cco
oss i t
t i
rnnθ θ
θ θ+−
=
SfP crux: Solve for theta
MIT Media Lab 43
Shape from Polarization
Old Principle [Fresnel 1819]
cos cosc oso cs
i
i t
tnrn
θ θθ θ⊥ +−
=
coscos cco
oss i t
t i
rnnθ θ
θ θ+−
=
SfP crux: Solve for theta
Need to know refractive index
MIT Media Lab 44
MIT Media Lab 45
MIT Media Lab 46
MIT Media Lab 47
MIT Media Lab 48
MIT Media Lab 49
MIT Media Lab 50
MIT Media Lab 51
MIT Media Lab 52
MIT Media Lab 53
MIT Media Lab 54
Can use Schechner 15 ICCP
MIT Media Lab 55
Image Formation Model
MIT Media Lab 56
Image Formation Model
( )( )max min max minpol pol( ) cos 2
2 2I II II φ φ φ+
+ −= −
MIT Media Lab 57
Image Formation Model
( )( )max min max minpol pol( ) cos 2
2 2I II II φ φ φ+
+ −= −
Suppose and 'φ φ φ π∃ = +
MIT Media Lab 58
Image Formation Model
( )( )max min max minpol pol( ) cos 2
2 2I II II φ φ φ+
+ −= −
Suppose and 'φ φ φ π∃ = +
Azimuthal Ambiguity problem with 2 solutionsP
MIT Media Lab 59
Why is Shape from Polarization Unpopular?
1. \pi Ambiguity in Surface Normal
MIT Media Lab 60
Why is Shape from Polarization Unpopular?
1. \pi Ambiguity in Surface Normal
MIT Media Lab 61
Why is Shape from Polarization Unpopular?
1. \pi Ambiguity in Surface Normal
2. Refractive Distortion
MIT Media Lab 62
Why is Shape from Polarization Unpopular?
1. \pi Ambiguity in Surface Normal
2. Refractive Distortion
MIT Media Lab 63
Why is Shape from Polarization Unpopular?
1. \pi Ambiguity in Surface Normal
2. Refractive Distortion
3. Low SNR for some geometries
MIT Media Lab 64
Why is Shape from Polarization Unpopular?
1. \pi Ambiguity in Surface Normal
2. Refractive Distortion
3. Low SNR for some geometries
4. Usual challenges of integrating surface normals..
MIT Media Lab 65
Shape from Polarization
Miyazaki ICCV 2003 Atkinson TIP 2006
Polarization Inverse Rendering Shape from Diffuse Polarization
MIT Media Lab 66
Shape from Polarization
Miyazaki ICCV 2003 Atkinson TIP 2006
Polarization Inverse Rendering Shape from Diffuse Polarization
SfP never as popular as shading or photometric stereo
MIT Media Lab 67
Potential of Polarization?
• Passive capture
• Break past Lambertian assumption
• No baseline
• Robust to Lighting
MIT Media Lab 68
Frequency Analysis
MIT Media Lab 69
Frequency Analysis
MIT Media Lab 70
Frequency Analysis
MIT Media Lab 71
Frequency Analysis
MIT Media Lab 72
Frequency Analysis
MIT Media Lab 73
Frequency Analysis
MIT Media Lab 74
Frequency Analysis
MIT Media Lab 75
Polarized 3D Fuses Depth and Polarization
Spanning Tree Integration
MIT Media Lab 76
Polarized 3D Fuses Depth and Polarization
Spanning Tree Integration
MIT Media Lab 77
Polarized 3D Fuses Depth and Polarization
Spanning Tree Integration
MIT Media Lab 78
Polarized 3D Fuses Depth and Polarization
Spanning Tree Integration
MIT Media Lab 79
Polarized 3D Fuses Depth and Polarization
Spanning Tree Integration
MIT Media Lab 80
Visual Debugging
Kinect 3 Polar Photos
MIT Media Lab 81
Visual Debugging
Kinect 3 Polar Photos SfP
MIT Media Lab 82
Visual Debugging
Kinect 3 Polar Photos SfP Grad. Corr.
MIT Media Lab 83
Visual Debugging
Kinect 3 Polar Photos SfP Grad. Corr. Spanning Tree
MIT Media Lab 84
Versus Expensive Laser Scanner?
Polarized 3D NextEngineMultistripe
MIT Media Lab 85
Assumptions
Unpolarized World Assumption
Dielectric or Low-frequency Material Transition
No specular interreflections
Diffuse-dominant or Specular-dominant surfaces with slack
MIT Media Lab 86
Break the Lambertian Assumption
MIT Media Lab 87
Break the Lambertian AssumptionKinect
MIT Media Lab 88
Break the Lambertian AssumptionShading [Wu 14]Kinect
MIT Media Lab 89
Break the Lambertian AssumptionPolarized 3DShading [Wu 14]Kinect
MIT Media Lab 90
Break the Lambertian AssumptionPolarized 3DShading [Wu 14]Kinect
MIT Media Lab 91
Break Lighting Assumptions
Kinect
MIT Media Lab 92
Break Lighting Assumptions
Polarized 3DShading [Wu 14]Kinect
MIT Media Lab 93
Break Lighting Assumptions
Polarized 3DShading [Wu 14]Kinect
MIT Media Lab 94
Break Lighting Assumptions
Polarized 3DShading [Wu 14]Kinect
MIT Media Lab 95
Polarized 3D
Collaborators:Vage Taamazyan
Boxin ShiRamesh Raskar
DIY Guide planned for 2016
Thanks to:Gary Atkinson
Terry BoultIn Kyu Park
media.mit.edu/~achoo/[email protected]
MIT Media Lab 96
MIT Media Lab 97
MIT Media Lab 98
MIT Media Lab 99
MIT Media Lab 100
Polarized 3DAchuta Kadambi, MIT Media Lab
Joint work with Vahe Taamazyan, Boxin Shi, and Ramesh Raskar
MIT Media Lab 101
Polarized 3DAchuta Kadambi, MIT Media Lab
Joint work with Vahe Taamazyan, Boxin Shi, and Ramesh Raskar
MIT Media Lab 102
Polarized 3D Power of Physics VisionAchuta Kadambi, MIT Media Lab
Joint work with Vahe Taamazyan, Boxin Shi, and Ramesh Raskar