bidi screen a thin, depth-sensing lcd for 3d interaction using light fields matthew hirsch ...
DESCRIPTION
http://web.media.mit.edu/~mhirsch/bidi/ Watch video: http://www.youtube.com/watch?v=kXuxK6IeQfo BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar 1MIT Media LabTRANSCRIPT
![Page 1: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/1.jpg)
BiDi ScreenDepth and Lighting Aware Interactive Display
Matthew HirschMIT Media Lab
Douglas LanmanBrown University
Ramesh RaskarMIT Media Lab
Henry HoltzmanMIT Media Lab
![Page 2: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/2.jpg)
BiDi Screen
![Page 3: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/3.jpg)
Inspiration
Light Sensitive Displays Depth Cameras
Multitouch
![Page 4: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/4.jpg)
Beyond Multi-touch: Hover Interaction
• Seamless transition of multitouch to gesture
• Thin package, LCD
![Page 5: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/5.jpg)
Beyond Multi-touch: Mobile
Laptops
Mobile
![Page 6: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/6.jpg)
Results: Analysis
![Page 7: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/7.jpg)
BiDi Screen
• Capture 3-D to enable depth and lighting aware interaction
• Collocated image capture and display
• Walk-up interaction
• Mobility and portability – thin device
![Page 8: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/8.jpg)
Design Overview
Dis
play
with
em
bedd
ed o
ptic
al s
enso
rs
Sharp Microelectronics Optical Multi-touch Prototype
![Page 9: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/9.jpg)
Design Overview
Dis
play
with
em
bedd
ed o
ptic
al s
enso
rs
LCD , displaying mask
Opt
ical
sen
sor a
rray
~2.5 cm~50 cm
![Page 10: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/10.jpg)
Design Vision
Object Collocated Captureand Display
Bare Sensor
Spa
tial L
ight
Mod
ulat
or
![Page 11: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/11.jpg)
Design Overview
Mask
Array of Virtual Cameras
Sensor
![Page 12: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/12.jpg)
Manipulating an object with 3D gesture
![Page 13: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/13.jpg)
Alternatives to capture depth from an LCD
• Adapted Touch
• Capacitive
• Resistive
• Optical
• Camera arrays
• Behind screen
• To sides
• Depth Sensors/Cameras
![Page 14: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/14.jpg)
Adapting Traditional Touch?
• Resistive touch screen
• Confined to screen
• Capacitive
• Not lighting dependant
• Calibration problems for gesture
• No existing technology trend for gesture
?
![Page 15: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/15.jpg)
Camera Arrays
• Cameras behind screen
• Interfere with backlight
• Expensive, Large
• Han , SecondLight, ThinSight
• No tech curve
• Cameras to side
• Transition region
• Second modality
?
![Page 16: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/16.jpg)
Depth Cameras and Sensors
• Depth Cameras
• Behind screen w/ projector (MSR TouchlLight)
• No longer a thin device
• No light sensitivity
• Touch difficult
![Page 17: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/17.jpg)
Theory: Depth from light-field capture
• LCD used for Spatial Heterodyne Light Field Capture and Display
• Pinholes or tiled broadband masks
• Separate Sensor and Mask
• Fourier Refocusing
• Depth from focus
• Blob tracking (gesture)
![Page 18: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/18.jpg)
Theory: Lightfield
θx
y θ
x
.
Sensor integrates these rays
(Sensor)
![Page 19: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/19.jpg)
Theory: Lightfield Frequency Domain
fθ
fx
θ
x
Lightfield FT
Fourier Slice Theorm
FT
![Page 20: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/20.jpg)
Theory: LF Skew in Free-Space Propogation
θ
x
x
y
mask
θ
mask
![Page 21: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/21.jpg)
Theory: Convolution with Delta Functions
![Page 22: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/22.jpg)
Theory: Transforms
fθ
fx
θ
x
Lightfield FT
fmaskmaskFT
Tiled Broadband Mask
![Page 23: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/23.jpg)
Theory: Spatial Heterodyning
x
y
mask
Multiplication in prime domain is convolution in frequency domain
![Page 24: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/24.jpg)
Theory: Spatial Heterodyning
fθ
fxfmask
fθ
fx
![Page 25: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/25.jpg)
Theory: Spatial Heterodyning
fθ
fx
Reconstructed LightfieldSpectrum
Band Limited
![Page 26: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/26.jpg)
Desired PrototypeActual Prototype
![Page 27: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/27.jpg)
Pipeline
Max ContrastOperator
![Page 28: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/28.jpg)
Pipeline
![Page 29: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/29.jpg)
Theory: Depth and Spatial Resolution
![Page 30: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/30.jpg)
Theory: Depth and Spatial Resolution
![Page 31: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/31.jpg)
Results: Analysis
![Page 32: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/32.jpg)
Manipulating an object with 3D gesture
![Page 33: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/33.jpg)
Navigating a 3D world with 3D gesture
![Page 34: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/34.jpg)
Lighting a vritual scene with a real light
![Page 35: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/35.jpg)
Limitations
• Requires separation between display and sensor (adds thickness)
• 2.5 cm for 50 cm range in our prototype
• 750mm for 8 cm range in iPhone like device
• Sensitive to room lighting or requires its own light source
• This is true of many optical systems
• Time multiplexing of display/capture
• Requires fast capture and screen refresh to stay below flicker fusion rate
• 240 Hz LCDs coming to market
![Page 36: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/36.jpg)
Conclusions
• Future Work
• Dynamic Masks
• Change frequency characteristics to match scene
• Video capture / Video chat (higher resolution)
• SIGGRAPH Asia 2009 Paper
• BiDi Screen: Depth and Lighting Aware Interaction and Display
![Page 37: BiDi Screen A Thin, Depth-Sensing LCD for 3D Interaction using Light Fields Matthew Hirsch Douglas Lanman Henry Holtzman Ramesh Raskar](https://reader031.vdocuments.mx/reader031/viewer/2022012918/5404fb7c8d7f729b768b49e2/html5/thumbnails/37.jpg)
Conclusions
• Enable multitouch and gesture interaction on a thin display screen
• Walk-up interaction does not require gloves / fiducials
• Capture depth using array of vitual cameras
• Thin portable devices possible with area sensor