siggraph 2014 course on computational cameras and displays (part 3)
DESCRIPTION
Recent advances in both computational photography and displays have given rise to a new generation of computational devices. Computational cameras and displays provide a visual experience that goes beyond the capabilities of traditional systems by adding computational power to optics, lights, and sensors. These devices are breaking new ground in the consumer market, including lightfield cameras that redefine our understanding of pictures (Lytro), displays for visualizing 3D/4D content without special eyewear (Nintendo 3DS), motion-sensing devices that use light coded in space or time to detect motion and position (Kinect, Leap Motion), and a movement toward ubiquitous computing with wearable cameras and displays (Google Glass). This short (1.5 hour) course serves as an introduction to the key ideas and an overview of the latest work in computational cameras, displays, and light transport.TRANSCRIPT
![Page 1: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/1.jpg)
Compressive Display Systems
Gordon Wetzstein
MIT / Stanford University
media.mit.edu/~gordonw
displayblocks.org
![Page 2: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/2.jpg)
![Page 3: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/3.jpg)
This slide has a 16:9 media windowEvolution?
1928 2014
![Page 4: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/4.jpg)
Nature
Image: National Geographic
Evolution!
![Page 5: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/5.jpg)
uberpixel
![Page 6: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/6.jpg)
Next-generation Devices
computation optics & electronics human visual
system
interaction
optics (compressive) computationsensing
Computational &
Compressive Displays
Computational
Imaging
![Page 7: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/7.jpg)
displayblocks.org
![Page 8: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/8.jpg)
Images: Wikipedia, Shinya Yoshioka
![Page 9: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/9.jpg)
This slide has a 16:9 media windowNature
Image: Desafio Monteverde and Arenal Volcano Tours Costa Rica
![Page 10: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/10.jpg)
Three-layer Tensor Display
![Page 11: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/11.jpg)
This slide has a 16:9 media windowCompressive Light
Field Displays
![Page 12: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/12.jpg)
This slide has a 16:9 media window
![Page 13: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/13.jpg)
This slide has a 16:9 media window
viewer moves right
vie
wer
moves d
ow
n
4D Light Field
![Page 14: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/14.jpg)
Display OpticsComputational
Processing
Compressive Displays
![Page 15: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/15.jpg)
![Page 16: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/16.jpg)
2D display
barrier
Parallax Barriers – Ives 1903
• low resolution & very dim
![Page 17: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/17.jpg)
Integral Imaging – Lippmann 1908
• low-res, but brighter than parallax barriers
2D display
len
sle
ts
![Page 18: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/18.jpg)
patent drawings - early 20th century
![Page 19: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/19.jpg)
multiplexing
nonlinear
compression
optimization
user experience
content
community
next
MN
O
C
U C
C
N
Structural Formula for Compressive Displays
![Page 20: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/20.jpg)
multiplexing
nonlinear
compression
optimization
user experience
content
community
next
MN
O
C
U C
C
N
Nonlinear Pixel Interaction & Coupling
![Page 21: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/21.jpg)
layer 2
layer 1
Conventional Parallax Barriers
![Page 22: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/22.jpg)
layer 2
layer 1
Conventional Parallax Barriers
blocked!
p1
p2
l = p1*p2
nonlinear
![Page 23: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/23.jpg)
layer 2
layer 1
Most Volumetric Displays / Additive Layers
e.g., LEDs or transparent OLEDs
not blocked!
p1
p2
l = p1+p2
linear
![Page 24: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/24.jpg)
Holography – Nonlinear Interaction
plane wave
hologram
emitted wavefront:
screen or retina
received intensity:
I(x) = T U(x){ }2
= Re T U(x){ }{ }2
+ Im T U(x){ }{ }2
U(x)
![Page 25: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/25.jpg)
Holography – Coupling
plane wave
hologram
W (x,u = l sin(q )) = t x+x '
2
æ
èç
ö
ø÷t x -
x '
2
æ
èç
ö
ø÷e
2pix 'u dx 'ò
Fourier transform of all points interacting
with each other!
nonlinear interaction & pixel coupling!
![Page 26: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/26.jpg)
attenuation layers with
spacers
backlight
Layered 3D – SIGGRAPH 2011
![Page 27: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/27.jpg)
Layered 3D – SIGGRAPH 2011
layer 2
layer K
layer 1
…
![Page 28: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/28.jpg)
Layered 3D – SIGGRAPH 2011
layer 2
layer K
layer 1
…
p1
p2
pK
l = p1*p2*…*pK
![Page 29: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/29.jpg)
Layered 3D – SIGGRAPH 2011
…
layer 2
layer K
layer 1
![Page 30: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/30.jpg)
Tensor Displays – SIGGRAPH 2012
directional backlight three layer
Reconfigurable
![Page 31: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/31.jpg)
![Page 32: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/32.jpg)
time
Tensor Displays – Directional Backlight
Perceptual Integration
layer 2
microlens array
layer 1
![Page 33: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/33.jpg)
LCD with directional backlight, rank 6
LCD + directional BL
conventional lenslets
view from above
![Page 34: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/34.jpg)
LCD with directional backlight, rank 6 (as seen by observer)
![Page 35: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/35.jpg)
front LCD
directional backlight
Filmed with High-speed Camera
![Page 36: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/36.jpg)
four stacked liquid crystal panels
two crossed polarizers
![Page 37: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/37.jpg)
LCD 1
LCD 2
LCD 3
backlight
Polarization Fields – SIGGRAPH ASIA 2011
polarizer
polarizercolor filter array
polarizercolor filter array
polarizercolor filter array
![Page 38: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/38.jpg)
backlight
Polarization Fields – SIGGRAPH ASIA 2011
LCD 1
LCD 2
LCD 3
polarizer
polarizer
![Page 39: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/39.jpg)
image formation
f1
f2
f3
L(x,q) = sin2 Q(x,q)( )
backlight
Polarization Fields – SIGGRAPH ASIA 2011
LCD 1
LCD 2
LCD 3
![Page 40: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/40.jpg)
multiplexing
nonlinear
compression
optimization
user experience
content
community
next
MN
O
C
U C
C
N
Compression
![Page 41: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/41.jpg)
Which Light Field is More Compressible?
a b
![Page 42: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/42.jpg)
Which Light Field is More Compressible?
a b
![Page 43: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/43.jpg)
This slide has a 16:9 media window
4D Light Field
Uniform or
Directional BacklightStacked Layers
Nonnegative Tensor
Factorization
Display-adaptive
Compression
Compressive
Computed Tomography
(LCDs or Transparencies)
Optics
Observer = Decoder
![Page 44: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/44.jpg)
Give those Pixels a Break!
Applied Mathematics
• sparse optimization
• low-rank factorization
• computed tomography
• …
Benefits for Optics & Electronics
• fewer pixels
• relaxation on refresh rate
• thinner form factors
• …
![Page 45: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/45.jpg)
Parallax Barriers
1903
Time-Shifted
Parallax Barriers 2007
t
HR3D
SIG Asia 2010
t tLayered 3D
SIGGRAPH 2011
Tensor Displays
SIGGRAPH 2012
Conventional Parallax Barriers
layer 2
layer 1
From Conventional to Compressive Displays
![Page 46: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/46.jpg)
…
…
Parallax Barriers
1903
Time-Shifted
Parallax Barriers 2007
t
HR3D
SIG Asia 2010
t tLayered 3D
SIGGRAPH 2011
Tensor Displays
SIGGRAPH 2012
…
time
Perceptual Integration
Tensor Displays – Multilayer & Directional Backlighting
From Conventional to Compressive Displays
![Page 47: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/47.jpg)
Parallax Barriers
1903
Time-Shifted
Parallax Barriers 2007
t
HR3D
SIG Asia 2010
t tLayered 3D
SIGGRAPH 2011
Tensor Displays
SIGGRAPH 2012
Perceptual Integration
Tensor Displays – Directional Backlighting
time
thin
!
From Conventional to Compressive Displays
![Page 48: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/48.jpg)
multiplexing
nonlinear
compression
optimization
user experience
content
community
next
MN
O
C
U C
C
N
Real-time Optimization
![Page 49: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/49.jpg)
This slide has a 16:9 media window
4D Light Field
Display-adaptive
Compression
CompressiveOptics
via optimization
![Page 50: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/50.jpg)
attenuation layers with
spacers
backlight
Layered 3D – SIGGRAPH 2011
![Page 51: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/51.jpg)
?
• limited baseline tomography
• use algebraic approaches!
Layered 3D – SIGGRAPH 2011
![Page 52: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/52.jpg)
This slide has a 16:9 media windowComputed Tomography (CT)
Image: W
ikip
edia
x-ray source
x-ray sensor
3D Reconstruction
Reconstructed 2D Slices
![Page 53: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/53.jpg)
Tomographic Light Field Synthesis
q
2D light field
x
xq
backlight
attenuation volume
virtual planesimage formation
L(x,q ) = e- m (r )dr
cò
log L x,q( )( ) = - m(r)drc
ò
2
20
P)log(argmin
L
tomographic synthesis
![Page 54: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/54.jpg)
Limits of Tomographic Light Field Synthesis
image formation
L(x,q ) = e- m (r )dr
cò
log L x,q( )( ) = - m(r)drc
ò
2
20
P)log(argmin
L
tomographic synthesis
log space
…
p1
p2
pK
…
p1
p2
pK
time
l = p1*…*pK
…
p1
p2
pK
log(l) = log(p1)+…+log(pK)
l = (p1*p2*…*pK)1+…+(p1*p2*…*pK)N
log
lin
???
![Page 55: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/55.jpg)
backlight
rear layer
front layer
two-layer light field display
Low-rank Light Field Factorization
light field
fm(1)(x1)
fm(2)(x2)
x1
x2
L(x1, x2)
![Page 56: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/56.jpg)
backlight
rear layer
front layer
two-layer light field display
fm(1)(x1)
fm(2)(x2)
x1
x2
L(x1, x2)
`
front layer
rear
laye
r
rank-1
Lanman et al. – SIGGRAPH Asia 2010
Low-rank Light Field Factorization
![Page 57: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/57.jpg)
backlight
rear layer
front layer
two-layer light field display
fm(1)(x1)
fm(2)(x2)
x1
x2
L(x1, x2)
` rank-4
Lanman et al. – SIGGRAPH Asia 2010
high-speed LCDs = perceptual average
Low-rank Light Field Factorization
![Page 58: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/58.jpg)
` rank-4F
G
L~
Lanman et al. – SIGGRAPH Asia 2010
Low-rank Light Field Factorization
high-speed LCDs = perceptual average
arg minF,G
L-FGW
2, for F,G ³ 0
objective function:
![Page 59: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/59.jpg)
fm(1)(x1)
fm(2)(x2)
fm(3)(x3)
x1
x2
x3
L(x2, x3)
multi-layer light field display
backlight
rear layer
middle layer
front layer
light field
Light Field Slice Representation
![Page 60: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/60.jpg)
fm(1)(x1)
fm(2)(x2)
fm(3)(x3)
x1
x2
x3
L(x1,x2,x3)
Rear
Layer
x3
x2
x1
L(x1,x2,x3)
light field tensor
backlight
rear layer
middle layer
front layer
light field
multi-layer light field display
Light Field Tensor Representation
![Page 61: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/61.jpg)
fm(1)(x1)
fm(2)(x2)
fm(3)(x3)
x1
x2
x3
Rear
Layer
x3
x2
x1
L(x1,x2,x3)
L(x1,x2,x3)
light field tensor
backlight
rear layer
middle layer
front layer
light field
multi-layer light field display
Light Field Tensor Representation
![Page 62: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/62.jpg)
fm(1)(x1)
fm(2)(x2)
fm(3)(x3)
Rear
Layer
light field tensor
backlight
rear layer
middle layer
front layer
light field
multi-layer light field display
Light Field Tensor Representation
![Page 63: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/63.jpg)
fm(1)(x1)
fm(2)(x2)
fm(3)(x3)
Rear
Layer
light field tensor
backlight
rear layer
middle layer
front layer
light field
multi-layer light field display
Light Field Tensor Representation
![Page 64: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/64.jpg)
fm(1)(x1)
fm(2)(x2)
fm(3)(x3)
Rear
Layer
light field tensor
backlight
rear layer
middle layer
front layer
light field
multi-layer light field display
Light Field Tensor Representation
![Page 65: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/65.jpg)
frame M
+ ... +
Target Light Field Tensor
frame 1
nonnegative tensorfactorization (NTF)
Rank-M Approximation
perceptualintegration
frame 2
+
Low-rank Tensor Factorization
![Page 66: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/66.jpg)
iterative update rules
nonlinear (multilinear)
optimization problem
Low-rank Tensor Factorization
• standard form – Tensor Compendium
• multiplicative update keep factors positive
• basically steepest descent with fixed step length
![Page 67: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/67.jpg)
“standard” formulationalternate formulation
Low-rank Tensor Factorization
![Page 68: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/68.jpg)
alternate formulation
forward projection (multiview rendering) back projection (projective texture mapping)
Efficient GPU Implementation
Tensor Factorization - Implementation
![Page 69: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/69.jpg)
![Page 70: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/70.jpg)
multiplexing
nonlinear
compression
optimization
user experience
content
community
next
MN
O
C
U C
C
N
Who cares?
![Page 71: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/71.jpg)
Vision-correcting Displayperceived image
displayed image
SIGGRAPH 2014 - Display Session, Tue morning
![Page 72: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/72.jpg)
Vision-correcting Display
iPod Touch prototype printed transparency
SIGGRAPH 2014 - Display Session, Tue morning
![Page 73: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/73.jpg)
prototype construction
300 dpi or higher
![Page 74: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/74.jpg)
vision-correcting displayconventional display
![Page 75: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/75.jpg)
multiplexing
nonlinear
compression
optimization
user experience
content
community
next
MN
O
C
U C
C
N
What’s next?
![Page 76: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/76.jpg)
resolution
contrast
3D display capabilities
![Page 77: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/77.jpg)
Compressive Multi-mode DisplayOptics Express 2014
![Page 78: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/78.jpg)
This slide has a 16:9 media window
High-speed LCD
High-speed LCD
Electronically-switchable Diffuser
![Page 79: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/79.jpg)
High-speed LCD
High-speed LCD
Electronically-switchable Diffuser OFF
3D Display Mode
![Page 80: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/80.jpg)
Light Field Factorization – LCD Patterns
Front Layer Rear Layer
![Page 81: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/81.jpg)
Light Field Factorization – Results
![Page 82: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/82.jpg)
High Dynamic Range Display Mode
High-speed LCD
High-speed LCD
Electronically-switchable Diffuser OFF
![Page 83: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/83.jpg)
Target HDR Image
![Page 84: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/84.jpg)
Solver = Light Field without Parallax
Front Layer Rear Layer
![Page 85: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/85.jpg)
ConventionalHigh Dynamic Range
![Page 86: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/86.jpg)
Superresolution Display Mode
High-speed LCD
High-speed LCD
Electronically-switchable Diffuser ON
![Page 87: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/87.jpg)
Results from Prototype
![Page 88: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/88.jpg)
Results from Prototype
![Page 89: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/89.jpg)
Light Field Projection SIGGRAPH 2014 – ETech & Paper
![Page 90: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/90.jpg)
Compressive Light Field Projection
![Page 91: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/91.jpg)
![Page 92: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/92.jpg)
![Page 93: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/93.jpg)
multiplexing
nonlinear
compression
optimization
user experience
content
community
next
N
O
C
U C
C
N
Structural Formula for Compressive Displays
M
Mobile Displays Optics Express 2014
Projection Displays SIGGRAPH 2014
Coded Illumination for
Microscopy & LithographyHead Mounted Displays
CGF 2010, SIGGRAPH 2014
Monitors / TVs SIG2011,2012,2013, SIGAsia 2009,2011
![Page 94: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/94.jpg)
Cool Displays at SIGGRAPH 2014
Technical Papers Sessions
Emerging Technologies
• Displays, Tuesday 10:45-12:15, Hall A
• Computational Sensing and Display, Tue 3:45-5:15, Hall A
• AR & VR Displays
• Light Field Projection, HDR Projection
• much more!
![Page 95: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/95.jpg)
Gordon Wetzstein
Massachusetts Institute of Technology
media.mit.edu/~gordonw
displayblocks.org
Matt Hirsch (MIT)
Doug Lanman (MIT/NVIDIA/Oculus VR)
Andrew Maimone (UNC)
Felix Heide (UBC)
Fu-Chung Huang (UC Berkeley)
Belen Masia (University of Zaragoza)
collaborators
sources of funding & hardware
Wolfgang Heidrich (UBC/KAUST)
Ramesh Raskar (MIT)
Diego Gutierrez (University of Zaragoza)
Brian Barsky (UC Berkeley)
Henry Fuchs (UNC)
![Page 96: SIGGRAPH 2014 Course on Computational Cameras and Displays (part 3)](https://reader033.vdocuments.mx/reader033/viewer/2022052623/55983a6e1a28ab28628b4774/html5/thumbnails/96.jpg)
Gordon Wetzstein
Massachusetts Institute of Technology
media.mit.edu/~gordonw
displayblocks.org
Stanford looking for:
• students, postdocs, interns
• (industry) collaborators