introduction to the light-field camera
TRANSCRIPT
![Page 1: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/1.jpg)
Introduction to the Light-Field
Camera
Speaker: Wei-Sheng Lai
Date: 2013/11/29
![Page 2: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/2.jpg)
Outline
1. Preface
2. Image Rendering
3. Light-Field History
4. Plenoptic 1.0
5. Plenoptic 2.0
6. Different Light-Field Camera
7. Conclusion
2
![Page 3: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/3.jpg)
Preface – What is Light-Field Camera?
1. Website: https://www.lytro.com/
2. Property: Refocusing, change orientation, display photo
in 3D
3
![Page 4: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/4.jpg)
Image Rendering
• A tradition image is formed by integrating rays from
every direction at each pixel.
4
𝐼 𝒒 = 𝒑
𝑟 𝒒, 𝒑 𝑑𝒑
𝒒: position (∈ 𝑅2)𝒑: direction (∈ 𝑅2)𝒓 𝒒, 𝒑 : radiance
𝑰(𝒒): pixel value
![Page 5: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/5.jpg)
Image rendering
• A tradition image is formed by integrating rays from
every direction at each pixel.
5
𝐼 𝑖, 𝑗 =1
𝑁2
𝑚
𝑛
𝑟[𝑖, 𝑗,𝑚, 𝑛]
𝒊, 𝒋: position
𝒎,𝒏: direction
𝒓 𝒊, 𝒋,𝒎, 𝒏 : radiance
𝑰[𝒊, 𝒋]: pixel value
![Page 6: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/6.jpg)
Light-Field History
• The first light-field camera – 1908 Lippmann’s Integral
Photographs
6
Lippmann’s Integral Photographs Lippmann
![Page 7: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/7.jpg)
Light-Field History
• 1968 Chutjian (First Digital Light-Field)
7
![Page 8: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/8.jpg)
Light-Field History
• 1992 Adelson - Plenoptic camera
8[1] Adelson, E. H., Wang, J. Y. “Single lens stereo with a plenoptic camera”. IEEE TPAMI, 14(2), 99-106, 1992.
Micro-lens
Main lens
![Page 9: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/9.jpg)
• Main lens focused on micro-lens, micro-lens focused on
infinity
Plenoptic 1.0
9[1] Adelson, E. H., Wang, J. Y. “Single lens stereo with a plenoptic camera”. IEEE TPAMI, 14(2), 99-106, 1992.
![Page 10: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/10.jpg)
Plenoptic 1.0
• Raw data
10
![Page 11: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/11.jpg)
Plenoptic 1.0
• Plenoptic image is a flat 2D array of 2D arrays
11
direction
position
![Page 12: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/12.jpg)
Plenoptic 1.0 - Rendering
• Light-field rendering
12
![Page 13: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/13.jpg)
Plenoptic 1.0 - Rendering
• Render different viewpoint
13
![Page 14: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/14.jpg)
Plenoptic 1.0 - Rendering
• Render different viewpoint
14
![Page 15: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/15.jpg)
Plenoptic 1.0 - Rendering
• Render different viewpoint
15
![Page 16: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/16.jpg)
Plenoptic 1.0 - Rendering
• Render different viewpoint
16
![Page 17: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/17.jpg)
Plenoptic 1.0 - Rendering
• Render different viewpoint
17
![Page 18: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/18.jpg)
Plenoptic 1.0 - Rendering
• Render different viewpoint
18
![Page 19: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/19.jpg)
Plenoptic 1.0 - Refocusing
• Change the distance from the lens to the sensor
19
sensor Main lens
![Page 20: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/20.jpg)
Plenoptic 1.0 - Refocusing
• Change the distance from the lens to the sensor
20
sensor Main lens
![Page 21: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/21.jpg)
Plenoptic 1.0 - Refocusing
• We capture radiance 𝒓𝟏. How to compute 𝒓𝟐 ?
21
sensor Main lens
Micro-lens
![Page 22: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/22.jpg)
Plenoptic 1.0 - Refocusing
• Ray travels distance 𝑡 through space
22
𝒓𝟏 𝒓𝟐 ray
• Refocusing is shearing transform: 𝑟′ 𝒒′, 𝒑′ = 𝑟(𝒒 + 𝑡𝒑, 𝒑)
𝒒′
𝒑′=1 𝑡0 1
𝒒𝒑 =
𝒒 + 𝑡𝒑𝒑
𝒒: position (∈ 𝑅2)𝒑: direction/slope (∈ 𝑅2 )
![Page 23: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/23.jpg)
Plenoptic 1.0 - Refocusing
• Separate raw data into different viewpoint images
23𝒑𝒙
𝒑𝒚
𝒒𝒙
𝒒𝒚
𝒚𝟎𝒑𝒙
𝒒𝒙
𝒑𝒙
𝒑𝒚
𝒒𝒙
𝒒𝒚
![Page 24: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/24.jpg)
Plenoptic 1.0 - Refocusing
• Shearing in 𝑞𝑥 ⇒ merge
24
𝒑𝒙
𝒒𝒙
𝒑𝒙
𝒑𝒚
𝒒𝒙
𝒒𝒚
![Page 25: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/25.jpg)
Plenoptic 1.0 - Refocusing
25
![Page 26: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/26.jpg)
Plenoptic 1.0 - Refocusing
26
![Page 27: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/27.jpg)
Plenoptic 1.0 - Refocusing
27
![Page 28: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/28.jpg)
Plenoptic 1.0 - Refocusing
• Direct refocusing in the spatial domain requires 𝑂(𝑁4)operations for each refocused image
• Ren Ng proposed the Fourier slice refocusing method:
28
Direct method
Initial
Refocusing
[2] Ng, Ren. "Fourier slice photography." ACM Transactions on Graphics (TOG). Vol. 24. No. 3. ACM, 2005.
![Page 29: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/29.jpg)
Is Plenoptic 1.0 good enough?
• Micro-lenses images are defocused relative to main lens
image
• Low resolution
29
![Page 30: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/30.jpg)
Plenoptic 2.0
• Lunsdaine and Georgiev 2008: Micro-lenses satisfy the
lens equation, and exactly focused on the main lens image.
30[3] Lumsdaine, Andrew, and Todor Georgiev. "Full resolution lightfield rendering."Indiana University and Adobe
Systems, Tech. Rep (2008).
![Page 31: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/31.jpg)
Comparison
31
![Page 32: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/32.jpg)
Comparison
32
![Page 33: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/33.jpg)
Comparison
• Resolution analysis
33
pixel
Micro-lens
![Page 34: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/34.jpg)
Plenoptic 2.0
• Rendering full aperture image
34
![Page 35: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/35.jpg)
Plenoptic 2.0
• Rendering one view
35
![Page 36: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/36.jpg)
Plenoptic 2.0
• Advantage of plenoptic 2.0:
1. Decouple resolution from the number of micro-lenses
2. Free to choose spatial-angular trade-off point
3. Each micro-lens is observing the scene as a slightly
shifted camera ⇒ good for super-resolution
36
![Page 37: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/37.jpg)
37
![Page 38: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/38.jpg)
38
![Page 39: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/39.jpg)
39
![Page 40: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/40.jpg)
40
![Page 41: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/41.jpg)
Super-resolution
• Plenoptic 1.0 result
41
![Page 42: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/42.jpg)
Other application
• We can put different filters on different micro-lenses
42
![Page 43: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/43.jpg)
Different Light-Field Camera
• Lytro – the first hand-held light-field camera
• Plenoptic 1.0, resolution = 1080 x 1080 pixels (1.2M)
• Software can run both on MAC and Windows
43
![Page 44: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/44.jpg)
Different Light-Field Camera
• Raytrix – target at industrial and scientific application
• Resolution from 1M to 3M
• Precise per-pixel depth information
44
Raytrix-R5 Raytrix-R11 Raytrix-R29 Raytrix-RX
![Page 45: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/45.jpg)
Different Light-Field Camera
• Nokia Pelican Imaging – implement on smart phone
45[4] Venkataraman, Kartik, et al. "PiCam: an ultra-thin high performance monolithic camera array." ACM
Transactions on Graphics (TOG) 32.6 (2013): 166.
![Page 46: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/46.jpg)
Conclusion
• What we have talked about:
1. The basic image rendering idea
2. The history of light-field camera
3. Property of Plenoptic camera 1.0
• Micro-lens focused on infinity
• Rendering and refocusing
4. Property of Plenoptic camera 2.0
• Micro-lens focused on main lens image
• Higher flexibility, suitable for super-resolution
5. Different kinds of light-field camera
46
![Page 47: Introduction to the Light-Field Camera](https://reader034.vdocuments.mx/reader034/viewer/2022051404/5871fde81a28ab02618bf13a/html5/thumbnails/47.jpg)
47