![Page 1: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/1.jpg)
CVPR 2018
Computational Imaging for
Self-Driving Vehicles
Jan Kautz--------Ramesh Raskar--------Achuta Kadambi--------Guy Satat
![Page 2: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/2.jpg)
Computational Imaging for Self-Driving Vehicles
Jan Kautz--------Ramesh Raskar--------Achuta Kadambi--------Guy Satat
Computational Imaging
Self Driving Cars
Novel Sensors LIDAR
Challenging Weather
Deep LearningOpen
Problems
![Page 3: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/3.jpg)
Course Schedule imagingav.media.mit.edu
13:30 – 13:50 Course IntroductionRamesh Raskar (MIT)
13:50 – 15:00 Existing Sensors and Their LimitsGuy Satat (MIT), Achuta Kadambi (UCLA)
15:00 – 15:10 Break
15:10 – 15:50 Emerging 3D SensorsAchuta Kadambi (UCLA)
15:50 – 16:30 Imaging in Bad WeatherGuy Satat (MIT)
16:30 – 16:40 Break
16:40 – 17:20 Deep Learning-based Computational ImagingJan Kautz (NVIDIA)
17:20 – 17:30 Conclusion and Open Problems
![Page 4: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/4.jpg)
![Page 5: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/5.jpg)
![Page 6: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/6.jpg)
![Page 7: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/7.jpg)
Radar
Sonar
Camera
LiDAR
Classification Resolution Localization Cost Any Weather
![Page 8: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/8.jpg)
Bit Hacking
Ph
oto
n H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Machine Learning
Computational Light Transport
Computational ImagingIllumination
![Page 9: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/9.jpg)
Plenoptic Light Transport
Viewpoint Diversity
(Light Field Cam)
Wavelength Diversity
(Hyperspectral Cam)Polarization Diversity
(Photos, Shape, Scatter)
Time of Flight
(3D, Scattering)
Bounce Index
(Scattering)
Adelson and Bergen “The Plenoptic Function…” MIT Press 1991
![Page 10: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/10.jpg)
![Page 11: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/11.jpg)
![Page 12: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/12.jpg)
X rays UV IR Microwave Radio Waves
Visible
Wavelength
• Resolution
• Optical Contrast
• Non ionizing
• Availability of fluorophores
![Page 13: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/13.jpg)
Wildlife Rehabilitation Center of Minnesota
Optical Contrast
Visible light X-Ray
![Page 14: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/14.jpg)
X rays UV IR Microwave Radio Waves
Visible
Wavelength
![Page 15: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/15.jpg)
Light Interaction
Object LensAbsorption Scattering
![Page 16: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/16.jpg)
DiffractionDynamic RangeDescattering
DefocusDepth
![Page 17: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/17.jpg)
1964
![Page 18: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/18.jpg)
MilliMicroNano
PicoFemto
Atto
Conquer .. Time
![Page 19: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/19.jpg)
MilliMicroNano
Pico
AttoFemto-graphy
Conquer .. Time NoiseSignal
![Page 20: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/20.jpg)
Light in Slow Motion ..
10 Billion x Slow
![Page 21: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/21.jpg)
![Page 22: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/22.jpg)
Velten et al Siggraph 2013
![Page 23: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/23.jpg)
Can you seearound corners ?
![Page 24: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/24.jpg)
2nd Bounce
1st Bounce
3rd Bounce
Velten et al. Nature Comm 2012
Multi-path Analysis
![Page 25: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/25.jpg)
Raw Data
![Page 26: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/26.jpg)
Femto-Camera
Wall
Door
Hidden Mannequin
Velten et al, Nature Communications 2012
![Page 27: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/27.jpg)
![Page 28: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/28.jpg)
![Page 29: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/29.jpg)
Collision avoidance, robot navigation, …
29
DARPA REVEAL Program by Dr. Predrag Milojkovic
![Page 30: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/30.jpg)
Identify Materials
![Page 31: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/31.jpg)
![Page 32: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/32.jpg)
Naik, Zhao, Velten, Raskar, Bala, (SIGGRAPH Asia 2011)
BRDF from a single viewpoint
![Page 33: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/33.jpg)
Single Photon Sensitive Imaging (SPAD)
Gariepy et al. Nature Comm (2015)
![Page 34: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/34.jpg)
![Page 35: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/35.jpg)
![Page 36: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/36.jpg)
Nanophotography
70 picosecond resolution
[Kadambi et al 2013]
![Page 37: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/37.jpg)
Multi-Dimensional Light Transport
5-D Transport
![Page 38: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/38.jpg)
Question: How does Imaging influence Autonomy?
Level 0: Driver only: the human driver controls everything independently,
Level 1: Assisted driving: assistance systems help during vehicle operation (Cruise Control,
ACC).
Level 2: Partial automation: the operator must monitor the system at all times.
Level 3: Conditional automation: the operator monitors the system and can intervene when
necessary.
Level 4: High automation: there is no monitoring by the driver required. Vehicles are designed
to operate safety-critical functions and monitor road conditions for an entire trip. However, the
functions do not cover all every driving scenario.
Level 5: Full automation: operator-free driving.
![Page 39: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/39.jpg)
![Page 40: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/40.jpg)
Kadambi, Bhandari et al
![Page 41: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/41.jpg)
Course Schedule imagingav.media.mit.edu
13:30 – 13:50 Course IntroductionRamesh Raskar (MIT)
13:50 – 15:00 Existing Sensors and Their LimitsGuy Satat (MIT), Achuta Kadambi (UCLA)
15:00 – 15:10 Break
15:10 – 15:50 Emerging 3D SensorsAchuta Kadambi (UCLA)
15:50 – 16:30 Imaging in Bad WeatherGuy Satat (MIT)
16:30 – 16:40 Break
16:40 – 17:20 Deep Learning-based Computational ImagingJan Kautz (NVIDIA)
17:20 – 17:30 Conclusion and Open Problems
![Page 42: CVPR 2018 Computational Imaging for Self-Driving Vehicles...CVPR 2018 Computational Imaging for Self-Driving Vehicles Jan Kautz-----Ramesh Raskar-----Achuta Kadambi-----Guy Satat](https://reader033.vdocuments.mx/reader033/viewer/2022042911/5f41fabd6b87eb19d80695e4/html5/thumbnails/42.jpg)
CVPR 2018
Computational Imaging for
Self-Driving Vehicles
Jan Kautz--------Ramesh Raskar--------Achuta Kadambi--------Guy Satat