autonomous vehicles sensors and limitations · sensor good resolution (small voxels) recovers depth...
TRANSCRIPT
![Page 1: Autonomous Vehicles Sensors and Limitations · Sensor Good resolution (small voxels) Recovers depth Good spectral resolution Day / Night Any Weather Cost Visible spectrum 360°](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e8562872925a05179761261/html5/thumbnails/1.jpg)
Autonomous Vehicles –
Sensors and Limitations
Guy Satat
Computational Imaging for Self-Driving Vehicles @ CVPR 2018
![Page 2: Autonomous Vehicles Sensors and Limitations · Sensor Good resolution (small voxels) Recovers depth Good spectral resolution Day / Night Any Weather Cost Visible spectrum 360°](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e8562872925a05179761261/html5/thumbnails/2.jpg)
Image source: Cognata
![Page 3: Autonomous Vehicles Sensors and Limitations · Sensor Good resolution (small voxels) Recovers depth Good spectral resolution Day / Night Any Weather Cost Visible spectrum 360°](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e8562872925a05179761261/html5/thumbnails/3.jpg)
Ideal Sensor
Good resolution
(small voxels)
Recovers depth
Good spectral
resolution
Any WeatherDay / Night
Cost
Visible spectrum
360°
Coverage
Recovers speed
Real time
Short Exposure
Sees low reflectivity
targets
© 2018 Guy Satat. All Rights Reserved.
![Page 4: Autonomous Vehicles Sensors and Limitations · Sensor Good resolution (small voxels) Recovers depth Good spectral resolution Day / Night Any Weather Cost Visible spectrum 360°](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e8562872925a05179761261/html5/thumbnails/4.jpg)
Common Sensors on AVs
• Ultrasonic
• RGB Camera
• Stereo Camera
• LiDAR
• RADAR
Image source: https://www.wired.com/story/self-driving-cars-perception-humans/© 2018 Guy Satat. All Rights Reserved.
![Page 5: Autonomous Vehicles Sensors and Limitations · Sensor Good resolution (small voxels) Recovers depth Good spectral resolution Day / Night Any Weather Cost Visible spectrum 360°](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e8562872925a05179761261/html5/thumbnails/5.jpg)
RADAR / Ultrasonic / LiDAR
Source
Detector
Target
𝑑 =1
𝑐
𝑇
2
© 2018 Guy Satat. All Rights Reserved.
![Page 6: Autonomous Vehicles Sensors and Limitations · Sensor Good resolution (small voxels) Recovers depth Good spectral resolution Day / Night Any Weather Cost Visible spectrum 360°](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e8562872925a05179761261/html5/thumbnails/6.jpg)
Ultrasonic Sensor
• Short range (~cm – few meters)
• Low cost
• Mostly for parking assist
• Being replaced by RADAR
Image source: Wikipedia© 2018 Guy Satat. All Rights Reserved.
![Page 7: Autonomous Vehicles Sensors and Limitations · Sensor Good resolution (small voxels) Recovers depth Good spectral resolution Day / Night Any Weather Cost Visible spectrum 360°](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e8562872925a05179761261/html5/thumbnails/7.jpg)
RADAR
Short Range Long Range
24 GHz 77 GHz
Image source: Wikipedia
© 2018 Guy Satat. All Rights Reserved.
![Page 8: Autonomous Vehicles Sensors and Limitations · Sensor Good resolution (small voxels) Recovers depth Good spectral resolution Day / Night Any Weather Cost Visible spectrum 360°](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e8562872925a05179761261/html5/thumbnails/8.jpg)
RADAR
Short Range Long Range
24 GHz 77 GHz
Smaller antennas More accurate
Image source: Wikipedia
© 2018 Guy Satat. All Rights Reserved.
![Page 9: Autonomous Vehicles Sensors and Limitations · Sensor Good resolution (small voxels) Recovers depth Good spectral resolution Day / Night Any Weather Cost Visible spectrum 360°](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e8562872925a05179761261/html5/thumbnails/9.jpg)
RADAR
Short Range Long Range
24 GHz 77 GHz
Smaller antennas More accurate
360 (complements blind spots) Forward facing
Image source: Wikipedia
© 2018 Guy Satat. All Rights Reserved.
![Page 10: Autonomous Vehicles Sensors and Limitations · Sensor Good resolution (small voxels) Recovers depth Good spectral resolution Day / Night Any Weather Cost Visible spectrum 360°](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e8562872925a05179761261/html5/thumbnails/10.jpg)
RADAR
Short Range Long Range
24 GHz 77 GHz
Smaller antennas More accurate
360 (complements blind spots) Forward facing
Used for:• Lane change assist• Parking assist• Cross traffic monitoring
Used for:• Break assist• Adaptive cruise control
Image source: Wikipedia
© 2018 Guy Satat. All Rights Reserved.
![Page 11: Autonomous Vehicles Sensors and Limitations · Sensor Good resolution (small voxels) Recovers depth Good spectral resolution Day / Night Any Weather Cost Visible spectrum 360°](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e8562872925a05179761261/html5/thumbnails/11.jpg)
LiDAR
• Raster Scan• Multiple beams / detectors
• MEMS
• Solid state
• Limited optical budget• Eye safety
• Usually NIR (905 / 1550nm)
• Cost
• Low reflectivity targets
• Interference
Image source: Wikipedia
Image source: Wikipedia© 2018 Guy Satat. All Rights Reserved.
![Page 12: Autonomous Vehicles Sensors and Limitations · Sensor Good resolution (small voxels) Recovers depth Good spectral resolution Day / Night Any Weather Cost Visible spectrum 360°](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e8562872925a05179761261/html5/thumbnails/12.jpg)
SPAD Pixel
Time
Pixel𝑇
1
2
2
3
4
3
2
1
1
2
© 2018 Guy Satat. All Rights Reserved.
![Page 13: Autonomous Vehicles Sensors and Limitations · Sensor Good resolution (small voxels) Recovers depth Good spectral resolution Day / Night Any Weather Cost Visible spectrum 360°](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e8562872925a05179761261/html5/thumbnails/13.jpg)
PixelPixel
SPAD Camera
Pixel
PixelPixel
Pixel
PixelPixel
Pixel
32x32 pixels
© 2018 Guy Satat. All Rights Reserved.
![Page 14: Autonomous Vehicles Sensors and Limitations · Sensor Good resolution (small voxels) Recovers depth Good spectral resolution Day / Night Any Weather Cost Visible spectrum 360°](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e8562872925a05179761261/html5/thumbnails/14.jpg)
Depth Sensing
© 2018 Guy Satat. All Rights Reserved.
![Page 15: Autonomous Vehicles Sensors and Limitations · Sensor Good resolution (small voxels) Recovers depth Good spectral resolution Day / Night Any Weather Cost Visible spectrum 360°](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e8562872925a05179761261/html5/thumbnails/15.jpg)
Camera
• 360 / Rear • Shorter range
• Front facing• Long range
• Traffic lights / signs
• Challenges• Dynamic range: direct sunlight / tunnels
• Night time
Image source: Wikipedia
© 2018 Guy Satat. All Rights Reserved.
![Page 16: Autonomous Vehicles Sensors and Limitations · Sensor Good resolution (small voxels) Recovers depth Good spectral resolution Day / Night Any Weather Cost Visible spectrum 360°](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e8562872925a05179761261/html5/thumbnails/16.jpg)
Radar
Ultrasonic
Camera
LiDAR
Classification Localization Night Availability Any Weather
© 2018 Guy Satat. All Rights Reserved.
![Page 17: Autonomous Vehicles Sensors and Limitations · Sensor Good resolution (small voxels) Recovers depth Good spectral resolution Day / Night Any Weather Cost Visible spectrum 360°](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e8562872925a05179761261/html5/thumbnails/17.jpg)
X rays UV IR Microwave Radio Waves
Visible
Wavelength
Visible Light:
• Good resolution (compared to RF)
• Optical contrast
• Non-ionizing radiation
Imaging Across the EM Spectrum
Wavelengths:
• EM Visible ~𝜇𝑚
• EM Radio ~𝑐𝑚
• Ultrasound: > 𝑐𝑚© 2018 Guy Satat. All Rights Reserved.
![Page 18: Autonomous Vehicles Sensors and Limitations · Sensor Good resolution (small voxels) Recovers depth Good spectral resolution Day / Night Any Weather Cost Visible spectrum 360°](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e8562872925a05179761261/html5/thumbnails/18.jpg)
Wildlife Rehabilitation Center of Minnesota
Optical Contrast
© 2018 Guy Satat. All Rights Reserved.
![Page 19: Autonomous Vehicles Sensors and Limitations · Sensor Good resolution (small voxels) Recovers depth Good spectral resolution Day / Night Any Weather Cost Visible spectrum 360°](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e8562872925a05179761261/html5/thumbnails/19.jpg)
Other Sensors
• IR / Thermal cameras
• Time gating
• Stereo Cameras
• mmWave RADAR
• Event cameras
Image source: Wikipedia
© 2018 Guy Satat. All Rights Reserved.
![Page 20: Autonomous Vehicles Sensors and Limitations · Sensor Good resolution (small voxels) Recovers depth Good spectral resolution Day / Night Any Weather Cost Visible spectrum 360°](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e8562872925a05179761261/html5/thumbnails/20.jpg)
Sensor Fusion
RADAR
Perceived Environment
Drive Control
Camera
Perceived Environment
Drive Control
LiDAR
Perceived Environment
Drive Control
Fusion
LiDAR RADAR Camera
Fusion
Drive Control
Perceived Environment
© 2018 Guy Satat. All Rights Reserved.
![Page 21: Autonomous Vehicles Sensors and Limitations · Sensor Good resolution (small voxels) Recovers depth Good spectral resolution Day / Night Any Weather Cost Visible spectrum 360°](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e8562872925a05179761261/html5/thumbnails/21.jpg)
Learning to Sense
• What sensors do we really need?
• How does sensors spec affect end-to-end performance
Hardware
Algorithms
Computational imaging
© 2018 Guy Satat. All Rights Reserved.
![Page 22: Autonomous Vehicles Sensors and Limitations · Sensor Good resolution (small voxels) Recovers depth Good spectral resolution Day / Night Any Weather Cost Visible spectrum 360°](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e8562872925a05179761261/html5/thumbnails/22.jpg)
Key Takeaways
• Different sensors for:• Complement limitations
• Different applications
• Ultrasonic, RADAR, LIDAR, Camera
• Sensor Fusion
Images source: Wikipedia© 2018 Guy Satat. All Rights Reserved.