new machine vision sensors for autonomous driving

NEW MACHINE VISION SENSORS FOR

AUTONOMOUS DRIVING

RONALD MUELLER

VISION MARKETS

EMVA BUSINESS CONFERENCE 2018, DUBROVNIK

2018-06-09

Trends

Full Screen image on trendsYour global growth in Imaging and Photonics

stronger

Brandmore

Salesbetter

Offeringsbigger

Company

Trends

Full Screen image on trends

51 customers served

since 2015:

• MV OEMs

• Private Equity Firms

• MV-Users

• System Integrators

• Distributors

• EMSs

• Market researchers

• Mobile Phone OEM

Trends

Full Screen image on trends

51 customers served

since 2015:

• MV OEMs

• Private Equity Firms

• System Integrators

• Distributors

• EMSs

• Market researchers

• Mobile Phone OEM

http://www.euclidlabs.it/event/

TRENDS

full screen image on welcome and or first things first

Market Trends

1893

175 15766 1,95

72

1855

177 161 982,25

72

0

200

400

600

800

1000

1200

1400

1600

1800

2000

Mobile phones Tablets Notebooks SurveillanceCams

MV Cams Other

Million

units

Shipments of Devices with Image Sensors

2016 2017

IMAGE SENSOR MARKET

av. 2 image sensors / phone & tablet

GROWTH IN SURVEILLANCE & MV

4092

1241

41997

5360

1340

604150

0

1000

2000

3000

4000

5000

6000

Hikvision FLIR Systems Cognex Basler

Turn

ove

r [m

€]

Surveillance & Machine Vision – Selected Players

2016 2017

AUTOMOTIVE

73

77

80

68

70

72

74

76

78

80

82

Cars

Million

units

Global Car Production

2015 2016 2017

Every new car sold in

US is equipped with

a rear-view camera

Up to 12

image sensors/car

AUTOMOTIVE VISION MARKET: ~25BN$ BY 2025

Image source: Autoliv

VISION BOM IN L4/5 CARS: 385 $

Automotive

requirements are

driving sensor

technology

developments

QUIZADAS & ADS

Image source: Youtube

Image source: Youtube

Image source: Youtube

Image source: Youtube

Image source: Youtube

Image source: Youtube

Image source: Youtube

BASICS OF ADAS & ADS

• Standards

• SAE International – J3016:

Taxonomy and Definitions for Terms

Related to Driving Automation Systems

for On-Road Motor Vehicles

• Terminology

• ADAS: Advanced Driver Assistance System

• ADS: Autonomous Driving System

• DDT: Dynamic Driving Task

• Levels: 0 – 5

Image source: SAE International

AUTONOMOUS DRIVING – SAE LEVELS 0 - 5Le

vel

NameLateral &

Longitudinal ControlMonitoring Fallback

Automated

Driving Modes

0 No Automation n/a

1 Driver Assistance Some

2 Partial Automation Some

3 Cond. Automation Some

4 High Automation Many

5 Full Automation All

Level

NameLateral &

Longitudinal ControlMonitoring Fallback

Automated

Driving Modes

0 No Automation n/a

1 Driver Assistance Some

2 Partial Automation Some

3 Cond. Automation Some

4 High Automation Many

5 Full Automation All

AUTONOMOUS DRIVING – SAE LEVELS 0 - 5

Level

NameLateral &

Longitudinal ControlMonitoring Fallback

Automated

Driving Modes

0 No Automation n/a

1 Driver Assistance Some

2 Partial Automation Some

3 Cond. Automation Some

4 High Automation Many

5 Full Automation All

AUTONOMOUS DRIVING – SAE LEVELS 0 - 5

IMAGE SENSORS IN SAE LEVEL 1&2

•Machine Vision

• Driving lane recognition

(departure warning / automatic steering)

• 3D localization of other road users

(driver alerting / emergency breaking,

lane changes, adaptive front lights)

• Street sign recognition

• Driver monitoring

•Human vision (surround view)

•Night vision

https://www.sensorsmag.com/components/three-sensor-types-drive-autonomous-vehicles

IMAGE SENSORS IN SAE LEVEL 3-5

Additional functions:

•3D sensing of the road

• Lanes, road curbs, parking cars,

junctions, construction work, etc.

•3D sensing of mobile objects

• Other vehicles, cyclists, pedestrians

• Pets and wildlife

• Obstacles

•2D recognition of traffic lights

Image source: Nvidia

CHALLENGES FOR AUTOMOTIVE IMAGE SENSORS

• Sunlight, head lights, reflections

• Low light

• Compactness

• Car speed

• High spatial & depth resolution

• LED Flicker

• Temperatures

• Reliability

• Security

Image source: Sensormag

Source: Ni78.com

SENSOR ARRAY IN MERCEDES S-CLASS

Image: https://blog.caranddriver.com/sensory-overload-how-the-new-mercedes-s-class-sees-all/

SENSOR ARRAY IN MERCEDES S-CLASS

Image: https://blog.caranddriver.com/sensory-overload-how-the-new-mercedes-s-class-sees-all/

4 Wide angle

cameras for 360°

surround view

NIR camera

for night vision

display

Infra-red camera to spot

people and wildlife up to

160 m

Visible spectrum stereo camera

for object detection up to 400 m

3D sensing up to 50 m

IMAGE SENSORS REQUIRED FOR ADS LEVEL 3 - 5

https://www.sensorsmag.com/components/three-sensor-types-drive-autonomous-vehicles

TECHNICAL DEVELOPMENTS IN AUTOMOTIVE

Compact front-view stereo

camera modules

•3D sensing at up to 100 m at

35° FOV

•Traffic sign/light recognition

Image: http://www.safecar.info/early-sunsets-require-different-driving-techniques/

Image source: Denso

Image source: Subaru

TECHNICAL DEVELOPMENTS IN AUTOMOTIVE

RCCC image sensors

•Max. sensitivity

•Color recognition for traffic

signs, traffic lights, lanes

Image: http://www.safecar.info/early-sunsets-require-different-driving-techniques/

Image source: Schematic Scholar

TECHNICAL DEVELOPMENTS IN AUTOMOTIVE

Split pixels / Multi-gain readout / Super-Exposure

•Single-exposure HDR: 95 dB

•Multi-exposure HDR: 140 dB

•Full-well capacity of 100 ke-

Image source: Sony

TECHNICAL DEVELOPMENTS IN AUTOMOTIVE

LED Flicker Mitigation in daylight

Image: http://www.safecar.info/early-sunsets-require-different-driving-techniques/

Image source: ON Semiconductor

TECHNICAL DEVELOPMENTS IN AUTOMOTIVE

LiDAR

(Light Detection and

Ranging)

Image: http://www.safecar.info/early-sunsets-require-different-driving-techniques/

TECHNICAL DEVELOPMENTS IN AUTOMOTIVE

SPADs:

Single Photon Avalanche Diodes

SiPM:

Silicon Photon Multipliers

Image: http://www.safecar.info/early-sunsets-require-different-driving-techniques/

TECHNICAL DEVELOPMENTS IN AUTOMOTIVE

Solid state LiDAR at

905 or 1500 nm

•SLAM up to 300 m

•Costs down to

few hundred $

Image: http://www.safecar.info/early-sunsets-require-different-driving-techniques/

ACHIEVEMENTS AT AUTOMOTIVE SENSORS

• Resolution: 2 MP stereo, 8 MP single

• Compactness: 2.25 – 3 µm

• Sensitivity: several ke-/lux s

• Full-well capacity: 100 ke-

• Dark noise: 1 e-

• High Dynamic Range:

multi-exposure: 140 dB

single-exposure: 95 dB

• Stacked sensors with on-chip processing,

e.g. motion compensation at ME-HDR

• Frame rate:

60 fps @ 1080p

40 fps @ 7.4 MP

• Heat resistance: 130°C

• Self-monitoring: ASIL B/C & IEC26262

• On-chip security features

• Solid State Lidar based on SPAD/SiPM

& MEMS

SUMMARY

•Market potential of Automotive Vision

drives image sensor innovations

• Tough requirements and first advancements:

• Imaging performance (HDR, Sensitivity, …)

• Compactness

• Reliability

• Security

• Cost

• Solid State LiDAR about to be solved

Tuesday, June 12, 2018© VISION MARKETS

Curious for more?

NEW MARKET REPORT – AVAILABLE FROM Q3/2018

• Technology challenges deriving from application requirements for ADAS and

autonomous vehicles.

• Scoring and ranking based on technical difficulty and impact

• Detailed analysis of the current status of solutions for each top ranked technology

challenge, and forecast for the state-of-the-art until 2023

Top 20 Technology Challenges for

Image Sensors in Automotive to 2023

CONTACT

Ronald MüllerPresident, Vision Markets UG

contact@markets.visionwww.markets.vision

t (DE): +49 89 21 553 665t (US): +1 302 231 1343

Your global growth in Imaging and Photonics