prof. dr. rer. nat. hermann winner dipl.-ing. walther ... · examples include electronic stability...

49
(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016 (How) Can Safety of Automated Driving be Validated? Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther Wachenfeld Philipp Junietz, M.Sc.

Upload: hoanghanh

Post on 11-Aug-2019

221 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

(How) Can Safety of Automated Driving be Validated?

Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther Wachenfeld Philipp Junietz, M.Sc.

Page 2: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

2

Considered Levels of Automated Driving

Levels ≥3: Highly Automated Driving (ref. BASt, VDA level 3)

Limited Self-Driving Automation (ref. NHTSA level 3)

Conditional Automation (ref. SAE level 3)

High Automation (ref. SAE level 4)

Fully Automated Driving (ref. BASt, VDA level 4)

Full Self-Driving Automation (ref. NHTSA level 4)

Full Automation (ref. SAE level 5)

Driverless Vehicle (ref. VDA level 5)

For all: No responsibility of human drivers during operation of automation

Sources: bast [1], VDA [2], SAE [3], NHTSA [4]

Vorführender
Präsentationsnotizen
NHTSA defines vehicle automation as having five levels: No-Automation (Level 0): The driver is in complete and sole control of the primary vehicle controls – brake, steering, throttle, and motive power – at all times. Function-specific Automation (Level 1): Automation at this level involves one or more specific control functions. Examples include electronic stability control or pre-charged brakes, where the vehicle automatically assists with braking to enable the driver to regain control of the vehicle or stop faster than possible by acting alone. Combined Function Automation (Level 2): This level involves automation of at least two primary control functions designed to work in unison to relieve the driver of control of those functions. An example of combined functions enabling a Level 2 system is adaptive cruise control in combination with lane centering. Limited Self-Driving Automation (Level 3): Vehicles at this level of automation enable the driver to cede full control of all safety-critical functions under certain traffic or environmental conditions and in those conditions to rely heavily on the vehicle to monitor for changes in those conditions requiring transition back to driver control. The driver is expected to be available for occasional control, but with sufficiently comfortable transition time. The Google car is an example of limited self-driving automation. Full Self-Driving Automation (Level 4): The vehicle is designed to perform all safety-critical driving functions and monitor roadway conditions for an entire trip. Such a design anticipates that the driver will provide destination or navigation input, but is not expected to be available for control at any time during the trip. This includes both occupied and unoccupied vehicles.
Page 3: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

3

Validation Challenge of Automated Driving

Challenge: Validation of promised safety level above the level of driving by humans: Evidence is needed that risk does not exceed today reference.

But what is the safety reference for validation?

Page 4: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

4

Safety References: Mortality

Abstract reference figures of mortality:

Criterion Yearly mortality Hourly mortality

Minimum Endogenous Mortality (MEM, EN 50126) 2·10-4/a 2.3·10-8/h

MEM adapted to Germany 2012/2014 (destatis) 0.7·10-4/a 0.8·10-8/h

Mean mortality (D, EU), (destatis) 10-2/a 1.1·10-6/h

Life expectancy (destatis) 1/90a 1.3·10-6/h

Page 5: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

5

Safety References: Today Road Traffic

Accidents per distance:

Numbers for Germany 2014 Total Autobahn

Distance travelled 726·109 km 225·109 km Total number of accidents 2.42·106 0.15·106 With personal injury 291·103 18.4·103

Distance between two accidents:

All accidents 0.34·106 km 1.67·106 km Involving personal injury 2.5·106 km 12·106 km Involving serious casualties >11·106 km >40·106 km Involving fatalities >200·106 km 660·106 km

Data from: [Statistisches Bundesamt, German Federal Statistical Office, 2014])

Page 6: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

6

Excursion: Risk Figures for Human Drivers

1 German person drives 55 years each year 14,000 km = 770,000 km/lifetime ≈ 15,000 h (at average speed 50 km/h) The average driver is involved every 340,000 km into a reported accident,

and self caused by 60%.

The average driver is involved every 210 mio. km into an accident with fatalities.

In average 1.4 reported accidents are caused by one human in his/her lifetime.

Nearly impossible to differentiate between good and bad drivers, just between lucky and unlucky drivers.

An accident with fatalities will be caused after 450 lifetimes of driving.

Page 7: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

7

Safety References: Time vs. Distance Normalization

Comparison: Basis: Mean velocity: 50 km/h (general)/100 km/h (Autobahn),

(total number of fatalities)/(accidents with fatalities) ≈ 1.3

Reference distance between two accidents with fatalities

Criterion Total (vav = 50 km/h)

Autobahn (vav = 100 km/h)

Minimum Endogenous Mortality (MEM, EN 50126) 2.85·109 km 5.7·109 km

MEM adapted to Germany 2012/2014 (destatis) 8.1·109 km 16.3·109 km

Mean mortality: ≈ 10-2/a (D, EU), (destatis) 57·106 km 114·106 km

Life expectancy: ≈ 90a, (destatis) 51·106 km 103·106 km Road traffic reality ≈ 260·106 km 660·106 km

Page 8: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

8

Safety References: Alternative Mobility

Comparison: Passenger distance per fatalities (in passenger-kilometers)

Mobility type

Motorcycles 30·106 pkm

Passenger Cars 500·106 pkm

Public Transport 6·109 pkm

Aviation 300 ·109 pkm

Data from: Verkehr in Zahlen 2015

Page 9: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

9

Safety References (Conclusion)

Reference variants: Possible safety references are within a wide bandwidth (several orders of

magnitude), much above today road safety as well as much below.

A progress in safety by automation has to be measured in comparison with today risk as reference.

At least two relevant categories have to be addressed as reference:

accidents with damage to persons and specifically

accidents with fatalities

Reference risk figures are far from today testing horizons by real driving tests, e.g. for Autobahn in Germany 2014

Distance between accidents with damage to persons 12·106 km

… with fatalities 660·106 km Data based on: Statistisches Bundesamt, German Federal Statistical Office, 2015, [5]

Page 10: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

10

Statistical Considerations

Poisson Distribution (independent random process) for the probability, that k events occur in case of an expected value of λ:

λ = ratio between observed test kilometers and system performance

The system performance describes the expected travel distance between two events

perf

test

ss

Ref.: Wachenfeld, W., Winner, H. [6], Winner, H. [7]

Page 11: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

11

Conclusion on statistics

Example: AD system might be twice as safe as human driver

If AD system drives about 10x the reference distance of human, probably 5 accidents would occur (instead of the mean of 10 from human).

So, with that result it is confident to about 95% that the system is not worse than human reference.

So, typically 10x the reference distance has to be expected for validation.

Validation Targets Depending on class of injury and looking for “self-caused” accidents only

more than 200·106 km (all injuries) or 10·109 km (fatalities) have to be expected for a first serious figure!

Demonstrating safety of automated driving in advance of introduction is nearly impossible => Approval Trap

Page 12: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

12

Supplementing information

Fundamental knowledge about ADAS, components and systems and development and testing methodology

What does engineering know up to partial automation?

Technical, Legal and Social

Aspects

A glance into the future

What does research know about autonomous driving?

Handbook of ADAS Autonomous Driving

Page 13: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

13

STOP!!!!!

For today’s vehicles (and more extreme for aviation) there is no requirement for such high testing distance, why here? What is the fundamental difference?

Page 14: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

14

Differences between conventional and automated vehicles

Vehicle

Longitudin. and

Lateraldyn.

Driver

Navigation

Guidance/ Conducting

Stabilization

Selected route Time schedule

Desired speed and trajectory

Steering Accelerating

Vehicle motion

Environment

Road network

Traffic situation

Road surface

Alternative routes

Range of safe motion states Actual trajectory and speed

Transport mission

according to Rasmussen [8] and Donges [9]

Knowledge-based Behavior

Rule-based Behavior

Skill-based Behavior

Sensory Input

Driving robot and vehicle

Current validation of vehicle doesn‘t cover the yellow area

Page 15: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

15

What do we know about Driving Safety Performance?

Statistics and Accident Research Reports on frequency of accidents and their causes

Figures about time gaps and exceeding speeds of some roads

Driver modeling Qualitative models for information processing and driving tasks

(Rasmussen, Donges, …) are able to explain the observed behavior.

Quantitative models for simple scenarios (car following, lane change, intersection crossing) are able to explain and predict traffic flow figures, but not accidents frequency and severity.

Human reliability models (Reichart, …) interpret the observed accidents frequency.

Page 16: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

16

Simple Probabilistic Accident Model

Image: https://en.wikipedia.org/wiki/ Swiss_cheese_model#CITEREFReason1990

surroundingE

pavementE

trafficdriver

egodriverCheese model idea from [10]

Swiss Cheese Model (adapted to human drivers)

, , , , /

, ,

; ( , )

( , )accidents hd crit hd transition hd crit hd ego traffic road

transition hd ego hd traffic

n n n f driver Ef driver driver

ρ

ρ

= ⋅ =

= n = frequency ρ = transition probability E = exposure of circumstances for potential hazards

Page 17: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

17

Knowledge about Driving Task and respective Safety

Lacks: Serious figure of the accident avoidance capability of human drivers

Frequency and type of non-standard situations (both self caused or innocently exposed)

Performance of human drivers in non-standard situations

Dark matter problem: We only know standard scenarios and the reported fail scenarios

(accidents), but do not know the probability for transition from accident free driving to real accident occurrence.

Avoiding the known human accident causes are not sufficient:

1. The accidents avoidance capability of humans is not recorded.

2. No quantitative figure about types of critical scenarios and their frequency where humans avoid accidents.

Page 18: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

18

Dark Matter Problem

Uncritical scenarios (very low potential for accidents)

Critical scenarios (potential for accident)

True accident scenarios

Page 19: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

19

Swiss Cheese Model (adapted to automated driving)

Accident Model for Automated Vehicles Automation Risks

, , , , , , , / /

, , / /

; ( , )

( , )accidents new crit ad new transition ad new crit ad old new ego traffic road

transition ad old new old new ego partner

n n n f robot Ef robot driver

ρ

ρ

= ⋅ =

=

egorobot

trafficdriver

, , , , ,

, , , , , ,

accidents ad accidents ad old accidents ad new

accidents ad old crit ad old transition ad old

n n nn n ρ

= +

= ⋅

Page 20: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

20

Dark Matter Problem

Uncritical scenarios (very low potential for accidents)

Critical scenarios (potential for accident, old type)

True accident scenarios (old type)

Automation risk exposure (new critical scenarios)

Automation accidents (new type)

Page 21: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

21

Knowledge Lack

For prediction of safety of automated driving we need: Valid quantitative number of critical scenarios (remaining and new critical

scenarios) and their specific characteristics

Valid models for capability of AD to control critical situations in a safe manner.

All figures have to be compared with the reference risk numbers of each relevant class.

With respect to the Swiss Cheese Model:

We have to model each slice in order to predict the risk of AD with high validity.

Page 22: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

22

First step of Risk Minimizing: Preventing of known human errors

Reduction of ncrit,ad,old by adequate careful driving behavior Today more than 90% accidents are caused by human mistakes.

Typical causes:

Inattentiveness, drowsiness

Exceeding speed, too short distances (time gaps)

Avoidable by design of vehicle guidance controller

For standard scenarios simply testable

Possible results of validation: Factors of improvement for these categories (e.g. 1000x less frequent time gaps below 0.5 s)

Most of human caused accident types will be removed from accident statistics by automation.

But, approach addresses only the “bright” matter of safety.

Page 23: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

23

Second step of Risk Minimizing: Limitation of Exposure in Hazardous Circumstances

Limitation of use case complexity for reduction of ncrit,ad,old Clear traffic rules

Defined or well known scenery

Motorways or similar

Defined areas

Speed limitation

Requirement for just a small set of behavioral strategies for vehicle guidance.

This approach will reduce frequency of critical scenarios, but there is no measure about the transition probability.

Page 24: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

24

First conclusion

The obvious safety gain: The functional design of automated driving promises higher safety by

reduction of frequency of known critical situations.

Still lacking: Capability of AD to avoid accidents in the remaining critical situations

Frequency of new critical situations generated by automated driving and the capability to control them safely.

Validation of automated driving has to cover both and has to gain all necessary knowledge prerequisites.

Page 25: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

25

Prerequisites for successful validation

We should know about … the representative worst case test cases,

the metrics for identification of critical situations,

the environmental influence on perception,

how the behavior can be tested as robust and safe,

whether the simulation models for MiL, SiL, HiL, ViL are valid and how to validate,

how representative the simulation has to be for approval purpose.

How can we gain that missing knowledge?

Page 26: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

26

Evolutionary Approach to Gain Testing Knowledge based on Functional Evolution

Functional evolution Predecessor function gains testing knowledge for successor function:

Data bases of situations, remarkable road parts, of sensor raw signals

Test cases (for real and virtual tests)

Virtual Assessment of Automation in Field Operation (VAAFO) [11]

More realistic simulation models

Statistical data for risk assessment

How does the functional evolution look like? It depends on the Use Case of Autonomous Driving [12]

Page 27: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

27

From ADAS to Autonomous Driving

Risk Speed

The Evolution Triangle towards Autonomous Driving

ACC: Adaptive Cruise Control LKS: Lane Keeping Support L²A: Longitudinal & Lateral Assist. FSR-ACC: Full-Speed-Range-ACC

AVP: Autonomous Valet-Parking AP: Automated Parking PSA: Park Steering Assist

Em-A: Emergency Assist CA-E: Collision Avoidance by Evading CA-B: Collision Avoidance by Braking CM-B: Collision Mitigation by Braking

(Source [Winner, H.: Quo vadis, FAS? In: Winner, H., Hakuli, S., Lotz, F., Singer, C. (eds.) Handbuch Fahrerassistenzsysteme, 3rd edn. Vieweg-Teubner-Verlag (2015)]

Ref.: Winner, H. [6]

Page 28: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

28

Evolutionary Approach to Gain Testing Knowledge based on Use Case Extension

Use case extension Functional start with full driverless automation

Limitation of operation area

From few routes to new driving area (incremental extension of potential risk).

At first, there is no comparable benchmark for the risk.

Field experience will make the autonomous driving more and more mature.

The operation is supervised (e.g. by a provider) and can be controlled (including shut down).

Page 29: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

29

Where and under what conditions is the automation available? Not only the level of automation and

the use case offer evolutionary paths

Also an evolution in availability is reasonable

Different approaches exist (most OEM vs. Google)

Availability vs. Degree of Automation

Availability

Something Everything Level of Automation

OEM

Google

Every- where

Some- where

Page 30: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

30

Virtualization of Tests for Validation

Enabling of efficient test tools Virtual testing has potential to accelerate approval, but with virtualization

always simplification takes place Validation of model?

Different ways of combining virtuality and reality exist and can be used

Sensor simulation for SiL and sensor stimulation for HiL/ViL is needed

Ref.: Wachenfeld, W., Winner, H. [7]

Page 31: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

31

Virtual Validation

Principle: Running tests by many paralleled systems (ViL, HiL, SiL) equivalent to

billions of km (109 km ≈ 20·106 hours = 2,280 years real time)

Validation of simulation models

component models (e.g. sensors) by component testing systems

environment conditions from test drives recording

vehicle dynamics by test maneuvers

Situation creation by permutation or stochastics (Monte Carlo approach).

Big challenge for modeling:

Validation of behavior models of other traffic participants.

Valid environment and sensor models

Page 32: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

32

Virtual Validation

Problem due to the manifold of situational combinations Many sceneries and circumstances

Many constellations of traffic partners

Various environment conditions (e.g. rain, fog, pavement condition, light brightness and direction, …)

Permutation of all influence parameters will overload any computer cluster Evaluation of safety in simulation just by counting occurred virtual

accidents needs the full combination of all variables

How can we come to a feasible virtual approval?

Page 33: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

33

Approval by Worst Case Testing

Systematic reduction of test cases Identify critical situations only

Assumption: If all critical situations could be handled, every less critical situation can be handled, too.

Need of a validated metric for criticality in situations

Even with only checking critical situations a huge variety of these critical situation must be tested (e.g. various weather conditions, brightness,…).

Challenge of reduction and virtualization

Real world Relevant world Artificial/virtual

tests Real driving

Page 34: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

34

Decomposition Approach for Test Case Reduction

Decomposing of the scenario into several layers Clustering of test cases (experimental as well as in simulation) related to

the layer

Situation Umgebung (dyn.) Situation Umgebung (dyn.) Situation subject vehicle

1: Information access

2: Information reception

3: Information processing

4: Decision (behavioral)

5: Action

Situation environment (dyn.) Situation statical

Object under test Test environment

1: Information access

2: Information reception

3: Information processing

4: Decision (behavioral)

5: Action

Layer after Graab et al. [13]

Page 35: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

35

Decomposition Approach for Test Case Reduction

Decomposition may lead to reduction of test effort by Modularisation of tests and no or less re-test in case of unchanged

functional modules

Reduction of redundant tests within the same layer

Pre-requisite: Pass/fail criteria (metrics) depending on decomposing layer

Open promise: Metrics for comparison on safety performance human vs. machine

Page 36: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

36

Virtual Approval

But: How can we be convinced that the test case set is sufficient? Check whether all situations of reconstructed accidents are part of the

test set does help just a little bit, because driving robot behavior has intentionally to differ from human driving.

And nobody knows how the behavior of human would change when they were confronted with autonomous driving.

Conclusion on virtual approval Simulation will be a very important part of the approval process, but it will

not help to overcome the approval trap due to the lack of validity for modeling and test case set in the beginning.

A virtual approval should be future objective for methodological and economical reasons.

So, the other test methods have to improve the model validity and the test catalogue.

Page 37: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

37

The new Role of Driving Tests

Driving tests for model validation Comparison of behavior between simulated and real situations

Driving tests for environment representation validation and behavior pattern of other traffic participants

Driving tests for test catalogue Situational statistics for importance ranking of test cases

Making the test case catalogue complete (for both virtual and real tests)

Only real drives are able to assess the validity of model and test catalogue.

Page 38: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

38

The new Role of Driving Tests

By the rate of “surprises” of new driving tests a careful extrapolation how valid the models are and how complete the current test catalogue might be possible.

Surprises per distance in 1/Mkm Number of counted new events per distance

Approx. trend line Extrapolation

1 10 100 1000

1000

100

10

1 Driven test kilometers 103 ⋅

Page 39: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

39

Field Data Assessment

How can knowledge be extracted from driving experience: 1. Recording of data, trigger by (test) driver, evaluation of triggered time

slices in Lab.

2. Transmission of all relevant situations within real-time to Lab. Labeling and assessment by automatic data analytics.

3. Virtual Assessment of Automation in Field Operation (VAAFO), triggering and extraction of data set onboard, records could be transmitted w/o real-time need.

All methods can be used with active or emulated automation or predecessor automation.

VAAFO is suitable for implementing in series fleet to assess potential upgrade functions.

Page 40: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

40

VAAFO Virtual Assessment of Automation in Field Operation

Combines the advantages of both methods (SiL and field test): Reality based test case generation

Test tool enabling tests without higher risk than usual real world driving

The limitations of the concept: Validity of the virtual world

Initialize (e.g. every 0,5 s) the virtual world on base of real world measurement.

Short (e.g. 2 s) simulation of the virtual world

Test distances don’t decrease

Start early in the development phase

Implement the tool in fleets field observation (data protection should be discussed)

Page 41: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

41

VAAFO Virtual Assessment of Automation in Field Operation

The strength of the concept: Using the tool in for example 10,000 vehicles with average milage of

14.000 km a year. [Numbers from DAT for 2013]

This leads to 140 million km driving in real traffic.

Concept can be applied on different steps of development and evolution

1st testing on road to

nth software-update

An Autobahn-Chauffeur equipped with sensors and processing power is capable to record data from driving in cities or on country roads.

Page 42: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

42

European Project:

European Initiative to Enable Validation for Highly Automated Safe and Secure Systems 05/2016 - 04/2019

Current Research Projects to overcome the Approval Trap of AD

National Project:

Project for Establishing Generally Accepted quality criteria, tools and methods as well as Scenarios And Situations for approval of highly automated driving functions 01/2016 - 06/2019

Page 43: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

43

Remaining Approval Trap Problem

Prognosis: For the first Autonomous Driving (Level ≥ 3) application the pre-requisites

for an approval for general and unlimited introduction will not be given (like a chicken & egg-problem).

The Approval Trap might be disarmed to some extend by the methodological work before, but not sufficient.

What is the alternative?

Page 44: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

44

Strategy: Risk limited Introduction: How statistics may help to introduce automated driving Risk limitation based on statistical figures (ref. [14]) Testing travel distance with the recorded mean value of accidents gives

an estimate for the expected accident value (depending on a given error probability one can calculate a best and a worst case factor).

Taking the worst case factor one can calculate the maximum expected risk for a given number of autonomous vehicles in the field.

Whether this worst case risk is below a detection limit in a statistical sense the vehicles can be introduced in order to record additional data helping the release for the next higher number of autonomous vehicles.

The driven travel distance increases the statistical basis and the fleet in traffic can be increased recursively.

Page 45: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

45

Conclusion (I)

The Approval Trap is still existing. There are different promising approaches Virtual Approval,

Virtual Assessment of Automation in Field Operation (VAAFO),

Decomposition, and

Worst Case Testing

overcoming the trap, but they need prerequisites which are far from today state-of-the-art.

Page 46: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

46

Conclusion (II)

Functional evolution or use case extension are strategies to get sufficient data. But there are still some doubts whether the quality of the methods for validation will be then sufficient. Test drives will play the key role for risk assessment and for development of all alternative test methodologies. A limiting risk introduction strategy will help to introduce autonomous driving by “tunnelling” the barrier or by skipping the trap for the first systems.

Page 47: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

47

Back to the Title of the Presentation

Can Safety of Automated Driving be Validated? Yes,

but not in advance of introduction on the level of reference figures coming from human drivers.

How can Safety of Automated Driving be Validated?

1. Data collecting by test driving and supervised risk limited introduction

2. Development and validation of simulation models for virtual tests

3. Certification of safety by combination of mainly virtual tests and recursive validation with real driving data.

Page 48: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking

(How) Can Safety of Automated Driving be Validated? | Prof. H. Winner | Virtual Vehicle Symposium Graz | May 24, 2016

(1) Gasser, T. M.; Arzt, C.; Ayoubi, M.; Bartels, A.; Bürkle, L.; Eier, J.; Flemisch, F.; Häcker, D.; Hesse, T.; Huber, W.; Lotz, C.; Maurer, M.; Ruth-Schumacher, S.; Schwarz, J.; Vogt, W.: Rechtsfolgen zunehmender Fahrzeugautomatisierung. Gemeinsamer Schlussbericht der BASt-Projektgruppe „Rechtsfolgen zunehmender Fahrzeugautomatisierung“ Dokumentteil 1. Wirtschaftsverlag NW, Bergisch Gladbach, 2012 (Heft F 83)

(2) Verband der Automobilindustrie: From Driver Assistance Systems to Automated Driving, VDA Magazine – Automation, 2015 (3) SAE: Levels of Driving Automation, Information Report J3016, 2014 (4) US National Highway Traffic Safety Administration (NHTSA): Preliminary Statement of Policy Concerning Automated Vehicles,

2013 (5) Statistisches Bundesamt / German Federal Statistical Office, 2014, https://www.destatis.de/DE/Publikationen/

Thematisch/TransportVerkehr/ Verkehrsunfaelle/VerkehrsunfaelleJ2080700147004.pdf?__blob=publicationFile (6) Winner, H.: ADAS, Quo Vadis?, in Winner, H.; Hakuli, S.; Lotz, F.; Singer, C. (eds.): Hand of Driver Assistance Systems, Springer

2016 (7) Wachenfeld, W., Winner, H.: Die Freigabe des autonomen Fahrens. In: Maurer, M., Gerdes, J.C., Lenz, B., Winner, H. (Hrsg.)

Autonomes Fahren, pp. 439-464. Springer Berlin Heidelberg (2015) (8) Rasmussen, J.: Skills, Rules, and Knowledge; Signals, Signs, and Symbols, and Other Distinctions in Human Performance

Models. IEEE Transactions On Systems, Man, and Cybernetics SMC-13(3), 257–266 (1983)] (9) Donges, Edmund: Fahrerverhaltensmodelle. In: Winner, Hakuli, Wolf (eds.) Handbuch Fahrerassistenzsysteme, pp. 15–23 (2011) (10) Reason, James (1990-04-12). "The Contribution of Latent Human Failures to the Breakdown of Complex Systems". Philosophical

Transactions of the Royal Society of London. Series B, Biological Sciences 327 (1241): 475–484. (11) Wachenfeld, W., Winner, H.: Virtual Assessment of Automation in Field Operation – A New Runtime Validation Method, FAS

Workshop in Walting 2015 (12) Wachenfeld, W., Winner, H., Gerdes, C., Lenz, B., Maurer, M., Beiker, S.A., Fraedrich, E., Winkle, T.: Use-Cases des autonomen

Fahrens. In: Maurer, M., Gerdes, J.C., Lenz, B., Winner, H. (eds.) Autonomes Fahren, pp. 9-37. Springer Berlin Heidelberg (2015)])

(13) Graab et al.: Analyse von Verkehrsunfällen hinsichtlich unterschiedlicher Fahrerpopulationen und daraus ableitbarer Ergebnisse für die Entwicklung adaptiver Fahrerassistenzsysteme, 2008

(14) Wachenfeld, W.; Winner, H.: The new role of road testing for the safety validation of automated vehicles. In Horn, M.; Watzenig, D. (eds.): Automated Driving – Safer and more efficient future driving; Springer International Publishing AG (2016)

References

Page 49: Prof. Dr. rer. nat. Hermann Winner Dipl.-Ing. Walther ... · Examples include electronic stability control \൯r pre-charged brakes, where the vehicle automatically assists with braking