1
Jace AllenBusiness Development Manager – Simulation, Test, and EEDM
dSPACE Inc. · 50131 Pontiac Trail · Wixom, MI · 48393 USA
Autonomous Automation: How do we get to a Million Miles of testing?
Agenda
1. Intro to V&V for ADAS/AV/HAD
Changing Environment of AV/HAD
ADAS V&V Process and ISO26262
2. Testing Toolchain for ADAS/AV/HAD
Models, Scenarios, and Sensors
HIL Testing and Sensor Fusion Needs
SIL Testing and Cluster Simulation
3. Testing Process and Autonomous Automation
Testing Methods and Tools
Real-time Testing and Observers
Test Management and Automation
Optimizing Testing
Challenges – Testing autonomous driving in real traffic
4
Unlimited number of real-life traffic scenarios
Many unknown factors, human driver no longer as a fallback … how to validate system robustness?
Exponential growth in testing effort …. hundreds of millions of test kilometers required.
Source: nik/pkbThe real world is complex The real world is unpredictable The real world is hazardous
Validate System Behavior with Simulation
5
MBD Testing = Simulation at All Levels
Advantages of simulation: reproducibility,
test beyond performance/endurance
limits and dangerous situations
ISO 26262 recommends MIL/SIL/HIL
simulation for conducting the software
safety requirements verification
What changes with testing Autonomous
Vehicles?
dSPACE-Internal6
ADAS/AV development process
System
conceptMIL
(traffic simulation)
Component HIL
(closed-loop)
ADAS HIL(closed-loop)
Test drives on
prov. ground
Test drives in
real traffic
Homolo-
gation
Prototyping
AlgorithmADAS/Sensor ECU(s)
under Test
Target
implementation
Requirements
Specification
SIL(closed-loop)
Driving
simulator
SIL(closed-loop)
Machine
learning
dSPACE-Internal7
ADAS/AV development process
System
conceptMIL
(traffic simulation)
Component HIL
(closed-loop)
ADAS HIL(closed-loop)
Test drives on
prov. ground
Test drives in
real traffic
Homolo-
gation
Prototyping
AlgorithmADAS/Sensor ECU(s)
under Test
Component HIL
(open-loop)
Target
implementation
SIL (open-loop,
data playback)
Requirements
Specification
SIL(closed-loop)
Driving
simulator
SIL(open-loop)
SIL(closed-loop)
dSPACE-Internal8
ADAS/AV development process
System
conceptMIL
(traffic simulation)
Component HIL
(closed-loop)
ADAS HIL(closed-loop)
Test drives on
prov. ground
Test drives in
real traffic
Homolo-
gation
AlgorithmADAS/Sensor ECU(s)
under Test
Component HIL
(open-loop)
Requirements
Specification
SIL(closed-loop)
SIL(open-loop)
Cloud
Driving
simulator
Machine
learningPrototyping
Target
implementation
SIL (open-loop,
data playback)SIL
(closed-loop)
PC cluster
PC cluster
Cloud
Measurement data(Sensor and vehicle data)
Test scenarios/cases
Model-, parameter-,
scenario management
9
Challenges – Changing validation process
Data & test management
Models, parameters,
tests, test results, …
• Vehicle, driver, sensors
• Road, road networks
• Traffic, roadside structures
• Environmental conditions (weather, …)
• Driving maneuvers
• …
Automated test execution,
test evaluation and control
Defined scenarios
Stochastic parameter variation
Test control to detect critical
scenarios based on metrics and
evaluation criteria
Requirements-based
testing not sufficient
Virtual ECUs Real ECUs
PC cluster
Traffic scenarios
(critical, representative, …)
Scenario databases, …
Support of open file
formats and standards
dSPACE Solutions: A Powerful ADAS Toolchain
11
Test
Scenarios
Visualize?
Maneuver
Control
&
Experiment
Test
Management
Tests
Simulate
Models
12
Automotive Simulation Models (ASM) for ADAS and autonomous driving
Road networks Roads and intersections
Lane support
Artificial/real world roads
Road import
Roadside structures
MotionDeskAnimation
ModelDeskParameterization
Traffic Objects Static and dynamic objects
Vehicles, trucks,
pedestrians
Traffic signs, traffic lights,
parking vehicles, …
…Environment Sensors 2-D/3-D sensors
Camera, radar, lidar, …
Line, lane and traffic sign
recognition
Object list simulation
Vehicle simulation Vehicle Dynamics
Drivetrain
Soft-ECU network
Driver model
Maneuver
Environment
Vehicle
Dynamics
Traffic
Test Scenario Definition
• ASM Traffic Scenario definition
• ASM Simulated Traffic flow and Tool automation
• Independent definition of fellow scenarios
• Demo scenarios for standards (Euro NCAP)
• Open API for Automation
13
Road Import
OpenDRIVE, OpenCRG
Measurement data/GNSS
Here/ADAS RP
OSM, Google Earth
Scenario Import
Manually defined based
on expert knowledge
GIDAS database
OpenSCENARIO (planned)
ModelDesk API – Stochastic, etc.
Real sensors
and “ADAS” ECUs
14
dSPACE HIL simulation technology for ADAS and autonomous driving
Road, environment,
driving maneuvers
Virtual driver
Vehicle dynamics
Sensor
models
Automotive Simulation
Models (ASM)
Real-time
HIL simulator
Soft
ECUs
Brake/engine torque,
steering angle, …
Real sensors
and “ADAS” ECUs
15
dSPACE HIL simulation technology for ADAS and autonomous driving
Road, environment,
driving maneuvers
Virtual driver
Vehicle dynamics
Sensor
models
Real-time
HIL simulator
Soft
ECUs
Sensor system
Automotive Simulation
Models (ASM)
16
ASM Traffic and sensor models – HIL or VEOS
Integration of sensor-specific error
models in Simulink possible due to
open Automotive Simulation Models
(ASM)
3-D object sensor
2-D contour sensor
Attribute sensor
Traffic
environment
Idealized
sensor data
Error model
(Noise, false
positive/negative
detections, …)
Sensor data
with realistic
errors
Simulink®
ASMSensor or customer
specific implementation
Netw
ork
Man
ag
em
en
t
Raw data
Target list
Object list
Sensor ECUSensor
Frontend
Application logic(Trajectory planning,
Motion control)
Object tracking
Detection, Data Proc.
Preprocessing
Options for testing Sensor ECUs
Option 1 & 2
Technology independent approach
Provide ideal ground truth based
information
Part of ASM
Calculated on SCALEXIO CN (CPU)
Option 3 & 4
Physics-based approach
More related to the measurement
principle of a sensor
Calculated on GPU
Option 5
Test Bench with real Sensor ECU
Confidential, Information are subject to change without notice
Opt. 2: Insert Object list
Opt. 1: Restbus
Vehicle network1) Automotive Real-Time Radar Scene Generator
Opt. 3: Insert target list
Opt. 4: Insert raw data
OTA-DeviceOpt. 5: Simulation over-the-air (OTA)
Netw
ork
Man
ag
em
en
t
Raw data
Target list
Object list
Sensor ECUSensor
Frontend
Application logic(Trajectory planning,
Motion control)
Object tracking
Detection, Data Proc.
Preprocessing
Confidential, Information are subject to change without notice
Options for testing Sensor ECUs
OTA-DeviceOpt. 5: Simulation over-the-air (OTA)
Opt. 1: Restbus
Opt. 2: Insert Object list
Opt. 3: Insert target list
Opt. 4: Insert raw data
Vehicle network
Option Camera Radar Lidar Ultrasonic
1
2
3 Under
development n/a n/a
4 Under
development3D point cloud n/a
5 ARSG1) n/a
1) Automotive Radar Scene Generator
Raw Sensor Data Generation –
Environment Sensor Interface Unit
Powerful high-end FPGA Xilinx Zynq UltraScale+ (MPSoC ZU9)
Synchronous output for >8 sensors (Cameras/Radar1/Lidar1) with 1 ESI Unit2
Up to 15.9 Gbit/s aggregated data rate
FPGA model (partly) open for customers (VHDL, Verilog, XSG, and HLS)
Flexible adaption of video interfaces via plug-in modules (FMC)
Direct camera I/F (GMSL, FPD-Link III, Ethernet AVB1 & GigE Vision1)
Customer-specific interfaces
ESI Unit Plug-on Device (POD) for short range video interfaces
parallel, HiSPI, CSI2, LVDS, etc.
High dynamic range (HDR) support: Up to 20 bit1
Feedback channel to ECUs (e.g. field of view, exposure time, …) from ESI Unit
Both Open loop and closed loop testing with the ESI1Under Development
2Depending on customer setup
FMC modules
Confidential
Camera Image Sensor
Source: http://www.onsemi.com/
Source: https://en.wikipedia.org/
Register InterfaceI²C
Bayer Pattern (RGGB)
Lens ImagerImage
processing
ESI Unit: FPGA Firmware Overview
Confidential
Video
Input
Environment Sensor Interface Unit (Xilinx Zynq ZU9EG FPGA)
HDMI Input
Configuration & Control
Video Data
Bayer
PatternGain FIU Aurora
Aurora
Output
Camera ECU
Ethernet
Pixel Pipeline Control
ESI
POD
4x ARM Cortex A53
Linux
Pixel Pipeline
ESI Unit: FPGA Firmware Overview
Confidential
Video
Input
Environment Sensor Interface Unit (Xilinx Zynq ZU9EG FPGA)
HDMI Input
Configuration & Control
Video Data
Aurora
Camera ECU
Ethernet
ESI
POD
4x ARM Cortex A53
Linux
Pixel PipelineAurora
Output
ESI Unit: FPGA Firmware Overview
Confidential
Video
Input
Environment Sensor Interface Unit (Xilinx Zynq ZU9EG FPGA)
HDMI Input
Configuration & Control
Video Data
Aurora
Camera ECU
Ethernet
ESI
PODPixel Pipeline
Aurora
Output
Video
Splitter
4x ARM Cortex A53
Linux
Radar PipelineGMSL
Output
Pixel PipelineGMSL
OutputCamera ECU
Radar ECU
GMSL
GMSL
Radar Data
Raw Data Generation for Cameras –
MotionDesk and multiple sensor models (Camera, Radar, Lidar)
Rest bus simulation
ASM
ModelDesk
…
Vehicle network
Sensor composition
combining multiple sensor
outputs incl. meta-data
ESI Unit
HDMI
Ethernet
Rear camera
Front camera
Laserscanner
Control data
Confidential
PREVIEW
PC with graphics card
and MotionDesk
Raw Data Generation for Cameras –
MotionDesk and multiple sensor models (Camera, Radar, Lidar)
Rest bus simulation
ASM
ModelDesk
…
Vehicle network
Sensor composition
combining multiple sensor
outputs incl. meta-data
ESI Unit
HDMI
Ethernet
CN+GPU
SensorSim
Rear camera
Front camera
Laserscanner
Control data
Confidential
PREVIEW
Overview Option 3 & 4 – HIL - today
29
Pro
du
ctEC
U s
pec.
pro
ject
Ethernet / IOCNet
Environment
Sensor Simulation
HDMIEthernet
Environment Sensor
Interface Unit
(FPGA)
Confidential
Netw
ork
Man
ag
em
en
t
Raw data
Target list
Object list
Sensor ECUSensor
Frontend
Application logic(Trajectory planning,
Motion control)
Object tracking
Detection, Data Proc.
Preprocessing
Environment Sensor
Interface POD
(FPGA)
Automotive
Simulation Models
(ASM, SCALEXIO CN)
High-end PC and GPU (Windows)
MotionDesk Host
ESI Unit
PODs
FPGA Firmware
38
Sensor simulation – HIL and SIL
PREVIEW
Integrated toolchain for SIL
and HIL use cases
Many sensor technologies
are adressed
Five options to test a
sensor toolchain
Toolchain with low latency
and synchronization
Use of several GPUs is
planned
API to integrate custom
models is planned
AutomationDesk Testing and Test Tools
40
Various methods of Test Development
Signal-based testing, XML and xIL-API Open Standards
Manage all tools in the ADAS Testing Process
• fit for purpose for
developing safety related
software according to IEC
61508 and ISO 26262.
• pre-qualified for all ASILs
according to ISO 26262
Real-Time Testing
41
Standard PC
Real-time test programming via Python scripts (with use of specific RTT Libraries)
Real-Time Test management (download, start, stop, pause...)
Real-time platform or VEOS
Execution/Scheduling of Python real-time tests
Synchronization between real-time tests and
Simulink® model
Observers and the RTT Observer LibraryCoupling of BTC Embedded Specifier and dSPACE Systems
42
Requirements
(textual)
Test specification
(informal, structured)
Simulator (VEOS / HIL)
Requirements
(formalized)
EmbeddedSpecifier
RTT-Observer
Observer
• Passed/Failed
• Req-Coverage
• Textual requirements =>
Formalized requirements
=> Simulation based
formal verification
(~ISO26262)
• Permanent verification of
safety-critical
requirements with
observers, e.g. in parallel
to execution of ‚classic‘
AutomationDesk tests
Drastic increase in test
depth and coverage for
safety-critical functions
Many additional benefits
for test automation users
Observer
Download
Test Management for ADAS: Keeping Track of Testing Activities
Traceability and Coverage
Traceability from requirement to test result
and overall requirements coverage
For all types of requirements, e.g. safety,
functional, performance or robustness
requirements
Test Scenario Traceability
Which test scenario is tested by which tests?
Has a given test scenario been tested
successfully?
43
Monitor progress across multiple
test platforms and different
test tools
For tests by means of simulation
(MIL/SIL/HIL) as well as real-world
tests
Test reports, results overviews
and test evaluation
Test Stimuli Traceability
Which parameters and inputs
were used for which tests during
a test execution?
SYNECT Test Management
44
Manage all MIL/SIL/HIL Testing
1. Requirement integration and
coverage analysis - Connect to
PLM/ALM Tools
2. Off-the-shelf integration with
common test tools such as
AutomationDesk, Simulink, and
BTC EmbeddedTester®
3. Open interface to connect other
test tools or custom test solutions
4. Monitor, analyze, visualize test
results during the test process
5. Dynamic Test Parameterization –
handle ADAS model/test
functionality
45
Integrated Tool Chain for Testing by Means of Simulation
Euro NCAP AEB Use Case
• Automatically execute
Euro NCAP tests and
generate score results
• Automated
parameterization,
execution and evaluation
of Euro NCAP tests
• Example AutomationDesk
NCAP AEB Test Demo
available online
• Solutions for all NCAP
tests available as an
engineering service
Test Automation (TA) Framework
46
SYNECT Project Navigator
Test Management Project
Test Cases
Test Steps Manager
PC cluster simulation with Virtual ECUs
50
Driving millions of kilometers on your PC
Testing at an early development stage
Highly scalable due to virtual ECUs
Deterministic and reproducible test execution
High test throughput through fast test execution
Simulation Cluster open to
Integrate with test generation methods
Cloud computing options
Simulation Cluster leverages
SIL tool chain in general (VEOS, xIL-API, ASM)
SYNECT Test Management
Real Time Testing
Parallel Testing and Execution Management
53
Plan and schedule test cases and
assign to specific cores (efficiency of
multi-core testing)
Offload test analysis from Test
resources (test resource efficiency)
Future – auto-optimization and API
for schedule/sequence customization
54
Optimizing HIL Testing Time
EXECUTION
EVALUATION
Test Case - 1
2 mins
4 mins
Offline PC
2 mins HIL Time
Iteration
Test case 1
Test case 2
Test case 3
Test case 4
. . .
Execution
Iteration
Test case 1
Test case 2
Test case 3
Test case 4
. . .
Post-Processing
Offline PC
Captured
DataResults
.mat / .mf4 1 Passed
.mat / .mf4 2 Failed
.mat / .mf4 3 Passed
.mat / .mf4 4 Passed
. . . . . .
HIL PC
Master Data ManagerSYNECT
dSPACE-Internal57
dSPACE - The Right Partner
for Autonomous Driving
Simulation
Exhaustive
testing
Validation
Sensors Algorithms
Proto-
typing
Virtual
test drives
58
Thanks for listening!
© Copyright 2017, dSPACE Inc.All rights reserved. Written permission is required for reproduction of all or parts of this publication. The source must be stated in any such reproduction.This publication and the contents hereof are subject to change without notice. Benchmark results are based on a specific application. Results are generally not transferrable to other applications.Brand names or product names are trademarks or registered trademarks of their respective companies or organizations.