humandrive: the most complex autonomously controlled

#CAM2019 @LCV_Event

HumanDrive: The Most Complex

Autonomously Controlled Journey in the UKNick Blake

Chief Innovation Strategist and Head of Big Data Labs – Hitachi

Research & Development Europe

CAM Seminar Hall Sponsor

© Hitachi Europe, Ltd. 2019. All rights reserved.

Data-Driven Perception and Planning

Methodologies for Autonomous Vehicles

04/09/2019

Nick Blake, Ph.D

Head of European Big Data Lab

European Big Data Lab

London, United Kingdom

Syed Adnan Yusuf, Ph.D

Senior Research Scientist




Presentation Contents

3

1. Hitachi Autonomous Vehicle European R&D Activities

2. HumanDrive Project

3. Autonomous Driving Software Paradigms

4. HumanDrive Software Paradigm – Hitachi Approach

5. Key Lessons

© Hitachi Europe, Ltd. 2019. All rights reserved. 4

1. Hitachi Autonomous Vehicle European R&D Activities


Hitachi Autonomous Vehicle European R&D Activities C

olla

bo

rativ

e re

se

arc

hK

ey techno

logie

s

Connected Autonomous Mobility & Advanced

Telematics Solutions (ATS)

Build

ing b

locks

Vehicle Dynamics

Cognition and Judgement

Communication & Cooperativeness

Adv. Telematics

Autonomous Vehicle Control & Monitoring

Sensor Fusion, Deep Learning, AI

Connected Car,Cooperative AD Data Analysis, Machine

Learning, AI, Pred. Maintenance, Fleet Opt.

Experimental

Analysis

Mathematics and

Data Analysis

Simulation

Machine Learning

and AI

European Project

activitiesDriver Characterisation

TCU

Standardization

TCU: Telematics Control Unit

2 test/demo vehicles

Sensor Fusion


2. HumanDrive Project

▪ Scope

▪ Consortium

▪ Video


HumanDrive Project – Scope

7

2017 2018 2019

Project start

Dynamic Trials

Static Env Trials

Grand Drive

Projectcomplete

OVERTAKING OF CYCLISTS

ROUNDABOUT

NATURAL ROAD POSITION

Technical challenges for human like driving using machine learning

The

BIG

Ambition

200

Miles

100%

AD

Collaboration between global and UK CAV experts in industry, academia and Government agencies.

Establish autonomous vehicle R&D team in Nissan UK.

Hitachi create CAV team in UK, with unique AI capability.

World Class cyber security capability developed between Atkins and SBD.

Using test UK test facilities at Horiba MIRA and Cranfield Univ

Benefit and Impact

SMOOTH AUTONOMOUS VEHICLE CONTROL

https://humandrive.co.uk/

https://humandrive.co.uk/


HumanDrive Project – Consortium

8

Our consortium is made up of 10 members, all of whom have specific responsibilities

and areas of expertise:

Sponsored by the Innovate UK (the UK’s innovation agency) and CCAV office:


HumanDrive Project – Video

9


3. Autonomous Driving Software Paradigms


Autonomous Driving Software Paradigms

11

Mediated Perception Behaviour Reflex

• Software Reconfigurability and Debuggability

• Component Level Safety Assurance

• High Software Complexity

• Fixed Driving Behaviour

• Low Software Complexity

• Customisable Driving Behaviour

• Safety Assurance

• Debuggability

Perception Control

PlanningLocalization AI Agent

• Construct a direct mapping from sensory inputs

e.g. cameras, lidars etc. to a driving action e.g.

accel/brake or steer.

• Problem is decomposed into separate sub-

modules to solve the perception, localisation,

planning and control tasks.

Source: http://deepdriving.cs.princeton.edu/paper.pdf

http://deepdriving.cs.princeton.edu/paper.pdf


4. HumanDrive Software Paradigm – Hitachi Approach

▪ Perception System

▪ Planning System

▪ Integration & Testing


HumanDrive Software Paradigm – Hitachi Approach (1)

13

Machine Learning to Develop

Natural Human-Like Control

Software Safety Assurance

Software Reconfigurability,

Debuggability, Maintainability

Cross-Platform Software

Integration

Challenges to Solve

1

2

3

4

Proposed Solution

• Combine the benefits of Mediated and

Reflex paradigms to enable natural

human-like control whilst providing the

required safety assurance.

• Perception and Planning are the two

key sub-components responsible for

“Cognition” and “Decision-Making”.

AI Perception Control

AI PlanningLocalization


HumanDrive Software Paradigm – Hitachi Approach (2)

14

Safety

Control

Software Diagnostics

➢ Numerical/Computational Issues ➢ Process Performance ➢ HW Health Status ➢ Log Info

Planning

Perception

Camera Based

Lidar Based

Camera + Lidar Based

➢ Occupied Area

➢ Detect/Track/Predict

➢ Detect/Track/Predict

➢ Drivable RoadLid

ars

+ C

am

era

sG

PS

+ O

do

m

Localization

➢ Manipulate actuators

to follow trajectory

➢ Localise on the map

➢ Determine desired

(coarse) route

Ego-carDesired route

Human-Like path

➢ Check if AI-based

path is within a set

of predefined

boundaries. If not,

then raise a flag to

indicate unsafe

operation.

Ego-car

Drivable-road

Desired route

Human-Like paths

Min/Max Limiter Rate Limiter

➢ Data-driven imitation

of human driving

behaviour

➢ Recurrent

Convolutional Neural

Network(s)

➢ Training using on pre-

recorder/ selected

human driving data


HumanDrive Software Paradigm – Perception System (1)

15

System Requirements

Occupancy Grid Drivable AreaDetection, Tracking,

Prediction

Real-Time

Execution

Top-Level Perception System Architecture

Sensing Area

Camera

LiDAR

Machine/Deep Learning Layer Engineering Layer Output Layer

Camera-Based

• CNN for

object

detection and

classification

Lidar-Based

• CNN for

object

detection and

classification

Lidar/Camera-

Based

• FCN for

semantic

segmentation

Fusion

Filtering/Tracking

PredictionPast states

Predicted states

PredictedMeasured

Optimal

• Reduce false positives

• Increase robustness

• Future predictions

• Data visualisation

Planning…

GridVector

• Data representation


HumanDrive Software Paradigm – Perception System (2)

16

Open source Hitachi labelling tool available on GitHub:

https://github.com/Hitachi-Automotive-And-Industry-Lab/semantic-segmentation-editor

Iterative Performance Improvement

Machine Learning Factory Off-Line & On-Line System Performance Evaluation

Data Engineering*

Training & Validation Pipeline

Occupancy Grid

Lidar Based Detection

Camera Based Detection

Camera View

BEV View

Truck 2

Car 2

Car 1Truck 1

https://github.com/Hitachi-Automotive-And-Industry-Lab/semantic-segmentation-editor


HumanDrive Software Paradigm – Planning System (1)

17

System Requirements

Human-Like Data-Driven Safety AssuranceReal-Time

Execution

Top-Level Planning System Architecture*

*Patent Pending

Planning Network (PlanNet)

Trajectory Generator

Occupancy Grid

Desired Route

Input Layer

Perception Sequence

t

t-1

t-n⋱

𝑥𝑦

𝜃

ሶ𝑥

ሶ𝜃

𝑡 + Δ𝑡 =

𝑥𝑦𝜃ሶ𝑥ሶ𝜃

(𝑡) +

ሶ𝑥

ሶ𝜃sin ሶ𝜃Δ𝑡 + 𝜃 −

ሶ𝑥

ሶ𝜃sin 𝜃

−ሶ𝑥

ሶ𝜃cos ሶ𝜃Δ𝑡 + 𝜃 +

ሶ𝑥

ሶ𝜃cos 𝜃

ሶ𝜃Δ𝑡00

Trajectory Generator

Planning Network (PlanNet)

RCNN LSTM

CO

NC

AT

INA

TE

FC

LSTM

FC

Vehicle Centric

Yaw Rate

Sequence

Speed

Sequence

Example

Network

Architecture

• Generate trajectory based on future desired

yaw rates and speeds

• Constant Turn Rate Vehicle (CTRV) model

can be used

Output Trajectory


HumanDrive Software Paradigm – Planning System (2)

18

Planning Network

Machine Learning Factory

Key Enabler: Driving Behaviour Analysis Toolchain

Imitation Learning: Important Consideration

Bias Variance Non-Smooth Driving


HumanDrive Software Paradigm – Integration and Testing

19

• Containerised (Docker) development

environment allows seamless integration

on multiple vehicles

• Robot Operating System (ROS) is used

as the main meta-operating system

System Integration System Testing


5. Key Lessons


Key Lessons

21

Data quality and quantity is key enabler for creating natural human-like vehicle control

Data science and engineering is a necessary step prior to any machine learning activity

Machine learning do not replace rigorous engineering – it is an enabler rather than a disruptor

Natural human-like vehicle control is not the same for every passenger

Data-driven engineering is essential to unlock personalised autonomous vehicles


Data-Driven Perception and Planning

Methodologies for Autonomous Vehicles

04/09/2019

Nick Blake, Ph.D

Head of European Big Data Lab



Syed Adnan Yusuf, Ph.D

Senior Research Scientist



humandrive: the most complex autonomously controlled

Documents