Leveraging AI for Self-Driving Cars at GM

Efrat Rosenman, Ph.D.

Head of Cognitive Driving Group

General Motors – Advanced Technical Center, Israel

Agenda

• The vision

• From ADAS (Advance Driving Assistance Systems) to AV (Autonomous Vehicles)

• AI for Self-Driving cars

• ADAS, AV and in-between

• Summary

10/19/2017 General Motors 2

The Vision

3

?

• Mobility – one of the most significant revolutions of modern times

• Self-driving cars will take mobility to a completely new phase…

”Zero Crashes, Zero Emissions, Zero Congestion” (Mary Barra, GM CEO)

10/19/2017 General Motors

The Vision

4

Increase Mobility: anywhere, anytime Increase Car Sharing & Reduce Road Capacity and Parking needs

10/19/2017 General Motors

Increase Safety Increase Productivity

5

From ADAS to AV

10/19/2017 General Motors

L5:Full

automation

Level 4: High automation

Level 3: Conditional automation

Level 2: Partial automation

Level 1: Driver assistance

Level 0: Driver in full control Info, warnings

Cruise control, lane position

Traffic jam assist

Anywhere, anytime

Fully autonomous specific scenarios

Highway driving (driver takes control with notice)

6

From ADAS to AV

• Will incremental steps get us to the top of this pyramid?

10/19/2017 General Motors

10/19/2017 General Motors 7

Sensing Mapping PerceptionDecision Making

Control

Components of self driving cars

Components of self driving cars

AI AGENT serves as the “brain” of the car

10/19/2017 General Motors 8

PerceptionDecision Making

Control

AI for Self-Driving Cars

9

AI in Perception

• Unsupervised learning

• Finding structure in point clouds

• Feature learning

• Supervised learning

• Object detection

• 2D object recognition (Classification)

• 3D scene understanding and modeling (3D objects pose)

• Semantic segmentation (boundaries of objects, free space)

10/19/2017 General Motors 11

AI in Perception - E2E trend

• Classification:

• Scene understanding:

• Perception:

10/19/2017 General Motors 12

Pixels Key Points ModelSIFT features Labels

Sensors 2D object detection

Pose estimationDepth estimation

3D World state

Pixels Segmentation Contextual relations

Object detection

Scene description

AI in Perception - E2E trend

• Classification:

• Scene understanding:

• Perception:

10/19/2017 General Motors 13

Pixels Key Points ModelSIFT features Labels

Sensors 2D object detection

Pose estimationDepth estimation

3D World state

Pixels Segmentation Contextual relations

Object detection

Scene description

DNN

DNN

DNN

Towards E2E: Sensors Fusion

10/19/2017 General Motors 14

• All sensors contribute

• Enables learning of complex dependencies “optimally”

• Sparse Vs. dense sensors

• Larger models, harder to learn

• Utilizes domain knowledge

• Model is explainable

• Based on tailored rules

• Suboptimal performance

Low Level: raw data combined in input stage

High Level: tailored hierarchy between sensors

Towards E2E: Multi-Task Learning

• Most our outputs are inter related• Objects, free space, lanes, etc.

• Cross regularization allows reaching a better local minima

• TPT• Major parts of the Deep Net are used for multiple tasks

• Data Efficiency

10/19/2017 General Motors 15

Mask R-CNN Facebook AI Research (FAIR); Apr 2017

What about data?

16

Automatic Data Annotation

• Data is the key contributor to perception accuracy – With no visible saturation

• How can we create annotated data• Manual annotation – Expensive and inaccurate

• Automatically

10/19/2017 General Motors 17

Revisiting Unreasonable Effectiveness of Data in Deep Learning Era, Google 2017

Automatic Data Annotation

• Technology• High end sensors (Lidar, IMU, etc.)

• High accuracy detectors (on behalf of computation time)

10/19/2017 General Motors 18

Example – AGT for StixelNet

• StixelNet - Monocular obstacle detection• Based on stixel representation

• Identify road free space

• Ground truthing is based on Lidar

10/19/2017 General Motors 19

Dan Levi, Noa Garnett, Ethan Fetaya. StixelNet : A Deep Convolutional Network

for Obstacle Detection and Road Segmentation. In BMVC 2015.Lidar (Velodyne HDL32) is used to identify obstacle on each stixel in the image

[Badino, Franke, Pfeiffer 2009]

Compact, local representation

Is Perception “solved”?

• Challenge of Cost• Sensors

• Mapping

• Computation

• Challenge of false positive & false negative• Data uncertainty (noise)

• Model uncertainty (confidence)

10/19/2017 General Motors 20

Label: Cyclist RGB: Pedestrian (0.56)

Decision Making

10/19/2017 General Motors 21

PerceptionDecision Making

Control

Learning Decision Making

Decision Making cannot learn from static examples

Need interactive domain

- > Reinforcement Learning (RL)

RL has seen some major successes in the recent years:

10/19/2017 General Motors 22

Go[Google deepmind] source: uk business insider

Poker [Bowling et al] source: wikipedia

Autonomous Helicopter Flight [Ng et al] source: ai.stanford.edu

Atari[Google Deepmind] source: nbcnews

RL challenges in Self-Driving agents

• Learn to act in a very high dimensional space

• Plan sequences of driving actions

• Predict long term behaviors of other road users

• Few sec

• Complicated situations

• Negotiate with other road user

• Guarantee safety

10/19/2017 General Motors 23

Simulation

• Advanced simulations are required• Multi-agent

• Various conditions

• Focus on “interesting miles”

• Drive billions of “virtual miles” (fuzzing)

“Any system that works for self driving cars will be a combination of more than 99 percent simulation.. plus some on-road testing.” [Huei Peng director of Mcity, the University of Michigan’s autonomous- and connected- vehicle lab]

10/19/2017 General Motors 24

Waymo simulation:https://www.engadget.com/2017/09/11/waymo-self-driving-car-simulator-intersection/

Safety Guarantees - From ADAS to AV

Will incremental steps get us to the top of this pyramid?

The technological heart is different in kind

10/19/2017

What’s the difference?

• For ADAS – Safety guarantee is based on the driver

• For autonomous – Safety guarantee should come from the system itself

10/19/2017 General Motors 26

Example: Highway Driving in Super Cruise™

10/19/2017 General Motors 27

The 2018 Cadillac CT6 will feature Super Cruise™ - a hands-free driving technology for the highway

It includes an Exclusive driver attention system to support safe operation

Safe Driving for level 4/5

• System should handle 100% of the cases

• Redundancy requires at all levels• Sensing• Algorithm• Computing• Control• Fallback strategies

• Guarantee of Safety is a must to the acceptance of AV• Statistical data-driven approach [miles-per-interrupts] requires driving billions of

miles to validate an agent• Should be repeated with every SW version

• Need safety constrains (rule-based/model-based)

10/19/2017 General Motors 28

Summary

• Advances in AI are key to success of self-driving cars

• AI-based features can bring ADAS to a new level in terms of accidence avoidance, productivity gain and saving in human lives

• Level 4/5 AV should be a parallel effort focus on redundancy and safety constrains

10/19/2017 General Motors 29

GM Advanced Technical Center in Israel (ATCI)

Thank you

10/19/2017 General Motors 31