23270: augmented reality for navigation and … · 23270: augmented reality for navigation and...
TRANSCRIPT
23270: AUGMENTED REALITY FOR NAVIGATION AND INFORMATIONAL ADAS
Sergii Bykov
Technical Lead Machine Learning
12 Oct 2017
Product Vision
Company Introduction
Apostera GmbH with headquarter in Munich, was established in March, 2017 together with 3 affiliated R&D centers to leverage 10+ years
engineering experience in complex software development for Automotive Industry.
Apostera GmbH engineering and business experience in Driver Experience, Navigation and Telecommunication domains together with unique
IP and mathematical talent guarantees creation of advanced product portfolio to bring mobility world to new era of autonomy.
Apostera GmbH today’s target is to reshape areas of Automotive Perception, Visualization, Path Planning, V2X and finally Autonomous Driving
in open and collaborative manner.
Perception: Advanced Surround View Monitoring, Software Smart Camera and Sensor Fusion
Visualization: Software Augmented Guidance (HUD and LCD)
Quality: A (AR, ADAS, AD) Testing Automated System Mobility: Software Managed Autonomous Driving
APOSTERA product lines - Basics
IA AAC
HAD
Informational ADAS
Active ADAS components
Highly AutomatedDriving
ADAS Platform
Representation For The Driver
Outdated
LCD screen
Smart Glasses
HUD in car
Real-depth HUD with wide FOV in car
Past
Alternative, fast developing market (today) On going development +2 years
Key Challenges For In-Vehicle AR
Usability – augmented reality subsystem should not disturb driver as it is continuously observed
Hardware limitations – computational, power consumption, zero latency (HUD)
Requirements for precise environmental model estimation for occlusion avoiding
Dependency on inaccurate map and navigation data
Distributed HW architectures, platform flexibility requirements
High precision absolute and relative positioning requirements
Components synchronization and latency avoidance
Embedded memory usage limitations, different memory models
Algorithms should be both configurable and efficient
Specific rendering requirements, not covered by general purpose frameworks
Variety of inputs under different platforms
Out-of-vehicle simulation (does not support natural simulation like classical navigation)
System Concept
Unique Automotive Augmented Reality Solution
Solution capable to create Augmented, mixed visual Reality for drivers and passengers based on Computer Vision, vehicle sensor, map data, V2X, navigation guidance using Data Fusion.
Automotive Cameras
Sensors/CAN
Navigation System/Map Data
Vehicle displaysProjection on wind shield - HUD
Telematics/V2XADAS
PlatformStep I
Integration of V2X information Motorbikes helmets
Path Planning and AR 360
ADASPlatform
In progress
ADASPlatform
Further Steps
Recognition and Tracking
• Road boundaries and lane detection
• Slopes estimation
• Vehicle recognition and tracking
• Distance & time to collision estimation
• Pedestrian detection and tracking
• Facade recognition and texture extraction
• Road signs recognition
Positioning
• Precise relative and absolute positioning
• Flexible data fusion and smooth Map Matching
• Automotive constrained SLAM
• Video-based digital gyroscope
Predictable Environmental Model, Safety Apps - V2X
• BSM transmitting/receiving
• Remote Vehicles trajectory prediction
• Basic safety applications based on collision detection
Integration with HD Maps
HD Maps utilization for Precise positioning, Map matching and Path planning, Junction assistance
Data generation for HD Maps
Contribution to ADAS attributes structure – NDS (HERE)
Augmented Reality
• LCD, HUD & further output devices
• Natural navigation hints & infographics
• Collison, Lane departure, Blind spots warnings, etc.
• POIs and supportive information (facades and parking slots highlighting, etc.)
Computer Vision Approaches
• Real-time feature extraction from video sensors
• Road scene semantic segmentation
• Adaptability and confidence estimation of output data
• GPU optimization for different platforms
Sensor Fusion
• Flexible fusion of data from internal and external sources
• LIDAR data merging
• 3D-environment model reconstruction based on different sensors
• Latency compensation & data extrapolation
Machine Learning Specifics
• CNN and DNN approaches
• Supervised MRF parameters adjustment
• CSP-based structure & parameters adjustment (both supervised and unsupervised)
• Weak classifiers boosting & others
Scientific and Engineering Expertise
System Overview
Live data from vehicle:- CAN data, Sensors- Video stream
ECU (e.g. Jetson TX2)
ADAS EngineSensor Abstraction LayerWeb InterfaceSW UpdateConfigurationDiagnostic
Video Stream withaugmented objects
ADAS/AR Engine
HUD/LCD
Head Unit
• Quick-install demonstration solution
• Platform for AR (allows to be portable)
• Integration with Head Units
• Integration with vehicle networks
• Using of own sensors if needed
Navigation data, preprocessedsensor data, etc.
Control/Settings
Perception Concept
Sensor Fusion: Data Inference
Optimal fusion filter parameters adjustment problem statement and solution developed to fit different car models
with different chassis geometries and steering wheel models/parameters.
Features:
Absolute and relative positioning
Dead reckoning
Fusion with available automotive grade sensors – GPS, steering wheel, steering wheel rate, wheels sensors
Fusion with navigation data
Rear movements support
Complex steering wheel models identification. Ability to integrate with provided models
GPS errors correction
Stability and robustness against complex conditions – tunnels, urban canyons
Sensor Fusion: Advanced Augmented Objects Positioning
Solving map accuracy problems
Placing:
•Road model
•Vehicles detection
•Map data
Position clarification:
• Camera motion model:•Video-based gyroscope
•Positioner Component
• Road model
• Objects tracking
Sensor Fusion: Comparing Solutions
Update frequency ~15 Hz (+extrapolation with any fps) Update frequency ~4-5 Hz
Apostera solution Reference solution
Lane Detection: Adaptability and Confidence
Lane Detection: 3D-scene Recognition Pipeline
Low level invariant features
• Single camera
• Stereo data
• Point clouds
Structural analysis
Probabilistic models
• Real-world features
• Physical objects
• 3D scene reconstruction
• Road situation
3D space scene fusion (different sensors input)
Backward knowledge propagation from high levels
Vehicle Detection
Convolutional neural network for vehicle detection
GPU Acceleration – CUDA
Running real-time on NVidia Jetson TX2
Inference speedup on embedded (TX2) GPU vs CPU is ~3x
More potential with new libraries (e.g. TensorRT)
Training speedup on desktop GPU vs CPU is ~20x
Classifier accuracy (about 50k, 960x540, ~55-60 deg
HFOV)
• Positive: 99.65%
• Negative: 99.82%
Size of detection down to 30 pix, detection range of
about 60 m
Figure – Vehicle detection examples
Road Scene Semantic Segmentation
Deep fully convolutional neural network for
semantic pixel-wise segmentation
Road scene understanding use cases: model
appearance, shape, spatial-relationship between
classes
Inference speedup GPU vs CPU is ~3x
Figure – Road scene segmentation examples
HMI Concept
Rendering Component Structure
Figure – Rendering component
Augmented Objects Primitives
Barrier Lane Line
Lane Arrow FishboneStreet Name
Augmented Objects Primitives And HMI
Head Up Display Concept. HUD vs LCD
Hardware limitation
• HUD devices are rarely
available on market
• FOV and object size
Timings
• Zero latency
• Driver eye position
Driver perception
• Virtual image distance
• Information balance
HUD Image Correction (Dewarping)
Figure – Corrected imageFigure – Uncorrected image
Need to correct slight distortion in the HUD image
A custom warp map was made by taking an image of a test pattern that was projected by the HUD and recorded
by a camera
Demo Application (LCD)
Summary: Key Technology Advantages
Proved understanding of pragmatic intersection and synergy between fundamental theoretical results and final
requirements
Formal mathematical approaches are complemented by deep learning
Solid GPU optimization
Automotive grade solutions integrated with all the data sources in vehicle – data fusion approaches
High robustness in various weather and road conditions, confidence is estimated for efficient fusion
Closed loops designed and implemented to enhance speed and robustness of each component
Integration with V2X and various navigation systems
System architecture supports distributed HW setup and integration with existing in-vehicle components if
required (environmental model, objects detection, navigation, positioner etc.)
Hierarchical Algorithmic Framework design highly optimizes computations on embedded platforms
Collaboration with scientific groups to integrate cutting edge approaches
Sergii Bykov
Technical Lead Machine Learning
BRINGING MOBILITY WORLD TO
NEW ERA OF AUTONOMY