connirotm...embedded system accurate performance evaluation and validation via in-vehicle ground...
TRANSCRIPT
ConniroTM Solution Brief
ConniroTM
Connected solutions with enhanced intelligence, personalization, integration and connectivity
2 | January 2020
ConniroTM Solution Brief
As the mobility environment evolves, the ways in which in-vehicle
user experiences are provided are also changing. Whereas drivers,
traditionally, had to be aware of and take control over the entire
environment inside the vehicle, in the future, an intelligent assistant
in a car will offer a variety of customized services in an integrated
manner, taking driver and passenger preferences into account.
Overview
LG Connected Solutions - ConniroTM
ConniroTM enables carmakers to provide a superior driving experience that is safe and convenient as well
as fun through a variety of interactions between vehicles and users, so that they can prepare for the era
of autonomous driving and innovation of mobility services based on connectivity technology.
ConniroTM provides a differentiated driving experience for the driver and passengers with integrated
solutions specially designed for the delivery of intelligent, personalized, integrated, and connected
experiences to respond flexibly to changes in the mobility environment.
Current Challenges Facing the Automotive Evolution
Technological advances in automotive industry introduce many challenges for carmakers:
● As autonomous driving has become the mainstream throughout all automotive sectors, the burden of driving
is reducing, and the demands on in-vehicle infotainment are increasing.
● Drivers and passengers seek to have more personalized experience from autonomous driving. Vehicle intelligence
is an essential means to power personalized and predictive services.
● UX/UI providing intuitive and natural interactions using multimodal input will play an important role in delivering
contextual safety and more convenient mobility experiences.
● An increased number of sensors in vehicles enables the collection of data within the vehicle relating to the
driver and passengers. How to turn the data into valuable functions and services optimized to the users will
be one key differentiating factor.
● Customer demand for better connectivity and high-quality connected services is increasing rapidly. Delivering new
types of connected services and seamless experience based on the connected services will be more important
than ever.
2 | January 2020
ConniroTM Solution Brief
January 2020 | 3
ConniroTM Solution Brief
ConniroTM provides a wide range of market leading connected
solutions for diverse requirements. It has already been
successfully applied and commercialized in many global car
models and the enhanced technical perfection allows carmakers
to reduce HMI development time and maintenance costs.
All Solutions of Conniro
TM
Driver Monitoring System for
driving safety by monitoring the driver’s
condition
Interior Monitoring System for
convenience and personalization by
monitoring a vehicle interior
Augmented Reality for intuitive
navigation and driver safety alerts
by augmenting the real world
Virtual Personal Assistants
for user interactions by combining
various HMIs and intelligent services
January 2020 | 3
ConniroTM Solution Brief
4 | January 2020
ConniroTM Solution Brief
ConniroTM AR (Augmented Reality) provides an intuitive user experience for navigation and driver safety alerts
by augmenting the real world with computer-generated visual information. The solution works with a variety of
vehicle sensors, including ADAS, front-facing cameras, and GPS. Driving-related information and safety warnings
are then displayed in the driver’s line of sight as augmented (3D) and non-augmented (2D) images.
● Support for various display such as HUD, AVN/CID, and mobile
● AR core engine with support for vehicle dynamic compensation, real-time eye position matching and time
warping by Unscented Kalman filter
● Sensor fusion which merges information received from in-vehicle sensors and predicts the future state of all
objects around the car to compensate message latency
● Advanced HMI including 3D carpet navigation guidance, ADAS object, and lane highlight, turn-by-turn overlay, pin hovering
for the destination, and POI highlight
● SDK with support for multiple HMI frameworks (CGI Studio, Kanzi, QT & Adobe After Effects), AR engine emulator, and
data simulator
Highlights Benefits
Key Features
● Applies the industry’s top-performing AR technology
● Provides intuitive navigation guide and Point of Interest
(POI) highlight
● Ensures active safety with various sensors
● Enables extended user experience with advanced POI
service integration
● Maximum flexibility and reusability of AR design
● Minimum use of CPU and memory resources
● Best performance through various optimization experiences
● Easy-to-update additional UX features depending on the
availability of extra vehicle sensors
Input/Output of AR Engine Use Cases
Navigation Guidance
Building POI Highlight
ADAS Warning
AR Parking Service
AR Engine
AVN/CID
ADAS Information
VehicleInformation
Sensor Data
NavigationInformation
HD MapInformation
HUD Mobile
Augmented Reality
January 2020 | 5
ConniroTM Solution Brief
ConniroTM DMS (Driver Monitoring System) identifies drivers and monitors the driver’s condition. The solution ensures
driving safety and maximizes convenience and personalization of the driving experience.
● Tracking driver’s head gesture, eye-gaze, and eyelid
● Detecting if the driver is distracted, drowsy, asleep, or speaking
● Anti-spoofing for face identification and autopilot mode
● Detecting blockages in the field-of-view of a camera
● Grid detection for camera calibration
Key Features
Highlights Benefits
● Driver state monitoring by tracking the driver’s facial
landmarks and eye-gaze for driving safety
● Driver identification and anti-spoofing achieved by deep
neural network-based features
● Ground Truth System that evaluates the performance on
various camera position and driving condition
● Lightweight tracking algorithm and applications on
embedded system
● Accurate performance evaluation and validation via in-vehicle
Ground Truth System
● Experienced experts support along the whole development
process - from algorithm adaptation and software
integration on a target device to application development
Personalization /
Convenience
- Anti-spoofing- Driver Identification- Area of Interest- Head Gesture
Safety
- Distraction Detection- Drowsiness Detection- Eyes on the Road- Yawn and Speaking
Detection
Camera Status
- Camera Blockage- Out-of-focus- Steering Wheel
Detection
Applications
Service Interface
Function Manager / Scheduler
Service
Engine
Ground Truth SystemCore Algorithms
Head TrackingHardware
- Head / Gaze tracking GT sys.
Software- Evaluation sys.- Labeling tool
Eyelid/Gaze Tracking
Input DMS Offering
IR Camera Configuration (Features, Camera)RGB Camera
DMS Software Architecture
Driver Monitoring System
6 | January 2020
ConniroTM Solution Brief
ConniroTM IMS (Interior Monitoring System) monitors a vehicle interior in real-time. The solution determines the
situation inside the vehicle and provides the driver and passengers with services for safety, convenience, and
personalization.
Interior Monitoring System
● AI-based detection and decision algorithm embedding
- To detect, track and classify the occupants and objects in the cabin
- To decide child presence, left-over items and occupant status (behavior)
● Flexible configuration
- IMS stand-alone (AI algorithm is embedded)
- Client-server (Al algorithm runs on a server)
- Head unit consolidation
Key Features
Highlights Benefits
● Object detection and occupant behavior analysis for
convenience services
● Occupant identification and analyzing their behavior
patterns for personalized services
● Flexible and highly scalable solution for various needs - range
from stand-alone installations to head unit consolidation
● Support for a video conference call in and out the car with
RGB-IR sensor
IMS Algorithm Flow
Detection Decision Control
Lost Items
Boarding Status
Face ID / Emotion
Child Presence
Body Pose Decision(Sleeping, lean-back, etc.)
Behavior Analysis
Personalization
Driver Safety
Sleeping Status
Gesture
Video conferencing
Object Tracking /Classification
Human Tracking /Classification
Head / Hand Tracking(Point of Interest) Audio / Video Haptic
Feedback
Vehicle Components(Airbag, AVN, …)
ConniroTM IMS is compatible with 2MP RGB-IR camera with Vertical Cavity Surface Emitting Laser (VCSEL)• RGB and IR Images output simultaneously with wide-angle (130 degrees)• About 70% less power consumption than legacy LEDs
January 2020 | 7
ConniroTM Solution Brief
ConniroTM VPA (Virtual Personal Assistants) helps the customer achieve maximum comfort and enjoy all connected
services and functions of vehicle by natural and intelligent interaction, providing valuable personalized services.
● Intelligent interaction to complete a task based on more clear user intent by interpreting various multimodal
inputs
● Connection to appropriate assistant based on user intent for consistent voice service experience
● Proactive and reactive suggestions for achieving maximum comfort and enjoying all the connected services and
vehicle functions
● Cloud assistant platform for seamless connected service experience (Microsoft)
● SDK package with prebuilt voice service & multimodal device adaptor on Linux with QT, ready-to-run sample
applications, documentation for HMI design
Virtual Personal Assistants
Key Features
Highlights Benefits
● Multi-modal interaction with the seamless and natural
combination of interactions such as GUI, touch, gestures,
and eye gazing based on voice
● Multi-assistants with a combination of different AI assistant
services
● Multi-Intelligence with proactive and reactive suggestions
provided for tailored information and entertainment
services
● White-labeled personal assistant with new multi-modal HMI
- Seamless integration into legacy systems
- Flexibility and scalability for extending new UX/UI
● Valuable personalized connected services
- Intelligence engine for data monetization
- Service arbitrator for providing valuable services
External assistants Connected services
OEM GUI Framework / Application
Dialog Design (OEM, 3rd party, …)
Multi-modal interaction Multi-assistants Multi-intelligence
Application Framework Adaptor
System Layer
VPA Software Architecture
VPAOffering
Intent interpretation Domain classification Learning / recommendation
Dialog management OEM/External assistant Rule engine
Hybrid context management Behavioral log management
Cloud Assistant Platform
chatbot
Touch Voice Eye gazing Gesture Car Data
QNA MAKER
LG Electronics, Inc. is a global leader and technology innovator in home
appliances, consumer electronics, mobile communications and vehicle
components. LG Electronics has been providing total solutions for vehicles based
on the experiences in delivering the best electronic devices to customers. From
next-generation vehicle parts such as OLED, wireless charger and battery for
electric cars to convergence systems like infotainment, telematics and digital
cluster, LG developed a global approach for an integrated customer experience.
Contact Us
For further information on ConniroTM,
please contact
Copyright ⓒ 2020, LG Electronics. All rights reserved. No express or implied warranties are provided for herein. All specifications are subject to change and any expected
future products, features or functionality will be provided on an if and when available basis. The product names and logos mentioned in this document are trademarks
or registered trademarks of their respective owners.
Please visit the LG Software Solutions website swsolutions.lge.com and
get more information about ConniroTM. The website offers useful resources
you need to get started with ConniroTM, along with other solutions for the
automotive industry.
Learn More