looking-in vehicle: selected investigations...

Looking-In and Looking-Out of a Vehicle: Selected Investigations inComputer Vision based Enhanced Vehicle Safety

Mohan M. Trivedi, Tarak Gandhi, and Joel McCallComputer Vision and Robotics Research Laboratory

University of California San Diego, La Jolla, CA, USA{mtrivedi,tgandhi,jmccall}ucsd.edu

Abstract- This paper presents an overview of investigationsinto the role of computer vision technology in developing saferautomobiles. We consider vision systems which can not only lookout of the vehicle to detect and track roads, avoid hittingobstacles or pedestrian, but simultaneously look inside thevehicle to monitor the attentiveness of the driver and evenpredict her intentions. In this paper, a systems-orientedframework for developing computer vision technology for saferautomobiles is presented. We will consider three maincomponents of the system, driver, vehicle, and vehicle surround.We will discuss various issues and ideas for developing modelsfor these main components as well as activities associated withthe complex task of safe driving. The paper includes discussion ofnovel sensory systems and algorithms for capturing not only thedynamic surround information of the vehicle but also the state,intent and activity patterns of drivers.

Index Terms- Intelligent vehicles, Driver support systems,Real-time machine vision systems, Active safety.

I. INTRODUCTION: RESEARCH MOTIVATION AND SCOPETRAFFIC related accidents are recognized as a serious

and growing problem with global dimensions.According to a recent study by World Health Organization(WHO) mentions that annually over 1.2 million fatalities andover 20 million serious injuries occur in the world [1].Enhancement of traffic safety is pursued with as a highpriority item not only by various government agencies such asNational Transportation Safety Administration (NTHSA) [2],but also by most major automobile manufactures. Universitybased researchers are also contributing to this importantmission. In this paper, we present an overview of a selectedresearch studies conducted in our research laboratory, wherenovel concepts and systems based upon computer visiontechnology are developed for enhancement of vehicle safety.

Recognition of computer vision as a critical technology forintelligent vehicles, can be traced to the earlier efforts dealingwith autonomous mobile robots and autonomous driving [3][4j[5] Such efforts helped to demonstrate the power of camerabased systems to support real-time control of vehicles. Theseand other earlier studies were not focused on the enhancementof automobile safety as the primary objective. However, a new

trend started emerging in the late nineties, where research incomputer vision got focused on enhancement of safety ofautomobiles [6][7][8][9][10][11]. Already, camera basedmodules with safety oriented features such as "back-up" (orreverse) viewing, lane departure warning, and blind spotdetection are offered in commercial vehicles.The research pursued in our Laboratory for Intelligent and

Safe Automobiles (LISA), considers development of visionbased systems with a wide range of application possibilities,including those for occupant safety, pedestrian safety, driverassistance, driver workload and "attention" monitoring, lanekeeping, and dynamic panoramic surround capture. Basically,these investigations have considered three types of viewingperspectives for the cameras.(1) Looking in the vehicle: to capture the important visual

context associated with the driver, occupant, and theiractivities and physical and mental state monitoring.

(2) Looking out ofthe vehicle: to capture the visual context ofthe vehicle, including that of the surrounding roadconditions and traffic, and

(3) Simultaneous Looking in and Looking out of the vehicle:to correlate the visual contextual information of vehicleinterior and vehicle exterior, driver's behavior and intentcan be systematically investigated which can lead toderivation of useful feedback mechanisms for managingdriver distraction.

A sensor system that is capable of maintaining dynamicrepresentations of the external world surrounding the vehicle,the state of the vehicle itself, and of the driver. Dynamiccontext capture for the Human-Centered Intelligent DrivingSupport System (HC-IDSS) requires analysis of multimodalsensory information and their fusion at multiple levels otabstraction. To develop a robust dynamic context capturesystem, computer vision and machine learning techniques playan important role. In our research we have purseddevelopment and evaluation of an active, multi-modal sensoryapproach for "dynamic context capture and situationalawareness" using cameras, radars, audio, etc for establishingrepresentations of the state of the environment, the vehicle,and the driver with accurate dynamic uncertaintycharacterization.

Manuscript received October 2, 2005. This work was supported in part bytheUC Discovery Grant's Digital Media Innovations Program, Nissan MotorCorporation, and the Volkswagen Corporation.

0-7803-9435-61051$20.00 ©2005 IEEE.29

Authorized licensed use limited to: Univ of Calif San Diego. Downloaded on August 06,2010 at 23:06:22 UTC from IEEE Xplore. Restrictions apply.

Figure I Framework for Multifunctional, "active" computer vision baseddynamic context capture system.

For dynamic context capture, vehicle-based state-of-the-artintegrated sensor suites are pursued. We propose ahierarchical structure, as shown in Figure 1, for contextprocessing capable of dynamically allocating computationaland sensor resources depending on the demands imposed bythe complexity of environment and the vigilance level andcognitive load of the driver. These can be captured withrelatively few resources and used to modulate the level ofdetail and integration in processing the vast amount of datafrom the multi-modal sensor network.

The overall objective of our studies is to seek answers tothe following important questions.- What sets of sensors are robust enough under a wide

variety ofenvironmental conditions?- What contexts can support a sufficiently complete

representation ofthe environment, the vehicle state, and thedriver state?

- What is the best computational framework to extractcontexts from sensor networks that is compatible withhuman perception?

- What are the best models of the environment, the vehiclestate, the driver state, and the knowledge that drivers mayhave ofthe driving ecology?

- How to classify and explain driver behavior according tothe tasks that the driver is engaged in, the tasks that thedriver is intending, and the safety margin of the driver toperform the task?

II. LISA TESTBEDSTo provide adaptable experimental testbeds for evaluating

the performance of various sensing modalities and theircombination, two test environments, based upon aVolkswagen Passat vehicle [Laboratory for Intelligent, SafeAutomobiles-P (LISA-P)] vehicle, and a Nissan Infinity Q-45vehicle (LISA-Q) were outfitted with a computer and amultitude of cameras and acquisition systems. Of principalimportance in the hardware specification and softwarearchitecture was the ability to capture and process data fromall the sensor subsystems simultaneously and to providefacilities for algorithm development and offline testing. Ashort description of these testbeds is provided below withreferences to relevant papers for details.

A. LISA-P: Occupant and Driver Posture Analysis andPedestrian DetectionThe LISA-P test bed shown in Figure 2 (a) is designed for

collecting and processing large amount of synchronized datafrom a number of different sensors, especially for monitoringdriver's state. Various sensory and computing modules usedin this testbed include:(1) A trinocular stereo system from Pt-Grey Research, which

provides 2-1/2-D stereo disparity maps is used forestimating the distance of the occupant.

(2) A miniature 2-D thermal long-wavelength infrared sensor,Raytheon model 2000 AS is mounted on the dashboard toobserve the face of the occupant. This device providesvideo response in the LWIR spectrum (7-14 m).

(3) An array of four color CCD cameras providing imagesused for obtaining 3-D voxel reconstruction throughshape-from-silhouette (SFS).

(4) A pair of omnidirectional cameras in front of the vehiclegiving panoramic images used for detection ofpedestrians and nearby vehicles.

(5) A pair of SICK LIDAR sensors on two sides of the carwhich can be used for detecting nearby objects anddeterrnining accurate distance to them.

30


capabilities of the LISA-Q intelligent vehicle include:The placement of the sensors is shown in Figs. 2 (a). These

sensors are supported by a synchronized video-stream-capturing hardware, and high-volume storage. The computingplatform consists of a commercial Xeon PC with a high-throughput disk subsystem for streaming video data. Thecomputing platform allows for a good deal of processing to bedone in real time as well as store data for off-line processing.LISA-P is outfitted with a power inverter to supply 120 voltsAC power. A detailed description of the testbed is provided in[8].

12VOtC-I1OC Res*w Mondor& lwyI,Vmerlt Wort CmIrolsh StrwcaCs t soWow Omni

I cameras

peal Xr non PC Near Ier,,ed 17wennalwARw/RMCIiMA10ummnmof Camea

(a)

(b)Fig. 2 (a) LISA-P testbed used for occupant and driver posture analysis, andpedestrian detection (b) LISA-Q testbed used for vehicle surround, lanetracking, and driver support system.

B. LISA-Q: Vehicle Surround, Lane Tracking and DriverSupport SystemThe LISA-Q intelligent test bed shown in Figure 2 (b) [12]

is designed as a system capable of collecting large amounts ofdata from a variety of modular sensing systems and processingthat data in order to be fed back to the human occupant.Sensor systems include rectilinear cameras, wide field-of-viewcamera systems, GPS and navigation systems, and the datafrom internal automobile vehicle state sensors. The systemcontains an array of computers that serve for data collection aswell as real-time processing of information. The key

(1) Eight NTSC hardware video compressors forsimultaneous capture.

(2) Controller-Area-Network (CAN) interface for acquiringsteering angle, pedals, yaw rate, and other vehicleinformation.

(3) Built-in 5-beam forward looking LASER RADAR rangefinder.

(4) WAAS enabled GPS.(5) Integration into car audio and after-market video displays

for feedback and alerts.Detailed information about this test bed is described in [12].

III. VISION SYSTEMS FOR ENHANCED SAFETY:ILLUSTRATIONS

A. Looking-In the Vehicle: Occupant Position and PostureThe objective of this research is the development of a

highly reliable and real-time vision system for sensingpassenger occupancy and body posture in vehicles, ensuringsafe airbag deployment and helping to prevent injuries. Thedesign of the "smart airbag" system can be divided into threeparts: 1) real-time scene sensing; 2) feature selection; and 3)body size, posture, and movement analysis, followed bydecision logic for various levels of airbag deployment asshown in Figure 3 (a). We propose using video cameras fortheir unobtrusiveness and potential for other purposes beyond"smart airbags."

For scene sensing, we consider emitted LWIR imaging andstereo depth imaging. For feature selection and analysis, weconsider both simple region occupancy features to detailedhuman body model pose estimation. Using stereo or multi-camera systems with high-level human body modeling wouldalso provide information useful for other applications withminimal extra effort. High-quality input data and detailedanalysis of body pose can also be used to enhance safety byanalyzing driver alertness and could also be used to buildintelligent interfaces to different in-car devices, such as themobile phone or radio.To determine whether a person is in the right position for

airbag deployment, the area between the back of the seat andthe dashboard can be divided into sections. A diagram of thein-position (IP), out-of-position (OOP), and critically out-of-position (COOP) areas in the passenger seat is shown inFigure 3 (b). By analyzing these regions, we can categoricallyexamine the human body under various positions that anoccupant can take in the passenger seat, including sitting in anormal position, leaning forward, reaching down, seated withthe seat advanced, reclined, slouched, and knees on thedashboard or the edge of the seat.

31


(e)

(a) can be useful for visualizing and analyzing the nearbysurroundings of the vehicle. In [13], we have introduced theconcept of Dynamic Panoramic Surround (DPS) map thatshows the nearby surroundings of the vehicle, and detects theobjects of importance on the road. We have demonstratedsuccessful generation of DPS in experimental runs on aninstrumented vehicle testbed using monocular as well asbinocular omni camera systems. These experiments prove thebasic feasibility and show promise of omni video based DPScapture algorithm to provide useful semantic descriptors of thestate of moving vehicles and obstacles in the vicinity of avehicle.

Figure 4 (b) shows the generation of 360 degree surroundmap using a monocular omni camera mounted on top of thevehicle. The motion of the road is modeled using a parametricplanar motion model whose parameters are estimated byoptimally combining camera calibration, vehicle speedinfornation from CAN bus, and the motion of features on theground. The features that do not satisfy the model areseparated as outliers. Using the model, the road motionbetween two frames is compensated, and normalized imagedifference is used to detect objects that are above the road orhave independent motion. The omnidirectional image withobject positions is unwarped to give the surround map.Details of the approach are described in [14].

Figure 4 (c) shows the detection of pedestrians in front ofthe vehicle using a stereo pair of omni-directional cameras.Video sequences are obtained from a pair of omni camerasmounted on two sides of the vehicle. Camera calibration isperformed off-line to determine the relationship between thevehicle and pixel coordinates. Using the calibration.information, the images are transformed to obtain virtualperspective views looking forward towards the road. Thistransformation, called rectification simplifies the stereogeometry making it easier to match corresponding featuresbetween the two images. Area-based correlation is then usedto perform stereo matching between features. The result is adisparity map showing the displacement of features from oneimage to another. Based on the disparity map, the features aregrouped into objects, and distance to the objects is computed.Details of this algorithm are described in [15].

Figure 3 Smart airbag system that makes airbag deployment decision based onposition of the occupant. (a) Block diagram (b) Areas for In-position (IP),Out-of-position (OOP) and Critically out-of-position (COOP) regions (c)Head detection using stereo cameras (d) Head detection using thermal cameras(e) 3-D voxel reconstruction ofupper part ofbody using multiple cameras.

B. Looking-out ofthe vehicle: Dynamic PanoramicSurroundAwareness of what surrounds a vehicle directly affects the

safe driving and maneuvering of an automobile. Surroundinformation or maps can help in studies of driver behavior aswell as provide critical input in the development of effectivedriver assistance systems. Omnidirectional cameras whichgive a panoramic view of the surroundings such as Figure 4

32


looking outside to detect and track lanes, but also from camerainside the vehicle that monitors driver's facial motion. Sincelane change is usually preceded by turning of head, the headmotion gives an advance indication of the lane change intent.The system composed of a the following key components asshown in Figure 5 (a):(1) Lane position tracking system that determines the lateral

position of the vehicle in the lane at any given time,(2) Driver head motion estimation module that uses facial

features detected from a camera in order to estimate theapproximate motion of driver's head,

(3) Vehicle parameter collection system that gives parameterssuch as vehicle speed, yaw rate, and acceleration,

(4) Lane change intent classifier based on sparse Bayesianlearning that combines the features from the abovecomponents in order to determine the probability of lanechange at any given time.

System details are described in [16]. Figure 5 (b) shows anexample of lane change intent detection. The top bar showsthe estimated probability of lane change using lane tracking,vehicle dynamics from CAN bus, as well as head motion. Thebottom bar is derived without using the head motion. It isobserved that the use of head motion gives an advantage of0.5 seconds in detecting the lane change intent, which iscritical for preventing accidents.

lc)

Figure 4 (a) Illustration of a dynamic surround map. (x; y): Coordinates ofother objects w.r.t. own vehicle. V: Velocity w.r.t. road. LP: Lateral positionof own vehicle w.r.t center of the lane (b) Generation of surround map usingmotion-based detection using a single omni camera on top of the vehicle (c)Pedestrian detection using stereo pair of omni cameras.

C. Looking-in and Looking-out ofthe vehicle: Lane changeintent analysisThis section gives an application that combines the use of

sensors that are looking-in as well as looking-out in order topredict driver's intended actions. Driver intent inferencepresents a challenging classification problem; namely, given adiverse array of multi-modal features, how to infer or classifydriver intentions.

This vision system estimates driver intentions in thespecific area of lane changes, arguably one of the mostimportant actions relevant to intelligent support systems. Forthis purpose, it uses the information not only from the camera

(b)

Figure 5 (a) Block diagram for inference of lane change intent usinginformation from lane tracking, head motion, and the CAN bus data (b)Example of detection of lane change intent. The top bar shows the estimatedprobability of lane change using all the three sources whereas the bottom barshows the probability using only lane tracking and CAN bus data. The lane

33


change is detected earlier when the head movement is used.

IV. CONCLUDING REMARKSDevelopment of a real-time, a robust dynamic context

capture system for an automobile, computer vision andmachine learning techniques play important roles. In thispaper, we presented a motivation and experimental support fordeveloping vision systems for "Looking-In and Looking-Out"(LILO) of a vehicle. Such an "active", multi-modal sensoryapproach for "dynamic context capture and situationalawareness using cameras, radars, audio, etc. allows forestablishing representations of the state of the environment,the vehicle, and the driver with accurate dynamic uncertaintycharacterization. It is believed that successful integration ofsuch powerful sensory suits in human-centric decision logicframework will have a significant impact on the safety of newgenerations of automobiles and telematics devices used for in-car communication, information access, business transactions,and entertainment.

ACKNOWLEDGMENTWe want to thank our colleagues at the Computer Vision

and Robotics Research Laboratory for their cooperation,assistance, and contributions. We are especially grateful to ourindustrial collaborators, who provided us with invaluableguidance and support to provide a "reality-check" for ourresearch. It is indeed our pleasure to acknowledge the positiveimpact of our interactions with such collaborators including,Dr. Yousuke Akatsu, Dr. Erwin Boer, Dr. Akhtar Jameel, Dr.Akio Kinoshita, Dr. Satoshi Kitazaki, Dr. Chris Pribe, Dr.Klaus Schaff, Dr. Arne Stoschek, Dr. Hiroshi Takahashi, andtheir research teams.

Measures ofDriver Alertness Conference, pp. 75-86, April 1999,FHWA-MC-99-136.

[8] M. M. Trivedi, S. Y. Cheng, E. M. C. Childers, S. J. Krotosky,"Occupant Posture Analysis with Stereo and Thenral Infrared Video:Algorithms and Experimental Evaluation", IEEE Trans. VehicularTechnology, 53 (6), 1698-1712, Nov. 2004.

[9] M. Bertozzi and A. Broggi, "GOLD: a Parallel Real-Time Stereo VisionSystem for Generic Obstacle and Lane Detection," IEEE Trans. onImage Processing, 7 (1), 62-81, Jan. 1998.

[10] W. Enkelmann, "Video-based driver assistance - from basic functions toapplications," International Journal ofComputer Vision, 45 (3), 201-221, December 2001.

[11] L. Matuszyk, A. Zelinsky, L. Nilsson, and M. Rilbe, "Stereo panoramicvision for monitoring vehicle blind-spots," in Proc. IEEE IntelligentVehicles Symposium, 31-36, June 2004.

[12] J. McCall, 0. Achler and M. M. Trivedi, "Design of an InstrumentedVehicle Testbed for Developing Human Centered Driver SupportSystem," In Proc. IEEE Intelligent Vehicles Symposium, 483-488, June2004.

[13] T. Gandhi and M. M. Trivedi, "Dynamic Panoramic Surround Map:Motivation and Omni Video Based Approach," In Proc. IEEEInternational Workshop on Machine Visionfor Intelligent Vehicles, June2005.

[14] T. Gandhi, M. M. Trivedi, "Parametric Ego-Motion Estimation forVehicle Surround Analysis Using Omni-Directional Camera," MachineVision and Applications, January 2005.

[15] T. Gandhi and M. M. Trivedi, "Vehicle Mounted Wide FOV Stereo forTraffic and Pedestrian Detection," In Proc. International Conference onImage Processing, September 2005.

[16] J. McCall, D. Wipf, M. M. Trivedi, B. Rao, "Lane Change IntentAnalysis Using Robust Operators and Sparse Bayesian Learning," InProc. IEEE International Workshop on Machine Visionfor IntelligentVehicles, June 2005.

[17] J. McCall, M. M. Trivedi, "Visual Context Capture and Analysis forDriver Attention Monitoring", In Proc. 7th IEEE Conf on IntelligentTransportation Systems, October 2004.

REFERENCES[1] "World report on road traffic injury prevention: Summary," World

Health Organization, Tech. Rep., April 2004,http://www.who.int/world-health-day/2004/infomaterials/worldreport/en

[2] "Traffic safety facts 2003 FARS/GES annual report (Final Edition)," USDept. ofTransportation, National Highway Traffic SafetyAdministration, Tech. Rep., January 2005, http://www-nrnhitsa.dot.gov/pdf/nrd-3ONCSA/TSFAnn/2003HTMLTSF/TSF2003%.HTM.

[3] E.D. Dickmas, B. Mysliwetz, T. Christians, "An integrated spatio-temporal approach to automatic visual guidance of autonomousvehicles," IEEE Transactions on Systems, Man and Cybernetics, 20 (6),1273-1284, Nov.-Dec. 1990.

[4] D. Pomerleau and T. Jochem, "Rapidly adapting machine vision forautomated vehicle steering," IEEE Expert: Special Issue on IntelligentSystem and theirApplications, 11 (2), 19-27, April 1996.

[5] U. Franke, D. Gavrila, S. Gorzig, F. Lindner, F. Puetzold, C. Wohler,"Autonomous driving goes downtown," IEEE Intelligent Systems andTheirApplications, 13 (6), 4048, Nov.-Dec. 1998.

[6] F. Heimes and H.-H. Nagel, "Towards active machine-vision-baseddriver assistance for urban areas," International Journal ofComputerVision, 50 (1), 5-34, 2002.

[7] Grace, R., Byrne, V. E., Legrand, J. M., Gricourt, D. J., Davis, R. K.,Staszewski, J. J., Carnahan, B., A Machine Vision Based Drowsy DriverDetection System for Heavy Vehicles, Proceedings ofThe Ocular

34


looking-in vehicle: selected investigations...

Documents