FE

Ta

b

c

d

a

ARR1AA

KsKFFVF

1

isaeottwtaDmvlmoa

a

1h

Applied Soft Computing 18 (2014) 25–38

Contents lists available at ScienceDirect

Applied Soft Computing

j ourna l h o mepage: www.elsev ier .com/ locate /asoc

ully automated real time fatigue detection of drivers through Fuzzyxpert Systems

ayyaba Azima,b, M. Arfan Jaffara,c,∗, Anwar M. Mirzad

National University of Computer & Emerging Sciences, Islamabad, PakistanCentre of Excellence in IT (CEIT), Institute of Management Sciences, Peshawar, PakistanCollege of Computer and Information Sciences, Al Imam Mohammad Ibn Saud Islamic University (IMSIU), Riyadh, Saudi ArabiaKing Saud University, Riyadh, Saudi Arabia

r t i c l e i n f o

rticle history:eceived 12 April 2010eceived in revised form0 November 2013ccepted 18 January 2014vailable online 25 January 2014

a b s t r a c t

This paper presents a non-intrusive fatigue detection system based on the video analysis of drivers. Thesystem relies on multiple visual cues to characterize the level of alertness of the driver. The parametersused for detecting fatigue are: eye closure duration measured through eye state information and yawninganalyzed through mouth state information. Initially, the face is located through Viola–Jones face detectionmethod to ensure the presence of driver in video frame. Then, a mouth window is extracted from the faceregion, in which lips are searched through spatial fuzzy c-means (s-FCM) clustering. Simultaneously, thepupils are also detected in the upper part of the face window on the basis of radii, inter-pupil distance

eywords:-FCMalman filteruzzy logicuzzy Expert Systemsigilance monitoring

and angle. The monitored information of eyes and mouth are further passed to Fuzzy Expert System(FES) that classifies the true state of the driver. The system has been tested using real data, with differentsequences recorded in day and night driving conditions, and with users belonging to different race andgender. The system yielded an average accuracy of 100% on all the videos on which it was tested.

© 2014 Elsevier B.V. All rights reserved.

atigue detection

. Introduction

Recent development in the computer hardware and softwarendustry has bestowed a number of value added services to theociety. One of such blessings is the idea of intelligent vehiclesnd driver support systems, which have the potential to greatlynhance the safety of drivers and passengers by alerting the driversf dangerous situations. The intelligent vehicles not only recognizehe situation, but also the driver’s intended actions. Statistics revealhat from 10% to 20% of all the traffic accidents are due to driversith a diminished vigilance level [1]. This problem has increased

he need of developing active safety systems that can prevent roadccidents by warning the drivers of their poor driving condition.rivers with diminished vigilance level and fatigue suffer from aarked decline in their abilities of perception, recognition, and

ehicle control, and, therefore pose serious danger to their ownife and to the lives of others. In case of fatigue, the driver is so

entally or physically exhausted that he/she either feels drowsyr cannot respond appropriately to the driving situation. Similarly,

crash is also imminent if the driver is inattentive rather than being

∗ Corresponding author.E-mail addresses: tayyaba.azim@imsciences.edu.pk (T. Azim),

rfan.jaffar@ccis.imamu.edu.sa (M.A. Jaffar), ammirza@ksu.edu.sa (A.M. Mirza).

568-4946/$ – see front matter © 2014 Elsevier B.V. All rights reserved.ttp://dx.doi.org/10.1016/j.asoc.2014.01.020

fatigued. The prevention of such accidents is a major focus of effortin the field of active safety research.

Most of the previous research has focused on the collection ofocular parameters to detect the fatigue level of driver [1–3]. Inmost of them, the system initializes with the detection of eyes togather the ocular parameters like blinking frequency, PERCLOS, eyestate etc. These systems may fail to predict the driver’s true state ifthe eyes are not detected due to varying light conditions or vibra-tions experienced in real driving situations. Hence, it cannot bedenied that an approach depending on multiple visual cues willwork more effectively than the one depending on just one facialfeature. For this reason, our proposed system incorporates multi-ple visual cues such as yawning and eye state analysis to determinethe fatigue level of the driver. The selection of the parameters isa tradeoff between accuracy and speed as more accuracy can beachieved by observing more features of the driver but this mightaffect the response of a real time system in terms of speed. Thusit is important to carefully choose these parameters such that thespeed of the system is not compromised for the sake of accuracy.There were a number of other code optimization steps that weretaken to enhance the computational speed of the system [4].

To the best of our knowledge, no one has ever investigatedthe best face and mouth detection algorithm for fatigue detectionapplications. A careful research was carried out on a number of algo-rithms that could best segment out the face and mouth of the driver.

2 ft Com

Ta

tkeSi

2

ris

2

c

••••

2

iphatttdeaThust

2

litscadfoca

2

bhtao

6 T. Azim et al. / Applied So

he expert system once tuned during the initialization phase, givesn accuracy of 100% on all the tested videos.

The paper is arranged as follows: In Section 2, a review ofhe previous studies is presented along with some backgroundnowledge. Section 3 describes the overall system architecture,xplaining its main modules. The experimental results are shown inection 4 and finally we present the conclusion and future studiesn Section 5.

. Background and related work

This section describes the background knowledge that isequired to understand the working of fatigue detection and vig-lance monitoring systems. Along with that, a literature review ofome of the existing fatigue detection techniques is also presented.

.1. Techniques of monitoring vigilance and fatigue

The techniques of monitoring vigilance and fatigue of the driversan be divided into following four categories:

Physiological measuresIndirect vehicle behaviorDirectly observable visual behaviorHybrid approaches

.1.1. Physiological behaviorsThis is the most accurate technique of fatigue detection as it

s based on physiological measures like brain waves, heart rate,ulse rate, respiration etc. These parameters are collected with theelp of various sensors placed either on the body of the driver orre embedded in a car. The main disadvantage of this technique ishat it is intrusive because it requires electrodes to be attached tohe drivers, causing annoyance to them. A representative project inhis line is the MIT Smart Car [5], where several sensors (electrocar-iogram, electromyogram, respiration, and skin conductance) arembedded in a car and visual information for sensor confirmationre used. In the advanced safety vehicle (ASV) project conducted byoyota [6], the driver must wear a wristband in order to measureis heart rate. Other techniques monitor eyes and gaze movementssing a helmet or special contact lenses [7]. These techniques, inpite of their accuracy, are still not acceptable in practice due toheir intrusiveness.

.1.2. Indirect vehicle behaviorThis technique involves the use of indirect vehicle behaviors like

ateral position, steering wheel movements and time-to-line cross-ngs to detect the driver’s vigilance and fatigue level. Although theseechniques are not intrusive, they are subject to several limitationsuch as vehicle type, driver experience, geometric characteristics,ondition of the road, etc. These procedures also require a consider-ble amount of time to analyze user behaviors and therefore, theyo not work with the so called micro-sleeps—when a drowsy driveralls asleep for a few seconds on a very straight road section, with-ut changing the lateral position of the vehicle [8]. In this line, wean find different experimental prototypes by Toyota, Mitsubishind other individuals but none of them has been commercialized.

.1.3. Directly observable visual behaviorsPeople experiencing fatigue show some easily observable visual

ehaviors from the changes in their facial features like the eyes,ead, and face. Typical visual characteristics observable fromhe images of a person, with a reduced alertness level include

longer blink duration, slow eyelid movement, smaller degreef eye opening (or even closed), frequent nodding, yawning,

puting 18 (2014) 25–38

gaze (narrowness in the line of sight), sluggish facial expression,and drooping posture. Computer vision can be a natural andnon-intrusive technique for extracting visual characteristics thattypically characterize a driver’s vigilance from the images taken bya camera placed in front of the user. An example of such a systemis FaceLAB developed by Seeing machines [5].

2.1.4. Hybrid techniquesHybrid techniques believe that driver’s state and performance

can improve the sensibility and reliability in fatigue detection.These methods make use of visual behaviors to notice the driver’sstate and then further combine this information with the indi-rect vehicle behavior information to predict the state of the driver.Doering et al. [8] used a range of measures like eye-closure, lanetracking and changes in physiological state to predict fatigue-related crashes. Recently, the European Union has completed anambitious project called AWAKE (System for Effective Assess-ment of Driver Vigilance and Warning According to Traffic RiskEstimation) [9] whose aim was to demonstrate the technologicalfeasibility of driver vigilance monitoring systems and to look at thenon-technical issues that can influence the use of such systems. Theproject estimates driver state through eyelid movement, changesin steering grip and driver behavior including lane tracking, use ofaccelerator and brake and steering position. These measures arethen combined and evaluated against an assessment of currenttraffic risk obtained from digital navigation maps, anti-collisiondevices, driver gaze sensors and odometer readings. Despite ofthe increased accuracy provided by such systems, they take moreresponse time due to the increased computations involved in gath-ering hybrid parameters and deducing fatigue.

2.2. Challenges posed by real time active safety systems

The challenges that are commonly faced by active safety systemsare:

• Lighting• Changing background• Vibration

Problems are encountered in fatigue detection due to the vari-ations in above parameters. The lighting conditions may vary dueto different weather conditions (sunny, stormy, cloudy, etc.) andday timings (night time, day time). Similarly, the background ofthe driver is also subjected to changes because the car is constantlymoving on the road with different scenes side-by, in reality. Thevibration of the car can also be a hindrance in the visual data col-lection process.

2.3. Critical requirements of active safety systems

2.3.1. Accurate and early detection of drowsinessThe critical issue that a drowsiness system must address is the

question of how to accurately detect drowsiness at an early stage.This is required because if the warning occurs early enough in thedevelopment of fatigue, such devices can enhance driver alertnesssufficient to avoid a collision [2].

2.3.2. Identifying right combination of parametersA single measure is not sufficiently reliable and sensitive enough

to quantify driver fatigue. One may encounter difficulty whenthe required feature cannot be acquired accurately. For example,drivers with glasses could pose serious problems to those tech-niques based on detecting eye-characteristics. Hence, there is a

ft Com

nr

2

safwspTeFs“hiapctb

lrtrsurfitartTcef

2

bFn

T. Azim et al. / Applied So

eed of identifying right number and combination of parametersequired for detection [2].

.4. Fuzzy logic

Fuzzy logic is a problem-solving methodology that provides aimple way to arrive at a definite conclusion based upon vague,mbiguous, imprecise, noisy, or missing input information. Theuzzy logic approach to control problems mimics how a personould make decisions, only much faster. Fuzzy logic incorporates a

imple, rule-based “IF X AND Y THEN Z” approach to solve a controlroblem rather than attempting to model a system mathematically.he fuzzy logic model is based empirically, relying on an operator’sxperience rather than their technical understanding of the system.or example, rather than dealing with temperature control in termsuch as “SP = 500F”, “T < 1000F”, or “210C <TEMP <220C”, terms likeIF (process is too cool) AND (process is getting colder) THEN (addeat to the process)” or “IF (process is too hot) AND (process is heat-

ng rapidly) THEN (cool the process quickly)” are used. These termsre imprecise and yet very descriptive of what must actually hap-en. Consider what you do in the shower if the temperature is tooold: you will make the water comfortable very quickly with littlerouble. Fuzzy logic is capable of mimicking this type of behaviorut at very high rate.

A Fuzzy Expert System is an expert system that uses fuzzyogic through a collection of fuzzy membership functions andules, instead of Boolean logic to reason about data. These sys-ems result in knowledge bases that are flexible and semanticallyich to be understood by the domain experts easily. Rule-basedystems offer three major advantages over other conventionallysed classification systems. First, a large training set is usually notequired. Secondly, most Fuzzy Expert Systems provide for parallelring of production rules, i.e. all rules are fired effectively at oneime, emulating a parallel computer. Parallel operation has severaldvantages for fuzzy systems as it makes programming easier anduns considerably faster than an equivalent sequential system andhis is what we desire in our real time fatigue detection system.hirdly, rule based expert systems possess the ability to learn byreation of new rules and addition of new data to the expert knowl-dge database. This feature makes the developed system adaptableor unseen situations and also extendable for future improvements.

.5. Related work

Different authors have adopted different algorithms and com-ination of parameters to resolve the fatigue detection problem.ollowing is a review of some of the notable computer vision tech-iques proposed by different authors:

Khan et al. [10] proposed a real time fatigue detection algorithm inwhich fatigue is detected using correlation function with an auto-matically generated online eye template. The proposed algorithmcan detect eyelids movement and can classify whether the eyesare open or closed by using normalized cross correlation functionbased classifier. If the eyes are closed for more than a specifiedtime, an alarm is generated. The algorithm showed an accuracy of90% on average. This system fails to work if the head rotation ismore than 30◦ in azimuth. Moreover, the generation of online eyetemplate badly influences the response time of the system whichmakes it less appealing for real time use.Park, Ahn and Byun [11] proposed an efficient method of measur-ing the eye blinking for drowsy driver detection. Since, while real

driving in daytime, driver’s face is exposed to various illumina-tions; the eyes are difficult to monitor. The authors made use ofSupport Vector Machines (SVM) as eye verification phenomenonto boost the accuracy of eye detection. Furthermore, for an

puting 18 (2014) 25–38 27

efficient measure of eye blinking, they defined a decision func-tion that was based on the measure of eye-lid movement and theweight generated by the eye classifier. The experiment showedreliable performance on their test data acquired during real driv-ing in various illumination conditions. The major drawback ofthe technique is that it relies on a single visual cue to determinedriver’s drowsiness level. If at any stage, the eye detection fails dueto challenging light conditions, the system loses its supervisingpower.Bergasa, Nuevo, Sotelo, Barea and Lopez [12] also proposed a non-intrusive computer vision system for monitoring driver’s vigilancein real time. They chose six parameters: Percent Eye closure (PER-CLOS), eye closure duration, blinks frequency, nodding frequency,face position, and fixed gaze. These parameters were combinedusing a fuzzy classifier to infer the level of inattentiveness of thedriver. The use of multiple visual parameters and the fusion ofthese parameters yielded a more robust and accurate inattentioncharacterization than by using a single parameter. The system wastested with different video sequences recorded in night and daydriving conditions on motorway and with different users. The sys-tem worked robustly at night, yielding an accuracy percentagenear to 100%. The performance decreased especially in bright days,and did not work with drivers wearing glasses. Moreover, since theparameters chosen were large in number, the increased accuracywas achieved at the cost of speed which is not bearable in real timesystems.Wang, Yang, Ren and Zheng [2] presented a comprehensive sur-vey of research on driver fatigue detection and provided structuralcategories for the methods that have been proposed so far. Themethods of fatigue detection mainly focused on measures of thedriver’s state, performance and the combination of the driver’sstate and performance. The measure of driver’s state includedPERCLOS (Percent Eye Closure), mouth shape and head position;the measures of driver’s performance included lane tracking andtracking of distance between vehicles. All these approaches werediscussed in detail along with the typical driver monitoring sys-tems like DAISY, Co-pilot, DAS (Driver assistance System), faceLAB and AWAKE (System for Effective Assessment of Driver Vigi-lance and Warning according to Traffic Risk Estimation). The paperemphasizes on the maintenance of two critical requirements for alldrowsiness and fatigue detection systems: (i) accurate and earlydetection of low attentiveness (ii) right number and combina-tion of parameters that lead us to accurate result in real time.Our proposed Fuzzy Expert System (FES) holds the privilege ofmaintaining both these characteristics simultaneously.Batisa [13] proposed a system that dealt with the computationof eye-lid movement parameters and head (face) orientation esti-mation. The system relied on pupil detection to robustly track thedriver’s head pose and monitor its level of fatigue. Visual informa-tion was acquired using a specially designed solution combining acharged coupled device (CCD) camera with an NIR (near infrared)illumination system. The system was fully automatic and classi-fied rotation in all-view directions, detected eye-blinking and eyeclosure and recovered the gaze of the eyes. Although, good resultswere achieved; the system was found to be highly dependent onthe image face ellipse detection. Further research is necessary toimprove the accuracy of the image face ellipse detection.Smith, Shah and Lobo [14] proposed a system that was ableto detect both visual attention and drowsiness. Visual atten-tion was defined as a process of visually looking forward andalertness/drowsiness as the state of fatigue and possibly fightingagainst sleeping bouts, micro-sleep, or other similar conditions.

They pointed out that there may be situations when the driver maybe fully awake but paying less visual attention to the road; henceboth the parameters should be monitored to warn the driver. Thesystem relied on estimation of global motion and color statistics

2 ft Computing 18 (2014) 25–38

3

cd

••••••

f

3

ffatidhabsol

symmetrically along two concentric and circular rings. The LEDs inthe inner ring are as close as possible to the camera lens in orderto maximize the “bright pupil” effect while the one in the outer

8 T. Azim et al. / Applied So

to track a person’s head and facial features. The system classi-fied driver’s rotation in all viewing directions, detected eye/mouthocclusion due to head rotation, detected eye blinking and eye clo-sure, and recovered the 3D gaze of the eyes. The system was testedin both stationary and moving vehicles with negligible differencesin its accuracy. It performed well under a wide variety of daytimeillumination levels, from strong daylight to a heavy cloud cover.However it was not tested at night time. Moreover, the methodused to detect eyes and lips does not always perform robustly dueto its reliance on color statistics.Qiang Ji and Yang [3] discussed a real time prototype computervision system for monitoring driver vigilance. The main compo-nent of the system consisted of a remotely located video CCDcamera, a specially designed hardware system for real time imageacquisition and for controlling the illuminator and the alarm sys-tem. There were two CCD cameras embedded on the dashboard ofthe vehicle; the first camera was a narrow angle camera, focusingon the driver’s eyes to monitor eye-lid movement while the sec-ond camera was a wide angle camera that focused on the driver’shead to track and monitor head movement. The system used visualbio-behaviors like eye-lid movement, face orientation and gazemovement (pupil movement) that typically characterize a driver’slevel of vigilance. The system was tested in a simulating envi-ronment with subjects of different ethnic backgrounds, differentgenders, ages, with/without glasses, and under different illumina-tion conditions, and it was found very reliable and accurate. Themajor limitation of this system is that it requires two image acqui-sition devices to determine the fatigue level; hence some time isrequired to fuse the results obtained from both devices and thenmanipulate the fatigue level. The testing was also done on sim-ulator rather than on real video sequences that truly portray thechallenges of real time systems. Hence, the performance of thesystem cannot be firmly assured for real time driving situations.

. Proposed approach

We present an algorithm that detects driver fatigue by analyzinghanges in the mouth and eye state. The method proposed can beivided into the following phases:

Image acquisitionFace detection and trackingPupil detectionMouth detectionYawning analysis and eye state analysisDriver state analysis

The outline of the proposed approach is shown in Fig. 1. In theollowing sections, each of the phases will be discussed in detail.

.1. Image acquisition

The purpose of this stage is to acquire video images of driver’sace so that directly observable visual cues could be gathered foratigue determination. The image acquisition system consists of

low-cost charged coupled device (CCD) micro camera sensitiveo near-infrared (NIR). The acquired images should be relativelynvariant to light conditions and should facilitate pupil and mouthetection. This purpose is served well by near-IR illuminator whichighlights our facial features of interest and reduces the effect ofmbient light [5]. Such a CCD camera can also be used to produce

right and dark pupil effects. This bright pupil effect is very muchimilar to the red eye effect in photography. If the NIR is illuminatedff the camera’s optical axis, a dark pupil image is generated as theight reflected by the pupils does not enter the camera lens.

Fig. 1. Block diagram of the proposed fatigue detection system.

Fig. 2 shows the image acquisition system configuration wherethe NIR illuminator is composed of two sets of LEDs, distributed

Fig. 2. Infrared sensitive camera used in the proposed system [3].

T. Azim et al. / Applied Soft Computing 18 (2014) 25–38 29

detect

rcsa

pprelina

3

factd

ndOat

ndfttitoosf

3

rofo

Fig. 3. (a) Original image. (b) Face region

ing are placed with some distance and their job is to enhance theontrast in the image, thus reducing the ambient light effects. Theymmetric position of the LEDs in rings around the camera opticalxis cancels the shadows generated by the LEDs [3].

Many of the eye trackers make use of bright pupil and darkupil effect to calculate the difference image in which pupils arerominent [5,7,15]. We do not make use of the difference images,ather the data is such acquired just to reduce the ambient lightffects and make it working at night time too. The video data is col-ected in real driving scenarios and is obtained from [5]. The beams non-intrusive to the driver driving; thus forming the basis of aon-intrusive fatigue detection system. The frame rate of the imagecquisition system is 25 frames per second.

.2. Face detection

Face detection can be defined as the process of finding humanaces in images taken as input. The face detection algorithm returns

set of image coordinates where human faces are located. In theurrent application, since only the video of driver is given as inputo the system, hence it is assumed that only a single face would beetected if it is in the camera view.

The task of face detection is carried out as a preprocessing step toarrow down the area of search for pupils and mouth. If a face is notetected in a frame, no time is spent in pupil and mouth detection.n the other hand, for a successful face detection instance, the eyesnd mouth are located in the face region returned by face detector,hus reducing the time spent for eye state and mouth state analysis.

We have examined a number of different face detection tech-iques to determine the method most suitable for active fatigueetection applications [16–19]. After comprehensive analysis, theace was detected using Viola and Jones [20,21] real time face detec-ion technique based on AdaBoost algorithm. The face detectionechnique in AdaBoost is comprised of three aspects: the integralmage, a strong classifier comprising of weak classifiers based onhe AdaBoost learning algorithm, and an architecture comprisingf a cascade of a number of strong classifiers. A 25-layer cascadef boosted classifiers is trained to detect multi-view faces. A set ofample face and non-face (termed as background) images is usedor training.

.2.1. Performance of the face detectorAdaBoost face detection algorithm detects faces in a rapid and

obust manner with a high detection rate at 15 frames per sec-nd. The face detector is trained such that it is able to detectaces that are tilted up to about ±15◦ in plane and about ±45◦ outf plane (toward a profile view). However, it becomes unreliable

ed. (c) Region of interest containing eyes.

with rotation more than this. We have also noticed that harsh back-lighting in which the faces are very dark while the background isrelatively light sometimes causes failures. Finally, the face detectorfails on significantly occluded faces since it is trained on frontal andupright face images only. If the eyes are occluded for example, thedetector will usually fail. The mouth is not as important and so aface with a covered mouth will usually still be detected. The systemcould be modified in future such that the camera rotates to get afrontal view of the driver if rotation of more than 15◦ in plane and45◦ out of plane takes place. Similarly, the training data set could beexpanded to facilitate rotations more than ±15◦ in plane and about±45◦ out of plane.

Face tracking is carried out using Kalman filter. The Kalman filteris an efficient recursive filter that estimates the state of a dynamicsystem from a series of incomplete and noisy measurements. It esti-mates the position and uncertainty of the moving objects in nexttime frame i.e. where to look for face and how large a region shouldbe searched in next frame, around the predicted position to be sureto find face with certain confidence. It works reasonably well withfrontal face orientations but fails if the face is not bright enoughfrom the background due to external illumination interferences.It also fails when a sudden head movement occurs because theassumption of a smooth head motion is violated. To overcome thisproblem, an adaptive search window could be used whose size isdetermined automatically based on the face position, face velocityand location error [1].

3.3. Pupil detection

The goal of pupil detection and tracking is to constantly monitorthe eyelid movement of the driver which eventually helps in deter-mining his/her eye state for vigilance and fatigue level monitoring.Eye closure and narrowing eyelids are the most obvious signs ofthe onset of somnolence. Hence, we lay our focus on eyes first todetermine the somnolence state.

Since eyes locate on the upper part of face, face detected iscropped and the upper half of the face region is searched for eyes asshown in Fig. 3(c). This further helps us in reducing the computingcost of the fatigue detection system and makes it a good candidatefor real time use.

The pupils are detected by searching the cropped image regionfor two bright blobs that satisfy certain geometric constraints.The cropped image is first binarized by using adaptive threshold

for detecting the bright blobs in image. This image further goesunder dilation and opening operations so that spurious bright pix-els in upper face region are eliminated. After that, a standard eightconnected component analysis is applied on the binarized image

30 T. Azim et al. / Applied Soft Computing 18 (2014) 25–38

Fig. 4. (a) Image obtained after dilation and opening of adaptively thresholded image. (b) Image obtained after eight connected component analysis. (c) The detectedc possibo sentel

tciddctf

3

ciemaopniIetftfibnmteprtupito

onnected components are labeled in sequence from 1 to 8. (d) The centroids of

btained after applying geometric constraints, the pupils finally detected are repreegend, the reader is referred to the web version of this article.)

o identify the binary blobs that satisfy certain size and shapeonstraints. The size and shape constraints checked are: radius,nter-pupil distance, and angle between the pupils. The best candi-ates are selected depending on their size, intensity, position andistance, whereas the circular regions that do not satisfy these sizeonstraints are removed. All the possible pairs are evaluated andhe pair with the highest aptitude is selected as pupils. See Fig. 4or illustration.

.3.1. Pupil trackingTo monitor the driver, pupils are continuously tracked in suc-

essive frames after locating the eyes initially. This monitoring tasks accomplished successfully by using two Kalman filters, one forach pupil. Kalman filter is an efficient recursive filter that esti-ates the state of a dynamic system from a series of incomplete

nd noisy measurements. It estimates the position and uncertaintyf the moving objects in next time frame i.e. the position whereupils locate and the range of the area that should be searched inext frame for true pupil positions. Around the predicted position,

t is to be assured that pupils are found with certain possibility.t works reasonably well with frontal face orientations with openyes. However, it fails if the pupils are not bright enough due toilted face orientations, eye closures or external illumination inter-erences. It also fails when a sudden head movement occurs becausehe assumption of a smooth head motion is violated. Since, Kalmanlter predicts the future position of the pupils, assuming linearehavior of the system velocity, hence when the system behaveson-linearly due to a jerk; the pupil positions are incorrectly esti-ated. The filter corrects the pupil location estimate by finding

he difference between true and estimated pupil positions but thisstimate stands useless as the person returns back to his originalosition soon after jerk. Hence, the tracker gives false predictionesults on those frames with sudden head movement. To overcomehis Kalman filter problem, an adaptive search window could besed, the size of which is determined automatically basing on the

upil position, pupil velocity and location error. When the filter fails

n tracking, the search window of the eyes is gradually increased;his assists in locating the eyes when they are closed or there is anblique face orientation.

le circular regions initially detected are marked through red asterisks. (e) Imaged by red plus symbol. (For interpretation of the references to colour in this figure

3.4. Mouth detection

Yawning is one of the most obvious signs of the onset of som-nolence. Hence after detecting the pupils, we next lay our focuson mouth detection to determine the somnolence state. To deter-mine yawning, mouth region needs to be extracted from the facewindow to further boost up the speed of fatigue detection system.The mouth is searched in the lower most part of the detected faceregion, thus further decreasing the computation cost of the system.The block diagram of mouth detection phase is shown in Fig. 5.

Given an input frame, the system first segments the face andextracts the face region containing mouth as a preprocessing step.Further preprocessing involves the enhancement of red channel,gamma channel adjustment and adaptive histogram equalizationsteps, which enhances the contrast of the image in all types oflight conditions posed in real driving. This region is then passedto the FCM module whose functionality is described graphically inFig. 6. Fuzzy c-means (FCM) is a method of clustering which allowsone piece of data to belong to two or more clusters. It is based onminimization of the following objective function:

Jm =N∑

i=1

C∑

j=1

�mij ‖xi − cj‖2, 1 ≤ m > ∞ (1)

where m is any real number greater than 1, uij is the degree of mem-bership of xi in the cluster j, xi is the ith data point of d-dimensionalmeasured data, cj is the d-dimension center of the cluster, and ||*||is any norm expressing the similarity between any measured dataand the center. Fuzzy partitioning is carried out through an itera-tive optimization of the objective function shown above, with theupdate of membership uij and the cluster centers cj by Eq. (2):

�ij = 1∑C

k=1(‖xi − cj‖/‖xi − ck‖)2/(m−1)

, cj =∑N

i=1�mij

xi∑N

i=1�mij

(2)

This iteration will stop when

maxij{|�(k+1)ij

�(k)ij

|} < � (3)

where � is a termination criterion between 0 and 1, whereas kare the iteration steps. Thus, the FCM algorithm creates a fuzzy

T. Azim et al. / Applied Soft Computing 18 (2014) 25–38 31

etect

pit

wdnvft(pti

P

vSasra

h

Fig. 5. Block diagram of mouth d

artition that properly describes the image, using only the pixelntensity feature. This phase involves the calculation of cluster cen-ers and membership function in the spectral domain only.

In order to keep the system totally autonomous, an automaticay to determine the right number of clusters, is needed. This wasone by iterating the FCM algorithm for a range of hypothesizedumber of clusters and choosing the best option, based on a clusteralidity measure. The cluster validity function evaluates the per-ormance of clustering through two representative functions forhe fuzzy partition: partition coefficient (PC) and partition entropyPE) defined in Eq. (4). The idea of these validity functions is that theartition with less fuzziness means better performance. As a result,he best clustering is achieved when the value PC is maximal or PEs minimal.

C = 1n

n∑

j=1

c∑

i=1

u2ij, PE = 1

n

n∑

j=1

c∑

i=1

uijloga(uij) (4)

After finding out optimal number of clusters through clusteralidity measure, we pass these optimal clusters to the spatial FCM.patial FCM also consider neighboring information of each pixellong with the spectral information and returns fuzzy member-hip matrix. Spatial information incorporated by FCM helps us toemove noise in the image. The spatial function in s-FCM is defined

s:

ij =∑

k ∈ NB(xj)

Uik (5)

Fig. 6. Block diagram showing lip s

ion and yawning analysis phase.

where NB(xj) stands for a square window centered on pixel xj inthe spatial domain and i, j are data points. A 3 × 3 window wasused while implementing this technique. Just like the membershipfunction, the spatial function hij stands for the probability that pixelxj belongs to jth cluster. The spatial function of a pixel for a clusteris large if the bulk of its neighborhood belongs to the same clusters.The spatial function is included into membership function as shownin Eq. (6):

U ′ij =

Upij

hqij∑c

k=1Upkj

hqkj

(6)

where p and q are parameters to control the relative importance ofboth functions and i, k, j are data points. In a homogenous region,the spatial function simply fortifies the original membership, andthe clustering result remains unchanged. However, for a noisy pixel,this formula reduces the weighting of a noisy cluster by the labelsof its neighboring pixels. As a result, misclassified pixels from noisyregions or spurious blobs can easily be corrected. The FCM iterationproceeds with the new membership that is incorporated with thespatial function.

The iteration is stopped when the maximum difference betweentwo cluster centers, at two successive iterations is less than anoptimal threshold. There are different ways of calculating an opti-

mal threshold. Mostly methods use mean as an initial thresholdbut it fails in case of noisy images because mean is not robustagainst noise. So we have used a method based upon fuzzy entropythat finds out the optimal threshold dynamically according to the

egmentation through s-FCM.

32 T. Azim et al. / Applied Soft Computing 18 (2014) 25–38

Fig. 7. Simultaneous detection of pupils and mouth from a video frame. The plus sign indicates the mouth and eye centroids, whereas the center of inter-pupil distance isindicated by red dot. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 8. Mouth centroid angle range with inter-pupil centroid.

clusters found out by fuzzy c-mean. The point where the entropy ismaximum is the optimal threshold. The fuzzy entropy based errorfunction used is known as logarithmic entropy, given as:

H(A) = − 1n ln(2)

∑

i

[�A(xi ln �A(xi)

+ (1 − �A(xi)) ∗ (1 − ln �A(xi)))] (7)

After the convergence, defuzzification is applied to assign eachpixel to a specific cluster for which the membership is maximal.Thus the output of FCM module is a segmented lip image in binaryform shown in Fig. 7.

The image further goes under two tests to verify that successfuldetection of mouth. The first test is based on eccentricity analysis.Since, the lips region is more like an ellipsoid; hence their eccen-tricity is closer to 1. Thus a check is made on the eccentricity ofdetected region, if it is closer to 1, it is checked further for the sec-ond test; else the next consecutive video frame is grabbed to detectthe correct mouth region. The second test checks the centroid of thedetected lip region for a specific angle range with the inter-pupilcentroid. In a frontal face pose, the centroid of the lip region shouldbe perpendicular to the point which is the center of inter-pupil dis-tance as shown in Fig. 8. But if the whole lip region is not detectedsuccessfully, then the centroid might be misplaced, in that case theflexibility in angle range ensures the true presence of lip region.The pupils’ locations and their centroids are calculated on the basisof radii, inter-pupil distance and angle between the two pupils [1].The formula for calculating the angle between centroids is given asshown in Eq. (8). The centroid point passing the two tests is a true

representative of lip region.

� = tan−1 |y2 − y1||x2 − x1| (8)

T. Azim et al. / Applied Soft Computing 18 (2014) 25–38 33

quire

3

im

••

e

Fp

Fig. 9. Mouth and eye model indicating the parameters re

.5. Eye and mouth state analysis

After successfully detecting the eye and mouth features, twomportant properties are calculated to determine the eye and

outh state of the driver. These two properties are:

Area

Width to height ratio

Area is defined as the total no of pixels that are found in eachye and mouth region, whereas the width to height ratio is defined

ig. 10. Fuzzy Expert System that monitors the fatigue level of driver by utilizing the resroduced by FIS at base level are employed by the FIS at level2 to deduce about driver’s fa

d to determine the status of mouth and eyes respectively.

as shown in Eq. (9) (Fig. 9):

ratio = w

h(9)

where w represents the width and h represents the height of thedetected eye and mouth regions respectively. Both the parametershelp us in evaluating the degree of eye and mouth openness through

fuzzy inference systems (FIS). The eye and mouth states once deter-mined through a FIS are passed in the next step to a different FISwhich determines the fatigue of the driver. This hierarchical imple-mentation of FIS can be understood from Fig. 10 where the low level

ults of fuzzy inference systems (FIS) working at two hierarchical levels; the resultstigue.

34 T. Azim et al. / Applied Soft Computing 18 (2014) 25–38

Table 1Details of input and output variables used by one of the fuzzy inference systems at base level 1. The range of area and ratio varies from user to user and is evaluated in thefirst 20 frames, each time the system is initialized.

Variable Type Range Labels Linguistic terms/fuzzy sets Membership functions

Eye area Input X 3 Small, medium, large Trapezoidal, triangularEye ratio Input X 3 Small, medium, large Trapezoidal, triangularEye state Output [0 100] 3 Open, half open, closed Trapezoidal, triangular

Table 2Details of input and output variables used by one of the fuzzy inference systems at base level 1. The range of area and ratio varies from user to user and is evaluated in thefirst 20 frames, each time the system is initialized.

Variable Type Range Labels Linguistic terms/fuzzy sets Membership functions

fsltd

oadSctmbwurm

ct

3

attavagd

TD

Mouth area Input X 3

Mouth ratio Input X 3

Mouth state Output [0 100] 3

eatures (area and ratio) are used to determine the eye and mouthtates; further this information helps in detecting the fatigue at FISevel 2. The system on the whole is regarded as a Fuzzy Expert Sys-em (FES) that determined the fatigue and vigilance level of theriver.

For each subject belonging to different ethnicity, the range ofpen, half open and closed eyes and mouth varies in terms of areand ratio. Similarly, these parameters are also influenced by theistance of the driver from the camera. Our proposed Fuzzy Expertystem (FES) provides a solution to both these problems withoutompromising at the speed and accuracy of the system. To avoidhe scaling of mouth and eyes in different frames due to move-

ent, we use the normalized area and ratio of mouth and eyeounding rectangles, to represent the degree of openness. Note thathile normalizing the eye area and ratio, the face region used is thepper cropped part of face containing eyes. Similarly, the lower faceegion containing mouth is used while normalizing the obtainedouth area and mouth ratio.This normalization makes the range of open, half open and

losed parameters, consistent for each driver. See Eq. (10) for assis-ance:

normalized mouth area = mouth areaface area

normalized mouth ratio = mouth ratioface ratio

normalized eye area = eye areaface area

normalized eye ratio = eye ratioface ratio

(10)

.5.1. Creating knowledge based fuzzy inference system (FIS)Once calculated, the eye and mouth parameters (area, ratio)

ssist the fuzzy inference systems (FISs) operating at base level 1,o determine the eye and mouth states respectively. This informa-ion is further employed by the fuzzy inference system operatingt upper hierarchical level (FIS2) to reveal the driver’s state of

igilance. The type of fuzzy inference systems (FIS) used herere Mandani type systems. These fuzzy inference systems, alto-ether build up our proposed Fuzzy Expert System that determinesriver’s fatigue level in real time.

able 3etails of input and output variables used by fuzzy inference system at level 2.

Variable Type Range Labels

Eye state Input [0 100] 3

Mouth state Input [0 100] 3

Fatigue level Output [0 100] 3

Small, medium, large Trapezoidal onlySmall, medium, large Trapezoidal, triangularOpen, half open, closed Trapezoidal, triangular

A Fuzzy Expert System (FES) is an expert system that uses a col-lection of fuzzy membership functions and rules, instead of Booleanlogic, to reason about data. The rules in a Fuzzy Expert System areusually of “If then else form” as described in Section 2.4. The firststep in developing such a knowledge based system is to define thenumber and nature of the variables involved in the diagnosis pro-cess according to the domain expert experience [1]. The variablesproposed after making an appropriate study of our system are EyeStatus, Mouth Status, Fatigue Level, Eye Area, Eye Ratio, Mouth Areaand Mouth Ratio. Since, the proposed system is adaptive in naturewith regards to the user, the ranges of the selected fuzzy inputsare decided after the analysis of first 20 frames during the initial-ization phase. Consequently, the expert system is not affected ifpeople belonging to different ethnicities use it. The range for theoutput variable ‘fatigue level’ is fixed and is manually chosen in therange [0–100]. The normal value of fatigue is maximum 45; whenthe value is in between the range 45 and 75, the fatigue is mediumand it is considered as high if the value exceeds the 75 markup. Inall the variables except for fatigue level, the fuzzy sets used weretriangular and semi trapezoidal at domain edges. In case of fatiguelevel, only trapezoidal fuzzy set was used, with semi trapezoidalversion at its domain edges.

For each of the fuzzy inference systems in hierarchy, the lin-guistic terms used to define each variable are shown in Tables 1–3respectively.

Based on the mentioned variables, experts state different piecesof knowledge (rules) to express the behavior of the system in anunderstandable way. Fuzzy linguistic rules are of form “If condi-tion Then conclusion” where both, the premise and conclusion uselinguistic terms. We have obtained a rule base with 27 rules whichare easily interpretable. According to these rules, eyes and mouthstatus which determines the degree of openness play an impor-tant role in determining the fatigue of the driver. The set of rulesused by fuzzy inference system (FIS) detecting fatigue are given inTable 4. The remaining 18 rules are used by fuzzy inference systemsat base level, where the input variables ‘area’ and ‘ratio’ are used todetermine the eye and mouth status respectively. The defuzzified(or crisp) value of eye and mouth state is then given as an input to

the fuzzy inference system resided at level 2 of hierarchy. The rulebase is also simplified after being defined to achieve compactnessand more interpretability, but without getting worse accuracy ofthe original knowledge base [22].

Linguistic terms Membership functions

Open, half open, closed Trapezoidal, triangularOpen, half open, closed Trapezoidal, triangularLow, medium, high Trapezoidal

T. Azim et al. / Applied Soft Com

Table 4Fuzzy rules used by fuzzy inference system working at level 2.

No. Rules Weight

1 If (eyestate is Open) or (mouthstate is Closed) then (fatigueis Low)

1

2 If (eyestate is Closed) or (mouthstate is Open) then (fatigueis High)

1

3 If (eyestate is Halfopen) or (mouthstate is Halfopen) then(fatigue is High)

1

4 If (eyestate is Open) and (mouthstate is Open) then (fatigueis Medium)

0.65

5 If (eyestate is Open) and (mouthstate is Halfopen) then(fatigue is Low)

1

6 If (eyestate is Closed) and (mouthstate is Closed) then(fatigue is High) (

1

7 If (eyestate is Closed) and (mouthstate is Halfopen) then(fatigue is High)

1

8 If (eyestate is Halfopen) and (mouthstate is Open) then(fatigue is High)

1

f

••••

3

acariocIew

9 If (eyestate is Halfopen) and (mouthstate is Closed) then(fatigue is Medium)

0.65

After defining the variables and rules properly, the process of auzzy inference system involves the following steps:

Fuzzification of Crisp Input ValuesEvaluation of all If-Then Rules in ParallelAggregation of the consequents across the rulesDe-fuzzification (conversion of fuzzy answer into a crisp value)

.6. Driver fatigue monitoring

The degree of eye and mouth openness is constantly monitoredt every fifth frame of the video sequence. The decision of nothoosing every frame for analysis is due to the pace at which vari-tions are expected in every successive frame. Secondly, speed andesponse time of the real time system are also at stake if every frames checked for fatigue. However, when a yawning or closed/halfpen eye state is observed, consecutive frames in the sequence are

hecked instead of every fifth to see for how long the state persists.f the FES observes that driver’s eyes are closed or half open for sev-ral consecutive frames or finds that the person is yawning alongith the closed or half open eyes, it immediately concludes that the

Fig. 11. Video clip of a drowsy driver whose degree of eye and mouth open

puting 18 (2014) 25–38 35

person is feeling fatigue and a warning is generated followed by analarm if the inattentiveness continues. This saves the system fromgenerating false alarm when he/she is talking while driving. Simi-larly, if just closed eyes are observed in several consecutive frames,then the system does not wait for the true yawning response by FIS,rather it immediately rings the alarm assuming the severity of sit-uation. In short, yawning alone cannot correctly predict the stateof the driver; this information must be combined with eye stateanalysis to infer the state of the driver.

4. Experimental results

The proposed system was implemented on Pentium 4, 1.7 GHz,using the Matlab environment. The video data was collected in realdriving scenarios and was recorded in day and night lighting con-ditions by Jesus Nuevo [1]. The input format of the video was RGBwith 400 × 320 size images. The frame rate of the video acquiredfrom camera was 15 frames per second.

We tested the system on different video sequences and its per-formance was found up to the mark unless some feature (face/eye)detection fails, or incorrect segmentation of some feature occurs,i.e. for instance if the segmentation result of an open eye turnsout to be half open in terms of area and ratio, the person mightbe falsely warned. Fig. 11 gives a glimpse of the video clip wherehigh fatigue of the driver is monitored by Fuzzy Expert System. Theobservations of the Fuzzy Expert System for this video are shown inFig. 12. The eye state of the driver is half-open during frames 1–38,57–67 and 80–100. The closed eye state is observed in the framesequence 39–56 and 68–79. The mouth of the driver is found openin the frames 1–20, half-open in the progression 21–34, 52–62 and83–90, and closed during the frames 35–51, 63–82 and 91–100. Asa result of the mouth and eye status information, fatigue is judgedin the range [0 100]. Notice the trend of the fatigue curve at frames2, 12, 19, 42, 60 and 64 where it is approaching to 0. This is becausein these frames face detection does not take place and as a resultfatigue is not evaluated. On the other hand, the frames where mouthis not detected (e.g. 37–41), but eyes are detected and segmented

properly, fatigue can be judged successfully.

Fig. 13 gives another example of a video where the driver isattentive and no fatigue is observed by our proposed Fuzzy ExpertSystem. The observations of the system for this video are shown in

ness are constantly being monitored by an image acquisition device.

36 T. Azim et al. / Applied Soft Computing 18 (2014) 25–38

0

10

20

30

40

50

60

70

80

90

100

1 969186817671666156514641363126211611 6

Deg

ree o

f O

pen

ness

s

Fatigue Detection Based on Mouth and Eye State Analysis

Eye state

Mouth State

Fatigue Level

driver

Feh15Tcmfinst

Frame

Fig. 12. Graph revealing the high fatigue level of

ig. 14. The mouth state of the driver is closed throughout the clipxcept for the frames in progression 81–92, where they are slightlyalf opened. The eye state is monitored as open during the frames–49 and 85–100. Note the peak of the fatigue curve in frames2–54 which is followed by medium and low fatigue inclination.his actually depicts the blinking phenomenon in which the eyeloses and then opens up gradually, causing fatigue from high toedium and then low successively. The fall of the fatigue curve in

fth frame is due to the fact that face is not detected and hence

o fatigue could be judged at this point. In such cases the system,kips the frame under consideration and grabs the next time frameo determine the fatigue level.

Fig. 13. Video clip of a driver who is n

due to critical eye and mouth state observations.

The performance of the fatigue detection system on differentvideos is shown in Table 5. After testing the system on differentvideos, it was found that the system fails to detect the fatiguewhen the driver rotates his head to an extent that face, mouth orpupil detection is missed. The system reinitializes when the drivermoves his head to front. The detection results are not affectedby the exposure of teeth due to the optimal number of clustercheck in spatial fuzzy c-means clustering. Moreover, the selectionof dynamic threshold and preprocessing contrast enhancement

steps assist in segmenting the lip region successfully and is noteffected by the lighting conditions unless the brightness is so severethat no contrast in image is present. In that case the frame under

ot fatigued and blinks normally.

T. Azim et al. / Applied Soft Computing 18 (2014) 25–38 37

0

10

20

30

40

50

60

70

80

90

100

1 999285787164575043362922158

De

gre

e o

f O

pen

ne

ss

Frames

Fatigue Detection Based on Eye and Mouth StateAnal ysi s Eye State

Mouth State

Fatigue Level

Fig. 14. Fatigue evaluation of a driver who

Table 5Fatigue detection results of Fuzzy Expert System on videos of different drivers.

Video 1 Video 2 Video 3

Total frames 100 100 100Open eyes 0 65 70Half open eyes 70 32 20Closed eyes 30 3 10Open mouth 20 0 20Half open mouth 33 12 7Closed mouth 47 88 73Correct warning 100 100 100False warning 0 0 0

ca

5

mtodaararcnwbmitaadutas

• The system also does not detect accurately the fatigue if the yawn-

Tracking failure 6 1 5Precision rate 100% 100% 100%

onsideration is skipped and next frame is checked for yawningnalysis.

. Conclusion

This paper proposes a real time fatigue detection system thatonitors eye closure duration and yawning behavior of the driver

o measure his/her level of inattentiveness. The system was testedn different sequences recorded in day and night lighting con-itions, with different users and yielded an accuracy of 100% onverage. Here, a 100% correct warning rate does not reveal anbsolute robustness of the algorithm; rather it reveals the accu-acy of the algorithm on movies on which it was tested. If during

period of dozing, the system can detect driver’s drowsiness anding an alarm for every challenging video then we consider this as aompletely successful fatigue detection system. The proposed tech-ique is non-intrusive in nature as it does not require the driver toear sensors on his/her body. A research was carried out on a num-

er of algorithms that could best segment out the face, pupils andouth of the driver. A notable feature of the proposed algorithm

s that all the proposed techniques of face, pupil and mouth detec-ion are implementable in real-time: many steps in the proposedpproach are specifically taken to reduce the feature extraction costnd thus improve the response time of the system. For example, theecision of face segmentation prior to pupil and mouth searching,se of Kalman filter, check on every fifth frame instead of consecu-

ive frames for eye and mouth state analysis, selection of fewer butppropriate number of parameters to determine fatigue; all theseteps make the system an ideal candidate for real time use.

is attentive throughout in the video.

The results obtained by the developed system were comparedto the results achieved by a fatigue detection system using sixdifferent parameters (Percent Eye closure (PERCLOS), eye closureduration, blink frequency, nodding frequency, face position, andfixed gaze) [1]. The authors in [1] reported an average accuracyof 98% on the same movies we have performed our experimentson. Our average performance was 100% just with PERCLOS andMouth state parameter whereas they have taken into account sixparameters to determine the vigilance level, thus increasing thecomputational cost and speed of a real time system.

To the best of our knowledge, no one has ever investigated theusage of different face detection techniques for fatigue detectionsystems. We have tested five different notable face detection tech-niques on driver’s videos and keeping in view the limitations of eachapproach, pointed out the most efficient face detection techniquefor real time fatigue detection systems. Another contribution of theapproach is that in driver vigilance and fatigue monitoring systems,mouth has never been detected through an s-FCM approach. Theapproach however was found very satisfactory because the resultswere not affected due to changing light conditions and vibrationsexperienced in real time. The selection of dynamic threshold in s-FCM segments the lip region successfully and is not effected by thelighting conditions unless the brightness is so high that no contrastin the image is present.

6. Future work

The implemented system serves as an extendible foundationfor the development of a completely robust fatigue detection andvigilance monitoring system. There are many future directions fordriver’s fatigue detection system:

• The system works robustly at night time because of the IR illu-minator being used. However, its performance decreases duringdaytime, especially in bright days. A possible solution to this prob-lem could be the incorporation of IR filters in the car’s glasses.

• At the moment, the system does not work if users wear glassesbecause different bright blobs appear in the image due to the IRreflections from glasses. Hence, in future, some work must bedone to resolve this issue.

ing driver puts hand on his/her mouth while yawning (a typicalyawning gesture). The mouth cannot be segmented in that frameand when the driver puts off his hand, the yawning gesture is

3 ft Com

•

•

•

A

wo

R

[

[

[

[

[

[

[

[

[

[

[

[

8 T. Azim et al. / Applied So

over too. Hence, some technique must be incorporated to resolvethis issue.The system should allow the driver to check side mirrors, look atthe radio/speedometer controls and converse easily without theactivation of alarm.Moreover, we should be looking for more efficient visual param-eters that form the right combination with existing cues to detectdriver’s fatigue and vigilance level. It should be remembered thatvery few or large selection of parameters will always require moretime to raise an alarm as the computational complexity to reachat some decision is increased.The system should be improved for more accurate segmenta-tion as wrong detections could lead to false alarms and systemblindness to fatigue.

cknowledgement

We are thankful to Jesus Nuevo for sharing the videos of driversith us. It would have been impossible to complete this effort with-

ut their continuous support.

eferences

[1] L. Bergasa, J. Nuevo, M. Soteli, R. Barea, M. Lopez, Real time system for monitor-ing driver vigilance, IEEE Transactions on Intelligent Transportation Systems 7(1) (2006) 63–77.

[2] Q. Wang, J. Yang, M. Ren, Y. Zheng, Driver fatigue detection: a survey, in: Proc.6th World Congress on Intelligent Control and Automation, Dalian, China, June21–23, 2006.

[3] Q. Ji, X. Yang, Real-time eye, gaze, and face pose tracking for monitoring drivervigilance, Real Time Imaging 8 (October (5)) (2002) 357–377.

[4] S. Wang, C. Chen, R. Chang, Optimizing code size for embedded real-time appli-cations, in: Proc. of Embedded and Ubiquitous Computing (EUC), LNCS 4096,2006, pp. 297–307.

[5] J. Healey, R. Picard, SmartCar: detecting driver stress, in: Proc. 15th Int. Conf.Pattern Recognition, vol. 4, Barcelona, Spain, 2000, pp. 218–221.

[6] A. Kircher, M. Uddman, J. Sandin, Vehicle control and drowsiness, in: SwedishNational Road and Transport Research Institute, Linkoping, Sweden Tech. Rep.VTI-922A, 2002.

[

puting 18 (2014) 25–38

[7] Anon., Perclos and eyetracking: challenge and opportunity, in: TechnicalReport, Applied Science Laboratories, Bedford, MA, 1999.

[8] M. Doering, D. Manstetten, T. Altmueller, U. Lasdstaetter, M. Mahler, Monitor-ing driver drowsiness and stress in a driving simulator, in: Proc. Int. DrivingSymp. Human Factors in Driver Assessment, Training and Vehicle Design,2001.

[9] S. Boveris, Driver fatigue monitoring technologies and future ideas, in: Proc.AWAKE Road Safety Workshop, Balocco, Italy, 2004.

10] M. Khan, A. Mansoor, Real Time Eyes Tracking and Classification for DiverFatigue Detection, vol. 5112, Springer Verlag, 2008, June, pp. 729–738.

11] I. Park, J. Ahn, H. Byun, Efficient measurement of the eye blinking by usingdecision function for intelligent vehicles, in: ICCS 2007, Part IV, LNCS 4490,2007, pp. 546–549.

12] L. Bergasa, J. Nuevo, M. Sotelo, R. Barea, M. Lopez, Real time system for mon-itoring driver vigilance, in: Proc. Intelligent Vehicle Symp., Parma, Italy, 2004,pp. 78–83.

13] J. Batista, A real time driver visual attention monitoring system, in: IbPRIA 2005,LNCS 3522, 2005, pp. 200–208.

14] P. Smith, M. Shah, N. Vitoria, D. Niels, Determining driver visual attentionwith one camera, IEEE Trans. Intell. Transp. Syst. 4 (December (4)) (2003)205–218.

15] H. Ueno, M. Kaneda, M. Tsukino, Development of drowsiness detection system,in: Proc. Vehicle Navigation and Information Systems Conf., Yokohama, Japan,1994, pp. 15–20.

16] T. Azim, M. Jaffar, A. Mirza, Skin detection in luminance images using thresholdtechnique, in: IEEE Proc. International Conference on Telecom Technology andApplications (ICTTA), Manila, Philippines, June 6–8, 2009.

17] T. Azim, M. Jaffar, A. Mirza, Automatic fatigue detection of drivers through realtime eye tracking, in: Proc. International Symposium on Management Engi-neering (ISME), Dalian, China, August 5–7, 2009.

18] T. Azim, M. Jaffar, A. Mirza, Automatic fatigue detection of drivers throughpupil detection and yawning analysis, in: Fourth International Conference onInnovative Computing, Information and Control (ICICIC), Kaohsiung, Taiwan,December 7–9, 2009.

19] T. Azim, M. Jaffar, A. Mirza, Automatic fatigue detection of drivers throughyawning analysis, in: International Conference on Signal Processing, ImageProcessing and Pattern Recognition, Jeju Island, Korea, December 10–12,2009.

20] P. Viola, M. Jones, Robust real time object detection, in: Proc. IEEE ICCV Work-shop on Statistical and Computational Theories of Vision, 2001.

21] P. Viola, M. Jones, Rapid object detection using a boosted cascade of sim-

ple features, in: Proc. IEEE Conf. Computer Vision and Pattern Recognition,2001.

22] J. Alonso, L. Magdalena, S. Guillaume, A simplification process of linguisticknowledge bases, in: Proc. Conf. Eur. Society Fuzzy-Logic and Technology(EUSFLAT), Barcelona, Spain, September, 2005, pp. 1045–1050.