8/7/2019 ICM1998

1/5

A Robsut Neural Network Multi-LaneRecognition System

SystemSCARF 1-31

ALVINN [M IRBFN U]Bristol [8,9]

A m er K. Dawoud', S a l a h G. Foda* and A h m ad S. Tolba**

low level medium high levellevelprobability *** match filter

back prop.radial basis

*****t** ***

pixel voting *** match filter

* Department of Electrical and Computer Engineering,** Department of Physics,Kuwait University, P.O. Box 5969, Safat 13060,KuwaitE-mail: sfoda@eng.kuniv.edu.kw

AbsZruct: In this paper, the design and neural imp-lementation of a vision based multi-lane highwaylane recognition system are presented. The designobjective of the system is to recognize the lane whicha test vehicle is currently driving through bydetermining its left and right lane boundaries. Whenthe proposed lane recognition system was tested itshowed very high percentage of correct results in verydifficult circumstances which suggests that itprovides the basis for a reliable road followingsystem.

I. IntroductionFor the past decade many institutes and researchcenters have been conducting research on visionbased guidance systems for the purpose ofautonomously navigating land vehicles [1-91. Thebuilding of such intelligen t systems is a complexand challenging task but with huge potentials inboth civil and m ilitary application domains.Autonomow land vehicle navigation system is asystem that can carry out tasks carried out by ahuman driver. These tasks include road following,speed control and tactical driving tasks such asobstacle avoidance, changing lane and negotiatingintersections. There are practical difficultiesrelated to the implementation of autonomousnavigation of an unmanned vehicle in the normalroads. Howev er, road following systems seem thesuitable and practical implementation where theroad following system can be used to monitor thevehicle's lateral position inside the road and todetect any unintended road departure .We believe that any real progress towardsthe implementation of vehicle navigation orroad following systems in normal roadsshould start with developing the recognitionsystem that could robustly produce correctjudgments regarding the posit ion of th e

vehicle. It should not show any tendency tobreak down when the environmentalvariables such as road width, road condit ionsand light conditions changes, and in thepresence of external noise such as nearbypassing vehicle.

Tab le 1: Existing navigation systems:11. Design of Lane Recognition System

The highway has been chosen as ourenvironment in this work because it is lesscomplicated than other road environments.The components of road environment includeonly asphalt road, lane markings, vehiclesand non-road surrounding areas l ike greenplants or sand areas. It does not includeintersections nor stop signalsThe expected result from the proposedrecognition system is to identify the left and

8/7/2019 ICM1998

2/5

right boundary lines of the lane that thevehicle is currently driving through.In the proposed lane recognition system, wehave chosen to follow the same steps of lanerecognit ion by human driver using neuralnetworks. It relies on image processing atall the three levels. The lowest level ofrecognition process involves classifying theinput image pixels into road and non-roadusing Learning Vector Quantizationtechniques. The medium level of recognitionprocess involves scanning the blocks of thebinary image resulting from the classificationprocess searching for the lane markings.Classical Back Propagation network wastrained fo r this purpose. Fig.1 shows themodules of the proposed lane recognitionsystem.

input color image+-Ie rocessinP=+---l[road and non-road ]

lane markingdetectionv

lane boundarydetermination

Fig.]: Block diagram for the proposed lanerecognition systemA S O N Y CCD video camera recorder is usedto captu re video recordings that are saved inthe form of successive frames. Thepreprocessing module involves twooperations: Light variation compensation andImage s ize reduct ion . The i w g e reductionreplaces a block of pixels in the originalimage with one pixel in the reduced image.It was empiricalJy round that the best resultsfor ima ge reddction is obtained by using sub-sampling and averaging.

111.Road and Non-road Classification NetworkThis part of the highway lane detection systemrepresents the low level of processing where eachpixel of the preprocessed color is classified intoeither road or non-road. The input space of thepreprocessed color image is of dimension 3, andeach pixel is represented by a vector in a hypercube of r ed green and blue components.Building a robust vision system capable ofrecognizing highway lane in many differentconditions requires the diversity of thehighway image scenes used to derive the roadand non-road color models required for theclassificatior.. The diversity of thes e highwayscenes is assured by taking intcconsideration the main factors that woulddifferentiate an image from another in ahighway environment. These main factorsare:(a) C olor of the highway asph alt,(b) Presence or absence of external noise of(c) Shape of the white lane markings whichcan be solid long line or dash ed line,(d) Nature of non-road area surrounding th ehighway which can widely vary.

nearby passing vehicle,

Many highway video recordings were madeavailable and from these recordings we haveselected 250 images that represent a wide varietyof highway scenes. Then each image ispreprocessed by light variation compensation,then sub-samptng and averaging 2x2 block ofpixels into one pixel in the reduced image asshown in the previous chapter. From each of theseimages 4 pixels were selected to represent roadinput vectors. The total number of road inputvectors is 1000. The same was repeated for non-road areas. Rozd input vectors included asphaltroad areas only, whiie the non-road input vectorsincluded all elements of highway environmentexcept the asphalt road such as:(1) Highway surrounding areas like green plantsareas and desert areas and the blue sky,(2)White highway lane markings,(c) Barriers between the two sides of the highway,(d) Obstacles like nearby passing vehicles ofdifferent colors.

8/7/2019 ICM1998

3/5

180%wmLer 14 -16 1998

Using Matlab Neural N etwork Toolbox, learningVector Quantization GVQ) [lo-121 w a s utilizedfor the purpose of optimizing the location of theweight vectors. After training is completed, theresulting LVQ network c an be used to clas sify thehighway road images.Each pixel of the preprocessed image is classifiedinto either road or non-road by presenting it to theLVQ network of Fig. 2 and the output will beeither [0 13 for road or [l 01 for non-road. Thesize of binary output image resulting from thisstage is 80x60 pixels.

Input Competitive ayer Linear layer Outputn l

Fig. 2: LVQ Classifying Network

Fig. 3: Input image and the classification binaryoutput.

Fig. 4: Road classifcation for highway imagewhich includes a dark vehicle

IV. Highway LaneDetectionNetworkThis part of lane recognition system representsthe mediumlevel of processing. The binary imageresulting from classification stage is scannedsearching for t he highway lane m arkings. M e d i mlevel means that blocks of pixel a r eas are beingprocessed not single pixel as in the previous lowlevel classification stage .To locate the boundaries of a highway lane weneed first to find the positions of the white lanemarkings which were classified as non-road areasinside road areas. These lane markings could bedefmed as:- white non-road areas surrounded byroad areas from both sides and have certainshapes. The shape of the lane marking could bealso used as a feature that distinguishes it fromother non-road areas such as nearby passingvehicles.The factors that contribute in variation of theshape and size of the white lane marking are:(1) The type of the lane marking whether solidwhite or dashed line,(2) The size of the lane marking changes withchange of the position of the marking with respectto input camera. The closer the lane markings tothe camera the larger they will appear in theimage,(3) The orientation of the lane rrxking withrespect to the input camera. Lane markingslocated on the left of input camera will havedifferent orientation compared to lane markingslocated on the right on input camera.Using a supervised neural network algorithm, theproposed system was trained to detect varioustypes of lane markings. The inputs to the lanemarking detection system a re blocks of the binarypixel image resulting from classlfylng thepreprocessed image into road and non-road. Theoutput is a binary number 0 r 1 indicating theabsence or presence of lane markmgs in thatblock. With this direct and defrnite input\outputrelationship the best choice was to use back-propagation Deural network algorithm withLevenberg-Marquardt approximation techniquefor updating the w eights and biases.

8/7/2019 ICM1998

4/5

The first stage of the lane detection system is tobuild models for the shapes of lane markings thatneed to be recognized.

Fig. 5(a):(b):(c):

Fig. 5(a)Fig. 5(b)non-road.

C

Original input imageBinary classification5x2 lane marking modelshows an original color image, whileshows the classification into road andFig. 5(c) shows a lane marking model,a box of white non-road area surrounded by roadarea from both sides. Since close lane marking tothe input camera have a width of 3 pixels, whilefar ones had a width of only one pixel.( 1 pixel =2x2 pixels of original image), the width of theblock model was chosen to be 5 pixels so that itcan fit all types of lane marking in the m idd e ofthe model plus road pixels from left and rightsides.

The architecture of the used back-propagationnetwork [13,14] is chosen as follows:a) The number of inputs to the network = 10,which represents the size of the 2x5 models inFig. 5(c). b) The number of neurons in the outputlayer is one, indicating the presence or absencelane marking.c) The number of neurons in the hidden layer waschosen to be 8. This number of hidden layerneuron was enough to reach the desired learningaccuracy.d) Log sigmoidal activation functions in hiddenand output layers neurons were chosen.V. Designof Lane Marking Detection NetworkLane markings could be defined as white non-roadareas surrounded by road areas from both sides.

Assuming that the trained feed foreword networkcould detect lane markings shapes of non-roadareas surrounded by road areas, then to satisfy thedefinition of lane markings we must make surethat non-road areas are white colored.In the previou:; section, the non-road color spacewas divided into 6 clusters, three of themrepresented light colors and the other threerepresented dark colors. The white color of lanemarking of course belongs to the light coloredclusters. Which means that all non-road pixels in5x2 window should belong to light coloredclusters. And if any of non-road pixels belongedto dark colored ciuster then that window will begiven output of 0 indicating the absence on lanemarking in that window.

To detect white lane marking, a 5x2 window willscan the binary image resulting from the road andnon-road classification. This window will be theinput to the back-propagation network that willhighlight the positions in the image which matchany of the 42 models of lane marking. Thiswindow will move with one pixel increment at atime.The white color detection neural ndwork checksclassification clusters of 5x2 window pixels and ifany non-road pixel does not bejong to lightcolored clusters, then the output of the trainedfeed-foreword will be forced to be 0 indicating theabsence of liane marking in that windowregardless the output of trained feed-forewordnetwork.

binaryclassificationinnge

Road classifisd pixe!Non-rozdclassified pixelDetectedlane marking

Fig. 6: Lane markingdetection system.

8/7/2019 ICM1998

5/5

Fig. 7: Examples of

References:[11 V. Graefe and K.D. Kuhnert Vision-basedautonomous road vehicles, In Vision-basedvehicle guidance, Masaki I.(eds), Springer-Verlag,NY, 992. .[2]J.D. risman and C.E. Thorpe, SCARF: Acolor vision system that tracks roads andintersection, IEEE Tram. on robotics andautomation, 1993.[3] M. Meng and A.C. Kak, Fast vision-guidedmobile robot navigation using neural networks,IEEE Proc. of system, Man and Cybernetic,1992.

original images and lane-marking detection output.VI. Conclusions and Future Work

The developed lane recognition systemperforms a task that originally is carried outby human driver. It can provide a skeletonfor future work in two main directionsrelated to active areas of ArtificialIntelligence a s follows:(a) real time implementation of lanerecognition system which involves theinterface with input color camera andbuilding dedicated hard war e that satisfiesthe real time requirements of processing eachframe.(b) The development of road following or lanekeeping system that utilizes the proposed fanerecognition system in finding the boundaries ofthe current lane and then guide the vehicle to themiddle of the lane. The development of lanekeeping system involves the genera tion of steeringangle decision, steering control mechanism andthe feedbackfrom the vehicle steering mechanismThe highest level of recognition processinvolves selecting the markings whichbelongs to the current lane and isolatingthem from the total detected lane markingset. This ca n be don e using Fuzzy logicwhich is the subject of current research.

[4] D.A. Pemerleau, Rapidly adapting artificialneural networks for autonomous navigation,Advances in neural information processing,Morgan Kaufrnann Publiihers, 1991.[5 ]T. Jochem and D. Pomerleau, Life in the FastLane: The Evolution of an Adaptive VehicleControl System,AI wgaz ine , 1996.[6] M. Rosenblum and L. Davis, An ImprovedRadial Basis Functiori Network for VisualAutonomous Road Following, IEEE Trans. onNeural Networks, 1996.[7] K. Dowling et a l., Nav lab: An autonomousnavigation testbed in Vision and Navigation:The Carnegie Mellon Navlab, C . Thorpe (Ed.),Norwell, MA: Kluwer, 1990.181 F.W.J. Gibbs and B.T. homas, The fusionof multiple image analysis algorithms for robotroad following, IEE Fij3h IntemtionalConference on Image Processing and it sApplications, 1995.[93 S. Spiegle, Autonomous road vehiclenavigation, Proc. Electronic Technology to theyear 2000, 1995.[ lo ] Y. inde, A. Buzo, and R. M. Gray, Analgorithm for vector quantizer design, IEEETrans. Commun., 1980.[113 J. Bryant, On the clustering of m ulti-dimensional pictorid data, Pattern Recognition,1979.[I21 T. Kohonen, Self-organizing andAssociative Memory, Spinger-Verlag, 1989.[13] M . H. Hasson, Fundamentals of ArtificialNeural Networks The MlTpress, 1995.Cl41 H. Demuth and M. Beale, Neural NetworkTOOLBOX for use with Matlab, The MathWorks Inc, 1994.