2nd international congress of technology,...

1

Indian Traffic Sign Detection and ClassificationUsing Neural Networks

Arun Nandewal Abhishek Tripathi Satyam ChandrraCSE Department IT Department EEE DepartmentNITK Surathkal NITK Surathkal NITK Surathkal

[email protected] [email protected] [email protected]

Abstract—This paper presents an automatic Indian Road Traf-fic Sign Detection and Classification system based on MultipleNeural Networks. Road safety being an indispensable factor fordriving and smooth traffic movement has been studied sincelong time and traffic signs provide the driver with necessaryinformation and warnings. Being very different in color andshape when compared to natural environment these signs canbe detected easily by humans.

The system proposed in this paper uses four stages.1)Imageprocurement and preprocessing images captured by the camerawhich might be blurred or corrupted due to environmentaldisturbances, we try to remove these by deblurring and en-hancing quality of the image.2)Color segmentation based onRGB ,YCbCr and NTSC color space to detect green blue andred color respectively then morphological operations to removeany unwanted noises that might be detected.3) Blob Detectionusing Binarization and Otsu Thresholding to obtain the regionof interest and shape classification.4) Classification using MultipleNeural Networks to decide the type of sign. From the results wecan conclude that when the neural network is trained over astandard database, the recognition of region of interest has highaccuracy and the proposed methodology works with real timeimages invariant to rotation, illumination and in many situationseven with partially distorted and occluded images.

The proposed method is validated on a standard data setof Indian Traffic Signs. As far as our knowledge is concernedno work has been done on the total data set considered by usalthough few works have been done on restricted data set of theone we considered and the data sets for other countries were alsoconsidered.

Index Terms—Neural Networks, segmentation, morphological,blob detection, thresholding

I. INTRODUCTION

Road safety and traffic maintenance is a very censoriousmodern day issue and a topic delve for research by expertsacross the globe. Everyday occurrence of fatal accidentsresulting in the loss of lives and other resources. Therecan be multiple stimulations leading to these misfortuneslike poor road maintenance, reckless driving, psychologicalstate of driver, casual attitude of pedestrians. Another majorreason leading to this might be the poor law enforcementand impromptu maintenance of road traffic signs. Occludedor deteriorated signs may mislead the driver. Here we proposea system which can detect and classify road signs. This hasfollowing advantages firstly it helps in providing the driverwith driving guidelines and warnings. alternatively it alsohelps is maintenance of smooth running traffic. Indian traffic

signs according to standard database of Delhi police [9] andChandigarh police [10] can be classified as 1) Compulsory2) Regulatory 3) Warning and 4) Informative in addition toSTOP sign and GIVE WAY sign, Figure1. Being distinctivein color and shape from natural environment these can beeasily isolated different type based on Table1. To detect thesesigns we must have information about the color and shape asmentioned in Table1.

Computer Vision based road sign detection and classifica-tion proposed here is a two step process

1) sign detection 2) sign classification.Initiation of detection is based on color segmentation .

Various techniques have been applied for color segmentation[1] uses RGB(Red Green Blue) color space and using colorthresholding with one color as a reference whereas [12] usesthreshold value in YUV(U and V are blueluminance andredluminance) space .Also in [8] uses HSI(Hue SaturationIntensity) threshold system because color information, whichis encoded with hue and saturation components, presents lowvariations for objects of interest with a similar color .Butresolving a threshold value is not very accurate as it maydepend on brightness, atmospheric condition , night vision, varying lighting may produce shadow or bright spots overthe image. Depending on varying environmental constraintsthe proposed algorithms may fail and in case the backgroundmatches with the color being segmented noise might beextracted within the region of interest. For shape matchingsome mechanisms employed are [1] Optimal Corner Detector ,[8] Neural Network and linear Support Vector Machine(SVM).Other possible techniques are [3] FFT (Fast Fourier Trans-form) and [1] circular and triangular contours matching.

The organization of this paper is as follows: Section IIfocuses on exhaustive description of algorithm proposed here,Figure2. After initial image preprocessing the color segmenta-tion is done to extract candidate blobs . Using RGB , YCBCRand NTSC color space green, blue and red color blobs areextracted respectively. By establishing the color first, onlysome geometric shapes are then liable to be considered inthe shape classification stage Figure3. Therefore, those objectswith similar colors as traffic signs, which hence representnoise are discarded from the consideration. Candidate blobsclassified as traffic signs at this stage are converted in form ofinput vector and inured to the first level of Neural Networkwhich concludes the category of the sign and appropriately

2nd International Congress of Technology, Management and Social Sciences-16 (ICTMS-16)

Research Cell : An International Journal of Engineering Sciences, Special Issue ICTMS-16 , 25-26 JUNE 2016 ISSN: 2229-6913 (Print), ISSN: 2320-0332 (Online) © 2016 MID Publications in association with Vidya Publications.

Authors are responsible for any plagiarism issues.

11

2

TABLE IMEANING OF INDIAN TRAFFIC SIGNS BASED ON COLOR AND SHAPE

Color Shape SignRed Triangle WarningRed Circle CompulsaryRed Inverted Triangle Give WayRed Octagon StopBlue Rectangle InformatoryBlue Circle Regulatory

Green Rectangle Informatory

Fig. 1. Indian Traffic Sign Database.

forwards it to the next level for exact classification. Section IIIincludes the result for all the images included in database andmany real time images tested on the system and demonstratesthe robustness of proposed algorithm including cases of someoccluded images. Finally Section IV gives conclusion andscope of future enhancements in the proposed system.

II. INDIAN ROAD TRAFFIC SIGNS

Indian Road Traffic Signs [2] are standardized and pertinentnationwide, these are broadly classified into the followingcategories.

1) Regulatory signs-These signs inform the road users aboutthe laws and regulations they have to follow. Violation of thesesigns is legal offence. They are circular in shape with redcircumference.

2) Compulsory signs-These signs are an extension to reg-ulatory signs and similar to the violation of regulatory signs,violation of these is a legal offence, which makes them most

important signs. They are circular in shape and are filled withblue color and white circumference.

3) Warning signs-These signs warn road users of certainhazardous conditions. They are triangular in shape and possessa red circumference. Violation of these is not a legal offence.

4) Informatory signs - These signs provide informationand guidance to road users. They are rectangular and mayvary in color, in some cases they might be green with whitecircumference whereas in others it might be white filledrectangle with blue circumference.

Further classification of each of these categories is basedon the information contained by these signs which may be apicture, alphanumeric string or a particular direction indicatingarrow.

III. SYSTEM OVERVIEW

Algorithm proposed for Traffic Sign Detection and classifi-cation as exemplified by Figure2 consists of three main stages.

1) Image color segmentation using RGB, YCbCr and NTSCcolor space

2) Blob Detection to attain the Region of Interest3) Classification using Multiple Neural NetworksThe following sections describe these stages. We use ex-

amples to demonstrate the actual viability of the proposedalgorithm on real time images.

A. COLOR SEGMENTATION

Any image present in any color space(RGB, YCbCr, HSI,HSV, NTSC) is defined by a triad of pixels. RGB colorspace being the most intuitive and at hand can be a veryconclusive selection for color segmentation but due to itslarge dependency on illumination any environmental conditionthat has poor lighting might not give very distinctive colorsin the image for segmentation. YCbCr being illuminationindependent can be selected for color segmentation. It is away of encoding RGB color space [7] here Y is the lumacomponent representing the illumination intensity of the imageand CB and CR are the blue-difference and red-differencechroma components which represent how close the pixel valueis to blue and red color component in RGB space. Whenimages taken for detection and classification are extracted fromvideo input, YCbCr color space is distinctively effective as itis used as a part of color image pipeline in videos. HSI or HSVcan be used competently to excerpt different color componentsby adapting to different threshold values associated with eachindividual color[8] but as mentioned, these threshold valuesmay fail depending on changing environmental conditionslike cloudy weather, low brightness, shadow or bright spotgeneration due to light reflection in night.

Red being the most prominent color with blue to be detectedfor successful detection of road signs. It can also be minedfrom NTSC color space, advantage of using NTSC overYCBCR red chroma component is that in some peculiar caseswhere the sign may have a red background which is not easilydistinguishable from the circumference of the sign, whichis also red in color may lead to extraction of noise withthe sign Figure4. Whereas when the sign is extracted from




12

Fig. 2. Color based Shape Classification.

Fig. 3. Comparison of YCbCr and NTSC colorspace used for Red Color Segmentation.

Fig. 4. Noise removal by Morphological operations.

NTSC space the red component is prospected with greaterdetail as in NTSC The first component, luminance, representsgrayscale information, while the last two components makeup chrominance (color information).Using the latter the pixelswhich actually appear red but have difference in their valuescan be distinguished and only the sign can be attained withoutnoise Figure4. So in the system proposed here we need tosegment three possible colors red green and blue to incorporatethe entire database of Indian Traffic Signs. After examinationthe culminating segmentation used is Green color signs fromRGB, Blue color signs from YCbCr and Red color signs fromNTSC color space Figure5.This segmentation is done basedon Otsu Thresholding which automatically perform clustering-based image thresholding [13].This gives output as an imagewith segmented color components. It is possible that some

noise may also be included with the sign being extracted whichmight be due to the constitution of pixels of same color orwithin threshold range in the background. To remove any suchsmall noises induced in image fundamental morphologicaloperations of Erosion and Dilation are performed.Figure5Erosion and Dilation thus performed with a line structuringelement also give finer and halcyon edges which increasesefficiency of shape classification.

B. BLOB DETECTION

With the input image as the output of color segmentationfollowed by morphological operations to remove any form ofnoise that may be comprehended as region of interest. Pro-ceeding towards the acquisition of region of interest. [11] Thegray scale image obtained is converted to a binary image usingstandard Otsu Thresholding for binarization. Then inhibitingthe intensity value of pixels and constraining the allowedregions to only connected components which is obtained byreconnoitering the adjacent pixels if the value in binary imagefor the neighbor is 1 it is considered to be connected and when0 disconnected. Thus all the connected components in binaryimage are obtained and are labelled with different colors andnumbers serving in distinguishing of all regions thus obtained.Using the MATLAB [5] functions which measure the area,location of centroid and intensity of connected components




13

Fig. 5. Neural Network Classification for each Level.

Fig. 6. Multiple Sign detection simultaneously.

Fig. 7. Disjoint components extracted in case of Prohibitory Signs.

in a binary image. The individual blobs can be excerpted andconsidered for shape classification. But all the candidate blobsare not to be tested for shape classification, the efficiency ofsystem can be increased in the two following ways such thatany unnecessary blobs which might be noise are not taken intoconsideration.

1) Confine the area, length and breadth of Blob2) Avoid overlapping blobsAs mentioned due to presence of background which might

be same in color as that being segmented some noise may beincurred within the image which is input to the Blob Detectionand as a result might also be present in the binary formof image and hence the extracted region of interest.Figure6such noises which are reported as a candidate blob may beremoved by constraining the area of blobs which are to beconsidered. As the traffic signs being considered at any timecan only occupy a fixed ratio of image the parameters for areaof the blob can be established also the candidate blobs withthe traffic sign are mostly squares or have the length to breadthratio within well defined limits. So using these as secondaryconditions only those blobs which actually contain the trafficsigns remain as candidate blobs and rest are removed beingconsidered as noise. Also as discussed earlier it might occurthat the background is similar in color to the segmented colorand overlaps with the sign being detected. In such case eitherthe sign may be detected as a candidate blob with or withoutnoise so if a conditional statement to check that no overlappingregion is included as a candidate blob then the candidate blobbeing considered are only the once which are needed.

After the candidate blobs are determined they are used asinput for shape classification. Initial shape classification isbasic template matching used by [2] using a standard database




14

Fig. 8. Blob extraction with measurement constraints.

Fig. 9. Different types of Informatory Signs.

which has the following shapes. Triangle, octagon, circle,inverted triangle and rectangle these are grouped into fiveclasses and each class has various similar shapes. To abrogatethe effect of size and rotation which might be present due toenvironmental conditions which may displace the sign or inthe image captured the size of sign may be small or large ascompared to the template in database, each class has variouspossible sizes of each shape and each shape is also savedin database after being rotated by a 10 degree measurementfrom 0 to 180 degree so that rotation of the sign doesnt causethe system to give inaccurate results. Formerly the blob hasbeen identified and needs to be shape classified but only someshapes are credible and must be contested as portrayed byFigure3. Those candidate blobs which are correctly classifiedaccording to color segmentation and shape classification andabide by the rules established in Figure3 are region of interestfor Neutral Network classification.

C. NEURAL NETWORK CLASSIFICATION

Region of Interest (RoI) procured is the input for NeuralNetwork (NN) classification [6]. The classification is a multi-ple NN approach and each level is as described below.

1) Level I. This level classifies the RoI as warning orregulatory or compulsory or informative sign.

2) Level II This level classifies the RoI as prohibitory ornon-prohibitory sign if the RoI is classified as compulsorysign after Level I.

3) Level III This level classifies the RoI as arrow or vehicleor human or others sign if the RoI is classified as compulsorysign or warning sign or regulatory sign after Level I.

TABLE IISTANDARD DATABASE IMAGES CLASSIFICATION

Type Total Color and Shape Level I Level II Level IIIProhibitory 56 56 54 54 51

Non Prohibitory 22 22 21 21 19Warning 67 66 64 - 60

Informatory 12 11 10 - -

4) Separate NN has been used for the signs stop and giveway.

The overview of NN classification is as shown in Figure7.NN implementation database is divided into three phases.

Learning, validation and testing. This has been carried outusing the MATLAB functions which significantly reduce thetime for neural network training, validation and testing. Theinput vector and target vector differ for the different levelsof NN. The RoI is resized to image of 25*25 pixels whichis then subdivided into 25 equal zones by doing this we get25 pixels in each zone which can be averaged to get a singlevalue for sample vector which needs to be tested. In trainingthe network, a similar procedure is followed for all the imagesin the database which constitutes images of all Indian TrafficSigns with a rotation of each image also stored with variationof angle of rotation from 0 to 180 degrees with an incrementalfactor of 10 degrees.

In such a case the input vector is of size 25*NUMBEROF IMAGES IN DATABASE*18. It is assumed that thereare Q input vector/target vector pairs. Each target vector hasK elements. One of these elements is 1 and the rest are 0.Thus, each input vector is associated with one of K classes.So the target vector is NUMBER OF CLASSES* NUMBEROF IMAGES*18 [4] . Individually Level I has 4 classes LevelII has 2 classes and Level III has 4 classes and independentNN has 2 classes one for give way sign and one for stop sign.Neural systems solves problem by adapting to the nature of thedata signals they receive. Here forward feed backpropagationwith MLP training is used in proposed system [6].The NNdata set is divided as 90

For prohibitory signs the input vector is 50*NUMBER OFIMAGES IN DATABASE*18 as there is loss of informationdue to overlapping red channel and the extracted informationconstitutes two different parts Figure8 using which two dif-ferent vectors are generated and then vertically concatenated.Similar procedure is followed for training the level III NNfor prohibitory sign. Eventually after the NN gives output andhence classifying the input sample vector as per Figure7 theinner part of each RoI can be extracted and input to a separateNN for individual classification of its type.

IV. RESULT

The proposed algorithm is experimentally tested on a im-ages incorporated within a standard database, real time imagesand video frames.

The results thus generated are summarized in Table2 andTable3. The frames excerpted from the video are also usedas real time images. The video is recorded with a mobilecamera 1920x1080. The real time images used for testing




15

Fig. 10. Multiple Disjoint components extracted in case of extraction of information within a sign.

TABLE IIIREAL TIME IMAGES CLASSIFICATION



Informatory 19 13 - -

TABLE IVPERFORMANCE OF PROPOSED SYSTEM WITH OCCULDED IMAGES



Informatory 6 4 2 - -

Fig. 11. Occluded Images Successfully Classified

vary in size 640x480, 1920x1080, 2592x1944, 3920x2204and 3920x2940.Figure12 illustrates the working of proposedalgorithm with partially occluded images.

The implementation of NN classification for different levelsis undergoing research to obtain the average time for classifi-

cation and misclassification of signs.

V. CONCLUSIONThis paper has described a complete method to detect and

classify the Indian Road traffic Signs. The algorithm proposedhas three main parts, the color segmentation, blob detectionand NN classification taking into consideration many outdoorenvironment difficulties like poor lighting and occlusion ofsigns.

The improvements are successful study of images withbackground similar to color being segmented, images withenvironmental distortions like poor lighting, partially occludedimages and introduction of Multiple NN for classification withclassification of STOP and GIVE WAY signs also. Multiplesigns present at same instant are also detected successfullyFigure10.

Experimental results indicate the proposed methodologyis efficient and accurate. Working with a standard databasemultiple real time images and images taken from a videosequence the algorithm gives result with high accuracy. Itallows us to detect different signs invariant to the rotation




16

7

and change of orientation. Some future work that can furtherenhance the robustness of system are as mentioned.

1) Informatory signs contain information which is notcorrectly classified due to its complexity. They may representvery distinctive information from distance of a particular placeto images of a hospital ahead. Figure9

2) When extracting information symbol within a sign blobdetection may result in multiple components wherever theseare not connected. Figure11

3) Images taken from night vision camera do not haveuniform illumination. Any shadow or bright spots generatedover the sign lead to misclassification of sign.

4) Real time implementation requires more robust systemwhich has reduced proceeding time. The classified sign mustbe associated with an audio signal or text information whichcan be the output to the driver for assistance in safe andautomated driving system.

REFERENCES

[1] A. De La Escalera, L. E. Moreno, M. A. Salichs, and J. M. Armingol.Road traffic sign detection and classification. Industrial Electronics,IEEE Transactions on, 44(6):848859, 1997.

[2] H. N. Dean and K. Jabir. Real time detection and recognition of indiantraffic signs using matlab.

[3] P. Gil-Jimenez, S. Lafuente-Arroyo,H. Gomez-Moreno, F. Lopez-Ferreras, and S. Maldonado-Bascon. Traffic sign shape classificationevaluation. part ii. fft applied to the signature of blobs. In IntelligentVehicles Symposium, 2005. Proceedings. IEEE, pages 607612. IEEE,2005.

[4] R. C. Gonzales and R. E. Woods. Digital image processing, 2-nd edition,2002.

[5] R. C. Gonzalez, R. E. Woods, and S. L. Eddins. Digital image usingmatlab processing. Person Prentice Hall, pages 183193, 2004.

[6] M. T. Hagan, H. B. Demuth, M. H. Beale, et al.Neural network design,volume 1. Pws Boston, 1996.

[7] K. Jack. Video demystified: a handbook for the digital engineer. Newnes,2005.

[8] S. Maldonado-Bascon, S. Lafuente-Arroyo,P. Gil-Jimenez, H. Gomez-Moreno, and F. Lopez-Ferreras. Road-sign detection and recognitionbased on support vector machines. Intelligent Transportation Systems,IEEE Transactions on, 8(2):264278, 2007.

[9] Indian Traffic Police Standard Division Delhi[10] Indian Traffic Police Database Division Chandigarh[11] MathLabcentral,image segmentation+blobs detection[12] J. Miura, T. Kanda, and Y. Shirai. An active vision system for real-

time traffic sign recognition. In Intelligent Transportation Systems, 2000.Proceedings. 2000 IEEE, pages 5257. IEEE, 2000.

[13] C. Yu, C. Dian-ren, Y. Xu, and C. Lei. Otsus thresholding methodbased on gray level-gradient two-dimensional histogram. In Informaticsin Control, Automation and Robotics (CAR), 2010 2nd InternationalAsia Conference on, volume 3, pages 282285. IEEE, 2010.




17

ANKUR RANA

Typewritten Text

ANKUR RANA

Typewritten Text

ANKUR RANA

Typewritten Text

2nd international congress of technology,...

Documents