CSB657Spring2016FinalProjectReportUsernames:

1.AdithyaNagarajTirumale(aditnaga)2.AkashRamGopal(agopal)3.SumitKumarDey(skdey)

Introduction:Nowadays,thereisalotofattentionbeinggiventotheabilityofthecartodriveitself.Oneofthemanyimportantaspectsforaselfdrivingcaristheabilityforittodetecttrafficsignsinordertoprovidesafetyandsecurityforthepeoplenotonlyinsidethecarbutalsooutsideofit.Thetrafficenvironmentconsistsofdifferentaspectswhosemainpurposeistoregulateflowoftraffic, make sure each driver is adhering to the rules so as to provide a safe and secureenvironmenttoallthepartiesconcerned. WehavefocusedourprojectontheUStrafficsignsandafewofthetrafficsignswhichwehaveinourdatasetisasshowninthefigurebelow.WeusedtheLISAtrafficsigndataset[3].Thedatasetconsistedof48differenttypesofUStrafficsings.About75%oftheframeswereingrayscaleandtherestincolor. The problemwe are trying to solve has some advantages such as traffic signs being uniquethereby resulting in object variations being small and traffic signs are clearly visible to thedriver/system[1].Theothersideofthecoinisthatwehavetocontendwithlightingandweatherconditions[1]. Themainobjectiveofourprojectistodesignandconstructacomputerbasedsystemwhichcanautomaticallydetecttheroadsignssoastoprovideassistancetotheuserorthemachinesothattheycan takeappropriateactions.Theproposedapproachconsistsofbuildingamodelusingconvolutionalneuralnetworksbyextractingtrafficsignsfromanimageusingcolorinformation.Wehaveusedconvolutionalneuralnetworks(CNN)toclassifythetrafficsignsandweusedcolorbasedsegmentationtoextract/cropsignsfromimages.Backgroundandrelatedwork:Manydifferenttechniqueshavebeenappliedtodetecttrafficsigns.MostofthesetechniquesarebasedonusingHOGandSIFTfeatures.In our approachweuse biologically inspired convolutional neural networks to build amodelwhichcanpredictthetypeoftrafficsign.Onesuchrelatedworkbasedonconvolutionalneural

networksispublishedin‘TrafficSignRecognitionwithMulti-ScaleConvolutionalNetworks’byPierreSermanetandYannLeCun[4].Methods:Theproblemoftrafficsignrecognitionistwofold:

1) Extractingapotentialtrafficsignfromanimage.Traffic signs are designed such that they appear unique and easily identifiable to thehumaneye.TrafficsignsintheUnitedStatesofAmericaareof3maincolors:Red,White,and Yellow. Other colors like orange and blue are also used. In our approach weconcentrateonRed,White,andYellowtrafficsigns.Sincethecolorofatrafficsignisuniqueinabackgroundwecanusethecolorinformationtonarrowdownourareasofinterest(partspotentiallycontainingthetrafficsign).Since RGB colored images are susceptible to variations in lighting, we use HSV (Hue,Saturation,andVariation)images.OncewehavetheHSVimageournextgoalistodefineourareasofinterest(i.e.rangeofYellow,RedandWhite)sothatwecansegmentourHSVimagebasedonthese3colors.Thecolorrangesusedareasfollows:

Color LowerRange(HSV) UpperRange(HSV)Yellow ([10,50,50]) ([30,255,255])Red ([170,50,50]) ([185,255,255])White ([0,0,50]) ([120,15,255])

Thenextstepistousethesecolorrangesandcreatebinarymasksforeachofthe3colors.ForExample,theredbinarymaskwillhave0assignedtoalltheregionswhicharenotintheredrangeand1assignedtoallregionswhichareintheredrange.TheRed,Yellowandwhitebinarymasksforanimageareshownbelow:

Asseenfromtheaboveexamplethetheoriginalimageissegmentedbasedoncolors.Weknowthattrafficsignsareusuallyoccur indifferentclosedshaped likerectangles,triangles,diamondsetc.Wecanusethispropertytoextractclosedshapedfromeachofthe3binarymasks.Thiscanbedonebyusing‘TopologicalStructuralAnalysisofDigitizedBinaryImagesbyBorder’[5].WeusedtheOpenCVimplementationoftisalgorithm[6].

Theextractedcontoursfromthebinarymasksareasfollows:

Aswecanseefromtheseimageswehavenarroweddowntheareasofinterestfromtheentireimage.Theseareasofinterestarefurtherrefinedbasedonthesizeofthecontourtoreducetheareasofinterest.

Once we have refined the set of areas of interest, we use the convolutional neuralnetworkwhichwearegoingtobuildinthenextsteptopredictthetypeofthissign(orifitisnotasign).

2) PredictingthetypeofExtractedtrafficsign.Fromtheextractedareasofinterestsinthepreviousstepwewanttodetermineifitisasignornotandifitisasignwewishtoknowwhatthetypeofsignitactuallyis.Forthispurpose,wecantrainaconvolutionalneuralnetwork.Thedatausedtotrainandtest the CNN was obtained from http://cvrr.ucsd.edu/LISA/lisa-traffic-sign-dataset.html.Ithadabout6000framesand49differenttypesoftrafficsigns.Foreachframe,thecoordinatepositionsforthetrafficsignintheimagewasgiven.FromthesepositionsthetrafficsignswerecroppedouttousefortrainingtheCNN.

ACNNisbasicallyinspiredbytheconnectionsbetweentheneuronsinthevisualcortexofanimals.[7]Sincetrafficsignshaveuniqueshapesinsidethemlikearrows,words,circlesand so on. It is useful to convert the traffic sign into amore useful form by using aLaplacian operation on the traffic sign. We can apply the Laplacian operation byconvolvingthefollowingkernelontheinputimage:

0 -1 0

-1 4 -1

0 -1 0

ConsiderthefollowingtrafficsignanditsLaplacian:

TheLaplacianisnowfedintotheCNNwhosearchitectureinshownbelow:

5x5Convolution

2x2Convolution

800FullyConnected

256FullyConnected

49Softmaxunits

ThelearningrateusedtotraintheCNNwas0.001andthemomentumusedwas0.9.TheCNNwastrainedfor200iterations(magicnumbers).OncetheCNNhasbeentrained,itisusedtopredictthesignofthecontoursobtainedinstep1.EachofthesecontoursareassignedthesignwiththemaximumprobabilitywhichistheoutputoftheCNN.Followingistheexampleimagewhichshowsthepredictedsignsforallthecontours:

WecanalsousethetrainedCNNtogettheAccuracy,Precision,RecallandF1scoremetricsonthetestset.Theseresultsarediscussedinthenextsection.Results:ThefollowingtablegivestheAccuracy,Precision,RecallandF1scoremetricsonthetestset.Thetestsetwasobtainedbysplittingthewholedatasetinto70%traindataand30%validationandtestdata.Outofthe30%validation,15%wasthetestdata.

Thefollowingsresultstakeintoconsiderationthetrafficsignthatisperfectlycroppedfromtheimage.Thismaynotbetruewhenweareextractingtrafficsignsfromtheimagewithoutthepriorknowledgeoftheirposition.

Metric ScoreAccuracy 86.9%Precision 0.8638Recall 0.8694

F1Score 0.8633Conclusion:FromthefollowingresultswecanseethattheCNNisdoingagoodjobinclassifyingdifferenttypesoftrafficsignswhentheextractedsignsarecroppedperfectlyfromtheimage.Ourapproachfailstogivegoodresultswhentheextractedsignsfromtestimagesarecroppedincorrectly.Anotherdrawbackofourapproach isthatwhenthecolorofthetrafficsignsvarywhichmaybeduetobadweatherconditionsandpoorcameraquality,theimagemasksobtainedarenotperfectandhencethesignsarenotdetectedproperly.Future improvements can bemade for extracting signs from test images by using advancedsegmentationmethodsReferences:[1]https://bartlab.org/Dr.%20Jackrit's%20Papers/ney/3.KRS036_Final_Submission.pdf[2]http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.695.3606&rep=rep1&type=pdf[3]http://cvrr.ucsd.edu/LISA/lisa-traffic-sign-dataset.html[4]http://yann.lecun.com/exdb/publis/pdf/sermanet-ijcnn-11.pdf[5] Suzuki,S.andAbe,K.,TopologicalStructuralAnalysisofDigitizedBinaryImagesbyBorderFollowing.CVGIP301,pp32-46(1985)[6]http://docs.opencv.org/2.4/doc/tutorials/imgproc/shapedescriptors/find_contours/find_contours.html[7]https://en.wikipedia.org/wiki/Convolutional_neural_network