A Vehicle License Plate Recognition System Based on Analysis of Maximally Stable Extremal Regions

Bo Li, Bin Tian, Qingming Yao*, Kunfeng Wang State Key Laboratory of Management and Control for Complex Systems

Beijing Engineering Research Center of Intelligent Systems and Technology Institute of Automation, Chinese Academy of Sciences

Beijing, China Abstract—Vehicle License Plate Recognition (VLPR) system is a core module in Intelligent Transportation Systems (ITS). In this paper, a VLPR system is proposed. Considering that license plate localization is the most important and difficult part in VLPR system, we present an effective license plate localization method based on analysis of Maximally Stable Extremal Region (MSER) features. Firstly, MSER detector is utilized to extract candidate character regions. Secondly, the exact locations of license plates are inferred according to the arrangement of characters in standard license plates. The advantage of this license plate localization method is that less assumption of environmental illumination, weather and other conditions is made. After license plate localization, we continue to recognize the license plate characters and color to complete the whole VLPR system. Finally, the proposed VLPR system is tested on our own collected dataset. The experimental results show the availability and effectiveness of our VLPR system in locating and recognizing all the explicit license plates in an image.

Keywords-license plate detection; license plate recognition; intelligent transportation systems

I. INTRODUCTION Research of Intelligent Transportation Systems (ITS) has

become a worldwide hot topic. With the rapid development of computer vision and pattern recognition, more and more vision-based sensors are applied in ITS to monitor the traffic scenes and collect traffic data for traffic control and management. License plate number is a unique identification for vehicles, so the vision-based vehicle license plate recognition (VLPR) system become a core module in ITS. The VLPR systems can be widely used in many applications such as urban road and freeway surveillance, electronic toll collection, law enforcement, etc.

The VLPR algorithms are generally composed of following three steps: (1) license plate localization to find the image areas which are related to the license plates; (2) plate character segmentation to segment the license plate image area into character images, and (3) plate character recognition to recognize the characters using the optical character recognition technique. The first two steps are mainly image processing techniques and the third step belongs to the pattern recognition problem. Among these three steps, the first stage is the foundation of the other two. When localizing license plate areas in images, many natural aspects (such as environmental illumination, weather, scale and rotation angle of license plates) must be taken into account, thus license plate localization is

usually thought as the most important and difficult module in VLPR.

Regarding the license plate localization, many methods have been proposed in literature [1]. Some are based on the dense texture characteristic of license plate [2]-[4]. They make an intuitive assumption that license plate has dense edges, so edge statistic and mathematic morphologic methods could be used. These methods may get good results in relatively simple environments, but they are not capable of dealing with complex situations and high definition images and often get high false alarm rates in some cases. Some methods are based on color of the license plate or fuse color information with other cues [5][6]. Fusing color information helps to get more precise results whereas the drawback is also obvious, that is the sensitivity of color with different illumination conditions and camera settings. Generally speaking, most of the proposed methods are constrained by some practical factors such as unstable environment, poor lighting condition, serious plate rotation, etc. Therefore, we aim to develop a robust VLPR system which is able to adapt to various environments and make less assumption of the surroundings and license plates.

In this paper, we mainly focus on the license plate localization step. We present a novel license plate localization method based on arrangement of the characters in standard license plates. To complete the whole VLPR system, we continue to segment and recognize the license plate characters using some simple methods. The performance of the proposed system is tested on our own collected image dataset. Some

Fig. 1. Some typical image examples in our own collected dataset.

This work was supported in part by NSFC projects 70890084, 60921061, and 90920305; and CAS projects 2F12F04, 2F09N05, 2F09N06, 2F10E08, and 2F10E10.

399978-1-4673-0390-3/12/$31.00 ©2012 IEEE

typical images in the dataset are shown in Fig. 1, and the brief framework of our VLPR system is shown in Fig. 2.

The remainder of this paper is organized as follows. Section II explains the license plate localization step in detail. The character segmentation and recognition as well as the color classification steps are presented in section III. Section IV describes the experiments which are performed to test the performance of the proposed system. Finally, we briefly make the conclusion in Section V.

II. LICENSE PLATE LOCALIZATION The motivation of our license plate localization method is

from [7]. Apart from that, we also consider the special structure of Chinese license plates. We make a definition of the license plates on the basis of their contents, that is, Chinese license plate is a rectangle region which consists of seven characters arranged as fixed layouts. Two types of character layout in typical license plates are shown in Fig. 3.

When we process the images, it is difficult and complicated to detect “characters” exactly. Therefore, we extract blobs that satisfy the constraints of attributes of characters such as size, area, aspect ratio, and we consider these blobs as characters. If we detect some blobs like characters in the image and their

positions are consistent with the character layout in standard license plates, then we can infer the position of the whole license plate. This method has two advantages. One is that it does not strictly restrict the lighting condition, surroundings and other factors. Another one is that when the license plates are located, character regions, namely, MSERs, are segmented simultaneously. The detailed processes are described below.

Fig. 2. Framework of the proposed VLPR system.

A. Candidate Character Regions Extraction The character regions can be considered as regions of

reasonable size, and gray levels of pixels in such regions are similar at the same time. In [7], many thresholds are set to get many binary images, and then candidate character regions are extracted from these binary images using connected components analysis. However, these gray level thresholds rely on the lighting condition, so extracted connected components are sensitive with illumination. In order to overcome such disadvantage, we employ a region feature detector, MSER (Maximally Stable Extremal Region) detector, to extract the region feature in images as candidate character regions.

Originally, MSER was proposed to settle the matching problem in wide baseline stereo vision in [8] with the benefit of their excellent affinity invariance. This characteristic can also help with the candidate character region extraction in the license plate localization algorithm. The MSER detection process is similar with the watershed algorithm. Generally speaking, the detection is to use some different thresholds to binarize the image, and then the binarized regions with the most stable area variation are defined as MSERs. Exactly, we use thresholds from 0 to 255 to binarize the image first of all. It is supposed that the pixels with gray level lower than the threshold are set to white and those with higher gray level are set to black. Then we can get a series of black and white regions called Extremal Regions, which can be denoted as

. Extremal means that all the pixels within those regions have the gray level higher or lower than the gray level of pixels in region’s boundary. If the area of an extremal region is stable in a wide range of gray level, then this region is a maximal stable extremal region (MSER). Using mathematical expression, that means if and only if

1 1,..., , ,...( )i i i iQ Q Q Q Q� � 1�

*i i� ,

\( ) i i

i

Q Qq i

Q� �� � � (1)

gets the local minimum, MSER is obtained. The symbol denotes the step length of the gray level threshold in this

formulation. Considering that gray levels can be adjusted in two opposite directions, after these two operations we can get two kinds of extremal regions respectively, i.e., bright extremal regions and dark extremal regions. The difference of these two regions also contributes to the inference of license plate location, as described in next subsection. The standard MSER detection is described above. In our implementation, linear time MSER detection [9] is utilized to increase the efficiency of MSERs extraction.

*iQ

�

(a) (b)

Fig. 3. Two kinds of characters layout in typical license plates. (a) The single line type. (b) The double line type.

Compared with other region features, MSER has many advantages and can get better performance in most applications [10]. In our task of detecting characters, the main superiority is

400

MSER’s invariance to lighting change. As long as the luminance in the image changes monotonously, the MSER can keep stable even illumination changes from daytime to nighttime.

In order to detect blobs even in some low-contrast image regions and ensure any MSERs not to be left, we slightly adjust the MSER detection process. First, we set the threshold step as a minimum value of 1. Then the diversity of extremal regions’ area in the same position is set to a very small value, so that we can get as many extremal regions as possible. Afterward, we only save the extremal regions with the minimum area variation among all regions in the same position as the final MSERs. Meanwhile, the MSERs are restricted by some preset license plate parameters. The remaining MSERs can be considered as candidate license plate characters. An example of MSERs extraction result in a test image is shown in Fig. 4.

�

B. License Plate Location Inference If the extracted MSERs just are the characters in a license

plate, we can infer the exact location of the license plate easily. However, in practical applications, just as the result shown in Fig. 4, some characters may not be detected as MSERs. The reason is various, such as the unequal illumination on license plate or the joint of characters with the license plate boundary. In addition, there are a large number of MSERs that meet the restriction in the background. Therefore, after we detect the MSERs in the image, we must check them and find those having similar layout with characters in standard license plates. The process of inference and analysis is described below.

The concepts of nodes and edges in the graphical model are introduced in our inference. We intend to set the candidate characters as nodes and build an edge between two nodes

which meet the geometric relationship and gray level relationship of two adjacent characters. First of all, we calculate the value of geometric relation and gray level relation between every two MSERs that were extracted before. Hence, the MSERs relationship matrix is built. The geometric relationship includes Euclidean distance, horizontal distance, vertical distance between two MSERs, difference of height and width between their bounding boxes, etc. The gray level relationship means that whether both blobs are bright or dark MSERs. Next we deal with each MSER to search in the left direction for another one that satisfies the geometric relation between two adjacent characters according to the value in the relationship matrix. If such is not found due to some undetected characters, then we search for two MSERs that have space of one character. Besides, the pair of matched MSERs must be the uniform bright or dark MSERs simultaneously. Then such two blobs can be set as nodes, and an edge is built between them. Meanwhile, the type of edge is labeled as either adjacent or at interval. The result of setting nodes and edges of Fig. 4 can be seen in Fig. 5, where yellow dots denote nodes and yellow short lines denote edges.

Although many MSERs exist in the background in Fig. 4, from Fig. 5 we can see that nodes in background become much fewer. Next we should find a cluster of linked nodes which satisfy the layout of characters in license plate. In consideration of two types of Chinese standard license plates (Fig. 3), we infer the location respectively. As to the single line license plate (Fig. 3(a)), the interval sign or five continuous MSERs are searched for. As to the double line license plate (Fig. 3(b)), not only at least four continuous MSERs are needed, an MSER in the above line is also necessary. A simple flowchart of inference on the basis of nodes and edges is shown in Fig. 6. Fig. 7 is the inference result of Fig. 5.

Fig. 5. Result of setting nodes and edges in Fig.4.

Fig. 4. An example of MSERs extraction result.

401

III. LICENSE PLATE RECOGNITION To complete our system, we continue to recognize the

located license plates. In this paper we mainly focus on the license plate localization procedure, so we merely finish the recognition task using simple methods in this section. After license plate recognition, we could get some information relevant to the license plate, including license plate number, color of the license plate, reliability value of the recognition result, and so on. Therefore, we divide the recognition task into two parts: color classification and character recognition.

A. Color Classification In order to recognize the color of license plates in China

(blue, yellow, and white), we convert the RGB color space of the license plate images into HSI (Hue, Saturation, and Intensity) color space. The advantage of HSI color space is that it separate intensity from hue and saturation, so that the analysis on hue or saturation space will be more stable with the lighting change. The histograms of hue and saturation in license plate image region are calculated and normalized. Then the color can be classified as blue, yellow, or white according to distribution of the two histograms. For blue and yellow, both of their hue histograms are unimodal generally, and the peaks of their hue histograms stay around 0.7 and 0.1 respectively. For color white, the hue histogram is not so discriminative, thus the saturation histogram is more critical. The saturation histogram of white color is also unimodal and usually gets its maximum around 0.1.

Fig. 6. Flowchart of the license plate location inference based on nodes and edges. (Single line license plate)

B. Character Recognition Before character recognition, the images of characters

should be segmented first from the license plate images. Because our license plate localization method is based on detection and analysis of MSERs, some of the detected blobs can be considered as segmented characters directly, while the undetected characters and the Chinese characters are inferred according to the character arrangement in a standard license plate.

After character segmentation, the character recognition is just a pattern recognition problem. We normalize those character images into gray level images with uniform size of 20*40 in our implementation. Then we use the Histogram of Oriented Gradients (HOG) [11] feature descriptors to represent the image as a feature vector. HOG computes histogram of gradient orientation on a dense grid of uniformly spaced cells and uses local contrast normalization in overlapped blocks for improved accuracy. It can describe the local object appearance and shape effectively. Since the HOG descriptor computes histograms on localized cells, the method keeps nice invariance to geometric and photometric transformations. In our implementation, we use 5*10 pixel cells and 2*2 cell blocks with 9 histogram channels. So we can get a feature vector of 324 dimensions to describe a character image.

In the classification stage, a certain number of character image samples are chosen as training samples. For each character class, the mean vector i� of all the feature vectors in the training set is calculated. Thus we can get the mean vector set � �1 2, ,..., C� � � , in which C means the number of classes. When we classify a new image, its HOG descriptor � is computed first of all, and then the class label c can be predicted as the class whose mean vector is the closest vector to � , that is

1,2,..,

arg min ii C

c � ��

� � . (2)

In our implementation, reliability value which is related to the value of Euclidean distance c� �� is introduced to supply additional information for recognition and localization.

Fig. 7. Inference result on the basis of nodes and edges in Fig. 5.

402

If more than one located license plates overlap, only the one with the maximum reliability value is kept. Meanwhile, the license plates which have the reliability value lower than a threshold are discarded. This strategy could increase the final recognition accuracy effectively.

IV. EXPERIMENT AND ANALYSIS We test the performance of the license plate localization

and VLPR system on our own collected license plate image dataset. The dataset consists of images captured from various surveillance cameras installed in crossings and roads. There are 520 images captured in daytime and 160 captured at night. Resolution of images are various, from 768*576 to 1936*2592. Some of the images have more than one license plate and the widths of all explicit license plates range from 100 pixels to 200 pixels. The type of license plates in our dataset including typical blue single line license plate, yellow double line license plate and some special types of license plate, such as police and army license plates.

All of the functional modules of the VLPR system are implemented and embedded in a dynamic link library (DLL), so it can be easily imported by other projects as a product. The DLL is developed on Visual Studio 2008 using C++ language and OpenCV 2.1.

To evaluate the performance of our system, we manually mark the correct location and license numbers of each explicit license plate in our dataset. In localization performance testing,

the overlap area between our localization result and the marked benchmark region is calculated. If the overlap area is over 90% area of the marked license plate, we can say the license plate is correctly localized. The parameters of license plate localization do not need previous training, and the parameters do not need to be changed from daytime to nighttime. That is the superiority of our system. In character recognition, we choose about 1000 character images to train the classifier. The experiment is performed on a PC with Intel Core2 2GHz CPU and 3GB RAM. Experimental results on performance of localization and whole recognition system are shown in Table I. Fig. 8 displays some of the localization results from our dataset. The total processing time can be under 100ms when recognizing an image with a resolution of 768*576 pixels without algorithm optimization. Most of the time is consumed on the MSERs extraction. This can be saved by parallel implementation of algorithm in the future. The experimental results show that the proposed system is available in practical applications.

TABLE I. EXPERIMENTAL RESULTS

DAYTIME NIGHTTIME

Localization Accuracy 96.0% 92.5% System Recognition Accuracy 91.9% 86.2%

Fig. 8. Localization result examples from our test dataset.

403

We have developed interactive software with C# language based on .net 2.0 framework to import the DLL and display recognition results. Fig. 9 shows the interactive application of the whole VLPR system running on a PC.

V. CONCLUSION In this paper, a vehicle VLPR system is proposed. The most

important and difficult module in this system is license plate localization, which make use of the contents in license plates. Firstly the MSER detector is applied to extract candidate character regions. Then the locations of license plates are inferred according to the arrangement of characters in standard license plates. This kind of localization makes no rigorous assumptions of the lighting condition, weather, rotation angles of the license plates and surroundings.

To complete our system, we continue to recognize the characters and color of license plates. The classification of license plate color is based on the HSI color space. Then HOG is utilized in the character recognition step to extract feature vectors in normalized character images. The whole system is evaluated on our own collected image dataset, and the results show that the system is available in practical application. In the

future, we would try detecting exact characters other than assuming blobs as characters. The character recognition algorithms should also be investigated in more depth.

Fig. 9. Interactive software of our VLPR system running on a PC.

ACKNOWLEDGMENT The first author would like to thank Professor Feiyue Wang

for his support to this work and Dr. Yuan Gu for his help in discussion and implementation of the algorithm.

REFERENCES [1] C.-N. E. Anagnostopoulos, I. E. Anagnostopoulos, I. D. Psoroulas, V.

Loumos, and E. Kayafas, "License plate recognition from still images and video sequences : a survey," IEEE Trans. Intelligent Transportation Systems, vol. 9, pp. 377-391, 2008.

[2] D. Zheng,Y. Zhao and J.Wang, “An efficient method of license plate location,” Pattern Recognition Letters, vol.26, pp. 2431-2438, 2005.

[3] H.Mahini,S.Kasaei and F.Dorri, “An efficient features-based license plate localization method,” in Proc. 18th Int. Conf. Pattern Recognition, Hong Kong, 2006, pp.841-844.

[4] H. Bai and C. Liu, “A hybrid license plate extraction method based on edges statistics and morphology,” in Proc. Int. Conf. Pattern Recognition, 2004, pp.831-834.

[5] V.Abolghasemi,A. Ahmadyfard, “ An edge-based color-aided method for license plate detection,” Image and Vision Computing, vol.27, pp.1134-1142, 2009.

[6] S. Chang,L. Chen,Y. Chung and S. Chen,”Automatic license plate recognition,” IEEE Trans. Intelligent Transportation System, vol.5, pp.42-53, 2004.

[7] D.Llorens,A.Marzal,V.Palazon and J.M.Vilar, “Car license plates extraction and recognition based on connected components analysis and HMM decoding,” in Proc. 2nd Iberian Conf. Pattern Recognition and Image Analysis, 2005.

[8] J. Matas, O. Chum, M. Urba and T. Pajdla, “Robust wide baseline stereo from maximally stable extremal regions,” in Proc. of British Machine Vision Conference, 2002, pp. 384-396.

[9] D.Nister and H.Stewenius, “ Linear time maximally stable extremal regions,” in Proc. European Conf. Computer Vision,2008, pp.183-196.

[10] K. Mikolajczyk, T. Tuytelaars, C. Schmid, A. Zisserman, T. Kadir and L. Van Gool, “A comparison of affine region detector,” Int. Journal of Computer Vision, vol.65, pp. 43-72,2005.

[11] N.Dalal and B.Triggs, “Histograms of oriented gradients for human detection,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2005, pp.886-893.

404