Unifying Approach for Fast License Plate Localization and Super-Resolution

Chu Duc Nguyen, Mohsen Ardabilian, Liming ChenUniversite de Lyon, CNRS

Ecole Centrale de Lyon, LIRIS, UMR5205, F-69134, France{chu-duc.nguyen, mohsen.ardabilian, liming.chen}@ec-lyon.fr

Abstract—This paper addresses the localization and super-resolution of license plate in a unifying approach.

Higher quality license plate can be obtained using super-resolution on successive lower resolution plate images. All ex-isting methods assume that plate zones are correctly extractedfrom every frame. However, the accurate localization needs asufficient quality of the image, which is not always true in realvideo. Super-resolution on all pixels is a possible but muchtime consuming alternative.

We propose a framework which interlaces successfully thesetwo modules. First, coarse candidates are found by an weak butfast license plate detection based on edge map sub-sampling.Then, an improved fast MAP-based super-resolution, usinglocal phase accurate registration and edge preserving prior,is applied on these regions of interest. Finally, our robustICHT-based localizer rejects false-alarms and localizes the highresolution license plate more accurately.

Experiments which were conducted on synthetic and realdata, proved the robustness of our approach with real-timepossibility.

Keywords-license plate; MAP super-resolution; interlacedlocalization and super-resolution;

I. INTRODUCTION

License Plate Recognition (LPR) is an active topic inautomatic video-surveillance, such as systems designed fortoll gates, stolen car detection, parking lots. Classically, aLPR system consists of three main modules: license platelocalization, character segmentation, and character recogni-tion. Among them, localization is considered as the mostcrucial stage, since a high accurate and fast recognitioncan be performed only if license plates had been correctlylocalized, and commercial systems for recognition of printedtext are available [1]. Recently, super-resolution (SR) tech-nology towards computer vision [2] is employed to enhancerecognition rate in LPR [3]. SR reconstruction aims toestimate a high-resolution (HR) image from multiple low-resolution (LR) images. Given a set of shifted observationsof one license plate, higher quality image of this platecould be generated using SR which facilitates the recognitionprocedure [4].

First approach which applied SR in LPR to identifymoving vehicles was proposed by Suresh [3]. The authorpresented a robust Maximum a posteriori (MAP) basedmethod with discontinuity adaptive Markov random fieldprior for enhancing edges in reconstruction process. Later, afast MAP-based SR algorithm was proposed by Yuan [4]. A

new cost function reduces the computation complexity thatallows SR reconstruction can be treated as a deconvolutionby using Wiener filter. But the fact that priors are removedmakes the method unstable, due to registration errors. Mostrecently, Gambotto [5] proposed a SR based on non-uniforminterpolation of registered LR images which was fast buteven more sensible to motion estimates.

All existing SR approaches for LPR need the plate cor-rectly cropped from successive frames, which is not a trivialtask due to uncontrolled imaging conditions. License platelocalization approaches diversify from rule based heuristicmethods to training based classifiers which differ aboutused features [6] [7]. One main drawback of all publishedmethods is that they generally make strong assumptionsabout license plate’s norm and context variability such asplate size, position, inclination and illumination condition.Moreover, the fact that only low features and their statistichave been exploited leads to high false-alarm rate. In [8], weproposed a robust and fast method which localizes licenseplate accurately with low false-alarm rate.

According to state-of-the-art on license plate localizationand SR for LPR, an “egg and chicken” problem may occurif these techniques are used in real applications. SR assumesthat license plates are localized and extracted correctlyfrom every frame. On another hand, the exact localizationrequires a sufficient quality of the image, which is not alwaystrue due to motion blur, noise or compression artifacts.SR on all pixels is a possible but much time consumingalternative which makes real-time processing unlikely. Whilelocalization and SR are still treated separately, the problempersists.

In this paper we propose a unifying approach that effi-ciently interlaces both localization and SR in a completeframework (Fig. 1). Firstly, given a video frame, all verticaledge zones in this frame, e.g. license plate regions, arefast segmented from a sub-sampled edge map. Then theseregions of interest (ROI) are tracked in sequences of thelast and next frames. Secondly, these tracked ROIs areregistered accurately under an affine model using local phaseinformation. Thirdly, registered ROIs are used as the inputof a fast MAP-based SR algorithm with edges preservingprior. Finally, we use our ICHT-based localizer [8] to rejectfalse-alarms and localize the license plate in high qualityROIs more accurately.

2010 International Conference on Pattern Recognition

1051-4651/10 $26.00 © 2010 IEEE

DOI 10.1109/ICPR.2010.100

380

2010 International Conference on Pattern Recognition

1051-4651/10 $26.00 © 2010 IEEE

DOI 10.1109/ICPR.2010.100

380

2010 International Conference on Pattern Recognition

1051-4651/10 $26.00 © 2010 IEEE

DOI 10.1109/ICPR.2010.100

376

2010 International Conference on Pattern Recognition

1051-4651/10 $26.00 © 2010 IEEE

DOI 10.1109/ICPR.2010.100

376

2010 International Conference on Pattern Recognition

1051-4651/10 $26.00 © 2010 IEEE

DOI 10.1109/ICPR.2010.100

376

Figure 1. Flowchart of the proposed framework.

The remainder of the paper is organized as follows. InSection II, we introduce the description of our approach andexplain analytically how it is robust and fast. In SectionIII, we present experiments and results which prove theeffectiveness of the proposed framework. Finally, SectionIV gives our conclusions.

II. PROPOSED APPROACH

A. Fast License Plate ROIs Localization

The idea of a coarse but fast candidates detection comesfrom observations on text in license plate: characters havedistinctive intensities from background; and characters al-ways respect a base-line. Therefore license plate text formsa dense region of quasi-vertical strokes, regardless of view-point change effect.

Vertical edge map is generated by applying one verticalSobel operator on a given video frame. The more resolutionof this map decreases, the more vertical edges becomeshorter and come close. So we reduce the map resolution, adense set of quasi-vertical strokes could be observed as onehomogeneous quasi-horizontal region. All regions havingthat proprieties are detected quickly using our improvedHough connective transform, which proved robust to noiseand quantification artifact [8]. Projection of these regionson the original frame gives some coarse rectangular ROIs.Therefore, the number of pixels passed to next steps is muchreduced; while keeping true license plate among ROIs. Then,a fast 2D tracking of these ROIs in sequences of the last andnext frames is yielded [9].

B. Accurate Local Phase Registration

SR algorithms use the information available from shiftedLR observations to reconstruct an HR image. Accurateregistration of these observations is important to the successof these algorithms [3]. License plate is a rigid body movingin a stable manner, only global transition, rotation and scalecaused by perspective projection needs to be considered [4].It means for LPR the affine model is sufficient under whichtwo views of an object are related by:{

x′ = xp1 + yp2 + p3

y′ = xp4 + yp5 + p6(1)

Given the model, sub-pixels accurate registration can beachieved with the exact knowledge of degradation parame-ters such as blur and non-uniform illumination. However, inpractice, this information is rarely available. We overcomethis problem using local phase to estimate registration pa-rameters. Local phase is robust towards noise and smoothlyvarying illumination [10].

1) Local Phase: can be computed using Gabor filters,which are popular band pass filters as they achieve thetheoretical minimum product of spatial width and band-width, desirable for better localization and accurate phasecomputation, respectively:

g(x, y) =1

2πσxσye−( x

2

2σ2x

+ y2

2σ2y

)ej(ωxx+ωyy), (2)

where (ωx, ωy) is the angular frequency, σx and σy controlthe spatial width of the filter. At each frequency (ωx, ωy),the image is convolved with g(x, y). The argument of thecomplex output is local phase.

2) Phase Difference: is computed by taking the differ-ence of phase values at each location of the image pairat the given angular frequency. 2D translation componentscan be computed from the basis of Fourier Shift theorem,according to which, a shift of (∆x,∆y) in the spatial domainwould produce a phase difference of 2π× (∆xωx + ∆yωy)at (ωx, ωy). Locations (x, y) and (x′, y′) in (1) are translatedby these shift values. Errors in phase difference computationmay occurs due to noise and the absence of the local fre-quencies with which the images are convolved. The degreeof match in the amplitude values is used as a measure ofconfidence [10]. The confidence is high if the amplitudes ofthe Gabor filter response at (x, y) in both the images areclose.

C. Improved MAP-based SR Reconstruction

The imaging process in SR reconstruction can be formu-lated under matrix form as:

yr = DHrWrx+ nr, 1 ≤ r ≤ m, (3)

where x is the original HR image (N1N2 × 1), yr isthe rth LR observation (M1M2 × 1), D is the down-sampling matrix (M1M2 × N1N2), Hr is the blur matrix

381381377377377

for rth frame (N1N2×N1N2) which describes motion blurand camera PSF, Wr is the geometric warp for rth frame(N1N2×N1N2), nr is the noise in rth frame, and m is thenumber of LR observations.Wr is estimated in the registration step and D,Hr are gen-

erally assumed to be known. Solving for x in (3) given theobservations yr is an ill-posed inverse problem. Hence, it isimportant to use a priori information about x that will reducethe space of solutions which conforms to the observed data.The Bayesian MAP formulation allows the incorporationof prior knowledge about x to improve robustness duringthe reconstruction process. The MAP estimate of x can bedescribed as:

x̂ = arg maxx{log[P (y1, ..., ym|x)] + logP (x)}. (4)

We propose a discontinuity adaptive Markov RandomField (DAMRF) prior in which the degree of interactionbetween pixels across edges is adjusted adaptively in orderto preserve discontinuities [3]. Assuming that the noise fieldsare independent of x and each other with variance σ2, theMAP estimate is equivalent to:

x̂ = arg maxx{m∑r=1

‖yr −DHrWrx‖2

2σ2+∑c∈C

g(dcx)}, (5)

where C is the set of cliques in the neighborhood system ofa pixel, dcx is a spatial activity measure within the data andg(n) = γ − γe−n2/γ is the adaptive interaction non-convexfunction. A deterministic annealing GNC algorithm is usedfor optimization [3].

In order to improve the robustness of SR algorithm, wepropose a process that verifies the quality of registrationbefore including an image in the reconstruction. Let {Xr}be the set of projections of {yr} onto the high-resolutiongrid using estimate Wr, D and Hr. {Xref} is the projectionof the target observation. We define a registration errordistance:

distr =1S

∑P

|Xr −Xref |, 1 ≤ r ≤ m, (6)

where P is all image patches and each patch is of size S.An image of index r is included in the MAP reconstructionif the corresponding distance satisfies:

distr ≤ k.min(distr), 1 ≤ r ≤ m, r 6= ref, (7)

where k is a constant which is empirically set. Sincemis-registered outliers can be removed, the SR algorithmconverges faster and yields higher quality output.

D. ICHT-based Localization and Verification

The detail of our license plate localizer can be foundin [8]. It should be noted that the weak localization in thesection II.A is a part of our localizer. Then we use highlevel line segment features which describe directionality,

(a) (b) (c)

Figure 2. Synthetic degraded images: (a) blurred and noisy down-sampled,(b) affine warped of (a), (c) affine warped of (a) with illumination change.

regularity, similarity, alignment and connectivity, to dis-criminate plate / no-plate patterns. An improved connectiveHough transform (ICHT) has been proposed for fast featuresextraction, and also for fine delimitation of plate pixels onimage. High quality ROIs issued of SR step are used asinput of the proposed localizer. False-alarm candidates arerejected and the boundary of the plate is more accuratelylocalized.

III. EXPERIMENTS AND RESULTS

In this section, we demonstrate the robustness of the pro-posed approach in terms of registration, SR reconstructionand framework performance. Experiments were conductedon synthetic and real data with a Centrino PC (2.2 GHz)using C++.

First of all, three LR observations of size 60× 15 pixelswere degraded from a good quality license plate image bydown-sampling, adding Gaussian blur, affine warping andGaussian white noise of variance σ2 = 5. Illuminationvariation was generated in the third LR observation (Fig. 2c).We compare our local phase registration with an invert com-putational gradient descent (ICGD) which uses point featurescorrespondence for registration [11]. The performances werecompared using absolute mean shift error (mse) [10]. TableI shows that the proposed algorithm performs much betterthan ICGD while illumination change exists which is morelikely to appear in real video-surveillance, thanks to theillumination invariance of phase information.

Next, since benchmark data for evaluation of SR forLPR do not exist, we gather video sequences from UCSDproject [12] as real data test. The input sequence for test con-sists of manually cropped license plates from 10 successiveframes. We compare the proposed MAP-based SR methodwith a non-linear interpolation (NLI) [5] and the originalDAMRF [3] and the results are shown in Fig. 3. The SRimage using the NLI is quite blurred and low contrasted dueto the high-frequency information lost during the process ofsampling are not recovered. The DAMRF and the proposedalgorithm visually yield the best result with more sharpededges. Nevertheless, our algorithm converges faster after 18iterations comparing to 30 for DAMRF. This comes from

Table IREGISTRATION PERFORMANCE IN mse.

Methods Const. illum. Illum. changeICGD [11] 0.254 2.813Proposed 0.247 0.587

382382378378378

(a) Zoomed LR observation (b) NLI [5]

(c) DAMRF [3] (d) Proposed DAMRF

Figure 3. Results of SR methods (up factor 2).

the automatic rejection of mis-registered images before thereconstruction stage.

Finally, we use a real sequence of 20 successive frameswhere there is so low quality license plate images that mostexisting localization methods [6] [7] [8] cannot localize(Fig. 3a). The proposed SR and localization [8] are testedin three different ways: SR on all pixels, then localizethe license plate on the HR image; localize the plate inevery frame, then do SR on cropped candidates; and theproposed framework (Fig. 1). The performance is evaluatedby visual quality, localization precision and computationtime. Fig. 4 shows that the output quality of Loc.→SR isinferior to those yielded by SR→Loc. and the proposedframework, due to errors in localization. SR→Loc. andthe proposed framework localizes the license plate moreaccurately since the localization is performed in high qualityimage. However, the proposed framework is much fasterthan SR→Loc. (Tab. II) for the reason that only ROIs pixelsare required to be estimated in SR reconstruction.

(a) Loc.→SR (b) SR→Loc. (c) Proposed

Figure 4. HR license plate outputs.

Table IICOMPUTATION TIME OF THREE FRAMEWORKS.

Framework Computation time(s)SR → Loc. 6.21Loc. → SR 2.05Proposed 1.32

IV. CONCLUSIONS

In this paper, we introduce a unifying approach of licenseplate localization and SR for LPR in video-surveillance.

Experiment results prove that our framework overcomes theinstability of existing SR methods for LPR which requirelicense plates are properly cropped from every frame; andvice versa localization methods expect image of sufficientquality. We also propose an accurate registration algorithmbased on local phase which performs well even if illumina-tion change happens. Another contribution is a robust MAP-based SR reconstruction excluding mis-registered images.In addition, our framework has real-time response (Tab. II)while tested with real video. In our further work, we proposeto investigate the strategy of localization and SR interlacingin the MAP cost function.

ACKNOWLEDGMENT

This work has been partially supported by the FrenchANR Omnia project under the grant ANR-07-MDCO-009-02.

REFERENCES

[1] F. Porikli and T. Kocak, “Robust license plate detection usingcovariance descriptor in a neural network framework,” inAVSS, IEEE, 2006, pp. 107–113.

[2] D. Capel and A. Zisserman, “Computer vision applied tosuper resolution,” SPM, IEEE, vol. 20, no. 3, pp. 75–86, 2003.

[3] K. V. Suresh, G. M. Kumar, and A. N. Rajagopalan, “Super-resolution of license plates in real traffic videos,” ITS, IEEE,vol. 8, no. 2, pp. 321–331, 2007.

[4] J. Yuan, S.-D. Du, and X. Zhu, “Fast super-resolution forlicense plate image reconstruction,” in ICPR, IAPR, 2008,pp. 1–4.

[5] J.-P. Gambotto, M. Denis, and R. Livier, “Super-resolutionof moving vehicles using an affine motion model,” in AVSS,IEEE, 2009, pp. 535–540.

[6] H. Mahini, S. Kasaei, and F. Dorri, “An efficient features- based license plate localization method,” in ICPR, IAPR,2006, pp. 841–844.

[7] H. Zhang, W. Jia, X. He, and Q. Wu, “Real-time license platedetection under various conditions,” UIC, LNCS, vol. 4159,pp. 192–199, 2006.

[8] C. D. Nguyen, M. Ardabilian, and L. Chen, “Robust carlicense plate localization using a novel texture descriptor,”in AVSS, IEEE, 2009, pp. 523–528.

[9] J. Y. Bouguet, “Pyramidal implementation of the lucas kanadefeature tracker: Description of the algorithm,” Intel ResearchLaboratory, Technical Report, 2002.

[10] H. Arora, A. Namboodiri, and C. Jawahar, “Robust imageregistration with illumination, blur and noise variations forsuper-resolution,” in ICASSP, IEEE, 2008, pp. 1301–1304.

[11] S. Baker and I. Matthews, “Lucas-kanade 20 years on: Aunifying framework,” IJCV, vol. 56, no. 3, pp. 221–255, 2004.

[12] S. Belongie and L. Dlagnekov, “Ucsd - car surveillanceproject,” http://vision.ucsd.edu/project/car-surveillance.

383383379379379