Ž .Pattern Recognition Letters 19 1998 629–641

A structuralrstatistical feature based vector for handwrittencharacter recognition

L. Heutte a,), T. Paquet a, J.V. Moreau b, Y. Lecourtier a, C. Olivier a

a PSI-La3i, UFR des Sciences et Techniques, UniÕersite de Rouen, F76821 Mont-Saint-Aignan Cedex, France´b Matra Systemes & Information, Velizy-Villacoublay, France` ´

Received 30 March 1996; revised 1 December 1997

Abstract

This paper describes the application of structural features to the statistical recognition of handwritten characters. It hasbeen demonstrated that a complete description of the characters, based on the combination of seven different families offeatures, can be achieved and that the same general-purpose structuralrstatistical feature based vector thus defined provesefficient and robust on different categories of handwritten characters such as digits, uppercase letters and graphemes. q 1998Elsevier Science B.V. All rights reserved.

Keywords: Handwritten character recognition; Structural features; Statistical features; Feature definition; Missing values; Statisticalclassification

1. Introduction

In the field of handwriting recognition, it is nowagreed that a single feature extraction method and asingle classification algorithm generally cannot yielda very low error rate, even if some highly reliablesystems have been developed recently, especially for

Žhandwritten numerals Smith et al., 1994; Shridhar et.al., 1997 . Due to the large variety of the available

Ž .feature extraction methods Trier et al., 1996 , manyresearchers have turned towards the use of severalfeature extractors with more complex structures ofclassification, such as multistage classification

Žschemes Rahman and Fairhurst, 1997; Cao et al.,.1995 or ‘‘parallel’’ combination of multiple classi-Ž .fiers Xu et al., 1992; Sabourin et al., 1993 . Even

) Corresponding author: E-mail: laurent.heutte@univ-rouen.fr.

though the results published in these papers are ofgreat interest, some problems still remain: ‘‘Howmany classifiers and what kind of classifiers shouldbe used? For each classifier, what types of featuresshould be chosen?’’. Moreover, this involves multi-ple tiresome learning steps for both the chosen clas-sifiers and the combination rules.

The authors believe that there have not beensufficient efforts made towards the use of severalfeature extractors in a same structure of classificationŽ .e.g., in a one-shot classifier . Three major interests

Ž .can however justify such an approach: i the use ofseveral types of features still ensures an accurate

Ž .description of the characters; ii the use of a singleŽ .classifier preserves fast and flexible learning; iii the

tedious tuning of combination rules is avoided.Based on this idea, the authors investigated the

possibility of combining both structural and statisti-cal features for the recognition of handwritten char-

0167-8655r98r$19.00 q 1998 Elsevier Science B.V. All rights reserved.Ž .PII: S0167-8655 98 00039-7

( )L. Heutte et al.rPattern Recognition Letters 19 1998 629–641630

acters using a single statistical classifier. The keypoint for doing so lies in the definition of a mapping

Ž .of the structural features qualitative features into afixed-size feature space to feed the statistical classi-fier. This leads to a 124-variable feature vector,called the StructuralrStatistical Feature Based Vec-

Ž .tor SSFBV . The statistical classifier is based on alinear discrimination technique. To show that theSSFBV provides a robust description of handwrittencharacters, the recognition system has been testedover large-scale samples including handwrittenwell-segmented numerals and handprinted charactersas well as graphemes which are the most currentpatterns generated by an handwritten cursive word

Žsegmentation graphemes can be either well-seg-mented, over-segmented or under-segmented charac-

.ters . This recognition scheme has proved to beŽefficient not only on numerals compared to some

.recent published works but also on more difficultrecognition problems such as those cited above.

The paper is organized as follows. Section 2presents the definition of the mapping retained:parametrization of the structural features and selec-tion of the replacement rules for missing values. Theseven families of features that make up the SSFBVare described in Section 3 as well as their extractionprocess. The various experiments conducted are pre-sented in Section 4 including a brief description ofthe statistical classifier and the collection of data forlearning and testing. Finally, some conclusions aredrawn in Section 5.

2. From shapes to features: the structuralrrrrrstatistical dilemma

Handwritten character recognition systems typi-cally involve two steps: feature extraction in whichthe patterns are represented by a set of features andclassification in which decision rules for separatingpattern classes are defined.

Features can be broadly classified into two differ-Ž .ent categories Govindan and Shivaprasad, 1990 :

Žstructural features like strokes and bays in variousdirections, end points, intersections of line segments,

.loops and stroke relations, . . . and statistical featuresŽderived from the statistical distribution of pointslike zoning, moments, n-tuples, characteristic

.loci, . . . . Structural and statistical features appear tobe complementary in that they highlight differentproperties of the characters. In order to cover thewithin-class and between-class variabilities of thepatterns, the combination of the two approaches hasbeen retained to provide an efficient and completedescription of the characters.

In the same way, there are broadly two mainŽ .approaches for classification Jain, 1990 : the statisti-

cal approach consisting in representing a pattern asan ordered, fixed-length list of numerical values andthe structural approach describing the pattern as anunordered, variable-length list of simple shapes. Thestatistical approach relies on firmly established ele-

Ž .ments of statistical decision theory Fukunaga, 1990 ,even though viewing a pattern in an n-dimensionalspace is rather difficult. On the contrary, the struc-tural approach is intuitively appealing because it

Žappears closer to human recognition strategy Bunke.and Sanfeliu, 1990 . Unfortunately, this approach is

usually difficult to implement in a fast, trainable androbust way over a large variety of shapes.

As a structural classifier is naturally well-suited tothe use of structural features but cannot easily handlestatistical features, we have chosen the frame ofstatistical classification to investigate the combina-tion of both structural and statistical features in aone-shot classifier. Now, in the field of statisticalclassification, a pattern must be represented by afixed-length list of n numerical variables called the

Žfeature vector. Therefore, the set of features either.statistical or structural describing a pattern must be

constrained to map the feature vector. This mappingconsists in a parametrization of each feature, i.e.defining numerical parameters for each feature.

The parametrization of statistical features is natu-rally trivial. On the contrary, structural features mustbe described by a set of numerical parameters whichrequire some ‘‘continuity’’ properties, e.g. a smallchange in perceived shape should always result in a

Ž .small displacement in parameter space Baird, 1988 .For example, suppose that F represents a structural

Žfeature extracted on the pattern such as an end-point,.a hole or a concave arc . . . . Most often, the

parametrization of such features is obtained using aŽ .zoning-like technique Trier et al., 1996 : a fixed

n=m grid is superimposed on the pattern image andthe feature is searched for in each of the n=m

( )L. Heutte et al.rPattern Recognition Letters 19 1998 629–641 631

zones, thus giving a binary feature vector of lengthn=m. Unfortunately, this leads to a loss of informa-tion on the feature position as well as to a loss ofcontinuity in the parameter space: small variations inthe pattern can lead to large variations in the parame-

Ž .ter space Cao et al., 1995 . To cope with this, theauthors have decided to parametrize the structuralfeatures by continuous numerical variables, such asthe ‘‘X–Y position’’ of F. Indeed, these variablesare calculated relative to the center of gravity of the

w xpattern and normalized in y1;q1 according to thewidth or the height of the pattern bounding box. Thisparametrization allows to locate accurately the struc-tural features and to respect the principle of continu-ity.

In the field of handwritten character recognition,the large variety of shapes which can be found in asame class of character, pose the problem of possiblemultiplicity of a same structural feature in the patternŽ .as shown for example in Fig. 1, for the end-points .

This is a priori incompatible with the necessity ofworking in a fixed n-dimensional feature space, butcan be overcome by over-sizing the feature space,i.e., by reserving a maximum number of occurrencesof the ‘‘position’’ variables for example. The choiceof the maximum number of ‘‘position’’ variables tobe reserved for each kind of feature is done beforethe learning phase by simply searching and keepingthe maximum number of occurrences of each featurewhatever the classes of characters. However, theover-dimensioning of the feature vector requires the

Fig. 1. Variability in the number of end points for a same class ofdigit.

definition of some decision rules, when the numberof detected features is not the expected one. There-fore, two replacement rules are applied, according towhether there are fewer or more features than ex-pected:

Ž .1 Replacement rule when there are fewer fea-tures than expected. Those X–Y position variablescorresponding to the missing features cannot be setto 0 because it would mean that these features areexactly on the center of gravity of the characterŽ .according to the normalization described above .However, a numerical value must be assigned tothese X–Y position variables. Several simple andinexpensive techniques for handling missing values

Ž .have been described in Dixon, 1979 . The onlyconfiguration which has provided good results in ourown experiment consists in assigning to the missingX–Y position variables the mean that these variables

Ž .take on the learning base Heutte, 1994 . In thefeature space, this means to replacing those featurevectors by the center of gravity of the clouds ofpoints.

Ž .2 Replacement rule when there are more fea-tures than expected. In this case, it is assumed thatthe extra features come from noise in the character,since the maximum number of occurrences for eachkind of features is known after the learning phase.Therefore, in order to avoid that these extra features

Žmight not be taken into account for example, by.directly removing them , the pairs of nearest features

are merged by deriving the average of their posi-tions.

This mapping has proved to be efficient to over-come the classical dilemma of combining both statis-tical and structural features in the same feature space.This will thus allow to handle the large variability of

Žhandwritten patterns digits, uppercase letters and.graphemes , using the same general-purpose struc-

turalrstatistical feature based vector, detailed in thefollowing section.

3. The StructuralrrrrrStatistical Feature Based Vec-( )tor SSFBV

The SSFBV has been built in order to provide awide range of identification clues over a large vari-

( )L. Heutte et al.rPattern Recognition Letters 19 1998 629–641632

ety of handwriting recognition systems, includingdigit, uppercase letter and grapheme recognition. Thefeature extraction methodology lies in the combina-tion of seven families of features selected frompreliminary work on pseudo-character recognitionŽ .Heutte et al., 1993 . The resulting SSFBV providesa complete description of the characters, mainlythanks to the combination of structural and statisticalfeatures that highlight global and local properties.The seven families of features presented below aresplit into two categories according to whether their

Žparametrization leads to retain position variables lo-. Ž .cal features or not global features .

3.1. Global features

Global properties of the patterns can be high-lighted, thanks to three different families: invariantmoments, projections and profiles, as presented be-low. Although the projection and profile familiesallow to describe structural properties of the patterns,the mapping of these characteristics into the featurespace does not require to manage missing values.

3.1.1. InÕariant momentsThe invariant moments used have been proposedŽ .in Hu, 1962 . These seven moments are well-known

to be invariant to position, size and orientation of thecharacter. They are pure statistical measures of thepixel distribution around the center of gravity of thecharacter and allow to capture the global charactershape information. They are the following:

f 1 s m qm ,Ž . Ž .20 02

2 2f 2 s m ym q4m ,Ž . Ž .20 02 11

2 2f 3 s m y3m q 3m ym ,Ž . Ž . Ž .30 12 21 03

2 2f 4 s m qm q m qm ,Ž . Ž . Ž .30 12 21 03

2f 5 s m y3m m qm m qmŽ . Ž . Ž . Ž .30 12 30 12 30 12

2y3 m qm q 3m y3m mŽ . Ž . Ž21 03 21 03 21

2 2qm 3 m qm y m qm ,. Ž . Ž .03 30 12 21 03

2f 6 s m ym m qmŽ . Ž . Ž .20 02 30 12

2y m qm q4m m qmŽ . Ž .21 03 11 30 12

= m qm ,Ž .21 03

2f 7 s 3m ym m qm m qmŽ . Ž . Ž . Ž .21 03 30 12 30 12

2y3 m qm y m y3m mŽ . Ž . Ž21 03 30 12 21

2 2qm 3 m qm y m qm. Ž . Ž .03 30 12 21 03

where

1m sp q Ž .pqq r2q1N N

2 2x yx q y yyŽ . Ž .Ý Ýi i

is1 is1

Np q

P x yx y yy ,Ž . Ž .Ý i iis1

N N1 1xs x , ys y ,Ý Ýi iN nis1 iy1

Ž .and Ns total number of black pixels x , y in thei i

image.

3.1.2. ProjectionsThese features are derived from histograms of

horizontal and vertical projections of black pixels insome particular areas of the character. They areextracted from the normalized image of the characterso as to obtain normalized histograms of black pixelsboth on the X-axis as well as on the Y-axis. In orderto locate the larger strokes of the character, relativemaxima of the histograms could have been extracted.However, extracting maxima from histograms is quitedifficult when the number of maxima is not known apriori. One way to tackle this problem is to derivethe cumulative histogram of the original one, assum-ing that each important gap on the Y-axis of thecumulative histogram corresponds to a relative maxi-mum on the Y-axis of the original histogram. Yet,when there is a gap on the Y-axis of the cumulativehistogram, the corresponding abscissas are closerthan in the case of a slow variation on the Y-axis.Therefore, dividing the Y-axis in enough equal partsand storing as features the corresponding abscissasallows to get information about the location of somestructural properties of the character like its larger

( )L. Heutte et al.rPattern Recognition Letters 19 1998 629–641 633

Ž . Ž .Fig. 2. Projection based features =1 to =10 . a An originalŽ .histogram of projections; b the cumulative histogram.

strokes. The feature extraction principle is illustratedon a basic example in Fig. 2.

In the present case, dividing the Y-axis in 11 parts– 10 abscissas are thus retained as features fromeach cumulative histogram – has been found experi-mentally to provide the most precise description ofthe characters.

3.1.3. ProfilesThe profile based features are partly inspired by

Ž .Shridhar and Badreldin, 1984 , in which the authorsuse these features, extracted from left and rightprofiles only, for the segmentationrrecognition ofhandwritten connected numerals. However, for hand-written character recognition, left and right profilesare not sufficient to sharply describe character vari-abilities. Top and bottom profiles are therefore addedto our description of the character.

The four profiles are derived from normalizedcharacters, in order to obtain normalized profiles

both on the X-axis and on the Y-axis. Processing theleft profile, for example, of a character consists inscanning each line of the normalized character fromtop to bottom and from left to right, and in storingthe first found black pixel of the character. Thisprofile is called the raw left profile of the character.Raw top, bottom and right profiles are obtained a

Ž .similar way see Fig. 3 . Features derived fromprofiles have been selected in order to give someinformation about the smoothness of the characterand to describe the character in terms of width andheight at particular locations of the character.

The smoothness of the character is obtained fromŽthe first difference profiles first order derivative of

.the raw profiles . From each first difference profile,the maximum amplitude is extracted and normalized

w x Žin 0;q1 according to the height for top and.bottom profiles or the width of the normalized

Ž .character for left and right profiles . These fourvariables thus correspond to some information aboutthe smoothness of the four profiles since a near to 0amplitude means that there is no ‘‘accident’’ in the

Ž . Ž . Ž .Fig. 3. Profile based features A: accident . a Raw profiles; bfirst difference profiles.

( )L. Heutte et al.rPattern Recognition Letters 19 1998 629–641634

profile, whereas a large amplitude points out a bro-Ž .ken profile see Fig. 3 .

The description of the character in terms of widthŽ .respectively height at particular locations is derivedfrom the difference between the left and the right

Ž .raw profiles respectively bottom and top at threeŽlocations: 1r5, 1r2 and 4r5 of the height respec-

.tively the width of the normalized character bound-w xing box. These variables are normalized in 0;q1

Ž .according to the width respectively the height ofthe bounding box. Complementary to the intersection

Ž .locations 1r3 and 2r3 of the height , the widthŽ .locations 1r5 and 4r5 privilege the extremities of

the character.

3.2. Local features

Four families are concerned that allow to take intoaccount complementary points of view of the patternstructure: intersections with straight lines, holes andconcave arcs, extrema and, end points and junctions.They all require a precise positioning in the patternbounding box and have been parametrized so as toensure continuity in the feature space. The mappingof these features therefore requires an over-dimen-

Žsioning of the feature vector as presented in Section.2 to handle position variables.

3.2.1. Intersections with straight linesNumber and position of intersections between the

character and straight lines have been retained asfeatures.

Since distortions on characters have usually higheramplitudes on the horizontal axis than on the verticalone, the number of intersections with a horizontalstraight line is more stable than the one with avertical straight line. Therefore, two horizontalstraight lines and only one vertical have been chosento describe the character in terms of intersectionbased features. The two horizontal straight lines arelocated at the first third and the second third of theheight of the character, thus providing a good de-scription of the lower and higher parts of the charac-ter. The vertical straight line goes through the centerof gravity of the character, in order to obtain a morestable location of the intersections for each class of

Ž .characters see Fig. 4 . Since straight line locationsare fixed a priori, only the X-position of the intersec-

Ž .Fig. 4. Intersection with straight lines based features. a 1stŽ . Ž . Ž . Ž .horizontal 1r3 height ; b 2nd horizontal 2r3 height ; c

Ž .vertical going through the center of gravity

tions with the horizontal straight lines and the Y-position of the intersections with the vertical straightline are retained as location variables.

3.2.2. Holes and concaÕe arcsThese features are extracted from the representa-

tion of the character by its polygonized contours.Two steps are needed for the extraction of thesefeatures: the first one is the polygonization of boththe external boundaries and the internal boundariesof the character in order to describe the character byan ordered list of vertices, the second one is theextraction of the features from this particular repre-

Ž .sentation. The retained variables correspond to: 1number, perimeter and location of concave arcs andŽ .2 number and location of holes. Therefore polygo-nization based features enable to get from the charac-

Ž .ter some local properties concave arcs and someŽ .global properties holes . Some examples are given

in Fig. 5.Concave arcs are only extracted from the ordered

list of vertices corresponding to the external bound-aries, and are obtained by calculating the interiorangle between two successive segments. If this angleis lower than 1, then these two successive segmentsform a concave vertex and the process goes on untilan angle superior to 1 is found. A concave arc is thus

Ž . Ž .Fig. 5. Holes H and concaves arcs CA from polygonizedcontours

( )L. Heutte et al.rPattern Recognition Letters 19 1998 629–641 635

Ž . Ž . Ž . Ž .Fig. 6. Extrema based features: a top; b bottom; c left; dright.

defined as an uninterrupted sequence of concavevertices. The perimeter of a concave arc is obtainedby adding the length of the segments and its positionis found by calculating the barycenter of the vertices.

Holes are directly defined by the ordered lists ofvertices corresponding to the internal boundaries.The position of a hole is obtained by calculating thebarycenter of its vertices.

3.2.3. ExtremaThis family of features includes top, bottom, left

and right extrema of the character. The retainedvariables correspond to number and X–Y position ofthe extrema.

The extrema extraction algorithm depends on thekind of extrema to be extracted. However, only thetop extrema extraction is described here, since theextraction principle for the bottom, left and right

Ž .extrema is similar only the search direction changes .The top extrema extraction algorithm consists insearching from top to bottom of the character bound-ing box the black pixels which have no 8-connectionwith upper pixels. In order to speed up and tofacilitate the search, the character is first encoded in

Žterms of its horizontal run lengths for the extraction.of top and bottom extrema or its vertical run lengths

Ž .for left and right extrema . A top extremum is thenfound when a run-length has no 8-connection withan upper run-length. A filtering step allows finally tokeep only the extrema coming from larger run-lengths.

Extrema based features enable to get some infor-mation about the convex envelope of the characterŽ .see Fig. 6 , and are thus complementary to those

Žextracted from polygonized boundaries i.e., the con-.cave arcs .

3.2.4. End points and junctionsŽThese pure structural features end points, Y and

.X-junctions are extracted on the skeletonized repre-sentation of the character – a beardless smoothed

Ž .8-connected skeleton see Fig. 7 – and the variablesretained are the number and the X–Y position ofthese features. The extraction principle of thesestructural features consists in trying to apply on eachblack pixel of the skeleton a collection of 3=3templates previously defined for each kind of fea-tures.

For the extraction of Y-junctions, a collection oftwelve 3=3 templates, corresponding to each con-figuration of an Y-junction, is applied on each blackpixel of the skeleton. In order to speed up thedetection of these features, the number of blackpixels in the 3=3 neighbourhood of the tested blackpixel must first contain 4 or 5 black pixels.

Extraction of X-junctions is rather different be-cause two levels are used to detect an X-junction.The first one is based on the application of twoparticular templates, in order to detect what is calleda real X-junction. Since the skeletonization processtends to split a real X-junction into two Y-junctionsŽ .thus generating a false stroke , a second level isapplied to merge the pairs of nearest Y-junctions into

Ž .X-junctions see Fig. 7 .A black pixel is considered to be an end point if

there is only one black pixel in its 3=3 neighbor-hood.

Ž . Ž .Fig. 7. End point and junction based features. a and c endŽ . Ž .points; b Y-junctions; c 2 Y-junctions transformed into an

X-junction.

( )L. Heutte et al.rPattern Recognition Letters 19 1998 629–641636

Overall, the SSFBV is made of 124 variables,distributed in the seven families of features as fol-lows: 7 invariant moments; 20 for the horizontal andvertical projections; 10 for the top, bottom, left andright profiles; 15 for the intersections with horizontaland vertical straight lines; 21 for the holes andconcave arcs; 30 for the top, bottom, left and rightextrema; 21 for the end points and junctions. In

w xaddition, variables are normalized either in y1;q1if they are X–Y position variables or invariant mo-

w xments or in 0;q1 if they are ‘‘number of feature’’variables, projection and profile based variables orperimeter of concave arcs.

4. Experiments

The efficiency and robustness of the SSFBV havebeen tested over a wide range of handwritten charac-ter recognition problems, using the same recognitionscheme based on a linear discrimination technique.Learning and recognition algorithms are briefly pre-sented, followed by the description of the databasesused for the experiments; experimental results arethen presented and analyzed.

4.1. Learning and recognition

The present character recognition system is di-vided into two different stages: the learning stageand the decision stage. Input of both these two stagesis the character image from which the 124-variablefeature vector described above is derived.

The learning stage is processed as follows. Let viŽ .be one of the K character classes. Let C v be a seti

of input vectors derived from a sample of charactersŽ Ž .belonging to v . For each pair of clusters C v ,i i

Ž ..C v , a linear discrimination technique is per-j

formed to derive a between-cluster hyperplane. Thew Ž .xK-class problem is thus converted into K Ky1 r2

2-class sub-problems of discriminating each possibleŽ .pair of classes Devijver and Kittler, 1982 . For any

pair of classes v and v , a hyperplane is calculated.i j

Each hyperplane partitions the feature space into twoareas: one area clusters the points located on thesame side as v in comparison with the hyperplane,i

the other area clusters the points located on the same

side as v . The output of the learning stage is finallyj

a set of hyperplanes corresponding to the separationof each pair of classes.

The decision algorithm is applied to each charac-ter as follows. The feature vector associated to thecharacter is first derived. Then its position is exam-ined with respect to each hyperplane separating apair of classes. If one class is better than the others, aconfidence level is associated to this class withrespect to the distance between the chosen class andthe second best class. All the classes with a non-zeroconfidence level with respect to the chosen class arealso kept in the list of possible solutions. The outputof the recognition stage is thus an ordered list ofcharacter classes associated to their decreasing confi-dence level.

The use of a linear discrimination based classifierhas two advantages over other statistical approachesin practice. The first one is that, unlike the non-para-

Žmetric approaches like k-NN or Parzen Window.based classifiers for example , this classifier does not

require large memory capacity and computation timeduring the decision stage. The second one is that thisclassifier is less sensitive to correlation betweenfeatures than a Bayesian classifier such as the maxi-

Žmum-likelihood classifier for example Heutte,.1994 . This last point is important in practice since

the efficiency of the SSFBV relies mainly on thecomplementarity of the seven families of featureswhich of course cannot guaranty a complete decorre-lation in the parameter space.

4.2. Data collection for learning and testing

The experiments have been conducted on threedifferent categories of handwritten characters: digits,uppercase letters and graphemes. These data havebeen collected without any constraint on the writingstyle or writing instrument.

The numeral samples have been extracted fromthe NIST Special Database 3 which contains a totalnumber of 223,125 digits representing 2100 writers.For the experiment reported here, an arbitrarily se-lected subset of 106,707 digits, evenly distributed inthe 10 natural classes from ‘0’ to ‘9’, has beenpartitioned into the two following non-overlapping

Ž . Ž .sets: 1 training set, 53,383 digits and 2 test set,

( )L. Heutte et al.rPattern Recognition Letters 19 1998 629–641 637

53,324 digits. Examples of the digit samples aregiven in Fig. 8a.

The uppercase letter samples have been collectedfrom the dead letter envelopes by the U.S. PostalServices at different locations. Among a total num-ber of 15,960 uppercase letter samples, distributed in

the 26 natural classes from ‘A’ to ‘Z’, 12,765 sam-Žples have been used as training data i.e., 80% of the

.full database without overlapping with 3195 sam-Ž .ples used as testing data i.e., the remaining 20% .

Fig. 8b shows examples of the uppercase letter sam-ples.

Ž . Ž . Ž . Ž . ŽFig. 8. Examples of digitized data for display purposes, normalized to a fixed size . a Digits 10 classes ; b uppercase letters 26. Ž . Ž .classes ; c graphemes 72 classes .

( )L. Heutte et al.rPattern Recognition Letters 19 1998 629–641638

The grapheme samples have been generated by ahandwritten cursive word segmentation performedon address word images provided by the U.S. Postal

Ž .Services Plessis et al., 1993 . Since graphemes canŽeither correspond to only one character succeeded

.segmentation or correspond to a part of a characterŽ .over-segmentation or to more than one characterŽ .under-segmentation , 72 classes of graphemes havebeen retained which cover both the variety ofgraphemes output by the segmentation and variancesin writing styles. These classes include: 52 well-seg-

Ž .mented lowercase letters two allographs per letter ,9 uppercase letters whose shape is not included inthe 52 lowercase letters, 8 graphemes correspondingto the most encountered over-segmented charactersand 3 graphemes corresponding to the two most

Žfrequently never-segmented characters double ‘t’,.double ‘o’, . . . . Examples of grapheme samples are

shown in Fig. 8c. The full database contains 26,382graphemes and has been partitioned into the two

Ž .following non-overlapping subsets: 1 training set,Ž21,108 graphemes i.e., 80% of the total number of

. Ž . Žsamples and 2 test set, 5,274 graphemes i.e., the.remaining 20% .

Finally, it must be noted that the choice of sam-ples to be in the training set or in the test set wasarbitrary for the three databases, and that no ambigu-ous, ill-segmented or misclassified character has beenremoved from either the training sets or the test sets.

4.3. Experimental results

The overall experimental results obtained fromthe testing sets are listed in Table 1, for the three

Ž .kinds of handwritten characters. TopM Ms1,2,3,4Ž .stands for the percentage of presence recognition or

Ž .absence substitution of the correct solution amongthe M best rated. For convenience, only the ‘‘recog-nition vs. substitution’’ index has been reported inTables 2–4. However, the particularity of the classi-

Table 1Ž . Ž .Overall ‘‘recognition % vs. substitution % ’’ results

Top 1 Top 2 Top 3 Top 4

Digits 97.84r1.85 98.96r0.85Uppercase 97.27r1.86 98.54r0.95 99.03r0.84lettersGraphemes 65.54r9.42 85.98r8.50 90.22r8.17 90.97r8.06

Table 2Ž .Detailed results as the confidence level Conf. increases: digits

Ž .10 classes

Ž . Ž . Ž . Ž .Conf. Recog. % Subst. % Reject. % Reliab. %

0.50 97.84 1.85 0.32 98.150.75 95.45 0.62 3.93 99.350.80 94.82 0.51 4.67 99.470.85 94.12 0.45 5.43 99.530.90 93.24 0.36 6.41 99.620.95 92.24 0.29 7.46 99.681.00 90.82 0.25 8.93 99.73

fier used leads us to define an ‘‘indecision rate’’according to the following relation:

recognition % qsubstitution % q indecision %Ž . Ž . Ž .s100% 1Ž .

Ž .Indeed, as mentioned in Devijver and Kittler, 1982 ,one drawback of the decision algorithm presented inSection 4.1 is that some decisions may be undefined

Žor ambiguous between two or more classes i.e., thesame confidence level is attributed to each of the

.classes involved . Therefore, TopM indecision raterepresents the percentage of characters for which

ŽMq1 classes at least including or not the correct.solution have been proposed with equal confidence

levels.From Table 1, we can see on the digit and

uppercase letter testing sets that the system canachieve a Top1 recognition rate of 97.84% and97.27%, respectively and a recognition rate of morethan 98.50% for both testing sets, if we consider the

( )L. Heutte et al.rPattern Recognition Letters 19 1998 629–641 639

Table 3Ž .Detailed results as the confidence level Conf. increases: upper-

Ž .case letters 26 classes

Ž . Ž . Ž . Ž .Conf. Recog. % Subst. % Reject. % Reliab. %

0.50 97.27 1.86 0.86 98.120.75 93.02 0.66 6.32 99.300.80 92.03 0.57 7.40 99.380.85 90.70 0.53 8.77 99.420.90 89.05 0.49 10.47 99.460.95 86.53 0.45 13.02 99.481.00 83.35 0.40 16.25 99.52

correct solution among the 2 best rated. On the otherhand, but not surprisingly, the Top1 performance ofthe system on the grapheme testing set is signifi-cantly lower: 65.54% recognition for 9.42% substitu-tion. Although this result seems to be poor comparedto those obtained on digits and uppercase letters, itmust be related to the large number of classes in-

Ž .volved i.e., 72 classes . Indeed, the larger the num-ber of classes, the larger the number of undefineddecision regions. This explains that 25% of thedecisions are undefined at the Top1 level. Besides,near 86% of the total number of graphemes arewell-recognized if we consider the correct solutionamong the two best rated: when replaced in thecontext of handwritten cursive word recognition, thislast result enables to generate more frequently the

Ž .correct word hypothesis Heutte, 1994 .The overall ‘‘recognition rate vs. substitution

rate’’ can give a broad idea of the SSFBV perfor-mance. However, this is not the only index forevaluating the efficiency and robustness of classifica-tion: performance can be indexed by the confidence

level which is used as a threshold for rejection. SuchŽperformances are listed in Tables 2–4 with a graphic

.representation of the results on the right for thethree kinds of handwritten characters. Recog., Subst.,Reject. and Reliab. are abbreviations of recognition,substitution, rejection and reliability rates, respec-tively. The reliability rate is defined by

Recog. Recog.Reliab.s s . 2Ž .

100%yReject. Recog.qSubst.

Since, as mentioned above, there is a trade-off be-tween the recognition performance and the numberof classes involved in the recognition process, thehighest performance is obviously achieved on thedigit and uppercase letter testing sets: with a rejec-tion rate of 8.9% of the total number of digitsŽrespectively 16.2% of the total number of uppercase

.letters , the system can achieve a substitution rate ofŽ .0.25% resp. 0.40% while still preserving a recogni-

Ž .tion rate as high as 90.8% resp. 83.4% . On theother hand, the lowest substitution rate achieved on

Table 4Ž .Detailed results as the confidence level Conf. increases:

Ž .graphemes 72 classes

Ž . Ž . Ž . Ž .Conf. Recog. % Subst. % Reject. % Reliab. %

0.50 65.54 5.77 28.69 91.910.75 60.85 3.48 35.67 94.590.80 59.76 3.30 36.94 94.760.85 58.59 3.17 38.24 94.880.90 57.06 3.02 39.92 94.980.95 55.52 2.79 41.49 95.221.00 52.81 2.59 44.59 95.32

( )L. Heutte et al.rPattern Recognition Letters 19 1998 629–641640

the grapheme testing set is only 2.6%, for a recogni-tion rate of 52.8%. However, if we take into accountthe wide range of shapes to be recognized in thegrapheme case, this last result appears to be good.

Finally, results given in Tables 2–4 demonstratethat the same SSFBV allows to achieve high perfor-mance on the three kinds of patterns, i.e., more than99.5% of reliability on digits and uppercase lettersand 95.3% on graphemes. These last results led tothe two following comments. The first one is that,even if the performance achieved on digits is broadlycomparable with other recent work using the same or

Žsimilar databases Xu et al., 1992; Sabourin et al.,.1993; Cao et al., 1995; Rahman and Fairhurst, 1997 ,

the results reported in this study have been obtainedby means of a simple and single structure of classifi-cation. This tends to prove that the combination ofmultiple classifiers may not be the only solution toyield a low error rate. The second comment is thatthe results for the uppercase letter set show only asmall deterioration, compared to those obtained ondigits, even though the class set has increased from10 to 26 classes. This is emphasized by the perfor-

Ž .mance achieved on the grapheme set 72 classes ,which is comparable or superior to the one obtainedon segmented characters in handwritten cursive word

Žrecognition problems Srihari, 1993; Chen et al.,.1994; Kimura et al., 1997 . This last remark suggests

that the combination of the seven families of featuresmay be more discriminatory for larger class prob-lems than other reported approaches, while keepinglittle advantage for smaller class sets.

5. Conclusions

We have presented in this paper a new featurevector for handwritten character recognition, whichcombines the strengths of both statistical and struc-tural feature extractors. Thanks to a combination of

Žseven complementary families of features rangingfrom pure structural to pure statistical and including

.both local and global features , a complete descrip-tion of the characters can be achieved, thus providinga wide range of identification clues. In order to feeda statistical classifier based on a linear discriminationtechnique, a parametrization of the structural featureshas been proposed and allows to get rid of the most

common-used transformation of these features intobinary values.

Even if this parametrization results in an over-di-mensioning of the feature vector and needs to definesome replacement rules for missing data, good per-formance can be achieved. The highest obtained arethe following ones:

Ž .i 0.25% substitution for 90.8% recognition on53,324 handwritten well-segmented digits extractedfrom the NIST Database,

Ž .ii 0.40% substitution for 83.4% recognition on3195 uppercase letters collected from US dead letterenvelopes,

Ž .iii 2.59% substitution for 52.8% recognition on5274 graphemes generated by an handwritten cursiveword segmentation performed on US address wordimages.

Finally, we have thus demonstrated that, by meansof a single structure of classification, high perfor-mance of recognition can be achieved on differentkinds of handwritten patterns using the same gen-eral-purpose feature vector, provided that this featurevector combines the strengths of both statistical andstructural feature extractors.

References

Baird, H.S., 1988. Feature identification for hybrid structuralrstat-istical pattern classification. Computer Vision, Graphics andImage Processing 42, 318–333.

Bunke, H., Sanfeliu, A., 1990. Syntactic and Structural PatternRecognition, Theory and Applications. World Scientific, Sin-gapore.

Cao, J., Ahmadi, M., Shridhar, M., 1995. Recognition of hand-written numerals with multiple feature and multistage classi-

Ž .fier. Pattern Recognition 28 2 , 153–160.Chen, M.Y., Kundu, A., Zhou, J., 1994. Off-line handwritten

word recognition using a hidden Markov model type stochas-tic network. IEEE Transactions on Pattern Analysis and Ma-

Ž .chine Intelligence 16 5 , 481–496.Devijver, P.A., Kittler, J., 1982. Pattern Recognition: A Statistical

Approach. Prentice-Hall, London.Dixon, J.K., 1979. Pattern recognition with partly missing data.

Ž .IEEE Transactions on Systems, Man and Cybernetics 9 10 ,617–621.

Fukunaga, K., 1990. Introduction to Statistical Pattern Recogni-tion, 2nd ed., Academic Press, New York.

Govindan, V.K., Shivaprasad, A.P., 1990. Character recognition –Ž .A review. Pattern Recognition 23 7 , 671–683.

Heutte, L., Moreau, J.V., Lecourtier, Y, 1993. A new featureextraction strategy for handwritten pseudo-character recogni-

( )L. Heutte et al.rPattern Recognition Letters 19 1998 629–641 641

tion based on multiple feature extractors. In: Proc. 6th IGSInternational Conference on Handwriting and Drawing, Paris,pp. 186–188.

Heutte, L., 1994. Reconnaissance de caracteres manuscrits: appli-`cation a la lecture automatique des cheques et des enveloppes` `postales. These de Doctorat de l’Universite de Rouen.` ´

Hu, M.K., 1962. Visual pattern recognition by moment invariants.IRE Transactions on Information Theory 8, 179–187.

Jain, A.K., 1990. Pattern Recognition. Tutorial on StatisticalPattern Recognition, in: Proc. 10th International Conferenceon Pattern Recognition, Atlantic City, NJ, pp. 8–20.

Kimura, F., Kayahara, N., Miyake, Y., Shridhar, M., 1997. Ma-chine and human recognition of segmented characters fromhandwritten words. In: Proc. 4th International Conference onDocument Analysis and Recognition, Ulm, pp. 866–869.

Plessis, B., Sicsu, A., Heutte, L., Menu, E., Lecolinet, E., Debon,O., Moreau, J.V., 1993. A multi-classifier combination strat-egy for the recognition of handwritten cursive words. In: Proc.2nd International Conference on Document Analysis andRecognition, Tsukuba, pp. 642–645.

Rahman, A.F.R., Fairhurst M.C., 1997. Introducing new multipleexpert decision combination topologies: a case study usingrecognition of handwritten characters. In: Proc. 4th Interna-tional Conference on Document Analysis and Recognition,Ulm, pp. 886–891.

Sabourin, M., Mitiche, A., Thomas, D., Nagy, G., 1993. Classifiercombination for hand-printed digit recognition. Proc. 2nd In-ternational Conference on Document Analysis and Recogni-tion, Tsukuba, pp. 163–166.

Shridhar, M., Badreldin, A., 1984. Recognition of isolated andconnected handwritten numerals. In: Proc. IEEE InternationalConference on Systems, Man and Cybernetics, pp. 142–146.

Shridhar, M., Houle, G., Kimura, F., 1997. Handwritten wordrecognition using lexicon free and lexicon directed wordrecognition algorithms. In: Proc. 4th International Conferenceon Document Analysis and Recognition, Ulm, pp. 861–865.

Smith, S.J., Bourgoin, M.O., Sims, K., Voorhees, H.L., 1994.Handwritten character classification using nearest neighbor inlarge databases. IEEE Transactions on Pattern Analysis and

Ž .Machine Intelligence 16 9 , 915–919.Srihari, S.N., 1993. Recognition of handwritten and machine-

printed text for postal address interpretation. Pattern Recogni-Ž .tion Letters 14 4 , 291–302.

Trier, O.D., Jain, A.K., Taxt, T., 1996. Feature extraction methodsfor character recognition – a survey. Pattern Recognition 29Ž .4 , 641–662.

Xu, L., Krzyzak, A., Suen, C.Y., 1992. Methods of combiningmultiple classifiers and their applications to handwritingrecognition. IEEE Transactions on Systems, Man and Cyber-

Ž .netics 22 3 , 418–435.