A Survey on Recognition of On-Line Handwritten Mathematical

Notation∗

Ernesto Tapia and Raul Rojas

Technical Report B-07-01

Freie Universitat Berlin, Institut fur InformatikTakustr. 9, 14195 Berlin, Germany

tapia,rojas@inf.fu-berlin.de

July 31, 2007

Abstract

This report describes recent advances in the area ofthe recognition of on-line handwritten mathematicalnotation. We describe architectures, symbol classi-fication methods, and techniques for the structuralanalysis of mathematical expressions. We also surveyapplications specialized for mathematical notation.

1 Introduction

Interest on developing pen-computing applicationshas been growing steadily during the last years, dueto, among other factors, the introduction of devicessuch as personal digital assistants (PDAs) or TabletPCs. The main characteristic of such devices is thatthey use the stylus as input tool, being a “natural”substitute for keyboards ans mouse, see Fig. 1.

The data generated by users writing with a styluson an electronic device is known as digital ink, andthe process of writing is called on-line handwriting.The minimal unit which forms digital ink are strokes,which are sequences of points generated between pen-down and pen-up events, at regular time intervals.

The main objective in pen-computing is not onlyhandling digital ink “as is” but also its conversioninto another data structure that can be automaticallyprocessed by computers. The elemental data struc-tures are alphanumeric characters, since PDAs wereoriginally developed to be personal organizers witha calendar or an address book. Most PDAs use the

∗A revised and extended version of this report is underpreparation for submission to a journal. We welcome any com-ments and suggestions to improve this work.

Figure 1: Pen-based devices.

stroke alphabet Graffiti [39] as the standard methodfor symbol recognition. The alphabet consists of aset of pre-defined stroke combinations that allow therecognition of letters and numbers, see Fig. 2.

The development of pen-based systems and appli-cations increased when Microsoft released the TabletPC version of the Windows operating system. It pro-vides a Software Development Kit (SDK) for process-ing, storing, and recognizing digital ink. These li-braries are specialized towards the recognition of let-ters and words of western languages and East Asianlanguages, allowing a more natural writing whencompared with Graffiti. One of its most attractivecharacteristics is that the recognizers are integratednatively in the operating system.

Even though pen-based mathematics has a po-tential application in scientific document process-ing, human-computer interaction and mathemati-cal knowledge management, the methods mentionedabove are only capable to recognize plain text, butno complex handwritten mathematics.

1

Figure 2: The Graffiti alphabet.

Figure 3: Maple’s recognition system.

Fortunately, pen-based mathematics has gained at-tention by the computer science community over thelast years. An example of this increasing interest isthe development of the symbol recognition tool forthe computer algebra system Maple 10. Its pen-basedinterface allows the recognition of isolated mathemat-ical symbols that are not available in the keyboard,see Fig. 3.

This report describes the advances in the areaof pen-based mathematics and on-line mathemati-cal recognition that have during recent years. It isorganized as follows. Section 2 describes the maincharacteristics which distinguishes the recognition ofon-line mathematics, as well as architectures for therecognition of mathematical notation. Section 3 de-scribes briefly the symbol recognition task. We givean overview of the most important techniques forstructural analysis in Sec. 4. Recent work on user in-terfaces for on-line handwriting is described in Sec. 5.

2 Recognizing mathematicalnotation

In comparison with plain handwritten text, recog-nition of on-line handwritten mathematics requiresmore elaborated symbol classifiers because mathe-matical notation uses a big set of characters includingLatin letters, Greek letters and mathematical sym-bols. Other difficulty relies also on the analysis ofthe intrinsic two-dimensional layout encountered informulas, arrays of symbols and diagrams, which re-quires a different treatment in comparison with hand-written plain text. These and other relevant char-acteristics of mathematical notation are enumeratedbelow, as suggested by Blostein [6], and Chan andYeung [9].

2.1 Characteristics of MathematicalNotation

2.1.1 Definition of mathematical notation

Mathematical notation is a specialized two-dimensio-nal notation which helps to communicate mathemat-ics and to visualize concepts and ideas. Althoughmathematical notation is a language used in many ar-eas of science, no formal definition in terms of syntaxand semantics as a two-dimensional language exists.Actually, this notation is not completely standardizedand many dialects are used by scientists. Some au-thors try to describe mathematical notation for solv-ing problems of typesetting [23] and for automaticprocessing of mathematical notation [29].

2.1.2 Scope of recognition systems

We have to consider which kind of mathematical ex-pressions we will recognize. Researchers normally re-strict themselves in the recognition of a subset ofmathematical notation, for example notation for aparticular area of mathematics, notation needed onlyfor high-school mathematics, etc. In addition, suchrestrictions also depend on the purpose of the recog-nition system: recognition of typeset scientific text,an input model for computer algebra systems, etc.

2.1.3 Grouping of basic symbols

When we read an expression, we group single symbolsto construct more complex objects, which plays animportant role in the interpretation of formulas. Forexample, the digits ‘3’, ‘8’, and ‘1’ have their ownmeaning as such, but if they are of the same size andlie in the same line, they can represent the integer

2

value ‘381’. By varying size and location they canalso represent ‘381’ or ‘381’. Similarly, we can groupsome letters to represent function names, like ‘log’ or‘sin’.

2.1.4 Explicit and implicit operators

Variation of size and location can also representgrouping criteria through mathematical relations be-tween symbols. In mathematical notation, we findexplicit and implicit relations between symbols. Sub-scripts, superscripts and tabular structures are de-scribed only by the location of operands; in contrast,addition, subtraction, and division use explicit oper-ator symbols. For example, in the expression ‘a + b’the explicit relation ‘addition’ between ‘a’ and ‘b’ isgiven by the symbol ‘+’. An example of an implicitrelation would be in the expression ‘x2’. In this casethe expression represents the relation ‘pow’ between‘x’ and ‘2’, a variation in the location results in a verydifferent relation, for example ‘x2’, representing therelation ‘subscript’.

2.1.5 Ambiguity in the role of symbols

We illustrate this characteristic by giving some ex-amples. The horizontal bar can represent the minussign and also the fraction bar. A dot can representmultiplication or the decimal point. The Greek let-ter sigma ‘Σ’ can also represent the operator sum.When grouping the symbols ‘dy’, they can representthe product of ‘d’ and ‘y’ as in ‘cx+ dy’ or the differ-ential operator when used in ‘

∫cos(y)dy’.

2.1.6 Irregular writing

Irregular handwriting aggravates ambiguities de-scribed before and makes it harder to group symbolsand to distinguish relations among them. It results inlayout problems affecting the recognition of the wholeexpression. A cause of this is due to unexperiencedusers, because they normally take excessive freedomwith the location and alignment of handwritten sym-bols. Other kinds of irregular writing arise during thecorrection, deletion, and insertion of symbols. Forexample, suppose an expression was entered by theuser and he decided to write some super-indices orsub-indices, but there was not enough space avail-able for this correction. Something like this couldgenerate a very complex expression, which could notbe recognized even by humans.

Figure 5: A typical on-line document.

2.2 Architectures and frameworks

This sections describes three architectures which areimportant for the development of pen-based math-ematics. From our point of view, they offer solu-tions for the same problem which can be consideredas mutually complementary. The architectures con-sider three core building blocks for pen-based math-ematical system: recognition, development, and user-system interaction.

2.2.1 Recognition

Most of the working systems and prototype programsfor the recognition of pen-mathematics follow, inessence, the steps proposed by Lee and Wang [36].Although his system is specialized for off-line math-ematical equations, i.e. expressions in scanned doc-uments, almost all of the procedures can be used aswell for the recognition of pen-based mathematics.We have only to substitute the “Optical Scanning”step for a “Ink Input” step in the flow diagram shownin Fig. 4.

The recognition of mathematical notation beginsby considering how the data is presented. Opticalscanning or ink input follows digitalization step thattransforms the given expression into a static (off-line)or dynamic (on-line) representation. In the segmen-tation step, mathematical expressions are extractedand isolated from text lines [36, 26]. When dealingwith on-line data, it is normally supposed that thedata only contain mathematical expressions. How-ever, it is not always the case, because modern on-line documents are also formed by a mix of plain text,graphics and mathematics, see Fig. 5

3

Figure 4: Digram flow for the recognition of mathematical expressions in off-line documents.

Once the document is segmented, the next step issymbol recognition. In this step, the label of sym-bols is established by means of a classifier. Duringthe expression formation step, the structure of therecognized symbols is analyzed to form a hierarchicalstructure that represents a mathematical expression.

In the last step, the internal hierarchical structureis processed and interpreted to obtain a final result.This result can be a character string which representsthe expression in LATEX, another structure used by acomputer algebra system, or an image representingthe plot of a function given by the expression, amongothers.

Their procedural framework can be divided in threemain modules:

1. Document Segmentation.

2. Symbol Recognition.

3. Structural Analysis.

Differences between recognition systems found in the

literature are generated by variations and improve-ments of these modules.

2.2.2 Development

Most of the systems created for the recognition ofmathematical notation have for objective the conver-sion of the handwritten expression into a data struc-ture, which can be used mainly by a text editor orfor some computer algebra system. Ideally, such ageneral system should be used for a wide range ofapplications such as the elaboration of mathemat-ical databases, text processing, symbolic computa-tion, elaboration of mathematical diagrams, etc. Itshould also be easily extended and provide a practicalGUI.

Smirnova and Watt [54] propose an architecturalframework for pen-based mathematics from the per-spective of software engineering. Their target deploy-ment is for document processing and mathematicalcomputing. Their objective is to define a platform-independent pen-based framework for mathematical

4

Figure 6: Framework components for mathematicalnotation.

entry, edition and calculation. These objectives leadthem to define two portability criteria for the frame-work. Longitudinal portability is the software life andusability given a platform, and lateral portability isthe software life across platforms.

They refer to the Microsoft’s Tablet PC SDK as anexample for longitudinal portability: this API evolvewith the time having a direct impact for the devel-opment of the framework. Thus, they suggest twomain components of their framework that have to re-main the same along longitudinal evolution: a) highlevel manipulation of mathematical objects and b)low level analysis of digital ink, see Fig. 6.

An abstract layer with platform-specific wrappersallows the lateral portability of the software. Thewrappers access natively resources and recognitionengines already contained in the platform (PalmGraffiti or Microsoft Tablet PC, for example). Thecomponents of their framework which shall vary, de-pending of the platform, are 1) basic software to col-lect digital ink, 2) low level processing module tointerconnect the components (a) and (b) mentionedabove, and 3) use of the framework in some hostingapplication. Figure 7 illustrates the complete archi-tecture that includes recognition modules.

2.2.3 User-System Interaction

LaViola [33] proposed a new paradigm for gesturalinteraction in pen-based mathematics. He considerpen-based mathematical documents to be constitutedfrom mathematical expressions and supporting dia-grams. His work concentrates in the development ofa gestural user interface, which allows a seamless in-teraction between these document constituting partsby generating actions from gestures. Figure 8 showsthe architecture he proposed.

The most relevant from this architecture are theUser Interface and Sketch Preparation modules. TheUser Interface Module has a Gesture analyzer compo-nent which decides whether the last written strokes

should be interpreted as a gesture. If it is decidedso, the corresponding action generates a connectionwith others modules. In the other case, the ink datais stored by the Data Entry component. The Cor-rection UI allows the user to correct the recognizedmathematical expression at the symbol level or at theparsing level.

The Sketch Preparation module has an Associa-tion Inference module, which associate label draw-ing elements with written mathematical expressions.These labels describe some property of supportingdiagrams, such as its location expressed as Cartesiancoordinates that are modifiable by means of the writ-ten expressions. After the analysis of the expressionby the Math Code Generation module, the diagramcan be animated. The Drawing Rectification mod-ule works in a similar way as association, but it isused to correct diagram parameters such as angles,lengths and location. As an example for rectification,imagine that a user draws a triangle to illustrate thePytagora’s Theorem, whose cathetus are not orthog-onal. This can be corrected by labeling the angle witha variable a to the angle, writing a formula a = 90 andusing the corresponding gesture which activates therectification module to correct the drawing. The Di-mension analysis component us used to calculate localcoordinates systems of suporting diagrams, which areuseful to generate animations in the ink document.

3 Symbol Recognition

3.1 Segmentation

Segmentation is the process of decomposing, group-ing, or isolating the data into classifiable units whichrepresent single symbols or words. In the case of on-line documents, the segmentation problem consists ofproperly classifying strokes as text, graphics.

An example of a simple heuristic-based segmenta-tion is the method used by Mandler [40]. He asso-ciates the input strokes with a predetermined thick-ness by dilating them with a rectangle. With thiscriterion, two strokes belong to the same symbol iftheir dilated versions intersect.

Sometimes, segmentation methods use the infor-mation given by the classifier. Winkler et al. [30, 65]generate a symbol hypothesis net from the givenstrokes. The hypothesis net generates all possiblecombinations of strokes in order to handle ambigu-ity during symbol grouping. The different combina-tions of strokes in the net, which represent poten-tial symbols, are classified using a Hidden MarkovModel (HMM). The probabilistic output of the classi-

5

Figure 7: The complete architecture for pen-based mathematics.

fier serves to associate a likelihood value to each com-bination of strokes, and the one with the highest like-lihood value is selected as the final segmentation re-sult. This method is sensitive to the order the strokesare written. If they are not written from left to rightand upside-down, segmentation and classification de-teriorate dramatically. Examples of classifier-basedsegmentation are used by some of the systems we de-scribe in Sect. 5.

Shilman et al. [51] present a method for the seg-mentation of digital ink based on a neighborhoodgraph and symbol classification. Strokes are regardedas nodes of a graph, where stroke are connected withan edge when they are “close” to each other. Oncethe graph is constructed, it is segmented into its con-nected components with a number of nodes up to apreviously selected number k. Each connected com-ponent is regarded as an isolated symbol, which isclassified using a previously trained model. A nega-tive log likelihood cost is assigned to every partitionbased on the classification results. Finally, dynamicprogramming is used to efficiently find the optimalstroke partition. The main advantage of this methodis that it is insensible to stroke order.

Ao et al. [2] present a framework based also in aneighborhood graph to find structures in digital inkdocuments. Their method is aimed to find baselinesstructures and to segment the document into textualand graphical parts. The main steps in their methodare patches creation, baseline extraction, separationinto text and graph, and word segmentation. Thepatches are the node subsets which are obtained byeliminating long edges from the neighborhood graph.They are grouped into text baselines my maximiz-ing their line strength. The line strength is basedon edge lengths (closely positioned), angle differences(linearly positioned) and sizes of neighboring blocks(comparable size). The classification of patches astext of graph is based in the supposition that graphi-cal parts do not present a “reasonable” layout struc-ture. The structure can be quantified by consideringwidth, height, stroke count, and density features onthe patch and their neighboring patches. This fea-tures are used to train a binary classifier which servesas model for the text/graph segmentation. Similarmethods to find text baselines and to classify digitalink as text or graph can be found in [12, 25, 52, 5, 67]

6

Figure 8: Architecture for mathematical sketching.

3.2 Preprocessing

Preprocessing is necessary to eliminate noise, to re-duce the amount of information, and to normalizehandwriting [59, 19].

An example of preprocessing to eliminate noise issimply the application of a Gaussian filter to a hand-written stroke. In this case, processed points in thestroke are the result of a weighted average of itself andits neighbors. Dehooking, point clustering, filtering,and stroke connection are useful preprocessing proce-dures to eliminate unnecessary information from thestroke. Examples of normalization for on-line dataare scaling, slant correction, and equidistant resam-pling.

3.3 Feature Extraction

Feature extraction means deriving measures andcharacteristics from the raw data which are useful inmaking predictions. It is common that feature extrac-tion methods are based on invariance, reconstruction,and expected distortions.

Features may be local or global. The source of infor-mation of on-line data are points and strokes. Whenwe talk about local features, we refer to character-istics of a specific point in strokes derived from itsneighboring points. Global features normally referto topological (morphological) characteristics of thestroke based on the trajectory it describes.

The first local feature we consider are the coordi-nates of a point. They serve, for example, to deriveimportant local characteristics, like curvature or di-rection. Points having a high curvature are important

to decompose strokes in static elements. This decom-position is a structural description of the underlyingshape of strokes [19].

Global features normally serve to obtain a cate-gorical (not numerical) classification of strokes. Forexample, the ratio of the distance between a stroke’send points and its length is a global feature, whichserves to determine if the stroke is closed or open,when comparing this ratio against a predeterminedthreshold. Similarly, when considering a symbolformed by two strokes, we can say that they crossor does not cross, or only touch. This kind of in-formation serves to construct a decision tree whichreduces the given data into a small set used in a lateranalysis using more complex features [59].

3.4 Symbol Classification

Symbol classification means the use of a certainmethod to obtain the symbol’s label. Actually, wecan potentially use any of the great number of exist-ing classification methods to such purpose. In thissection we review some of such methods which werespecially developed for symbol classification. Otherinteresting results for classification of isolated sym-bols is [46].

3.4.1 Similarity Measures.

Similarity measures for strokes allow the use ofdistance-based classifiers. Generally, unknown sym-bols are matched against stored references (proto-types), and the label corresponding to the unknownsymbol is the same as that of the symbol which best

7

matches it. Similarity measures are normally definedat stroke (curve) level.

Tappert [58] uses elastic matching as a similaritymeasure. The procedure to calculate it is based onthe alignment of points which constitute the strokeusing the dynamic time warping (DTW) algorithm.The optimal matching corresponds to the minimumsum of matching costs, which are normally the dis-tance between strokes’ points obtained by dynamicprogramming. This algorithm allows the comparisonof strokes having a different number of points. In thiscase, the feature-extraction step can be skipped.

Li and Yeung [37] also use time warping to calcu-late similarities between strokes. Their method con-verts strokes into a character string. The string de-scribes the stroke as a sequence of eight directions.They find the distance between two string sequencesby calculating costs of three string transformations,namely compression, expansion, and substitution.

Schwenk [50] proposes a similarity measure basedon constrained tangent distance. The distance islocally invariant under the following local transfor-mations: translation, scale, axis deformation, rota-tion, diagonal deformation, and slope transformation.This method tries to incorporate knowledge about ge-ometrical variations of handwriting.

Other similarity measures for curves which can beuseful for on-line symbol classification can be foundin [61].

3.4.2 Structural Methods.

Structural methods are based on the assumption thathandwriting is made up of some elementary or primi-tive shapes, also known as allographs. They describethe morphological characteristics of strokes. For ex-ample the symbol ‘6’ can be describes through twoprimitives: a descending curve and a loop.

Parizeau and Plamondon [45] construct a set ofbasic allographs using a fuzzy syntactic approach tomodel handwriting. For example, a primitive can beof type ‘c’ or ‘ci’, which corresponds to stroke seg-ments with a shape similar to the letter ‘c’ where the‘i’ indicates “inversion”, i.e. a horizontal or verticalreflection of the shape. In this way, they considertwenty allographs, which correspond to i-shapes, c-shapes, t-crossings, loops, etc. A symbol is coded us-ing a grammar formalism as a sequence of these prim-itives and the similarity between symbols is measuredby using a sequence distance. Similar approaches areused in [7, 48, 66].

3.4.3 Neural Network and Statistical Mod-els.

Guyon et al. [22] developed a neural network systemfor the recognition, based on so-called time delay neu-ral networks. The system integrates recognition andsegmentation in the same neural network architec-ture. The network is used to estimate a posterioriprobabilities for characters in a word. One of thelayers of the network converts the temporal infor-mation, point position, curvature, and direction ina two-dimensional image representation. Similar ap-proaches are used in [38], which have been speciallydeveloped for the recognition of on-line handwrittenwords.

One of the most recent statistical approaches usedfor on-line recognition are Hidden Markov Models(HMM). They use dynamic programming to find theoptimal alignment which performs character segmen-tation and recognition simultaneously. Bellegarda [4]uses local features, point positions and curvature,to feed a simple two-state HMM. Sin and Kim [53]also uses a HMM to model inter-letter relations, alsoknown as ligatures. They train a HMM for each letterand ligature type. Schenkel et al. [49] uses a hybridneural-network-HMM system for on-line word recog-nition.

Shwenk [50] uses an auto-encoder neural networkto construct a model for each symbol class. Thisauto-encoder network actually describes each sym-bol class by finding the principal components thatdescribe the class population. New symbols are fedinto each network and projected to a hyperplane gen-erated by the principal components. The tangent dis-tance between the encoded (projected) vector and theoriginal one is calculated. The class label assigned tothe new vector corresponds to the one where the min-imum distance is reached.

Bahlman et al. [3] combine dynamic time (DTW)warping and support vector machines (SVM) to clas-sify isolated symbols. They use a Gaussian kernelto train the SVM-classifier and substitute the vec-tor metric in the exponential argument of the kernelwith the DTW distance. Theoretically, this substi-tution can lead to some numerical problems, becausethe DTW distance is not a metric, a strong assump-tion within the SVM approach to construct a kernelfunction. However, it was shown experimentally thatonly in an small set of training examples this assump-tion is not fulfilled, an issue that could be avoided bynot considering that set.

Cho and Kim [14, 13] classify isolated symbolsby modeling strokes internal relationships within aBayesian network framework.

8

3.4.4 Clustering Methods.

Some of the above recognition approaches were devel-oped in a user-dependent manner. Clustering meth-ods in on-line handwriting are developed to remedythis situation. They try to model intra-class varia-tions caused by different writing styles found in largedatabases. Vuori [62] uses a self-organizing map tocluster writing styles. She uses variations of the elas-tic matching method as a similarity measure. Herclustering algorithms can be used for prototype se-lection, which serves to classify characters accordingto the k-Nearest Neighbors (k-NN) rule. The recog-nition system adapts to new writing styles by modi-fying its prototype set. In the same fashion, Connelland Jain [15] use also elastic matching to measureintra-cluster and inter-class measures, to be used ina hierarchical clustering algorithm.

Watt and Xie [63] propose a clustering method forthe classification of large on-line symbol databases.The proposed methods can recognize four symbolssets used in mathematical formulas which include dig-its, uppercase and lowercase Latin letters, Greek let-ters, mathematical operators and arrows, summingup a total of 277 symbol classes. They find clusters inthe symbol database based on the following features:number of strokes, initial stroke position, width toheight stroke ratio, and the initial and end directionas north, north-west, etc. They selected these fea-tures not only by their effectiveness, but also by theirlow computational cost. The cluster in the databaseare pruned by locating prototypes via a elastic match-ing distance. A new symbol is classified by locatingit in a cluster and assigning it the label of the clos-est prototype using elastic matching. The methodreduces the computation time with a small degrada-tion of the classification rate.

4 Structural Analysis

4.1 Expression Formation

Expression formation consists of translating the re-sults obtained in the symbol recognition step into atree which describes the relations between symbols ofthe mathematical expression. The nodes that consti-tute this tree are a data structure which stores thesymbol’s label and other numerical parameters, forexample size and location. Nodes are a compact rep-resentation of the original raw symbol. After an anal-ysis of the relative positions of symbols, the nodesin the tree are linked to each other depending onwhich one of the spatial relations –above-left, above,superscript, right, subscript, below, below-left, and

subexpression– they satisfy [6, 36]. As we will see,most of the methods described in this section rely onthis principle.

Fateman et al. [17] developed a recognizer for type-set mathematical expressions. The recognized sym-bols are grouped into box tokens representing num-bers, function names, and variables. They use abottom-up recursive descent parser to obtain the finalresult as a lisp expression.

Lavirotte and Pottier [35] define a context-sensitivegraph grammar to represent mathematical formulasand to remove ambiguities during structural analy-sis. Rules in graph grammars consist of collapsing asubgraph in a node, if they satisfy a given condition.They define a set of rules to describe the mathemati-cal relation and apply a bottom-up parsing algorithmto obtain the final result. If we consider the expres-sion ‘ex+1’, a rule is defined to collapse the two nodesrepresenting the symbols ‘e’ and ‘x’. They now repre-sent a single token, and the process can be continuedby collapsing the new tokens ‘ex’ and ‘1’ to constructthe final result.

Miller and Viola [42] use a generative model ap-proach based on a stochastic-free grammar. Therecognition process of the structure consists in find-ing the productions of the grammar with the highestprobability when considering the overall probabilityof generating the expression. They use a convex hullcondition to prune the searching space of the gram-mar productions. A production in the grammar isnot admissible if the convex hull of a symbol groupintersects a symbol not belonging to the group.

Winkler et al. [65] used HMMs for segmentationand classification of symbols. Using a soft-decisionapproach, they construct a hypothesis net of all pos-sible segmentations of strokes. They store the in-formation of the possible spatial relations betweensymbols in a directed graph, and a set of alternativeinterpretations of the expression is given.

Kosmala et al. [32] also present a statistical ap-proach based on the application of HMMs. The sys-tem allows simultaneous segmentation and symbolclassification, as well as the interpretation of the sym-bols’ spatial relationships. One of the conditions forcorrect segmentation and recognition is that the ex-pression be written from left to right and from topto bottom. Delayed symbols are not allowed, aswhen first writing an expression and then enclosingit in parenthesis. In an posterior work [31] they usegraph rewriting for the structure analysis to avoid theabove-mentioned restrictions of handwriting.

Chan and Yeung [8] propose the use of a definite-clause grammar (DCG) to define precise replacement

9

rules when parsing mathematical expressions. Theyuse DCG because a grammar expressed through thisformalism is declarative and easy to maintain and ex-tend. They use a parsing scheme using left-factoringto reduce the time for the expression’s interpretation.

Eto and Suzuki [16] use a weighted graph to rep-resent spatial relations between symbols. The nodesin the tree represent the symbols obtained during thesegmentation stage. They are connected dependingon whether they satisfy the superscript, subscript, orright relations. Because of the multiple results ob-tained during the recognition step, nodes have mul-tiple edges which store a symbol’s label and weight.They generate a set of spanning trees of the initialgraph, which result from the minimization of the edgecosts. If they result in an admissible (not contradic-tory) mathematical structure, one of them is selectedas the final result if it minimizes a global cost crite-rion.

Zanibbi et al. [68] describe a mathematical expres-sion as a structure of nested baselines, a so-calledbaseline structure tree. Baselines constitute horizon-tal arrangements of symbols. The first step in theprocedure recursively constructs the baseline tree inthe handwritten expression by handling irregular orpoor horizontal layout structures. The second stepconverts the structure tree into an operator tree,which is further processed by applying tree transfor-mations using the TXL language. Tree transforma-tions consists of locating patterns in the tree struc-ture and applying replacing rules on them. Definingdifferent replacing rules helps to define a particulardialect or recognition scope of mathematical nota-tion.

Zahng et al. [70] use fuzzy logic to define spatial re-lationships among mathematical symbols. They usefuzzy membership functions to define the superscript,subscript and horizontal symbol relationships. Theyproduce recursively different alternatives for the in-terpretation of a expression in the case of non-zeromembership values. To illustrate de procedure, con-sider the expression abcd, where a is the dominantsymbol in the expression. The fuzzy regions providetwo interpretation for the spatial relations between aand b, generating the two alternatives ab and ab. Themethod proceeds recursively with these two subex-pressions, generating another two interpretations abcand a[bc] for the latter. The square brackets indicatethat the symbols they embrace must be analyzed aswell. A final interpretation is a symbol group whichcan not generate any alternative. Finally, confidencevalues are assigned to final alternatives by taking theminimum membership value among all the fuzzy spa-

tial relationships in the expression. The interpreta-tions are presented to the user as list ordered by theconfidence values.

4.2 Error Correction

Most of the approaches mentioned before convert atwo-dimensional representation of the expression intoa one-dimensional one, a string of characters, whichrepresents the expression in a certain computer lan-guage. Erroneous symbol segmentation and classifi-cation can generate incorrect results during structuralanalysis.

Lee and Wang [36] propose heuristics for the cor-rection of the most common errors derived by falserecognized symbols. They consider four heuristicrules. One of them is “for every binary operator theremust exist two operands”. By applying this rule theexpression ‘/ < i < n’ is corrected to ‘1 < i < n’.They also define a rule to consider errors encoun-tered during the construction of function names. Anexpression like ‘5inx’ should be corrected to ‘sinx’.Numerals with subindexes are considered as an error.Expressions like ‘12’ and ‘5b’ are corrected to ‘l1’ and‘Sb’ respectively. The last rule considers the correc-tion derived by case confusion of letters and othercharacter properties, considering whether it is italic,bold, etc., because letters forming a determined groupin the expression are similar. Expressions like ‘3Pqr’and ‘x · y’ are transformed to ‘3pqr’ and ‘x · y’.

Chan and Yeung [10] extend their system basedon definite-clause grammar to handle lexical, syn-tactic, and semantic errors. They implement somerules to identify common errors in string grammar,namely substitution, deletion, and insertion. Theyalso incorporate rules for the correction of some com-mon syntactic errors, like incorrect implicit opera-tors, missing binding and fence symbols, or missingoperands. The semantic errors they consider are cor-rected heuristically, similar to the way Lee and Wangdo. The lexical errors they consider take place at therecognition and segmentation level, caused by poordigitalization or irregular writing. They also proposesome performance measures that concern the recog-nition of expressions, symbols, and operators, as wellas an integrated performance measure.

Actually, not much effort is expended on the errorcorrection task. Some other authors does not handleautomatically the correction of errors. For this pur-pose, they offer instead user-friendly interfaces whichallow immediate feedback and have undo-redo andvisualization capabilities, as we will see in the nextsection.

10

Figure 9: The Natural Log system.

5 User Interfaces

5.1 The Natural Log System

The Natural Log system is a user-dependent systemdeveloped by Matsakis [41]. The system was writtenin Java and is only available on-line as an applet onthe internet. See Fig. 9.

To classify on-line symbols, he constructs a high-dimensional normal distribution, which describes thepopulation of each class. The symbol label corre-sponds to the class that has the maximum probabil-ity. Low probability values are used to reject symbolswhich can represent potential errors in the handwrit-ing. The procedure to recognize a given mathemati-cal expression begins by finding an optimal groupingof the written strokes into isolated symbols. The fi-nal grouping of stroked is determined by evaluatingall possible groupings and taking the one which min-imizes a sum-cost function. This function is the sumof the log likelihood of the classifier output of eachsymbol in the current partition. To make the opti-mization of the cost function manageable, its evalua-tion is constrained by the minimum spanning tree ofstrokes, considering the centers of strokes’ boundingboxes as nodes of a completely connected weightedgraph. Different combinations of subtrees of the min-imum spanning tree are evaluated and the optimalone is taken as the final segmentation result.

The structural analysis in this system consists oflocating a “key” symbol usually an explicit mathe-matical operator. Once the key symbols are located,the parse algorithm proceeds to find their correspond-ing operands, and partial subexpression are formed.The procedure is applied recursively until no morekey symbols are found. The algorithm is extended tosupport parsing of superscripts, i.e. to non-explicitoperators, but no support for subindexes is offered.

Figure 10: Free Hand Formula Entry System

5.2 Free Hand Formula Entry System

The Freehand Formula Entry System is a pen-based equation editor developed by Smithies andNorvins [55] and distributed under the GNU Gen-eral Public License. The program runs under Linuxand MacOS X platforms.

The classification of symbols is done by using thenearest-neighbor method. The developers use confi-dence information supplied by the classifier to groupstrokes. Their method proceeds by generating all pos-sible combinations of a fixed number of strokes (bydefault they take a maximum of four strokes), whichpotentially can constitute a single symbol. Once sin-gle symbol is classified, the confidence level of a com-bination correspond the lowest output of the classi-fier. Finally, the group with the highest confidence istaken and the first symbol in the group is returnedand considered a correctly recognized character. Theprocedure is repeated, once again, when a fixed num-ber of input strokes is reached.

For the structural analysis, they first used amethod based on graph rewriting, similar to themethod developed by Lavirotte and Pottier [35]. Fig-ure 10 shows a version of their program, modifiedby Zanibbi [68], which uses the structural analysismethod he developed.

11

Figure 11: Infty editor

5.3 Infty Editor

Infty Editor [44, 56] is a commercial system special-ized for creating mathematical documents. The edi-tor is linked to the computer algebra system Mathe-matica by Mathlink. It also supports input and out-put of expressions in TEX format. The editor containsan real-time recognition system for mathematical ex-pressions. See Fig. 11.

The recognition system combines segmentationand recognition of characters to remedy difficultiesin structural analysis due to irregular symbol posi-tion and size [27]. The rewriting puts symbols intoextendable symbols and unextendable ones. The for-mer are extended to form other symbols by addingmore strokes and the latter are written with onlyone stroke. For example, F can be extended into E.When a stroke is classified as unextendable, the clas-sification result is rewritten by the computer in thedrawing area using a predefined prototype. If a strokeis classified as an extendable character, the systemwaits for the next strokes. The classification result iswritten automatically if a predetermined time inter-val has elapsed or the expected number of strokes isreached.

5.4 MathJournal

Wenzel and Dillner [64] describe another commercialproduct, MathJournal, developed for the Tablet PCversion of the Windows platform. The interface isvery similar to Microsoft’s Journal program, which isincluded in the operating system, see Fig. 12. Ap-parently, the program is still under development andonly descriptions of it are given. The developers alsooffer the xThink Calculator as an evaluation program.The recognition capabilities of this program are sim-

Figure 12: MathJournal.

ilar to the ones of MathJournal. It operates as anormal pocket calculator, the operations are done byrecognizing a handwritten arithmetical expression.

MathJournal uses the recognizer integrated in theoperating system for the classification of isolatedhandwritten characters. Although it is possible torecognize special mathematical symbols and con-stants (square root, calculus operators, and the con-stant π), the system does not recognize Greek letters.The symbols recognized by the system are limited tothe ones recognized by the Microsoft API.

In the description of the system, they mention thatheuristics of graph rewriting and, when required, aminimum spanning tree construction are used duringstructural analysis.

The most relevant aspects of this system are its“solution engines”. They process the recognized ex-pressions in numeric, graphic, or symbolic formats.Diagrams, such as function tables, are processed andplotted by using curly braces and arrows as gestures.Similar gestures are used for the solution of equationsystems or for plotting functions.

5.5 MathPad

MathPad is a system for the creation and explorationof mathematical sketches. The main objective in thisuser interface is to facilitate the user-editor interac-tion using sketches. This gestural user interface is oneof the most advanced in terms of pen-based design.

The set of gestures are used to interact with therecognition and processing engine included in the sys-tem. The basic interaction is the drawing of a lassogesture followed by a tap. This action indicates thesystem the recognition of the selected strokes. It isused also for the association of variables and strokegrouping. Drawing the equals sign followed by a tapor by the minus sign allows the solution of equations

12

Figure 13: Gestures used in the mathematical sketch-ing paradigm.

or factorization of expressions. Other gestures areused for deletion of ink, association of recognized ex-pressions, and angle association. Figure 13 showa thedifferent gestures used in MathPad.

The system is writer-dependent to guarantee thea more accurate symbol recognition. Handwriting isrecognized using hybrid recognizer. First, a dynamicclassifier calculates the similarity with prototypes [37]is combined with an statistical classifier [55], to ob-tain information to use once again dynamic program-ming for the fine classification [15]. The structuralanalysis uses the techniques described in [6].

References

[1] L. Anthony, J. Yang, and K.R. Koedinger. Eval-uation of Multimodal Input for Entering Mathe-matical Equations on the Computer. Conferenceon Human Factors in Computing Systems, pages1184–1187, 2005.

[2] X. Ao, J. Li, X. Wang, and G. Dai. Structuraliz-ing digital ink for efficient selection. Proceedingsof the 11th international conference on Intelli-gent user interfaces, pages 148–154, 2006.

Figure 14: MathPad.

[3] C. Bahlmann, B. Haasdonk, and H. Burkhardt.Online handwriting recognition with supportvector machines-a kernel approach. In Proceed-ings of the Eighth International Workshop onFrontiers in Handwriting Recognition, pages 49–54, 2002.

[4] E. J. Bellegarda, J. R. Bellegarda, D. Nahamoo,and K.S. Nathan. A Probabilistic Framework forOn-Line Handwriting Recognition. In Proceed-ings of the Third International Workshop Fron-tiers of Handwriting Recognition, pages 225–234,May 1993.

[5] J. Blanchard and T. Artieres. On-line hand-written documents segmentation. Proceedings ofNinth International Workshop on Frontiers inHandwriting Recognition., pages 148–153, 2004.

[6] D. Blostein and A. Grbavec. Recognition ofMathematical Notation. In P. S. P. Wang andH. Bunke, editors, Handbook on Optical Charac-ter Recognition and Document Analysis. WorldScientific Publishing Company, 1996.

[7] K. Chan. A Simple Yet Robust Structural Ap-proach for Recognizing On-Line HandwrittenAlphanumerical Characters. In Proceedings ofthe sixth international workshop on frontiers inhandwriting recognition, 1998.

[8] K. F. Chan and D. Y. Yeung. An Efficient Syn-tactic Approach to Structural Analysis of On-Line Handwritten Mathematical Expressions.Pattern Recognition, 33(3):375–384, March 2000.

[9] K. F. Chan and D. Y. Yeung. Mathematical Ex-pression Recognition: a Survey. InternationalJournal on Doucument Analysis and Recogni-tion, 3(1):3–15, 2000.

13

[10] K. F. Chan and D. Y. Yeung. Error Detection,Error Correction and Performance Evaluationin On-Line Mathematical Expression Recogni-tion. Pattern Recognition, 34(8):1671–1684, Au-gust 2001.

[11] B. B. Chaudhuri and U. Garain. An Approachfor Recognition and Interpretation of Mathemat-ical Expressions in Printed Documents. PatternAnalysis and Applications, 3(2):120–131, 2000.

[12] P. Chiu and L. Wilcox. A dynamic groupingtechnique for ink and audio notes. Proceedingsof the 11th annual ACM symposium on User in-terface software and technology, pages 195–202,1998.

[13] S.J. Cho and J.H. Kim. Bayesian networkmodel-ing of strokes and their relationships for on-linehandwriting recognition. Pattern Recognition,37:253–264, 2004.

[14] H. Choi, S.J. Cho, and J.H. Kim. Writer De-pendent Online Handwriting Generation withBayesian Network. In Proceedings of the NinthInternational Workshop on Frontiers in Hand-writing Recognition, pages 130–135, 2004.

[15] S. D. Connell and A. K. Jain. Writer Adaptationfor On-Line Handwriting Recognition. IEEETransactions on Pattern Analysis and MachineIntelligence, 24(3):329–346, March 2002.

[16] Y. Eto and M. Suzuki. Mathematical FormulaRecognition Using Virtual Link Network. InProceedings of the Sixth International Confer-ence on Document Analysis and Recognition,pages 430–437, Seattle, 2001.

[17] R. J. Fateman, T. Tokuyasu, B. P. Berman, andN. Mitchell. Optical Character Recognition andParsing of Typeset Mathematics. Journal of Vi-sual Communication and Image Representation,7(1):2–15, 1996.

[18] U. Garain and BB Chaudhuri. Recognitionof online handwritten mathematical expressions.IEEE Transactions on Systems, Man and Cyber-netics, Part B, 34(6):2366–2376, 2004.

[19] W. Guerfali and R. Plamondon. Normalizingand Restoring On-Line Handwriting. PatternRecognition, 26(3):419–431, 1993.

[20] C. Guy, M. Jurka, S. Stanek, and R. Fateman.Math Speak & Write, a Computer Program toRead and Hear Matematical Input. Universityof California Berkeley, August 2004.

[21] C. Guy and S. Stanek. Math Speak & Write:A Programmer’s Guide. University of CaliforniaBerkeley, August 2004.

[22] I. Guyon, P. Albrecht, Y. LeCun, J. S. Denker,and Hubbard W. Design of a Neural Net-work Character Recognizer for a Touch Termi-nal. Pattern Recognition, 24(2):105–119, 1991.

[23] N. Higham. Handbook of Writing for the Math-ematical Sciences. SIAM, 1993.

[24] T. Igarashi, S. Matsuoka, and T. Masui. Adap-tive recognition of implicit structures in human-organized layouts. Proceedings of 11th IEEEInternational Symposium on Visual Languages,pages 258–266, 1995.

[25] A.K. Jain, A.M. Namboodiri, J. Subrahmonia,and Y. Heights. Structure in On-line Docu-ments. Proceedings of the Sixth InternationalConference on Document Analysis and Recogni-tion, pages 844–848, 2001.

[26] A. Kacem, A. Belaid, and M. Ben Ahmed. Auto-matic Extraction of Printed Mathematical For-mulas Using Fuzzy Logic and Propagation ofContext. International Journal on DocumentAnalysis and Recognition, 4(2), 2001.

[27] T. Kanahori, K. Tabata, W. Cong, F. Tamari,and M. Suzuki. On-Line Recognition of Math-ematical Expressions Using Automatic Rewrit-ing Method. Advances in Multimodal Interfaces–ICMI2000, pages 394–401, 2000.

[28] L.B. Kara and T.F. Stahovich. HierarchicalParsing and Recognition of Hand-Sketched Di-agrams. In Proceedings of the 17th AnnualACM Symposium on User Interface Softwareand Technology, pages 13–22. ACM Press NewYork, NY, USA, 2004.

[29] D. Knuth. Mathematical Typography. Bulletinof the American Mathematical Society, 1979.

[30] M. Koschinski, H. Winkler, and M. Lang. Seg-mentation and Recognition of Symbols withinHandwritten Mathematical Expressions. InProceedings of the International Conference onAcoustics, Speech, and Signal Processing, vol-ume 4, pages 2439–2442, Detroit, MI, 1995.

[31] A. Kosmala, S. Lavirotte, L. Pottier, andG. Rigoll. On-Line Handwritten Formula Recog-nition using Hidden Markov Models and Context

14

Dependent Graph Grammars. In Fifth Inter-national Conference on Document Analysis andRecognition, Bangalore, India, 1999.

[32] A. Kosmala and G. Rigoll. On-Line HandwrittenFormula Recognition Using Statistical Methods.In International Conference on Pattern Recogni-tion, page PR33, 1998.

[33] J. LaViola. Mathematical Sketching: A New Ap-proach to Creating and Exploring Dynamic Illus-trations. PhD thesis, Brown University, Depart-ment of Computer Science, May 2005.

[34] J. LaViola and R. C. Zeleznik. MathPad2:a system for the creation and exploration ofmathematical sketches. ACM Trans. Graph.,23(3):432–440, 2004.

[35] S. Lavirotte and L. Pottier. Optical FormulaRecognition. In International Conference onDocument Analysis and Recognition, pages 357–361, 1997.

[36] H. J. Lee and J. S. Wang. Design of a Mathemat-ical Expression Understanding System. PatternRecognition Letters, 18(3):289–298, March 1997.

[37] X. Li and D.Y. Yeung. On-Line Handwritten Al-phanumeric Character Recognition Using Dom-inant Points in Strokes. Pattern Recognition,30(1):31–44, 1997.

[38] R. Lyon and L. Yaeger. On-Line Hand-PrintingRecognition with Neural Networks. In FifthInternational Conference on Microelectronicsfor Neural Networks and Fuzzy Systems, Lau-sanne, Switzerland, 1996. IEEE Computer Soci-ety Press.

[39] I.S. MacKenzie and S. Zhang. The immediateusability of Graffiti. Proceedings of Graphics In-terface, 97:129–137, 1997.

[40] E. Mandler. Advanced Preprocessing Techniquefor On-Line Recognition of Handprinted Sym-bols. In C. Y. Suen R. Plamondon and M. L.Simmer, editors, Computer Recognition and Hu-man Production of Handwriting, pages 19–36.World Scientific, 1989.

[41] N. Matsakis. Recognition of Handwritten Math-ematical Expressions. Master’s thesis, Mas-sachusetts Institute of Technology, Cambridge,MA, May 1999.

[42] E. G. Miller and P. A. Viola. Ambiguity andConstraint in Mathematical Expression Recog-nition. In The Fifteenth National Conference onArtificial Intelligence, pages 784–781, 1998.

[43] K. Nakagawa and M. Suzuki. MathematicalKnowledge Browser with Automatic HyperlinkDetection. In MKM, pages 190–202, 2005.

[44] H. Okamura, T. Kanahori, M. Suzuki,H. Fakuda, R. Cong, and F. Tamari. Handwrit-ing Interface for Computer Algebra Systems. InProceedings of Graphics Interface, 1999.

[45] M. Parizeau and R. Plamondon. A Fuzzy-Syntactic Approach to Allograph Modeling forCursive Script Recognition. IEEE Transactionson Pattern Analysis and Machine Intelligence,17(7):702–712, July 1995.

[46] E. H. Ratzlaff. Methods, reports and surveyfor the comparison of diverse isolated charac-ter recognition results on the UNIPEN database.In Proceedings of the Seventh International Con-ference on Document Analysis and Recognition,pages 623–628, 2003.

[47] E.H. Ratzlaff. A scanning n-tuple classifier foronline recognition of handwritten digits. In Pro-ceedings of the Sixth International Workshop onFrontiers in Handwriting Recognition, 2001.

[48] E. G. Sanchez, J. A. G. Gonzalez, Y. A. Dim-itriadis, J. M. C. Izquierdo, and J. L. Coro-nado. Experimental Study of a Novel NeuroFuzzy System for Online Handwritten UnipenDigit Recognition. Pattern Recognition Letters,19(3-4):357–364, March 1998.

[49] M. Schenkel, I. Guyon, and D. Henderson. On-line Cursive Script Recognition Using Time-Delay Neural Networks and Hidden Markov-Models. Machine Vision and Applications,8(4):215–223, 1995.

[50] H. Schwenk and M. Milgram. Constraint Tan-gent Distance for On-Line Character Recogni-tion. In International Conference on PatternRecognition, August 1996.

[51] M. Shilman, P. Viola, and K. Chellapilla. Recog-nition and grouping of handwritten text in di-agrams and equations. In Ninth InternationalWorkshop on Frontiers in Handwriting Recogni-tion, pages 569–574, October 2004.

15

[52] M. Shilman, Z. Wei, S. Raghupathy, P. Simard,and D. Jones. Discerning structure from freeformhandwritten notes. Proceedings of the SeventhInternational Conference on Document Analysisand Recognition, pages 60–65, 2003.

[53] B. K. Sin and J. H. Kim. Ligature Modeling forOnline Cursive Script Recognition. IEEE Trans-actions on Pattern Analysis and Machine Intel-ligence, 19(6):623–633, 1997.

[54] E. Smirnova and S. M. Watt. A context for pen-based computing. In Maple Conference 2005,pages 409–422, Waterloo Canada, July 2005.

[55] S. Smithies, K. Novins, and J. Arvo. AHandwriting-Based Equation Editor. In Graph-ics Interface, pages 84–91, 1999.

[56] M. Suzuki, F. Tamari, R. Fukuda, S. Uchida,and T. Kanahori. INFTY: an integrated OCRsystem for mathematical documents. In Pro-ceedings of the 2003 ACM symposium on Doc-ument engineering, pages 95–104. ACM PressNew York, NY, USA, 2003.

[57] M. Suzuki, S. Uchida, and A. Nomura. AGround-Truthed Mathematical Character andSymbol Image Database. Proceedings of theEigth International Workshop on Frontiers inHandwriting Recognition, pages 675–679, 2005.

[58] C. C. Tappert. Cursive Script Recognition byElastic Matching. IBM Journal of Research andDevelopment, 26(6):765–771, 1982.

[59] C. C. Tappert, C. Y. Suen, and T. Wakahara.The State of the Art in On-Line HandwritingRecognition. IEEE Transactions on PatternAnalysis and Machine Intelligence, 12(8):787–808, August 1990.

[60] S. Toyota, S. Uchida, and M. Suzuki. StructuralAnalysis of Mathematical Formulae with Ver-ification Based on Formula Description Gram-mar. In Workshop on Document Analysis Sys-tems (DAS06), pages 153–163, 2006.

[61] R. Veltkamp and M. Hagedoorn. State-Of-The-Art in Shape Matching. Technical ReportUU-CS-1999-27, Utrecht University, the Nether-lands, 1999.

[62] V. Vuori. Clustering Writing Styles with a Self-Organizing Map. In Proceedings of the 8th Inter-national Workshop on Frontiers in HandwritingRecognition, pages 345–350, 2002.

[63] S.M. Watt and X. Xie. Recognition for LargeSets of Handwritten Mathematical Symbols. InProceedings of th Eighth International Confer-ence on Document Analysis and Recognition,pages 740–744, 2005.

[64] L. Wenzel and H. Dillner. MathJournal– An Interactive Tool for the Tablet PC.http:\\www.xthink.com.

[65] H. Winkler, H. Fahrner, and M. Lang. ASoft-Decision Approach for Structural Analysisof Handwritten Mathematical Expressions. InProceedings of the International Conference onAcoustics, Speech, and Signal Processing, vol-ume 4, pages 2459–2462, Detroit, MI, 1995.

[66] L. P. Yang and R. Prasad. Online Recogni-tion of Handwritten Characters Using Differen-tial Angles and Structural Descriptors. PatternRecognition Letters, 14(12):1019–1024, Decem-ber 1993.

[67] M. Ye, H. Sutano, S. Raghupathy, C. Li, andM. Shilman. Grouping text lines in freeformhandwritten notes. In International Conferenceon Document Analysis and Recognition, 2005.

[68] R. Zanibbi, D. Blostein, and J. Cordy. Recogniz-ing Mathematical Expressions Using Tree Trans-formation. IEEE Transactions on Pattern Anal-ysis and Machine Intelligence, 24(11), November2002.

[69] R. Zanibbi, D. Blostein, and J.R. Cordy. Di-rections in Recognizing Tabular Structures ofHandwritten Mathematics Notation. Proceed-ings of the Fourth International IAPR Workshopon Graphics Recognition, pages 493–499, 2001.

[70] L. Zhang, D. Blostein, and R. Zanibbi. UsingFuzzy Logic to Analyze Superscript and Sub-script Relations in Handwritten MathematicalExpressions. In International Conference onDocument Analysis and Recognition, 2005.

[71] L. Zhang and R. Fateman. Survey of User In-put Models for Mathematical Recognition: Key-boards, Mice, Tablets, Voice. Technical report,University of California, 2003.

16