classifying glyphs - ntnu

111
ABCDEFGHIJKLMNOPQRSTUVWXYZ234+? TDT4500 Intelligent Systems, Specialization Project Autumn 2010 Evolving Neural Networks Classifying Glyphs Comparing Evolution and Learning Tiril Anette Langfeldt Rødland Supervisor: Keith L. Downing abcdefghijklmnopqrstuvwxyz567

Upload: others

Post on 14-Jan-2022

10 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Classifying Glyphs - NTNU

ABCDEFGHIJKLMNOPQRSTUVWXYZ234+?

TDT4500 Intelligent Systems, Specialization Project

Autumn 2010

Evolving Neural Networks

Classifying GlyphsComparing Evolution and Learning

Tiril Anette Langfeldt Rødland

Supervisor:Keith L. Downing

abcdefghijklmnopqrstuvwxyz567

Page 2: Classifying Glyphs - NTNU
Page 3: Classifying Glyphs - NTNU

Abstract

This dissertation investigates the classification capabilities of artificial neuralnetworks. The goal is to generalize over the features of a writing system, andthus classify the writing system of a previously unseen glyph.

Evolution and learning were both used to tune the weights of a network. Thiswas to investigate which of the methods is best suited for creation of this kind ofgeneralization and classification networks. The difference was remarkable, as theevolved network displayed absolutely no ability to classify the glyphs. However,the learned network managed to clearly separate between the writing systems,resulting in correct classifications.

It can thus be concluded that artificial neural networks are able to correctlyclassify the writing system of a glyph, and that learning by backpropagation isthe ultimate method for creating such networks.

i

Page 4: Classifying Glyphs - NTNU

ii

Page 5: Classifying Glyphs - NTNU

Preface

This dissertation is the pre-study project of the master dissertation. It iswritten at the Department of Computer and Information Science, NTNU, duringthe autumn semester of 2010.

The original plan was to find an Egyptological dissertation subject, due to mylong-lasting interest in the field. I appreciate the assistance from Vidar Edland,Norwegian Egyptology Society, for helping me in the search for a suitable subject.However, the search came up empty. Inspired by Dehaene (2009), Egyptologyturned to hieroglyphs, and then other writing systems joined in. Classificationseemed to be a natural choice of task, with regard to different writing systems andneural networks. This revealed an interesting subject on which there seeminglyhad been done no previous research. I like to think it still is a relevant problem.At least it can be used to show off the generalization abilities of artificial neuralnetworks.

I wish to thank my supervisor Professor Keith L. Downing at the Depart-ment of Computer and Information Science, Norwegian University of Science andTechnology, for his invaluable assistance.

Special thanks are given to John A. Bullinaria, for his immense help in debug-ging, and for sharing of his infinite wisdom regarding neural networks.

Gratitude is given to Arne Dag Fidjestøl and Aslak Raanes in the SystemAdministration Group, Department of Computer and Information Science, forgiving me access to hardware on which I could run the evolution.

I am also very grateful to Cyrus Harmon, Edi Weitz, David Lichteblau, and allthe other Common Lisp programmers out there, whose libraries saved me frommonths of extra work.

Trondheim, December 2010

Tiril Anette Langfeldt Rødland

iii

Page 6: Classifying Glyphs - NTNU

iv

Page 7: Classifying Glyphs - NTNU

Contents

Chapter 1 : Introduction 1

Chapter 2 : Background 32.1 Writing Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.2 Classification of Glyphs . . . . . . . . . . . . . . . . . . . . . . . 52.3 Artificial Neural Network . . . . . . . . . . . . . . . . . . . . . . 6

2.3.1 The XOR Example . . . . . . . . . . . . . . . . . . . . . . 62.4 Training ANNs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.4.1 Training ANNs Using Supervised Learning . . . . . . . . . 92.4.2 Backpropagation . . . . . . . . . . . . . . . . . . . . . . . 102.4.3 Fast Backpropagation . . . . . . . . . . . . . . . . . . . . 122.4.4 Tricks for Multi-Class Classification Problems . . . . . . . 13

2.5 Evolving ANNs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.5.1 Evolving Neural Networks . . . . . . . . . . . . . . . . . . 15

Chapter 3 : Methodology 173.1 Glyphs and Writing Systems . . . . . . . . . . . . . . . . . . . . . 173.2 Neural Network Design . . . . . . . . . . . . . . . . . . . . . . . . 203.3 Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.4 Evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Chapter 4 : Results and Analysis 254.1 Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

4.1.1 Pair Classification . . . . . . . . . . . . . . . . . . . . . . 254.1.2 Full Classification . . . . . . . . . . . . . . . . . . . . . . . 274.1.3 Writing System Classification Quality . . . . . . . . . . . 27

4.2 Evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Chapter 5 : Discussion 355.1 Learning or Evolution? . . . . . . . . . . . . . . . . . . . . . . . . 355.2 Writing Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . 355.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

References 39

Software Library 43

Appendix A : Unicode Charts A-1

v

Page 8: Classifying Glyphs - NTNU

Appendix B : Classification B-1B.1 Pair Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . B-1

B.1.1 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . B-1B.1.2 Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . B-6

B.2 Full Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . B-29B.2.1 Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . B-29B.2.2 Evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . B-31

Appendix C : Data Set Distribution C-1

Appendix D: Pixel Comparison D-1

vi

Page 9: Classifying Glyphs - NTNU

List of Figures

2.1 A simple ANN . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2 Activation function . . . . . . . . . . . . . . . . . . . . . . . . . . 72.3 XOR ANN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.4 Backpropagation ANN . . . . . . . . . . . . . . . . . . . . . . . . 112.5 The evolution cycle . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.1 ANN: From Unicode to output values . . . . . . . . . . . . . . . . 19

4.1 Learning: full classification . . . . . . . . . . . . . . . . . . . . . . 284.2 Correlation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304.3 Evolution: full classification . . . . . . . . . . . . . . . . . . . . . 32

A.1 Coptic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-2A.2 Runic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-3A.3 Hebrew . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-4A.4 Arabic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-5A.5 Hangul jamo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-6A.6 Hiragana . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-7A.7 Thai . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-8A.8 Canadian Aboriginal syllabics . . . . . . . . . . . . . . . . . . . . A-9A.9 Han unification . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-10A.10 Egyptian hieroglyphs . . . . . . . . . . . . . . . . . . . . . . . . . A-11

C.1 Approach 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-3C.2 Approach 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-4C.3 Approach 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C-4

vii

Page 10: Classifying Glyphs - NTNU

viii

Page 11: Classifying Glyphs - NTNU

List of Tables

2.1 XOR truth table . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.2 Evolutionary methods . . . . . . . . . . . . . . . . . . . . . . . . 14

3.1 Writing systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173.2 Evolution parameters . . . . . . . . . . . . . . . . . . . . . . . . . 223.3 The gene . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

4.1 Learning: pair classification . . . . . . . . . . . . . . . . . . . . . 264.2 Learning: pair classification probabilities . . . . . . . . . . . . . . 26

C.1 Number of glyphs . . . . . . . . . . . . . . . . . . . . . . . . . . . C-1C.2 Comparison of combinations of relative set size . . . . . . . . . . C-2

D.1 Glyph similarity . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-2D.2 Glyph similarity . . . . . . . . . . . . . . . . . . . . . . . . . . . . D-2D.3 Pixel comparison using Richard’s curve (128) . . . . . . . . . . . D-3D.4 Pixel comparison using Richard’s curve (200) . . . . . . . . . . . D-3D.5 Pixel comparison using Richard’s curve (250) . . . . . . . . . . . D-3D.6 Pixel comparison using difference ≤ 40 . . . . . . . . . . . . . . . D-4D.7 Pixel comparison using difference ≤ 20 . . . . . . . . . . . . . . . D-4D.8 Pixel comparison using difference ≤ 10 . . . . . . . . . . . . . . . D-4D.9 Pixel comparison using difference ≤ 5 . . . . . . . . . . . . . . . . D-5

ix

Page 12: Classifying Glyphs - NTNU

x

Page 13: Classifying Glyphs - NTNU

Abbreviations and Acronyms

ANN Artificial neural networks

BP Backpropagation

CJK Chinese, Japanese, Korean

EA Evolutionary algorithm

EEC Evolving efficient connections

EP Evolutionary programming

ES Evolutionary strategies

GA Genetic algorithm

GP Genetic programming

TSS Total sum squared error

XOR Exclusive or

xi

Page 14: Classifying Glyphs - NTNU

xii

Page 15: Classifying Glyphs - NTNU

CHAPTER 1

Introduction

33 000 years ago, the first Homo sapiens discovered that they could draw rec-ognizable pictures on the wall of the Chauvet cave, in southern France (Dehaene,2009). There were drawings of animals, as well as nonfigurative shapes such asdots, lines and curves. Pictography appeared in ancient Sumeria and Egypt,creating easy-to-read writings.

The art of writing mutated as it spread throughout the cultures. Even thoughpictograms are relatively easy to read, they are — due to their many details— very slow to write. They were thus simplified in different ways, and thesymbols were tied to different sounds. For example, the Chinese character isa simplification of a horse, and the letter A was originally an ox (Dehaene, 2009,Figure 4.3). This is visible in the Greek letter α (alpha), as aleph means just that.Similarly, m symbolizes waves (mem), k symbolizes a hand with outstretchedfingers (kaf), and R symbolizes a head (res) (Dehaene, 2009).

Even though both and A originally came from large, four-legged animals,the resulting glyphs are very different. For each step of simplification, somefeatures were extracted for use in the simplified glyph. Different civilizationsfocused on different features. The simplification was also depending on the writingtools available; pencils and paper enable glyphs of many different shapes, whileglyphs carved in stone ought to consist of straight lines.

Today, 33 000 years after the first drawings in a cave, we have numerous writingsystems, in different stages of evolution. The human brain seems to be rathergood at separating between these different writing systems, depending on theirrelative placement on the family tree. It might be difficult to separate betweenthe Latin, Greek and Cyrillic alphabets, if one is familiar with neither. However,it is trivial to separate these from Egyptian hieroglyphs. After having learnedthat m, l and w are hieroglyphs, it is easy to see that R also is of thiswriting system.

Even though the glyphs within one writing system differ from each other,they are also very similar. One tend to reuse the same basic shapes for differentglyphs within the writing system (Dehaene, 2009, Figure 4.1), thus creating asubconscious expectation on how glyphs from that writing system shall look.

When humans separate between writing systems like that, it is due to the

1

Page 16: Classifying Glyphs - NTNU

CHAPTER 1. INTRODUCTION 2

neural network of our brain. We are able to generalize common features over aset of glyphs, and classify unseen glyphs based on that. Since the natural neuralnetwork is able to do this separation, an artificial neural network ought to be aswell.

We already know that artificial neural networks are able to recognize letters(Bishop, 1996). This dissertation will investigate whether an artificial neuralnetwork can generalize over the patterns and shapes common for the differentwriting systems, and thus find the writing system of an unseen glyph.

It was evolution that gave humans a brain capable of reading. However, noone can read without having learnt it first. The main question of the dissertationis thus whether learning or evolution of the neural net will give the best result.That is, the ultimate solution with respect to both success rate and executiontime, or rather how easy it is to attain said solution.

The dissertation is structured as follows. This introductory chapter intro-duces the problem, and writing systems in general. Chapter 2 demonstrates theconcepts of classification, evolution, and artificial neural networks, including com-mon methods and algorithms. Also writing systems are discussed in more detail.Chapter 3 presents the methodology used, focusing on differences from the stan-dard approach given in Chapter 2. Chapter 4 reveals the experiments and theirresults, including a low-level analysis. A more general discussion can be found inChapter 5, in addition to future work.

Additional information can be found in the appendices. Appendix A includesUnicode charts from the writing systems used, and Appendix B contains the fullresults from the experiments. The optimal data set distribution is researched inAppendix C, while a comparison of the writing systems based on pixel values canbe found in Appendix D.

Page 17: Classifying Glyphs - NTNU

CHAPTER 2

Background

This chapter introduces the concepts used in this dissertation. First there is abrief overview of writing systems, followed by an introduction to artificial neuralnetworks.

2.1 Writing Systems

A writing system is the set of glyphs used to represent the writing of a specifichuman language (FOLDOC, 1998b). In this case, a glyph is an image used tovisually represent the characters of the language (FOLDOC, 1998a).

There are many different classes of writing systems, depending mainly on whateach glyph represent. Below is the classification used by The Unicode Consortium(2009, Ch. 6), as well as a few examples.

Alphabets are writing systems consisting of letters, where each letter can beeither a consonant or a vowel. The two have the same status as letters, giving ahomogeneous collection of glyphs.

The Latin alphabet is the most well-known example, as it is used to writethousands of different languages. Another example is the old runic alphabet.

Abjads are writing systems which have only the consonants and the long vowels.All other sounds are either left out entirely from the writing, or simply indicatedby specific marks on the consonants.

Semitic languages are often abjads, with Hebrew and Arabic as well-knownexamples.

Syllabaries are writing systems where each symbol represent both a consonantand a vowel, or sometimes more than one.

The Japanese kana systems — hiragana and katakana — are both syllabaries.The glyphs there describe syllables like for example ka (か), ki (き), ku (く), ke(け), ko (こ), ra (ら), and ri (り). Here, each glyph is unique in shape, such thatか andき look nothing alike, even though they both start with the sound k.

3

Page 18: Classifying Glyphs - NTNU

CHAPTER 2. BACKGROUND 4

Other syllabaries, like the Korean Hangul jamo, have glyphs functioning morelike conglomerates. That is, they consist of one part for the initial consonant orconsonants, one part for the vowel, and an optional consonant part in case of afinal consonant.

Abugidas are writing systems which mix the syllabic and alphabetic character-istics. Each consonant has an inherent vowel connected to it (usually a). Thereare also independent vowels, and when one of these follows a consonant, it over-rides the inherent vowel. For example, when i follows ka, it results in ki. Often,there is also a symbol that, when it follows a consonant, kills the inherent vowelwithout adding another sound, thus resulting in a bare consonant sound.

The Devanagari script, which is used to write for example Hindi and Sanskrit,is an abugida.

Logosyllabaries are writing systems representing words or morphemes of words,instead of only the sounds.

The Han script — which is used to write Chinese — is a logosyllabary. Theseglyphs are borrowed by other East Asian languages, e.g. Japanese and Korean.As a consequence, it is possible to understand written Chinese when you knowJapanese, even though the spoken languages are nothing alike.

Japanese is special, in that it uses four different writing systems in the samecomposite system (Banno et al., 1999). That is, kanji (logosyllabary) for thingsand meanings, hiragana (syllabary) for e.g. grammatical postfixes and particles,katakana (syllabary) for e.g. foreign words, and Latin (alphabet) for romaji.Romaji is a transliteration of the sound, for use when the other systems will notbe understood, e.g. when working with computers and foreigners.

Because of this separation, the kanji is used only for meaning, not sound.Each kanji can have many different sounds connected to it. As a result, thesame sequence of kanjis can be translated to different strings of kana, dependingon the context. 曜, meaning Sunday, is pronounced nichiyoubi, にちようび. Note that the first and the second have different sounds. This is becausethey have different meanings; the first means Sun, while the second meansday.1 The basic meaning is the same, however, as the day can be seen as afunction of the Sun. Similarly, means both Moon and month. When it comesto logosyllabaries, context is everything. A hiragana て, on the other hand, willalways be a te.

Egyptian hieroglyphs, however, can double as both kanji and kana. While themajority of the hieroglyphs are logographs, displaying the meaning, there is alsoa subset of these that behaves like an alphabet. Other glyphs represent sequencesof consonants. In other words, hieroglyphs are complicated.

One can take j (pr) as an example (Collier and Manley, 2003). This lookslike the floor plan of a simple one-room house, and is thus the symbol used forhouse.

j| means an actual house, as | is a sign used to indicate that this indeedis an ideogram. When used in

jrL (pr), on the other hand, j has nothing todo with houses. Similarly, r (r) does no longer mean mouth. j is now usedfor the sound pr, which means to go out, leave. The r is there to clarify the

1曜 means weekday.

Page 19: Classifying Glyphs - NTNU

5 2.2. CLASSIFICATION OF GLYPHS

reading of j, not to add an extra r. The walking legs at the end, L, are usedas a determinative, an unspoken character placed at the end to give the reader ageneral idea of the meaning. Which in this case is movement.

2.2 Classification of Glyphs

Glyph classification is an important field of research, with many applicationareas. Systems that can recognize and classify glyphs can be used to e.g. sortmail automatically based on the hand-written zip codes (or similar area-specificnumbers), and convert printed documents into digital text. A good example isDetexify, created by Kirsch. It is a LaTEX symbol recognizer, driven by an artificialneural network. The user draws a symbol on the screen, and the system displaysa list of possible symbols and their LaTEX command.

All of this necessitates the recognition of each glyph, i.e. to separate an a froma b, and to recognize a’s of different fonts and quality, printed or handwritten.Character recognition is thus a much researched area, and an important part ofgeneral pattern recognition (Bishop, 1996). Mantas (1986) and Govindan andShivaprasad (1990) present an overview of the methods at the time. However,there is still research going on in the field. For example, Ramos-Garijo et al.(2007) has created an engine to recognize handwritten characters, and Rochaand Pavlidis (1994) propose a method of recognizing characters written withdifferent fonts.

This dissertation, however, is not about recognizing neither a handwrittennor a printed a. It is about recognizing the a as a Latin character. Which is aproblem without many application areas, as the writing system tend to be knownbased on context. As a result, there is not much research going on in this part ofthe field. Alas, there is no available research material on actually classifying thewriting system of a glyph. The most relevant research might be by Knight andYamada (1999), who propose an approach to decipher unknown scripts. That is,historically forgotten scripts where either the written or the spoken language isunknown. Unfortunately, that is of no use for this problem.

The main difference between recognizing a character and classifying the char-acter is the data set. When trying to recognize an a, the system has already seenmany a’s. They might be different, with different fonts or noise. Nonetheless, thesystem has seen an a before. The task is then to generalize over all the a’s andextract the important features, to also recognize a’s which look a bit different.This luxury is not allowed when classifying the writing system. The point is notto recognize already seen glyphs, but to generalize important aspects of a writingsystem and recognize other glyphs based on these features. That is, to recognizeaか as a hiragana after learning thatは,て andる are hiragana.

However, both character recognition and glyph classification are pattern recog-nition problems. As artificial neural networks are an important tool in the fieldof pattern recognition (Bishop, 1996), there is reason to believe they will solvethe glyph classification problem.

Page 20: Classifying Glyphs - NTNU

CHAPTER 2. BACKGROUND 6

2.3 Artificial Neural NetworkAn artificial neural network (ANN) is an artificial constructed brain, designed

to benefit from its parallelism (Callan, 1999). An ANN is more or less a collec-tion of many interconnected processors, similar to the neurons in a brain. Eachprocessor — or neuron — is very simple. It receives an input signal, and sends anoutput signal, which is a function of the input signal. Even though one neuronis simple, an ANN can solve quite complex problems due to the large numbers ofthese simple neurons.

These nodes or neurons can be structured in many different ways. The ANNmust have at least one input node, and at least one output node. There may alsobe nodes in other layers, so-called hidden nodes. These are the nodes that areneither input nor output, and are thus hidden in the black box of the ANN.

All these nodes can be connected to each other in multiple ways. One commonapproach is the layered model, with one layer of input nodes, one layer of outputnodes and zero, one or more layers of hidden nodes. Here, each layer is fullyconnected to all the nodes in the layer above. There can also be connectionsinternally in the layers. Another approach is to allow any node to be connectedto any other node, without any layered structure. The ANN in Figure 2.1 on thefacing page is an example of a three-layered ANN, fully connected between thelayers. There are two blue input nodes, two red output nodes and one layer offour green hidden nodes.

A link is the collection of connections between two layers. In the ANN inFigure 2.1, there are two links; one between the input layer and the hidden layer,and one between the hidden layer and the output layer.

netj =n∑

i=0

xiwij, (2.1)

For all nodes but the input nodes, the input signal is given as Equation 2.1,where xi is the input weight from node i in the layer below, and wij is the weighton the arrow from node i to this current node, j. So the input is the sum of theoutput from all the incoming nodes, times their weight.

The output is calculated by sending the input signal through an activationfunction. This function takes in the input, and returns a number between either0 and 1, or -1 and 1. A commonly used activation function is the sigmoid function(Equation 2.2 and Figure 2.2 on the next page).

f(net) =1

1 + exp(−net)(2.2)

The output of an ANN is thus a function of the input signals, but also of theweights. These can be manually set on small networks, but on larger networksthey ought to be automatically configured.

2.3.1 The XOR ExampleThe XOR ANN in Figure 2.3 on page 8 is a good example of how an ANN

works. The XOR function (exclusive or, Table 2.1, (WorldLingo, 2010n)) is abinary operation that is true only if one input is true and the other false.

Page 21: Classifying Glyphs - NTNU

7 2.3. ARTIFICIAL NEURAL NETWORK

.

.O1 .O2

.H2.H1 .H3 .H4

.I1 .I2

Figure 2.1: A simple ANN with one hidden layer.

00.10.20.30.40.50.60.70.80.9

1

-4 -2 0 2 4

f(n

et)

net

The sigmoid activation function

f(net) = 11+exp(−net)

Sigmoid

Figure 2.2: The sigmoid activation function.

Page 22: Classifying Glyphs - NTNU

CHAPTER 2. BACKGROUND 8

Table 2.1: Truth table for the XOR function.

Input OutputA B A XOR B0 0 00 1 11 0 11 1 0

.

.I1 .I2

.H2act = 2 .H3act = 1.H1act = 1

.H4act = 1

.O.

.1 .1.-2

.1 .1 .1.1

Figure 2.3: An ANN that solves XOR.

Page 23: Classifying Glyphs - NTNU

9 2.4. TRAINING ANNS

Say I1 = 1 and I2 = 0. The input signal for the three nodes in the first hiddenlayer of Figure 2.3 is then

H1in=I1 × 1 = 1 × 1 = 1

H2in=I1 × 1 + I2 × 1 = 1 × 1 + 0 × 1 = 1

H3in=I2 × 1 = 0 × 1 = 0

The activation values are 1, 2 and 1, respectively. As a result, only H1 isactivated, due to the activation values.

H1in≥ H1act ⇒ H1out = 1

H2in< H2act ⇒ H2out = 0

H3in< H3act ⇒ H3out = 0

The input signal to H4 is thus

H1out × 1 + H2out × −2 + H3out × 1 = 1,

meaning the output signal is 1. The output node will then be 1, as True XORFalse is True.

It is thanks to the weight values that this ANN actually manages to solvethe XOR function. In this case, the weights are possible to set by hand, as itis a rather small network. For larger networks, however, we must find a way totune in the correct weight values automatically. This can be done by for examplelearning or evolution.

2.4 Training ANNsThere are three main types of learning in an ANN:

Supervised learning is learning with a teacher that for each step informs thesystem if it was right or wrong, and gives the correct answer (Downing,2009c; Floreano and Mattiussi, 2008). This is often used in classification,where each new training object has a correct (known) class.

Unsupervised learning is learning with no external feedback, neither from ateacher nor the environment. The system itself ought to realize what was agood or a bad move (Downing, 2009d).

Reinforcement learning is learning with only infrequent feedback and super-vision (Downing, 2009b). The system is told the overall result, but not whatit did wrong and what it should have done.

This section will focus on supervised learning, as that is the main method foruse in classification. All the glyphs do have a correct and known answer, and thisshould be used to guide the learning.

2.4.1 Training ANNs Using Supervised LearningUsing supervised learning, the weights of the network are learned by exploiting

the difference between the output of the network and the known solution.

Page 24: Classifying Glyphs - NTNU

CHAPTER 2. BACKGROUND 10

For each pattern µ of the M total training patterns, there is a pair of inputvectors ~xµ and desired output vectors ~tµ. The goal is to find the set of synapticweights such that the actual output ~yµ of a two-layered network is as close aspossible to the desired output ~tµ, for all of the M patterns.

The weights of the network should change gradually, until it is performingacceptably. The performance can be described using an error function, e.g. themean quadratic error between the desired and the actual output (Equation 2.3).The error function is, just as the output ~yµ, depending on the synaptic weights.The error can thus be reduced by changing the weights accordingly.

EW =1

2

∑µ

∑i

(tµi − yµ

i )2 =

1

2

∑µ

∑i

tµi −

∑j

wijxµj

2

(2.3)

The weight change is given by a learning rule. Differentiating (2.3) with respectto the weights gives the learning rule in Equation 2.4, where η is the learning rate(Wagstaff, 2008; Floreano and Mattiussi, 2008).

∆wij = η (ti − yi) xj (2.4)

This algorithm is known both as the Widrow-Hoff rule — after its authorsWidrow and Hoff (1960) — and as the delta rule, because of the difference δ =t − y. It is often written as simply ∆wij = ηδixj.

This ∆wij is then added to the weight wij, either online, i.e. after each andevery pattern, or in batch mode, i.e. adding the accumulated weight change afterall the M training patterns have been run through the net.

While the delta rule works fine for training patterns that are linearly separable,it does not work for more complex patterns, e.g. the XOR example in Section2.3.1. As this function needs hidden layers (Figure 2.3 on page 8), a more complexversion of the delta rule is necessary. The solution — the generalized delta rule— was presented by Rumelhart et al. (1986). This algorithm, best known asbackpropagation of error, propagates the error back through the network, fromthe outer layer back through all the hidden layers, and to the input layer. Asa result, it can handle a neural network with an arbitrary number of nodes andlayers.

2.4.2 Backpropagation

Backpropagation can be explained using the three-layered ANN in Figure 2.4on the facing page.

The backpropagation algorithm given in Floreano and Mattiussi (2008) foronline weight updating, is as follows:

1. Initialize all weights, vjk and wij, to random values close to 0.

2. Set the values of input nodes to the current training pattern s:

xµk = sµ

k (2.5)

Page 25: Classifying Glyphs - NTNU

11 2.4. TRAINING ANNS

.

. .

.. . .

. .

.yi

.wij

.hj

.vjk

.xk

Figure 2.4: An ANN with the backpropagation syntax. xk is the input layer, with k nodes.Similarly, hj and yi are the hidden layer with j nodes and the output layer with i nodes,respectively. vjk is the link between the input layer and the hidden layer. Each node inthe input layer is connected to every node in the hidden layer, so the link consists of k × jconnections. Similarly, wij is the link between the hidden layer and the output layer.

3. Compute the values of the hidden nodes, based on the input nodes and thelink from the input to the hidden layer:

hµj = Φ

(∑k

vjkxµk

)(2.6)

where

Φ(ai) =1

1 + e−kai(2.7)

4. Compute the values of the output nodes, based on the hidden nodes and thelink from the hidden to the output layer:

yµi = Φ

∑j

wijhµj

(2.8)

5. Compute the delta error for each output node, based on the hidden nodesand the link from the hidden to the output layer, as well as the differencebetween the expected and the actual output:

δµi = Φ

∑j

wijhµj

(tµi − yµ

i ) (2.9)

Page 26: Classifying Glyphs - NTNU

CHAPTER 2. BACKGROUND 12

It can be noted that the derivative of the sigmoid function can be expressedin terms of the output of the sigmoid function (Wagstaff, 2008):

Φ

∑j

wijhµj

= yµi (1 − yµ

i ) (2.10)

As a result, the delta error can be written solely depending on the actualand the expected output:

δµi = yµ

i (1 − yµi ) (t

µi − yµ

i ) (2.11)

6. Compute the delta error for each hidden node, based on the input nodes,both links, and the delta error for the output nodes:

δµj = Φ

(∑k

vjkxµk

)∑i

wijδµi (2.12)

Thanks to the differentiated of the sigmoid function, however, the error isindependent of the input:

δµj = hµ

j

(1 − hµ

j

)∑i

wijδµi (2.13)

7. Compute the changes in synaptic weights, based on the delta errors and theoutput values of the first two layers:

∆wµij = δµ

i hµj (2.14)

∆vµjk = δµ

j xµk (2.15)

8. Update the weights by adding the changes to each weight, multiplied withthe learning rate η:

wtij = wt−1

ij + η∆wµij (2.16)

wtjk = wt−1

jk + η∆wµjk (2.17)

The algorithm goes through this procedure multiple times for every trainingcase, until the total sum squared error is sufficiently small. This error (Equation2.18) is computed over all the output nodes (i), for all the training patterns (µ).

TSS =1

M

M∑µ

(1

N

N∑i

(tµi − yµ

i )2

)(2.18)

2.4.3 Fast BackpropagationThe backpropagation algorithm fulfils its task, but is not specifically designed

to be as efficient as possible. Ngolediage et al. (1993) thus proposed a moreefficient variant of the algorithm, the fast backpropagation algorithm.

The fast backpropagation algorithm is similar to the standard algorithm, butis continuously adapting the learning parameter η for individual synapses. Thisis done using the Fermi-Dirac Distribution (Ngolediage et al., 1993), which againis based upon quantum principles.

Page 27: Classifying Glyphs - NTNU

13 2.4. TRAINING ANNS

The probability P (w) that a specific electron has an energy between w andw + δw is given by the Fermi-Dirac Distribution function (Equation 2.19).

P (w) =1

1 + exp(

w−wF

KT

) (2.19)

K is the Boltzmann constant (Encyclopædia Britannica, 2010b), T is the ab-solute temperature (Encyclopædia Britannica, 2010a), and wF is the Fermi levelof the material (Encyclopædia Britannica, 2010c).

Inspired by this, the fast backpropagation updates the learning rate by Equa-tion 2.20.

η = λ exp(

−|β(t)|γT

)(2.20)

β(t) =1

2(||∆w(t)| − |∆w(t − 1)|| + β(t − 1)) (2.21)

Here, T is a parameter pretending to be the system temperature, and β(t)(Equation 2.21), is the mean difference between the step sizes at two consecutiveepochs (t − 1 and t). λ is the step size parameter, and γ ≥ 0 is the scale factorfor T . The final weight update rule for fast backpropagation is thus given asEquation 2.22, where E is the error between the desired and the actual output,and α is the momentum of the change.

∆w(t) = −λ exp(

|β(t)|γT

)δE

δw(t)+ α∆w(t − 1) (2.22)

2.4.4 Tricks for Multi-Class Classification Problems

Bullinaria (2009) proposed the use of cross-entropy error function and thesoftmax activation function for multi-class classification problems. Accordingto Dunne and Campbell (1997), there is a natural pairing between these twofunctions, and they should be used together.

Softmax The softmax activation function is only for use in the output layer(Wikibooks, 2010). It is specifically designed for the output nodes, where eachnode is representing one class. Softmax is giving each of these classes (or nodes)a value, which is the probability that the input pattern belongs to this class(Bishop, 1996). It thus ensures that all the output values are in the range [0, 1],and that all the output nodes sum up to 1.

pi =eqi∑n

j=1 eqj(2.23)

The softmax function is represented by Equation 2.23, where pi is the value ofoutput node i, qi is the net input to the output node i, and n is the number ofoutput nodes (Bishop, 1996).

Page 28: Classifying Glyphs - NTNU

CHAPTER 2. BACKGROUND 14

Cross-entropy error function The cross-entropy error function (Equation 2.24)is the output error function. It is used to specify if the error if low enough tostop learning (Bullinaria, 2009; Bishop, 1996). It is used together with the deltafunction (2.25), instead of the more common delta function (2.11) (Bullinaria,2009).

Error = − (tµi log(yµ

i ) + (1 − tµi ) log (1 − yµ

i )) (2.24)

δµi = tµ

i − yµi (2.25)

2.5 Evolving ANNsEvolutionary computation is a search method, or family thereof, deeply in-

spired by the concept of evolution (Darwin, 1859). One starts off with a bunchof randomly generated solutions — or individuals — and use these to evolve agood solution. As long as one has a fitness function — a way to grade how wellthe individual solves the problem — the fitness can be used to kill off the leastfit individuals, and enhance the best. As in real world evolution, the survivingindividuals mate and create new individuals, with aspects from both parents.Mutations can also occur, changing one individual slightly. This might be a badmove, in which case the individual is likely to die. It is also possible that themutation increased the fitness. In that case, the individual is allowed to mate,and the mutation will spread down through the generations.

Evolutionary computation can be divided into four subfields: a) evolutionarystrategies (ES), b) evolutionary programming (EP), c) genetic algorithms (GA),and d) genetic programming (GP). They all follow the same basic principle, butdiffer in their traditional characteristics (Downing, 2009a).

According to Downing (2009a, Table 1), the characteristics of the four evolu-tionary methods can be summarized as in Table 2.2. The main difference betweenthe four is the choice of genotype. ES use a list of real values, EP a list of integers,and GA use a bit vector. GP is used to evolve programs, and the genotype isthus a code tree.

Table 2.2: Brief overview of the four evolutionary methods (Downing, 2009a, Table 1). Over-Prod is selection among an overproduction of children, GenMix is selection among both currentand last generation, GenRep is full generational replacement, and FitPro is fitness proportion-ate.

Method Genotype Mutation Crossover Adultselection

Parentselection

Mateselection

ES Realvector Yes Sometimes OverProd,

GenMix Truncate Uniform

EP Int vec-tor Yes No OverProd,

GenMix Truncate Uniform

GA Bit vec-tor Yes Yes GenRep Uniform FitPro

GP Code tree Yes Yes GenRep Uniform FitPro

Page 29: Classifying Glyphs - NTNU

15 2.5. EVOLVING ANNS

.

.Development.Reproduction

.Adult.selection

.Parent.selection

Figure 2.5: The evolution cycle can be described as these stars. The green four-tagged starsare the representation of the genotypes, and the blue eight-tagged stars are the phenotypes.

Even though the four methods differ in the implementation details, the mainalgorithm is the same (Figure 2.5). Simple representations are created (the geno-types), and these develop into phenotypes. Phenotypes are more complex datastructures that can be tested for fitness. Each phenotype will thus have a fitnessvalue. Based on this fitness, some (or all) of the individuals in the population areselected to become adults. Of these, some (or all) are selected to become parents.From this pool of parents, pairs of individuals are selected to mate, creating newindividuals with genotype parts from both parents (or as a copy of one). Someof these will also be mutated, with a small change in genotype. Now we have anew population of genotype-children, which develops into phenotypes, and so itcontinues.

2.5.1 Evolving Neural NetworksWhen evolving a neural network, one must find a genotype to represent the

network (the phenotype). The fitness function will be depending on the outputof the neural network.

A bit vector genotype gives the most freedom when it comes to translation ofthe genotype to phenotype. A bit vector can represent almost anything, and isthus a good choice when evolving a neural network (Whitley et al., 1990). Thatpoints to the use of a genetic algorithm (Table 2.2), and thus also the use ofboth mutation and crossover, full generational replacement as adult selection, a

Page 30: Classifying Glyphs - NTNU

CHAPTER 2. BACKGROUND 16

uniform parent selection and a fitness proportionate mate selection.When it comes to evolution of neural networks, there are many ways to do it.

One can use a predefined topology and only evolve the weights, or one can evolvethe topology as well (Jones, 2009; Dewri, 2003).

Since this network shall be changed by both the learning and the evolution,the topology ought to be predefined. That would make the implementation lesscomplicated, and more modular. However, it is still possible to slightly adapt thenetwork. Shi and Wu (2008) designed EEC (Evolving Efficient Connections), amethod to evolve both connection weights and a switch to turn the weight on oroff. Otherwise, one must wait for all the bits to become 0 for the connection tobe removed; with EEC just one bit needs to be turned off.

2.5.1.1 Genetic Algorithm

A genetic algorithm used to evolve a neural network has the same main ap-proach as in Figure 2.5 (De Jong, 2006):

1. Randomly generate a population of m parents.2. Repeat:3. Compute and save the fitness u(i) for each

individual i in the current parent population.4. Define selection probabilities p(i) for each

parent i so that p(i) is proportional to u(i).5. Generate m offspring by probabilistically

selecting parents to produce offspring.6. Select only the offspring to survive.7. End repeat

The uniform parent selection from Table 2.2 can be seen in line 3, where allthe adults are seen as potential parents. No individuals are excluded from thepossibility of reproduction; their probability of reproduction is, however, depend-ing on their fitness (line 4 and 5). There is also full generational replacement, asall of the offspring — and nothing but the offspring — is allowed to survive (line6).

The most important, however, is the use of a bit vector as genotype.

Page 31: Classifying Glyphs - NTNU

CHAPTER 3

Methodology

3.1 Glyphs and Writing SystemsThe writing systems to be classified are chosen based on the classification in

Section 2.1. There are two systems from each type, as seen in Table 3.1.

Table 3.1: The writing systems to be classified, and their respective type and Unicode values(The Unicode Consortium, 2009). Example glyphs from the writing systems can be found inAppendix A.

Type Script Code point range Image

Alphabet Coptic U2C80-U2CFF

Runic U16A0-U16FF

Abjad Hebrew U0590-U05FFArabic U0600-U06FF

Syllabary Hangul U1100-U11FF

Hiragana U3040-U309F

Abugida Thai U0E00-U0E7F

Canadian Aboriginal syllabics U1400-U167F

Logosyllabary Han U4E00-U9FCF

Egyptian hieroglyphs U13000-U1342F

The Egyptian hieroglyphs were chosen because of a strong interest in theancient Egypt, and because — unlike in most other writing systems — the originalmeaning of the glyph is still visible within it (WorldLingo, 2010f). This is aninteresting feature that adds to the complexity of the glyph.

Coptic was chosen because of the same Egyptian interest, as Coptic was usedto write the Egyptian language (WorldLingo, 2010b). Also, it is quite close to

17

Page 32: Classifying Glyphs - NTNU

CHAPTER 3. METHODOLOGY 18

Greek, which also was under consideration.

As the other alphabet, runic was chosen. Runes are also historically interest-ing, as they were used here in Norway (WorldLingo, 2010j).

Abjads are mostly used in the Middle East (WorldLingo, 2010m). Hebrew andArabic were chosen mainly because they are both well known and not too similar(WorldLingo, 2010e,a).

Han — also known as the CJK ideographs, the unified logosyllabaries of Chi-nese, Japanese and Korean — were chosen as a modern logosyllabary. Thiswas because of their differences from the Egyptian hieroglyphs, their widespreaduse today, an interest in Japanese, and because it is a writing system used forthree very different spoken languages (WorldLingo, 2010c). While this might becommon for other writing systems as well (e.g. Latin (WorldLingo, 2010i)), thefact that Han is a logosyllabary makes the same text more or less readable forboth Chinese, Japanese and Koreans. The most important reason for choosingHan/CJK and not just kanji, however, is because the system relies on Unicode,and CJK is the Unicode way of saving storage on three very similar writing sys-tems (The Unicode Consortium, 2009, CJK Unified Ideographs).

Since the Japanese kanji were added as part of Han, hiragana was chosen. Partbecause of the interest in Japanese, but mainly to see if the system would haveless trouble separating the kanji from the hiragana than from the other scripts.After all, hiragana is a simplification of kanji, for use by the uneducated women(WorldLingo, 2010g). Katakana — another Kanji simplification (WorldLingo,2010h) — was also under consideration, but hiragana was chosen; part becauseof its softer fonts (as to not interfere with the runes), and part because hiragana isthe most common of the two. Katakana is mostly used for loan words, transcribedfrom foreign languages (WorldLingo, 2010h).

The other syllabary was chosen for similar reasons. Hangul jamo is the nativewriting system of the Korean language (WorldLingo, 2010d), in contrast to Han.It is also not too different from Han, even though Hangul is not a logosyllabary.

Thai was chosen to have another complex set of glyphs from East Asia, ofyet another type (WorldLingo, 2010l). Then the abugida was topped off bythe Canadian Aboriginal syllabics, mainly because they look a bit alien andhave fascinating vowels. That is, the independent vowel following a consonant isrepresented by the rotation of the consonant (WorldLingo, 2010k).

Page 33: Classifying Glyphs - NTNU

19 3.1. GLYPHS AND WRITING SYSTEMS

..x1 .x2 .... .x400

.h1 .h2.... .h50

.y1 .y2 .... .y10

.

.0.02 .0.05 .0.11 .0.82

.∑

.1.00

.

. ..

.I.#13001

.0.5

.0.5

Figure 3.1: The path from glyph to output values is long and complex. The starting pointis the Unicode value, i.e. #13001, which is converted to a glyph image. This is resized to a20 × 20 image. Each of the 400 pixels are given to an input node in the ANN, after goingthrough a sigmoid function (Equation 3.2 on page 21). The output of the ANN is a list ofnormalized number, one for each output node; thus one for each writing system. The value isthe probability that the glyph in question is of that specific writing system.

Page 34: Classifying Glyphs - NTNU

CHAPTER 3. METHODOLOGY 20

3.2 Neural Network DesignThe neural networks used have the same three-layered architecture. There are

400 input nodes plus one bias node, 50 hidden nodes plus one bias, and up to tenoutput nodes.

Nodes The number of input nodes is the number of pixels in the glyph images.This must be balanced between enough pixels to separate the glyphs and fewenough to have an acceptably low execution time. The trick is thus to go aslow as possible without compromising the image quality too much. By creatingglyphs of different resolutions, those around 20 times 20 were found to be a goodcompromise.

The number of hidden nodes was chosen a bit more ad hoc. 50 was chosenbecause it gave an acceptable result in acceptably short time. 100 hidden nodesadded more to the execution time than to the quality of the results.

The number of output nodes, on the other hand, is undoubtedly correct. Thereis one output node for each writing system to be classified in that run. Eachoutput node has thus its own writing system, and the value of the node describesthe probability that the glyph in question is of that specific writing system. Thisoutput design is to help analyze the relationship between the writing systems.By using the probability for each of the writing systems, it is easier to pick upon common misclassifications and other patterns, e.g. if it really is harder toseparate between Han and hiragana than, say, Han and Thai.

The two bias nodes were chosen to add a threshold to the nodes (Jones, 2009;Floreano and Mattiussi, 2008; Callan, 1999). The bias was originally 1, butexperimentation on the values 0, 0.5 and 1 indicated that 0.5 was the best valuefor both bias nodes.

Architecture The final architecture, as well as the whole path from Unicodepoint to output values, can be seen in Figure 3.1 on the previous page.

The starting point is the Unicode value, e.g. #13001. This is converted to aglyph image, which then is resized to a 20 × 20 image. This results in a list of400 pixels, each with an integer value between 0 and 255, inclusive. Each of thesepixels are run through a sigmoid function (Equation 3.2 on the facing page), suchthat the final value is a floating point between 0 and 1, inclusive. In additionto reducing the numbers to a more ANN friendly value, the sigmoid functionwidens the gap between the light and the dark pixels, creating more of a binarysituation. The 400 floating points are then given to one input node each, and thevalues propagate through the network in the normal ANN manner (Section 2.3on page 6).

The final output is a normalized list of the values of the output nodes. That is,they sum to 1.0, or 100%. Each of these values is the probability that the glyphin question is of the writing system coupled with that specific output node. Aperfect classification of #13001 would thus be (0.000 0.000 0.000 0.000 0.0000.000 0.000 0.000 0.000 1.000), as the glyph1 is an Egyptian hieroglyph,

1The glyph used in Figure 3.1 isI. This is a determinative, or meaning-sign, used to illustrate that the wordit follows has something to do with the mouth, either literally or metaphorically. That is, it follows words forwhat can be taken in or expelled through the mouth. As it is a determinative, the glyph has no sound of its

Page 35: Classifying Glyphs - NTNU

21 3.3. LEARNING

which is coupled to the last output node. The writing systems are coupled withnodes in the order given in Table 3.1 on page 17.

3.3 Learning

The learning algorithm is based on the standard backpropagation algorithmon page 10, but using the tricks in Section 2.4.4.

Weight Initialization All the weights are initialized according to Equation 3.1,where n is the number of nodes in the link’s input layer. This gives a numberbetween -0.3 and 0.4, before it is divided on the number of input nodes n. Theweights are divided on n to make sure that the input values for the nodes inthe next layer will be at an acceptably low level, even given the large number ofincoming weights.

w =random(70) − 30

100n(3.1)

One might think that using 35 instead of 30 — thus initialize to a numberbetween -0.35 and 0.35, with an average of 0 — would be a better solution. How-ever, experimentation showed that this led to a slowly increasing error, insteadof a faster decreasing error. Initialization between -0.4 and 0.3 led to a fasterincreasing error. A number between -0.3 and 0.4 seemed thus like best option.

From Glyph to Input Node For each of the training patterns, the pixels aremapped onto the input nodes. However, the pixel values are not directly copied.Both copying, some different sigmoid functions, and bucket classification weretried. The best result came when sending the pixels through Equation 3.2 (Ta-ble D.3 on page D-3). The input values are thus in the range [0, 1].

Φ(n) =1

1 + e−0.1(n−127)(3.2)

From Input Node to Output Node The values go from the input layer tothe hidden layer through Equation 2.6, and further to the output layer by usingEquation 2.8. The sigmoid function used, however, is not Equation 2.7.

For the last link, into the output nodes, the softmax activation function (2.23)is used. As the softmax function is specifically designed for multi-class classifica-tion problems (which the glyph classification problem is), it is the natural choice.However, as it is only for use in the output layer, Equation 3.3 was used for thehidden layer. (3.3) is a specialized version of (2.7), with k = 1.

Φ(ai) =1

1 + e−ai(3.3)

own; it is simply a written construct to give the reader more information and context. (Collier and Manley,2003)

Page 36: Classifying Glyphs - NTNU

CHAPTER 3. METHODOLOGY 22

Table 3.2: Parameter values for the evolution.

Parameter ValueCrossover rate 0.30Mutation rate 0.02Elitists 2Number of children 60Culling rate 0.20Maximum number of generations 200Error threshold 0.1

Error and Weight Change The errors for the output layer and the hidden layerare calculated using Equations 2.25 and 2.13, respectively. (2.25) is used becauseof the softmax function and the cross-entropy error function.

The weight change is based on (2.14) and (2.15). In addition, there is amomentum factor, adding a momentum term m that determines the effect thelast weight change shall have on this weight change (Cho et al., 1991).

∆wµt

ij = δµi hµ

j + m∆wµt−1

ij (3.4)∆vµt

jk = δµj xµ

k + m∆vµt−1

jk (3.5)

In reality, however, it is the same as in (2.14) and (2.15). Experimentationhas shown that m = 0 gives the best results, so the whole term is in practiceremoved.

The final weight is then updated as in (2.16) and (2.17), by increasing theweight by the product of the weight changes and the learning rate η = 0.01.

Error and Stopping Criteria The final error is the cross-entropy error function(2.24), as proposed by Bullinaria (2009). The learning continues until the error(3.6) is less than 0.01. When the error is this low, the accuracy is acceptablyhigh, and it does not take too long to run.

Error = |− (tµi log(yµ

i ) + (1 − tµi ) log (1 − yµ

i ))| < 0.01 (3.6)

3.4 Evolution

As a genetic algorithm (GA) is a good choice of evolutionary computationmethod when it comes to evolving neural networks (Downing, 2006), that is themethod used here.

That includes a bit vector genotype, both mutation and crossover, full gen-erational replacement, uniform parent selection and fitness proportionate mateselection (Table 2.2 on page 14). The parameters used in the evolution can befound in Table 3.2.

Development As this is a GA, the genotype is a bit vector. Each chromosomeconsists of one gene for each weight, where each gene consists of 8 bits (Table3.3). The first bit is the connection bit, inspired by EEC (Shi and Wu, 2008). Ifit is 0, there is no connection; i.e. the weight is automatically set to 0. If 1, the

Page 37: Classifying Glyphs - NTNU

23 3.4. EVOLUTION

Table 3.3: The gene consists of 8 bits, and specifies whether there is a connection (bit 0), thesign of the weight (bit 1) and the weight value (bit 2-7).

0 1 2 3 4 5 6 7Connection Sign 100 10−1 10−2 10−3 10−4 10−5

weight is as specified by the later bits. The second bit is the sign. If it is 0, theweight is positive; if 1, the weight is negative. The rest of the bits specify theweight value. The first of these bits specifies the ones, then follows the tenths, thehundreds and so on. This results in a weight range from −1.96875 (11111111)to 1.96875 (10111111).

Fitness Testing The fitness of the individual is depending on the classificationresults. The weights created during development are pushed on an ANN, andthen the ANN is used to classify the glyphs in the training set. The error iscalculated as the average error for all the glyphs.

The error function used is not the same as during learning, but simply thetotal sum squared error (Equation 2.18).

The fitness of an individual is calculated based on the actual error and themaximal error. As the error is the total sum squared error (2.18), and the possiblevalues for each node is between 0 and 1, the error must necessarily be between 0and 1. The fitness function is thus:

fitness(error) =Errormax − error

Errormax

= 1 − error (3.7)

Adult Selection As specified by the algorithm (Section 2.5.1.1), GAs shall havea full generational replacement, where all of the children — and only the children— survive into adulthood. As a result, there is virtually no selection pressure.This gives all the individuals the possibility to stay and maybe develop into avery good individual in some generations. However, this is a slow process andsuccess is in no way guaranteed.

To speed up the process, culling was introduced (Baum et al., 2001; McQuestenand Miikkulainen, 1997). Real-life culling is the process of removing animals froma group based on specific criteria, either to reinforce good characteristics or toremove undesired characteristics (Merriam-Webster, 2010). The same processis implemented here; the individuals are sorted based on fitness, and a certainpercentage of the best children are kept, while the rest are killed off. The adultpopulation consists thus of only 20% of the best children.

Parent Selection There is a uniform parent selection, as specified by the algo-rithm (Section 2.5.1.1). It is not, however, as uniform as it was meant to be, asthe suboptimal individuals were killed off in the adult selection. Instead of a uni-form selection between all the individuals, it is thus a uniform selection betweenthe best individuals.

Reproduction The reproduction follows the algorithmic guidelines given in Sec-tion 2.5.1.1. The parents are chosen from the adult pool with a probabilityproportional to their fitness. There is a 30% chance the parents are mating using

Page 38: Classifying Glyphs - NTNU

CHAPTER 3. METHODOLOGY 24

crossover, otherwise they are just directly copied into the next generation. Incase of crossover, their chromosomes are split on the border between two genes,to prevent the destruction of two potentially good genes. Then, for each gene ineach of the new chromosomes, there is a 2% chance of a mutation, i.e. a bit flipof a random bit.

The newborn children are not the only ones competing for a place in the nextgeneration. There are also two elitists; the very best from the last generation. Inaddition to having a high probability of giving their genes to the next generationthrough mating, they are allowed to join in themselves. This is to prevent goodindividuals from dying off without contributing to even better children; if thenext generation is not superior to the last, at least the best individual is not anyworse.

Stopping Criteria The evolution was meant to continue until one of the indi-viduals had reach the error threshold of 0.01. However, the evolution was a slowprocess, with a very slowly decreasing error. The available memory, on the otherhand, tended to disappear very quickly. Alas, the whole process always died ofmemory fault, long before this error was reached.

To solve this problem, the evolution continues only until the error has reached0.1, but for no more than for 200 generations. Then, a neural network is createdbased on the best individual, and that ANN is used for testing.

Page 39: Classifying Glyphs - NTNU

CHAPTER 4

Results and Analysis

For both learning and evolution, full comparisons using all ten writing systemswere performed. In addition, the writing systems were compared in pairs usinglearning. This was not retested using evolution, as it turned out that learningworked while evolution did not.

4.1 LearningLearning gave generally good results, and was used for both pair classification

and full classification.

4.1.1 Pair ClassificationAll combinations of writing systems were compared in pairs. The graphs can

be found in Appendix B.1. For the training, all the runs that contain a specificwriting system are drawn in the plot for that writing system. As a result, all runsare plotted twice; once for each writing system. The test graphs, on the otherhand, contain only two writing systems at once.

These classifications were performed using learning, with a learning rate η =0.05 and a momentum m = 0. There were 30 glyphs from each writing system,and a training ratio of 80%. There were a maximum of 500 iterations. However,the graphs end at 300 iterations, as there were no notable change in error for thelast 200 iterations.

It is worth noting that there was only one run per combination, so this datais not scientifically significant. It might, however, work as a proof of concept.

Each comparison had a test set with 12 glyphs, 6 from each writing system.Table 4.1 displays, for each of the comparisons, how many of these 12 that werecorrectly classified.

The test plots in Appendix B.1.2 display the accumulated output from theruns, separated on the actual writing system. This does not say anything aboutthe final classification, as that is depending on the difference in output (prob-ability) on each glyph. As a result, even if the accumulated values are close,the classification might still be correct. The test plots do, however, give a good

25

Page 40: Classifying Glyphs - NTNU

CHAPTER 4. RESULTS AND ANALYSIS 26

Table 4.1: The number of correctly classified glyphs, of a total of 12 test glyphs.

Cop

tic

Run

ic

Heb

rew

Ara

bic

Han

gul

Hir

agan

a

Tha

i

Sylla

bics

Han

Hie

rogl

yphs

Ave

rage

Coptic - 11 10 7 12 11 9 11 10 12 86.1%Runic 11 - 12 9 12 12 12 10 12 6 88.9%

Hebrew 10 12 - 11 10 11 12 10 11 12 91.7%Arabic 7 9 11 - 10 10 10 9 11 12 82.4%Hangul 12 12 10 10 - 10 11 9 10 12 88.9%

Hiragana 11 12 11 10 10 - 12 9 9 12 88.9%Thai 9 12 12 10 11 12 - 11 12 12 93.5%

Syllabics 11 10 10 9 9 9 11 - 10 12 84.3%Han 10 12 11 11 10 9 12 10 - 12 89.8%

Hieroglyphs 12 6 12 12 12 12 12 12 12 - 94.4%

Table 4.2: The maximal number of accumulated misclassification probabilities (Figures inAppendix B.1.2). At 3, there is an equal probability for both writing systems, i.e. below 3there is a very good chance of correct classification — perfect classification at 0 — while above3 the classification is not good. At 6 there is a perfect misclassification, the worst possibleresult.

Cop

tic

Run

ic

Heb

rew

Ara

bic

Han

gul

Hir

agan

a

Tha

i

Sylla

bics

Han

Hie

rogl

yphs

Coptic 1 1 1 3 1 2 1 2 0Runic 1 0 3 0 0 0 2 0 4

Hebrew 1 0 1 1 1 0 2 1 0Arabic 1 3 1 1 3 2 2 2 0Hangul 3 0 1 1 2 1 3 2 0

Hiragana 1 0 1 3 2 1 2 2 0Thai 2 0 0 2 1 1 1 1 0

Syllabics 1 2 2 2 3 2 1 2 0Han 2 0 1 2 2 2 1 2 0

Hieroglyphs 0 4 0 0 0 0 0 0 0

picture on the quality of the classification. The ultimate classification has 100%probability for the correct writing system.

Table 4.2 displays the highest misclassification from these plots (AppendixB.1.2). It shows that Egyptian hieroglyphs are not just correctly classified byaccident; there is actually a very clear separation between the hieroglyphs andthe other writing system. The exception is with runic, but that was a run whichnever got an error below 0.6 (Appendix B.1.1, Runic). Otherwise, runic does verywell, with multiple 0-runs. Also Thai and Hebrew do well, with mainly 1’s.

The fact that both hieroglyphs and runic do very well in general but are adisastrous combination, points to the possibility of their glyphs being too similar.The example images in Table 3.1 might help explain this; both the hieroglyphs andthe runic glyph have very light colours. While the other images are mainly blackand white, hieroglyphs and runic are on the grey-scale. The theory is supportedby the pixel comparison in Appendix D. While the two writing systems are verysimilar when the acceptable difference is larger (Tables D.3 and D.6), they are

Page 41: Classifying Glyphs - NTNU

27 4.1. LEARNING

not so similar when the acceptable difference is smaller (Tables D.2 and D.9).With larger spans, their grey-scale pixels are seen as identical. The other writingsystems are not caught in this similarity, as their white and black pixels are toofar off in value.

4.1.2 Full ClassificationThe main classification consists of all ten writing systems, using 30 glyphs

from each. This was, according to the tests in Appendix C, the fairest data setdistribution. That is, as many glyphs as possible while using the same numberfrom each writing system, and without duplicates.

As the system is highly random, e.g. when selecting the glyphs to use, theclassification of all ten writing systems were run twenty times. All the twentyerror graphs can be seen in Figure 4.1a on the following page. The runs thatstart their flattening phase first, are among those that need the most iterationsto reach the error threshold. This is not unnatural, assuming the error decreaseswith roughly the same speed in all of their flattening phases.

Figure 4.1b on the next page displays the percentage of correctly classifiedglyphs for all the runs. The success rate is ranging from 58.33% to 86.67%, withan average of 73.17%. That is, almost 44 of 60 test glyphs correctly classified.

73% correctly classified means there is still 27% that is incorrectly classified.That is not a very impressive figure. However, these figures represent the situationin black and white. The grey-scale can be seen in Figure 4.1c on page 29. Thisplot shows the average of the accumulated probability that each of the test glyphsis a member of each of the writing systems. Even though none of the writingsystems are perfectly classified, all of them have a much higher probability ofbeing classified as the correct one, than as any one of the others.

It is worth noting that the similarity between runic and Egyptian hieroglyphscontinues. It is not as visible in the full classification as in the pair classification,where they never reached a low error (Appendix B.1.1). In this case (Figure 4.1c),both runic and hieroglyphs have a higher success rate than average. However,while the other writing systems are misclassified as a seemingly random otherwriting system, runic and hieroglyphs are (almost) always misclassified as eachother. That does make sense, as all the other writing systems share the sameblack or white similarity. If they are misclassified, it ought to be as a writingsystem that share their pixel value distribution.

4.1.3 Writing System Classification QualityFigure 4.2 on page 30 points to a certain correlation between the pair clas-

sification and the full classification. The five writing systems with the lowestaccumulated probability of correct classification during the full classification —Hangul, hiragana, Coptic, syllabics and Arabic — also had the lowest averageof correct classification in the pair classification. The most notable exception isrunic, which had precisely the same average in the pair classification as Hanguland hiragana, but did much better in the full classification.

Otherwise, there is a clear separation between the five best writing systemsin the top right corner, and the five worst in the bottom left corner. It seemslearning is better suited to separate Egyptian hieroglyphs, Thai, Hebrew, Han

Page 42: Classifying Glyphs - NTNU

CHAPTER 4. RESULTS AND ANALYSIS 28

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 500 1000 1500 2000 2500

Erro

r

Iteration

Training

Error

(a) Training

020406080

100

0 2 4 6 8 10 12 14 16 18 20

Cor

rect

lycl

assifi

edgl

yphs

[%]

Run

Testing

Success rate(b) Success rate

Figure 4.1: Classification of all ten writing systems. 20 runs with learning rate η = 0.01,momentum m = 0, error threshold of 0.01, 30 glyphs from each writing system, and a trainingratio of 80%. The glyphs are selected anew for each of the twenty runs.Figure 4.1a contains the 20 error curves for the run. They are smoothed using a Bezier function,and then drawn on top of each other. Figure 4.1b displays the percentage of correctly classifiedglyphs for each of the runs. This is a matter of correct or not correct, while Figure 4.1c containsthe accumulated probability that each of the glyphs is a member of each of the writing systems.Here, the data is averaged over the 20 runs.

Page 43: Classifying Glyphs - NTNU

29 4.1. LEARNING

0

1

2

3

4

5

6

Coptic

Runic

Hebrew

Arabic

Hangul

Hiragana

Thai

Syllabics

Han

Hieroglyphs

Cla

ssifi

edas

Writing system

Testing

CopticRunic

HebrewArabicHangul

HiraganaThai

Canadian Aboriginal syllabicsHan

Egyptian hieroglyphs(c) Testing

Figure 4.1: (cont.) The test results.

Page 44: Classifying Glyphs - NTNU

CHAPTER 4. RESULTS AND ANALYSIS 30

and runic, than Hangul, hiragana, Coptic, Canadian Aboriginal syllabics andArabic.

There is no clear correlation between these results and the pixel similarityin Appendix D. According to Table D.3 — as Richard’s curve with maximumgrowth at 128 was used during learning — runic, Thai and Egyptian hieroglyphshave all higher similarity with itself than with most other writing systems. Hanand Hebrew, on the other hand, do not. It is thus difficult to say if there is anycorrelation between pixel similarity and ease of classification.

82

84

86

88

90

92

94

96

3 3.5 4 4.5 5 5.5 6

Aver

age

corr

ectly

clas

sified

inpa

ircl

assifi

catio

n[%

]

Accumulated probability in full classification

Correlation between pair classification and full classification

Coptic

Runic

Hebrew

Arabic

HangulHiragana

Thai

Syllabics

Han

Hieroglyphs

Figure 4.2: Correlation between the full classification and the pair classification. The x axis isthe accumulated probability that each of the writing systems is correctly classified in the fullclassification (Figure 4.1c), while the y axis is the average of correctly classified glyphs in thepair classification (Table 4.1).

4.2 EvolutionAs the evolution was slow and memory consuming (Section 3.4), it was difficult

to get good results. The evolution had to stop after 200 generations to avoidmemory problems, but by then the fitness had not yet reached an acceptablelevel (Figure 4.3). Because of the time consumed and the overall bad results, theevolution was only run four times. This is not enough to make sure the results arenot just random fluctuations. However, the results are significantly worse thanfor learning, and there is no reason to believe the next sixteen runs would be any

Page 45: Classifying Glyphs - NTNU

31 4.2. EVOLUTION

different in that matter.The fitness for the four runs can be seen in Figure 4.3a. This is an interesting

graph with respect to the minimal and average fitness. In all four cases, theaverage fitness is just above the minimal fitness. This points to a homogeneouspopulation, where the average is pulled up because of some individuals that aremuch more fit than the rest. These individuals would be the elitists, making surethat the best individual in the population is as good as or better than the bestindividual in the last generation.

One theory was that the mutation was destructive, causing potentially goodindividuals to change too much. The mutation rate was thus reduced from 2%to 0.2%, and then further down to 0.02%. There are no official data for any ofthese runs, but there were no positive change in the first hundred generations.Then the crossover rate was changed from 30% to 0.3%, but still no change.

As the error at the end of training was this high (more than ten times higherthan during learning), the testing results come as no surprise (Figure 4.3b). Thetest results range from 15% correct to 3.33% correct, with an average of 9.58%.Compared to the 73% average of learning (Figure 4.1b on page 28), this is clearlysuboptimal.

The problem can be seen in Figure 4.3c. While the test results from thelearning (Figure 4.1c) clearly separate between the writing systems, the evolutiondoes not. The probability of being classified as one specific writing system isalmost the same for each of the writing systems. All of the writing systems havea high probability of being classified as hiragana or hieroglyphs. Hebrew andCanadian Aboriginal syllabics follow not far behind. It seems the few glyphsthat were correctly classified, were either hiragana or hieroglyphs, or just lucky.These results clearly show that the evolved brain is not able to separate betweenthe writing systems.

Page 46: Classifying Glyphs - NTNU

CHAPTER 4. RESULTS AND ANALYSIS 32

0.80.810.820.830.840.850.860.870.880.890.9

0.91

0 20 40 60 80 100 120 140 160 180 20000.0020.0040.0060.0080.010.0120.014

Fitn

ess

Stan

dard

devi

atio

n

Generation

Training

Minimal fitnessMaximal fitnessAverage fitness

Average standard deviation(a) Training

020406080

100

0 0.5 1 1.5 2 2.5 3

Cor

rect

lycl

assifi

edgl

yphs

[%]

Run

Testing

Success rate(b) Success rate

Figure 4.3: Classification of all ten writing systems. 4 runs with crossover rate of 30%, mutationrate of 5%, 60 children, 2 elitists, 20% culling rate, an error threshold of 0.1, and a maximumof 200 generations. The glyphs are selected anew for each of the four runs.Figure 4.3a contains the fitness for the four runs, smoothed using a Bezier function. Thisgraph contains both the maximal, minimal and average fitness, as well as the average standarddeviation. Figure 4.3b displays the percentage of correctly classified glyphs for each of theruns. This is a matter of correct or not correct, while Figure 4.3c contains the accumulatedprobability that each of the glyphs is a member of each of the writing systems. Here, the datais averaged over the 4 runs.

Page 47: Classifying Glyphs - NTNU

33 4.2. EVOLUTION

0

1

2

3

4

5

6

Coptic

Runic

Hebrew

Arabic

Hangul

Hiragana

Thai

Syllabics

Han

Hieroglyphs

Cla

ssifi

edas

Writing system

Testing

CopticRunic

HebrewArabicHangul

HiraganaThai

Canadian Aboriginal syllabicsHan

Egyptian hieroglyphs(c) Testing

Figure 4.3: (cont.) The test results.

Page 48: Classifying Glyphs - NTNU

CHAPTER 4. RESULTS AND ANALYSIS 34

Page 49: Classifying Glyphs - NTNU

CHAPTER 5

Discussion

The main goal of this dissertation is to see if a neural net can generalizeenough to classify the writing system of an unseen glyph. The results from thelearning (Figure 4.1 on page 29) point to a neural net that is capable of this.The results are far from perfect. However, with more than 50% correct from tenpotential answers, it is clear that the network is able to separate between thewriting system.

5.1 Learning or Evolution?The results indicate that learning is far better suited for this task than evo-

lution is. The best network created from learning had an average success rateof 73% (Figure 4.1), while the best evolved network achieved less than 10% onaverage (Figure 4.3). As Figure 4.1c reveals, learning had a success rate far abovewhat is expected of random classification. Evolution, however, was unsuccessful.There is nothing in Figure 4.3c that indicates any form of intelligent classification.

It is only natural that the learned network perform better than the evolved net-work. Learning stopped when the error was below 0.01, while evolution stoppedalready at 0.1, or after a maximum of 200 generations. This is the main problemwith evolution; it uses too much memory, too much processing power, and toomuch time. No available hardware is able to keep the evolution going for morethan this. Alas, the results are a natural consequence.

However, the fitness graphs in Figure 4.3a do not look promising. The maximalfitness is rising because of elitists; there is no clear improvement in the averagefitness. Parameter tweaking might be able to solve this. The evolutionary algo-rithm in its current state, however, will probably never reach an error of less than0.01; except by a potential strike of pure luck in the random generator.

5.2 Writing SystemsIt seems like there is no specific correlation between any of the writing systems.

This includes the closely related Han and hiragana (Figure 4.1c, Table 4.2, Section3.1). Egyptian hieroglyphs and runic are tightly coupled (Figure 4.1c, Tables 4.1

35

Page 50: Classifying Glyphs - NTNU

CHAPTER 5. DISCUSSION 36

and 4.2), but that is probably just a result of the grey-scale of their images(Section 4.1.2).

It is worth noting that hieroglyphs, Thai, Hebrew, Han and runic are easier toclassify than Hangul, hiragana, Coptic, syllabics and Arabic (Figure 4.2). How-ever, there is no obvious reason why these specific systems are easier. It does notseem to have anything to do with pixel similarity (Section 4.1.3).

5.3 Future WorkThere are many ways to go from here. There is still fine-tuning needed on

the learning algorithm. On the other hand, it is too early to dismiss evolutionjust yet. There are still untested tweaks and algorithms that might improve theresults.

Perfecting the Learning While the results from the learning are good, there isstill room for improvement. Since learning is not especially memory consuming,nor has a very long execution time, it might be a good idea to create moreinput nodes. That is, higher resolution on the glyphs. It might be that thecurrent resolution is generalizing just enough, but it is something one ought toinvestigate.

More layers might help in the generalization. More than two hidden layers areprobably not necessary, but one extra layer ought to be tested.

Fast backpropagation or other versions of the algorithm might also improve theresults. Either way, the parameter values ought to be better configured. Thesecan for example be found using evolution, or just manual trial and error.

Giving the Evolution a Second Chance Another possibility is to focus on theevolution, and try to achieve better results. As this would be done with no regardfor learning, also the topology can be evolved. Hopefully this will create a networkmore suited for the task. Also here, some extensive parameter tweaking ought tobe performed.

Neuroevolution of augmenting topologies (NEAT) is a method for evolvingneural networks, including the topology. Stanley and Miikkulainen (2002) claim itis very efficient, which is a very good thing regarding the execution time problemsof the standard evolution. The specialization HyperNEAT (Gauci and Stanley,2010) might be an even better solution, as HyperNEAT introduces geometry. Theglyph classification can be seen as a geometrical problem, as the pixels’ relativelocation is crucial for the representation of the glyph. The possibilities of usingNEAT or HyperNEAT to evolve the network ought to be explored.

However, extensive evolution necessitates a cluster to run on, or correspondinghardware.

General Improvements It might be advantageous to create the network cumu-latively. For example, one can create a subnet for each of the writing systems,and then merge them into one larger network. This method should work for bothlearning and evolution, but is probably most worthwhile during evolution. Itis during evolution the execution time is most problematic, and thus there thisdivision of labour is most important.

Page 51: Classifying Glyphs - NTNU

37 5.3. FUTURE WORK

The input might need some redesign. The pixel values used in this version areprobably not the best solution when it comes to capturing the lines and circles ofthe glyph. It is worth investigating more advanced image processing techniques,to create another means of representation.

Page 52: Classifying Glyphs - NTNU

CHAPTER 5. DISCUSSION 38

Page 53: Classifying Glyphs - NTNU

References

Eri Banno, Yutaka Ohno, Yoko Sakane, and Chikako Shinagawa. Genki 1: AnIntegrated Course in Elementary Japanese. 1999. ISBN 4789009637.

E.B. Baum, D. Boneh, and C. Garrett. Where Genetic Algorithms Excel.Evolutionary Computation, 9(1):93–124, 2001. ISSN 1063-6560. URL http://www.mitpressjournals.org/doi/abs/10.1162/10636560151075130.

BioSS. The generalised logistic curve. URL http://www.bioss.ac.uk/smart/unix/mgrow/slides/slide02.htm.

Christopher M. Bishop. Neural Networks for Pattern Recognition. Oxford Uni-versity Press, Oxford, England, 1996. ISBN 0198538642.

John A. Bullinaria. John Bullinaria’s Step by Step Guide to Implementing aNeural Network in C, 2009. URL http://www.cs.bham.ac.uk/~jxb/INC/nn.html.

Robert Callan. The Essence of Neural Networks. Prentice Hall Europe, Harlow,Essex, Enland, 1999. ISBN 0-13-908732-X.

T.-H. Cho, R.W. Conners, and P.A. Araman. Fast backpropagation learningusing steep activation functions and automatic weight reinitialization. Confer-ence Proceedings 1991 IEEE International Conference on Systems, Man, andCybernetics, 3:1587–1592, 1991. doi: 10.1109/ICSMC.1991.169915. URL http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=169915.

Mark Collier and Bill Manley. How to read Egyptian hieroglyphs: a step-by-step guide to teach yourself. University of California Press, Los Ange-les, 2003. ISBN 0520239490. URL http://books.google.com/books?id=aroQ0Zrl8J4C&pgis=1.

Charles Darwin. On the Origin of Species by Means of Natural Selection, or thePreservation of Favoured Races in the Struggle for Life. Murray, London, Eng-land, 1859. URL http://www.literature.org/authors/darwin-charles/the-origin-of-species/.

Kenneth A. De Jong. Evolutionary Computation : A Unified Approach. The MITPress, Cambridge, MA, USA, 2006. ISBN 0-262-04194-4.

Stanislas Dehaene. Reading in the Brain : The Science and Evolution of a HumanInvention. Penguin Group, London, England, 2009. ISBN 978-0-670-02110-9.

39

Page 54: Classifying Glyphs - NTNU

REFERENCES 40

Rinku Dewri. Evolutionary Neural Networks: Design Methodolo-gies - AI Depot, 2003. URL http://ai-depot.com/articles/evolutionary-neural-networks-design-methodologies/.

Keith L. Downing. Combining Evolutionary Algorithms and Neural Net-works. Science And Technology, 2006. URL http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.100.5913&amp;rep=rep1&amp;type=pdf.

Keith L. Downing. Introduction to evolutionary algorithms. 2009a. URL http://www.idi.ntnu.no/emner/it3708/lectures/evolalgs.pdf.

Keith L. Downing. Reinforcement Learning in Neural Networks. 2009b. URLhttp://www.idi.ntnu.no/emner/it3708/lectures/learn-rl.pdf.

Keith L. Downing. Supervised Learning in Neural Networks. 2009c. URL http://www.idi.ntnu.no/emner/it3708/lectures/learn-sup.pdf.

Keith L. Downing. Unsupervised Learning in Neural Networks. 2009d. URLhttp://www.idi.ntnu.no/emner/it3708/lectures/learn-unsup.pdf.

R. A. Dunne and N. A. Campbell. On The Pairing Of The Softmax ActivationAnd Cross-Entropy Penalty Functions And The Derivation Of The SoftmaxActivation Function. In 8th Australian Conference on Neural Networks, pages181–185, Melbourne, Australia, 1997.

Encyclopædia Britannica. Absolute temperature scale, 2010a. URL http://www.britannica.com/EBchecked/topic/1801/absolute-temperature-scale.

Encyclopædia Britannica. Boltzmann constant, 2010b. URL http://www.britannica.com/EBchecked/topic/72417/Boltzmann-constant.

Encyclopædia Britannica. Fermi level, 2010c. URL http://www.britannica.com/EBchecked/topic/204779/Fermi-level.

Dario Floreano and Claudio Mattiussi. Bio-Inspired Artificial Intelligence : The-ories, Methods, and Technologies. The MIT Press, Cambridge, MA, USA, 2008.ISBN 978-0-262-06271-8.

FOLDOC. Glyph, 1998a. URL http://foldoc.org/glyphs.

FOLDOC. Writing System, 1998b. URL http://foldoc.org/writing+system.

Jason Gauci and Kenneth O. Stanley. Autonomous evolution of topographicregularities in artificial neural networks., July 2010. ISSN 1530-888X. URLhttp://www.ncbi.nlm.nih.gov/pubmed/20235822.

V.K. Govindan and A.P. Shivaprasad. Character Recognition - A Review. PatternRecognition, 23(7):671–683, 1990.

M. Tim Jones. Artificial Intelligence : A Systems Approach. Jones and BartlettPublishers, Sudbury, MA, USA, 2009. ISBN 978-0-7637-7337-3.

Daniel Kirsch. Detexify LaTeX handwritten symbol recognition. URL http://detexify.kirelabs.org/classify.html.

Page 55: Classifying Glyphs - NTNU

41 REFERENCES

Kevin Knight and Kenji Yamada. A computational approach to deci-phering unknown scripts. In ACL Workshop on Unsupervised Learningin Natural Language Processing, number 1, pages 37–44. Citeseer, 1999.URL http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.43.7982&amp;rep=rep1&amp;type=pdf.

J Mantas. An Overview of Character Recognition Methodologies. Pattern Recog-nition, 19(6):3–8, 1986.

Paul McQuesten and Risto Miikkulainen. Culling and teaching in neuro-evolution. In Proceedings of the 7th International Conference on Genetic Al-gorithms, pages 1–8, 1997. URL http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.49.8014&amp;rep=rep1&amp;type=pdf.

Merriam-Webster. Cull, 2010. URL http://www.merriam-webster.com/dictionahttp://www.merriam-webster.com/dictionary/culling.

J.E. Ngolediage, R.N.G. Naguib, and S.S. Dlay. Fast backpropagation for super-vised learning. Proceedings of 1993 International Conference on Neural Net-works (IJCNN-93-Nagoya, Japan), 2(4):2591–2594, 1993. doi: 10.1109/IJCNN.1993.714254. URL http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=714254.

Rafael Ramos-Garijo, Sergio Martın, Andres Marzal, Frederico Prat, Juan MiguelVilar, and David Llorens. An input panel and recognition engine for on-linehandwritten text recognition, pages 223–232. IOS Press, 2007.

Jairo Rocha and Theo Pavlidis. A shape analysis model with applications to acharacter recognition system. IEEE Transactions on Pattern Analysis and Ma-chine Intelligence, 16(4):393–404, April 1994. ISSN 01628828. doi: 10.1109/34.277592. URL http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=277592.

D.E. Rumelhart, G.E. Hinton, and R.J. Williams. Learning Representations byBack-Propagation of Errors. Nature, 1986.

Min Shi and Haifeng Wu. Evolving Efficient Connection for the Design ofArtificial Neural Networks. In Proceedings of the 18th International Con-ference on Artificial Neural Networks, pages 909–918, Trondheim, Norway,2008. URL http://www.springerlink.com/content/au45qn6753k42360/fulltext.pdf.

Kenneth O. Stanley and Risto Miikkulainen. Evolving Neural Networks throughAugmenting Topologies. Evolutionary Computation, 10(2):99–127, 2002.

The Unicode Consortium. The Unicode Standard, Version 5.2.0. The UnicodeConsortium, Mountain View, CA, USA, 2009. ISBN 978-1-936213-00-9. URLhttp://www.unicode.org/versions/Unicode5.2.0/.

Kiri Wagstaff. ANN Backpropagation : Weight updates for hidden nodes.pages 1–3, 2008. URL http://www.wkiri.com/cs461-w08/Lectures/Lec4/ANN-deriv.pdf.

Page 56: Classifying Glyphs - NTNU

REFERENCES 42

D. Whitley, T. Starkweather, and C. Bogart. Genetic algorithms and neuralnetworks: optimizing connections and connectivity. Parallel Computing, 14(3):347–361, August 1990. ISSN 01678191. doi: 10.1016/0167-8191(90)90086-O.URL http://linkinghub.elsevier.com/retrieve/pii/016781919090086O.

B. Widrow and M.E. Hoff. Adaptive Switching Circuits. In Proceedings of the1960 IRE WESCON Convention, pages 96–104, New York City, NY, USA,1960. IRE.

Wikibooks. Artificial Neural Networks/Activation Functions - Wik-ibooks, collection of open-content textbooks, 2010. URL http://en.wikibooks.org/w/index.php?title=Artificial_Neural_Networks/Activation_Functions&oldid=1942655.

WorldLingo. Arabic alphabet, 2010a. URL http://www.worldlingo.com/ma/enwiki/en/Arabic_alphabet.

WorldLingo. Coptic alphabet, 2010b. URL http://www.worldlingo.com/ma/enwiki/en/Coptic_alphabet.

WorldLingo. Han unification, 2010c. URL http://www.worldlingo.com/ma/enwiki/en/Han_unification.

WorldLingo. Hangul, 2010d. URL http://www.worldlingo.com/ma/enwiki/en/Hangul.

WorldLingo. Hebrew alphabet, 2010e. URL http://www.worldlingo.com/ma/enwiki/en/Hebrew_alphabet.

WorldLingo. Egyptian hieroglyphs, 2010f. URL http://www.worldlingo.com/ma/enwiki/en/Egyptian_hieroglyphs.

WorldLingo. Hiragana, 2010g. URL http://www.worldlingo.com/ma/enwiki/en/Hiragana.

WorldLingo. Katakana, 2010h. URL http://www.worldlingo.com/ma/enwiki/en/Katakana.

WorldLingo. Latin alphabet, 2010i. URL http://www.worldlingo.com/ma/enwiki/en/Latin_alphabet.

WorldLingo. Runic alphabet, 2010j. URL http://www.worldlingo.com/ma/enwiki/en/Runic_alphabet.

WorldLingo. Canadian Aboriginal syllabics, 2010k. URL http://www.worldlingo.com/ma/enwiki/en/Canadian_Aboriginal_syllabics.

WorldLingo. Thai alphabet, 2010l. URL http://www.worldlingo.com/ma/enwiki/en/Thai_alphabet.

WorldLingo. Writing system, 2010m. URL http://www.worldlingo.com/ma/enwiki/en/Writing_system.

WorldLingo. Exclusive or, 2010n. URL http://www.worldlingo.com/ma/enwiki/en/Exclusive_or.

Page 57: Classifying Glyphs - NTNU

Software Library

alexandria. Alexandria. http://www.cliki.net/Alexandria.http://common-lisp.net/~loliveira/tarballs/inofficial/alexandria-2008-07-29.tar.gz, version 2008-07-29.

atdoc. atdoc : Common Lisp documentation generation. http://www.lichteblau.com/atdoc. http://www.lichteblau.com/atdoc/download/atdoc.tar.gz, version 2008.11.30. David Lichteblau.

babel. Babel. http://www.cliki.net/Babel. http://common-lisp.net/project/babel/releases/babel_latest.tar.gz, version 0.3.0.

ch-image. ch-image. http://www.cliki.net/ch-image. http://cyrusharmon.org/static/releases/ch-image_0.4.1.tar.gz.

ch-util. ch-util. http://cyrusharmon.org/static/releases/ch-util_0.3.10.tar.gz.

cl-jpeg. cl-jpeg. http://www.cliki.net/cl-jpeg. http://www.common-lisp.net/project/mcclim/cl-jpeg.tar.gz.

cl-ppcre. CL-PPCRE : Portable perl-compatible regular expressions for CommonLisp. http://www.weitz.de/cl-ppcre. Ubuntu package cl-ppcre, http://weitz.de/files/cl-ppcre.tar.gz, version 2.0.1-2. Edi Weitz.

cl-unicode. CL-UNICODE : A portable unicode library for Common Lisp. http://www.weitz.de/cl-unicode. http://weitz.de/files/cl-ppcre.tar.gz,version 0.1.1. Edi Weitz.

cl-yacc. CL-Yacc. http://www.cliki.net/CL-Yacc. http://www.pps.jussieu.fr/~jch/software/files/cl-yacc-0.3.tar.gz, version 0.3.

clem. clem. http://cyrusharmon.org/static/releases/clem_0.4.1.tar.gz.

closer-mop. Closer to mop. http://www.cliki.net/closer-mop. http://www.common-lisp.net/project/closer/ftp/closer-mop_latest.tar.gz, ver-sion 0.61.

closure-common. closure-common. http://www.cliki.net/closure-common.http://www.common-lisp.net/project/cxml/download/closure-common.tar.gz, version 2008-11-30.

43

Page 58: Classifying Glyphs - NTNU

SOFTWARE LIBRARY 44

closure-html. closure-html. http://www.cliki.net/closure-html. http://www.common-lisp.net/project/closure/download/closure-html.tar.gz,version 2008-11-30.

cxml. CXML. http://www.cliki.net/CXML. http://www.common-lisp.net/project/cxml/download/cxml.tar.gz, version 2008-11-30.

cxml-stp. cxml-stp. http://www.cliki.net/cxml-stp. http://www.lichteblau.com/cxml-stp/download/cxml-stp.tar.gz, version 2008-11-30.

flexi-streams. flexi-streams. http://www.cliki.net/Flexi-streams. http://weitz.de/files/flexi-streams.tar.gz, version 1.0.7.

pango. Pango. http://www.pango.org. Ubuntu packages gir1.0-pango1.0,libpango1.0-0, libpango1.0-0-dbg, libpango1.0-common,libpango1.0-dev, libpango1.0-doc, libsdl-pango-dev, libsdl-pango1,version 1.28.0.

parse-number. Parse-number. http://www.cliki.net/PARSE-NUMBER. http://www.common-lisp.net/project/asdf-packaging/parse-number-latest.tar.gz, version 1.0.

plexippus-xpath. plexippus-xpath. http://www.cliki.net/plexippus-xpath.http://www.common-lisp.net/project/plexippus-xpath/download/plexippus-xpath.tar.gz, version 2008-12-07.

puri. Puri (Portable Universal Resource Identifier). http://www.cliki.net/Puri. http://files.b9.com/puri/puri-latest.tar.gz, version 1.5.5.

Salza2. Salza2. http://www.cliki.net/Salza2.

sbcl. SBCL (Steel Bank Common Lisp). http://www.sbcl.org. Ubuntu packagesbcl, version 1.0.29.11.debian.

split-sequence. Split-sequence. http://www.cliki.net/SPLIT-SEQUENCE. http://ftp.linux.org.uk/pub/lisp/experimental/cclan/split-sequence.tar.gz.

trivial-features. trivial-features. http://www.cliki.net/trivial-features.http://common-lisp.net/~loliveira/tarballs/trivial-features/trivial-features_latest.tar.gz, version 0.5.

trivial-gray-streams. trivial-gray-streams. http://www.cliki.net/trivial-gray-streams. http://common-lisp.net/project/cl-plus-ssl/download/trivial-gray-streams.tar.gz, version 2008-11-02.

xuriella. Xuriella. http://www.cliki.net/xuriella. http://www.common-lisp.net/project/xuriella/download/xuriella.tar.gz, version2009-04-04.

ZPNG. Zpng. http://www.cliki.net/ZPNG. http://www.xach.com/lisp/zpng.tgz.

Page 59: Classifying Glyphs - NTNU

APPENDIX A

Unicode Charts

Below are examples of glyphs from the ten writing systems used. All the glyphsare not present, as there are over 600 pages in total. This should, however, beenough to get the characteristics of each writing system.

The Unicode tables are copied from the Unicode charts (The Unicode Con-sortium, 2009), with the blessing of the Unicode Terms of Use (The UnicodeConsortium, 2009, http://www.unicode.org/copyright.html):

Any person is hereby authorized, without fee, to view, use, reproduce,and distribute all documents and files solely for informational purposesin the creation of products supporting the Unicode Standard, subjectto the Terms and Conditions herein.

A-1

Page 60: Classifying Glyphs - NTNU

APPENDIX A. UNICODE CHARTS A-2

Figure A.1: Coptic

Page 61: Classifying Glyphs - NTNU

A-3

Figure A.2: Runic

Page 62: Classifying Glyphs - NTNU

APPENDIX A. UNICODE CHARTS A-4

Figure A.3: Hebrew

Page 63: Classifying Glyphs - NTNU

A-5

Figure A.4: Arabic

Page 64: Classifying Glyphs - NTNU

APPENDIX A. UNICODE CHARTS A-6

Figure A.5: Hangul jamo

Page 65: Classifying Glyphs - NTNU

A-7

Figure A.6: Hiragana

Page 66: Classifying Glyphs - NTNU

APPENDIX A. UNICODE CHARTS A-8

Figure A.7: Thai

Page 67: Classifying Glyphs - NTNU

A-9

Figure A.8: Canadian Aboriginal syllabics

Page 68: Classifying Glyphs - NTNU

APPENDIX A. UNICODE CHARTS A-10

Figure A.9: Han unification

Page 69: Classifying Glyphs - NTNU

A-11

Figure A.10: Egyptian hieroglyphs

Page 70: Classifying Glyphs - NTNU

APPENDIX A. UNICODE CHARTS A-12

Page 71: Classifying Glyphs - NTNU

APPENDIX B

Classification

This appendix contains the training and testing graphs for all the runs. Af-ter the pair classification follows the full classification, with both learning andevolution.

B.1 Pair Classification

The pair classification was only performed with learning. The training plotsare collected per writing system, while the test results are plottet for pair.

B.1.1 Training

0

0.2

0.4

0.6

0.8

1

0 50 100 150 200 250 300

Erro

r

Iteration

Coptic

RunicHebrewArabic

Hangul jamoHiragana

ThaiCanadian Aboriginal syllabics

Han unificationEgyptian hieroglyphs

B-1

Page 72: Classifying Glyphs - NTNU

APPENDIX B. CLASSIFICATION B-2

0

0.2

0.4

0.6

0.8

1

0 50 100 150 200 250 300

Erro

r

Iteration

Runic

CopticHebrewArabic

Hangul jamoHiragana

ThaiCanadian Aboriginal syllabics

Han unificationEgyptian hieroglyphs

0

0.2

0.4

0.6

0.8

1

0 50 100 150 200 250 300

Erro

r

Iteration

Hebrew

CopticRunic

ArabicHangul jamo

HiraganaThai

Canadian Aboriginal syllabicsHan unification

Egyptian hieroglyphs

Page 73: Classifying Glyphs - NTNU

B-3 B.1. PAIR CLASSIFICATION

0

0.2

0.4

0.6

0.8

1

0 50 100 150 200 250 300

Erro

r

Iteration

Arabic

CopticRunic

HebrewHangul jamo

HiraganaThai

Canadian Aboriginal syllabicsHan unification

Egyptian hieroglyphs

0

0.2

0.4

0.6

0.8

1

0 50 100 150 200 250 300

Erro

r

Iteration

Hangul jamo

CopticRunic

HebrewArabic

HiraganaThai

Canadian Aboriginal syllabicsHan unification

Egyptian hieroglyphs

Page 74: Classifying Glyphs - NTNU

APPENDIX B. CLASSIFICATION B-4

0

0.2

0.4

0.6

0.8

1

0 50 100 150 200 250 300

Erro

r

Iteration

Hiragana

CopticRunic

HebrewArabic

Hangul jamoThai

Canadian Aboriginal syllabicsHan unification

Egyptian hieroglyphs

0

0.2

0.4

0.6

0.8

1

0 50 100 150 200 250 300

Erro

r

Iteration

Thai

CopticRunic

HebrewArabic

Hangul jamoHiragana

Canadian Aboriginal syllabicsHan unification

Egyptian hieroglyphs

Page 75: Classifying Glyphs - NTNU

B-5 B.1. PAIR CLASSIFICATION

0

0.2

0.4

0.6

0.8

1

0 50 100 150 200 250 300

Erro

r

Iteration

Canadian Aboriginal syllabics

CopticRunic

HebrewArabic

Hangul jamoHiragana

ThaiHan unification

Egyptian hieroglyphs

0

0.2

0.4

0.6

0.8

1

0 50 100 150 200 250 300

Erro

r

Iteration

Han unification

CopticRunic

HebrewArabic

Hangul jamoHiragana

ThaiCanadian Aboriginal syllabics

Egyptian hieroglyphs

Page 76: Classifying Glyphs - NTNU

APPENDIX B. CLASSIFICATION B-6

0

0.2

0.4

0.6

0.8

1

0 50 100 150 200 250 300

Erro

r

Iteration

Egyptian hieroglyphs

CopticRunic

HebrewArabic

Hangul jamoHiragana

ThaiCanadian Aboriginal syllabics

Han unification

B.1.2 Testing

0

1

2

3

4

5

6

Coptic

Runic

Cla

ssifi

edas

Writing system

Coptic, Runic

Coptic Runic

Page 77: Classifying Glyphs - NTNU

B-7 B.1. PAIR CLASSIFICATION

0

1

2

3

4

5

6

Coptic

Hebrew

Cla

ssifi

edas

Writing system

Coptic, Hebrew

Coptic Hebrew

0

1

2

3

4

5

6

Coptic

Arabic

Cla

ssifi

edas

Writing system

Coptic, Arabic

Coptic Arabic

Page 78: Classifying Glyphs - NTNU

APPENDIX B. CLASSIFICATION B-8

0

1

2

3

4

5

6

Coptic

Hangul

Cla

ssifi

edas

Writing system

Coptic, Hangul jamo

Coptic Hangul jamo

0123456

Coptic

Hiragana

Cla

ssifi

edas

Writing system

Coptic, Hiragana

Coptic Hiragana

Page 79: Classifying Glyphs - NTNU

B-9 B.1. PAIR CLASSIFICATION

0

1

2

3

4

5

6

Coptic

Thai

Cla

ssifi

edas

Writing system

Coptic, Thai

Coptic Thai

0123456

Coptic

Syllabics

Cla

ssifi

edas

Writing system

Coptic, Canadian Aboriginal syllabics

CopticCanadian Aboriginal syllabics

Page 80: Classifying Glyphs - NTNU

APPENDIX B. CLASSIFICATION B-10

0

1

2

3

4

5

6

Coptic

Han

Cla

ssifi

edas

Writing system

Coptic, Han unification

Coptic Han unification

0123456

Coptic

Hieroglyphs

Cla

ssifi

edas

Writing system

Coptic, Egyptian hieroglyphs

CopticEgyptian hieroglyphs

Page 81: Classifying Glyphs - NTNU

B-11 B.1. PAIR CLASSIFICATION

0

1

2

3

4

5

6

Runic

Hebrew

Cla

ssifi

edas

Writing system

Runic, Hebrew

Runic Hebrew

0

1

2

3

4

5

6

Runic

Arabic

Cla

ssifi

edas

Writing system

Runic, Arabic

Runic Arabic

Page 82: Classifying Glyphs - NTNU

APPENDIX B. CLASSIFICATION B-12

0

1

2

3

4

5

6

Runic

Hangul

Cla

ssifi

edas

Writing system

Runic, Hangul jamo

Runic Hangul jamo

0123456

Runic

Hiragana

Cla

ssifi

edas

Writing system

Runic, Hiragana

Runic Hiragana

Page 83: Classifying Glyphs - NTNU

B-13 B.1. PAIR CLASSIFICATION

0

1

2

3

4

5

6

Runic

Thai

Cla

ssifi

edas

Writing system

Runic, Thai

Runic Thai

0123456

Runic

Syllabics

Cla

ssifi

edas

Writing system

Runic, Canadian Aboriginal syllabics

RunicCanadian Aboriginal syllabics

Page 84: Classifying Glyphs - NTNU

APPENDIX B. CLASSIFICATION B-14

0

1

2

3

4

5

6

Runic

Han

Cla

ssifi

edas

Writing system

Runic, Han unification

Runic Han unification

0123456

Runic

Hieroglyphs

Cla

ssifi

edas

Writing system

Runic, Egyptian hieroglyphs

RunicEgyptian hieroglyphs

Page 85: Classifying Glyphs - NTNU

B-15 B.1. PAIR CLASSIFICATION

0

1

2

3

4

5

6

Hebrew

Arabic

Cla

ssifi

edas

Writing system

Hebrew, Arabic

Hebrew Arabic

0

1

2

3

4

5

6

Hebrew

Hangul

Cla

ssifi

edas

Writing system

Hebrew, Hangul jamo

Hebrew Hangul jamo

Page 86: Classifying Glyphs - NTNU

APPENDIX B. CLASSIFICATION B-16

0123456

Hebrew

Hiragana

Cla

ssifi

edas

Writing system

Hebrew, Hiragana

Hebrew Hiragana

0

1

2

3

4

5

6

Hebrew

Thai

Cla

ssifi

edas

Writing system

Hebrew, Thai

Hebrew Thai

Page 87: Classifying Glyphs - NTNU

B-17 B.1. PAIR CLASSIFICATION

0123456

Hebrew

Syllabics

Cla

ssifi

edas

Writing system

Hebrew, Canadian Aboriginal syllabics

HebrewCanadian Aboriginal syllabics

0

1

2

3

4

5

6

Hebrew

Han

Cla

ssifi

edas

Writing system

Hebrew, Han unification

Hebrew Han unification

Page 88: Classifying Glyphs - NTNU

APPENDIX B. CLASSIFICATION B-18

0123456

Hebrew

Hieroglyphs

Cla

ssifi

edas

Writing system

Hebrew, Egyptian hieroglyphs

HebrewEgyptian hieroglyphs

0

1

2

3

4

5

6

Arabic

Hangul

Cla

ssifi

edas

Writing system

Arabic, Hangul jamo

Arabic Hangul jamo

Page 89: Classifying Glyphs - NTNU

B-19 B.1. PAIR CLASSIFICATION

0123456

Arabic

Hiragana

Cla

ssifi

edas

Writing system

Arabic, Hiragana

Arabic Hiragana

0

1

2

3

4

5

6

Arabic

Thai

Cla

ssifi

edas

Writing system

Arabic, Thai

Arabic Thai

Page 90: Classifying Glyphs - NTNU

APPENDIX B. CLASSIFICATION B-20

0123456

Arabic

Syllabics

Cla

ssifi

edas

Writing system

Arabic, Canadian Aboriginal syllabics

ArabicCanadian Aboriginal syllabics

0

1

2

3

4

5

6

Arabic

Han

Cla

ssifi

edas

Writing system

Arabic, Han unification

Arabic Han unification

Page 91: Classifying Glyphs - NTNU

B-21 B.1. PAIR CLASSIFICATION

0123456

Arabic

Hieroglyphs

Cla

ssifi

edas

Writing system

Arabic, Egyptian hieroglyphs

ArabicEgyptian hieroglyphs

0123456

Hangul

Hiragana

Cla

ssifi

edas

Writing system

Hangul jamo, Hiragana

Hangul jamo Hiragana

Page 92: Classifying Glyphs - NTNU

APPENDIX B. CLASSIFICATION B-22

0

1

2

3

4

5

6

Hangul

Thai

Cla

ssifi

edas

Writing system

Hangul jamo, Thai

Hangul jamo Thai

0123456

Hangul

Syllabics

Cla

ssifi

edas

Writing system

Hangul jamo, Canadian Aboriginal syllabics

Hangul jamoCanadian Aboriginal syllabics

Page 93: Classifying Glyphs - NTNU

B-23 B.1. PAIR CLASSIFICATION

0

1

2

3

4

5

6

Hangul

Han

Cla

ssifi

edas

Writing system

Hangul jamo, Han unification

Hangul jamo Han unification

0123456

Hangul

Hieroglyphs

Cla

ssifi

edas

Writing system

Hangul jamo, Egyptian hieroglyphs

Hangul jamoEgyptian hieroglyphs

Page 94: Classifying Glyphs - NTNU

APPENDIX B. CLASSIFICATION B-24

0123456

Hiragana

Thai

Cla

ssifi

edas

Writing system

Hiragana, Thai

Hiragana Thai

0123456

Hiragana

Syllabics

Cla

ssifi

edas

Writing system

Hiragana, Canadian Aboriginal syllabics

HiraganaCanadian Aboriginal syllabics

Page 95: Classifying Glyphs - NTNU

B-25 B.1. PAIR CLASSIFICATION

0123456

Hiragana

Han

Cla

ssifi

edas

Writing system

Hiragana, Han unification

Hiragana Han unification

0123456

Hiragana

Hieroglyphs

Cla

ssifi

edas

Writing system

Hiragana, Egyptian hieroglyphs

HiraganaEgyptian hieroglyphs

Page 96: Classifying Glyphs - NTNU

APPENDIX B. CLASSIFICATION B-26

0123456

Thai

Syllabics

Cla

ssifi

edas

Writing system

Thai, Canadian Aboriginal syllabics

ThaiCanadian Aboriginal syllabics

0

1

2

3

4

5

6

Thai

Han

Cla

ssifi

edas

Writing system

Thai, Han unification

Thai Han unification

Page 97: Classifying Glyphs - NTNU

B-27 B.1. PAIR CLASSIFICATION

0123456

Thai

Hieroglyphs

Cla

ssifi

edas

Writing system

Thai, Egyptian hieroglyphs

ThaiEgyptian hieroglyphs

0123456

Syllabics

Han

Cla

ssifi

edas

Writing system

Canadian Aboriginal syllabics, Han unification

Canadian Aboriginal syllabicsHan unification

Page 98: Classifying Glyphs - NTNU

APPENDIX B. CLASSIFICATION B-28

0123456

Syllabics

Hieroglyphs

Cla

ssifi

edas

Writing system

Canadian Aboriginal syllabics, Egyptian hieroglyphs

Canadian Aboriginal syllabicsEgyptian hieroglyphs

0123456

Han

Hieroglyphs

Cla

ssifi

edas

Writing system

Han unification, Egyptian hieroglyphs

Han unificationEgyptian hieroglyphs

Page 99: Classifying Glyphs - NTNU

B-29 B.2. FULL CLASSIFICATION

B.2 Full Classification

Here follows the graphs from the full classification, with all ten writing systems.This was done with both learning and evolution. There are three graphs for eachmethod; first the training graph displaying the error, then a statistical graphdisplaying the success rate, and finally the classification results.

B.2.1 Learning

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 500 1000 1500 2000 2500

Erro

r

Iteration

Training

Error

020406080

100

0 2 4 6 8 10 12 14 16 18 20

Cor

rect

lycl

assifi

edgl

yphs

[%]

Run

Testing

Success rate

Page 100: Classifying Glyphs - NTNU

APPENDIX B. CLASSIFICATION B-30

0

1

2

3

4

5

6

Coptic

Runic

Hebrew

Arabic

Hangul

Hiragana

Thai

Syllabics

Han

Hieroglyphs

Cla

ssifi

edas

Writing system

Testing

CopticRunic

HebrewArabicHangul

HiraganaThai

Canadian Aboriginal syllabicsHan

Egyptian hieroglyphs

Page 101: Classifying Glyphs - NTNU

B-31 B.2. FULL CLASSIFICATION

B.2.2 Evolution

0.80.810.820.830.840.850.860.870.880.890.9

0.91

0 20 40 60 80 100 120140160 18020000.0020.0040.0060.0080.010.0120.014

Fitn

ess

Stan

dard

devi

atio

nGeneration

Training

Minimal fitnessMaximal fitnessAverage fitness

Average standard deviation

020406080

100

0 0.5 1 1.5 2 2.5 3

Cor

rect

lycl

assifi

edgl

yphs

[%]

Run

Testing

Success rate

Page 102: Classifying Glyphs - NTNU

APPENDIX B. CLASSIFICATION B-32

0

1

2

3

4

5

6

Coptic

Runic

Hebrew

Arabic

Hangul

Hiragana

Thai

Syllabics

Han

Hieroglyphs

Cla

ssifi

edas

Writing system

Testing

CopticRunic

HebrewArabicHangul

HiraganaThai

Canadian Aboriginal syllabicsHan

Egyptian hieroglyphs

Page 103: Classifying Glyphs - NTNU

APPENDIX C

Data Set Distribution

The chosen writing systems have a different amount of glyphs. As can beseen in Table 3.1 on page 17, their Unicode point range varies from 95 (Hebrewand Thai) to 20943 (Han). It can be noted that the actual numbers are not thishigh. Many code points do not yet have a glyph, and are thus removed. Thevowel marks in the abjads are also removed, as they cover only a few pixels inthe final image. They would give the abjads an unfair advantage, as they all lookvery much the same. The available glyphs thus range from 32 (Hebrew) to 20940(Han) (Table C.1).

Table C.1: The actual number of glyphs in the writing systems, after removal of the emptycode points.

Script Number of glyphsCoptic 111Runic 90Hebrew 32Arabic 190Hangul 254Hiragana 93Thai 87Canadian Aboriginal syllabics 363Han 20940Egyptian hieroglyphs 1071

There are, however, not this many glyphs that actually can be used. For someglyphs, something has gone wrong either when drawing the glyph or when readingit, resulting in no usable image. These must be removed during runtime. Alas, itis hard to predict precisely how many glyphs there are available in each writingsystem. Either way, there is such a large difference that some writing systemswill have an advantage compared to others. Which writing systems depend onhow the glyphs are selected. Three different approaches were tested.Approach 1 One solution is to use more glyphs in one writing system than in

another. This is a bit problematic, as the ANN will be better trained on thelarger writing systems than on the smaller. This will thus give the smaller

C-1

Page 104: Classifying Glyphs - NTNU

APPENDIX C. DATA SET DISTRIBUTION C-2

writing systems a disadvantage.Approach 2 One could also use the same number of glyphs for all writing sys-

tems, and allow duplicates. Then, the smaller writing systems would havemultiple instances of the same glyphs. Those systems will have a huge ad-vantage, as they probably will have the same glyphs in both the training setand the test set.

Approach 3 One can use the same number of glyphs from every writing system,but without duplicates. Alas, one cannot use more glyphs than the smallestwriting system has available. There are probably many important aspectsof the larger systems that are lost in the unchosen glyphs. This approachwill thus give the larger writing systems a disadvantage.

Table C.2: Comparison of Hebrew, Hangul, Thai and Han with different combinations ofnumber of glyphs. All runs used a learning rate η = 0.02 and used 80% of the glyphs fortraining, the rest for testing.

Approach Writingsystems

Number ofglyphs

Correctlyclassified

Total correctlyclassified

Number ofiterations

1

Hebrew 30 0/6

88.24% 31Hangul 250 41/50Thai 80 0/16Han 1000 199/200

2

Hebrew 300 53/60

64.17% 16Hangul 300 11/60Thai 300 60/60Han 300 30/60

3

Hebrew 30 5/6

58.33% 158Hangul 30 1/6Thai 30 3/6Han 30 5/6

The outputs of the runs are summarized in Figures C.1 to C.3. This test wasrun with an earlier version of the learning algorithm than what was used for thereal classification (Appendix B). Alas, the graphs function a bit different. Theoutput in this case is a number between 1 and -1, not 1 and 0. The optimal resultwill thus have the correct writing system far above zero, and the three other farbelow zero.

At first glance, the first approach seems to be the best, with 88% correct.However, the predicted advantage for the larger writing systems has occurred.While Han — with its 1000 glyphs — has 199 of 200 correct, Hebrew and Thai— which had far fewer glyphs available — had none of their glyphs correctlyclassified (Table C.2, Figure C.1).

This approach still did good on total, because Han was almost completelycorrectly classified, and there were more testing glyphs of Han than of the otherthree in total. Adding on the rather good Hangul gives 250 glyphs that has ahigh probability of being correct, of a total of 272. Alas, since Hebrew and Thaihave so few glyphs involved, their classification is disastrous. But since they haveso few glyphs, the disaster is not noticeable in the grand scheme.

This is clearly an approach that is best for the larger writing systems. It is

Page 105: Classifying Glyphs - NTNU

C-3

020406080

100120140160180200

Hebrew

Hangul

Thai

Han

Cla

ssifi

edas

Writing system

Approach 1

Hebrew Hangul Thai Han

Figure C.1: Comparison of what the different writing systems were classified as, using Approach1; as many glyphs from each writing system as that writing system has.

also a good solution if one only care about the percentage of correctly classifiedglyphs. If the system actually is to be used, however, this is not the way to go.

The second approach does also a rather good job, with 64% (Table C.2). Ascan be seen in Figure C.2, the result is not as biased as that of Approach 1.However, it is clear that the small writing systems (Hebrew and Thai) are farbetter classified than the larger. This is quite natural; with 300 glyphs, eachHebrew glyph is used roughly 10 times.. As a result, the ANN can train againand again on the same Hebraic glyphs. The main problem is that the testingglyphs of Hebrew must necessarily be present also in the training glyphs, toavoid only being able to test on one or two different glyphs. Then, the ANN isnot generalizing over the Hebraic glyphs as it does over the Han glyphs; the ANNhas actually seen the Hebraic glyphs before. Then it naturally does a better jobclassifying them.

Even though Hebrew and Thai get an unfair advantage, the solution does seemto work. It is far from perfect, but almost all of the writing systems are classifiedmore often as itself than of any of the others. Hangul is more often classified asThai than of Hangul, but just a bit. It is an even greater problem in the lastapproach (Figure C.2).

Approach 3 is good at classifying both Hebrew and Han, and is thus not biasedtowards smaller or larger writing systems (Figure C.3). It is, however, not verygood at separating Hangul and Thai. The percentage is not that impressive(Table C.2), but it is the most unbiased, as it performs well on both small andlarge writing systems.

Approach 3 — to use only 30 glyphs from each of the writing systems — is theoverall best solution. This is the most unbiased approach, and thus the fairest.

Page 106: Classifying Glyphs - NTNU

APPENDIX C. DATA SET DISTRIBUTION C-4

-505

101520253035

Hebrew

Hangul

Thai

Han

Cla

ssifi

edas

Writing system

Approach 2

Hebrew Hangul Thai Han

Figure C.2: Comparison of what the different writing systems were classified as, using Approach2; as many glyphs as possible, with repeating glyphs if necessary.

-4-3-2-101234

Hebrew

Hangul

Thai

Han

Cla

ssifi

edas

Writing system

Approach 3

Hebrew Hangul Thai Han

Figure C.3: Comparison of what the different writing systems were classified as, using Approach3; as many glyphs as possible without repetition and with the same amount from all writingsystems.

Page 107: Classifying Glyphs - NTNU

APPENDIX D

Pixel Comparison

To get a better grasp on how the system interprets the writing systems, apixel comparison test was performed. The writing systems were compared toeach other, glyph by glyph, based on their pixel values. That is, based on theirinput values to the neural network. Two glyphs are compared by going throughthe image, pixel for pixel, testing the two pixels for similarity. The total similarityis then the average number of pixels that are similar. Precisely what it means tobe similar, is the difference between the comparisons below.

The colours in the comparison tables are the grey-scale of LaTEX, as a functionof the comparison values. Hence, a lighter colour means a higher value, thushigher similarity.

As can be seen in Table D.1 on page D-2, the similarity is not that high, noteven for the same writing systems. Even though a glyph have a similarity of 1.0compared to itself, the writing systems have a lower similarity value because ofthe differences between the glyphs of that writing system.

There are some interesting similarities (Table D.2 on page D-2). For example,the Egyptian hieroglyphs have a very low similarity with all the writing systems,itself included. Thai, on the other hand, has a clear similarity with itself. Thesame goes for Hebrew, Hangul, hiragana, Canadian Aboriginal syllabics and Han,but not as much.

It is also clear that Hangul, hiragana, Canadian Aboriginal syllabics and Hanare quite similar. There are four lighter areas where these writing systems meet.Thai is also not that far from. It creates a darker cross in the middle, but it isstill far lighter than the other writing systems. Hebrew is not far from either, butis more similar to Thai than the others.

Coptic, runic, Arabic and hieroglyphs can be seen as darker lines, with lowsimilarity to the rest. However, while Coptic and runic has acceptably highsimilarity with itself, Arabic and hieroglyphs have very low internal similarity.

Tables D.1 and D.2 do not, however, show the whole truth. As the pixels arecompared on a same-value basis, the pixel values 253 and 254 are just as differentas 0 and 255. More tests were thus executed.

Tables D.3 to D.5 compare the pixels by first sending them through Richards’curve, or the generalized sigmoid function (BioSS). The time of maximal growth

D-1

Page 108: Classifying Glyphs - NTNU

APPENDIX D. PIXEL COMPARISON D-2

is 128, 200 and 250, respectively.Tables D.6 to D.9 compare the pixels by comparing the difference in their

values. The glyphs are considered similar if the difference is less than 40, 20, 10,and 5, respectively.

Considering all the tables, we see some patterns. An expected pattern is thatthere is a higher general similarity in the cases with a more forgiving similarityfunction.

Another pattern is how different Arabic is from all the other writing systems.This is the case in all of the tables. Another of these black lines can be seen inall the tables. This one, however, is moving from Han to the hieroglyphs. In thecases with strict similarity the hieroglyphs are different from the rest, while Hanis different in the cases with a more forgiving similarity function.

The crossing line where the writing systems are compared to itself can also beseen as more or less lighter than the rest.

Table D.1: Similarity between the glyphs, compared by pixel values.

Cop

tic

Run

ic

Heb

rew

Ara

bic

Han

gul

Hir

agan

a

Tha

i

Sylla

bics

Han

Hie

rogl

yphs

Coptic 0.221 0.180 0.212 0.182 0.230 0.228 0.212 0.221 0.210 0.154Runic 0.180 0.217 0.198 0.180 0.215 0.209 0.194 0.198 0.192 0.154

Hebrew 0.212 0.198 0.285 0.209 0.248 0.247 0.261 0.223 0.210 0.177Arabic 0.182 0.180 0.209 0.195 0.216 0.212 0.210 0.199 0.188 0.156Hangul 0.230 0.215 0.248 0.216 0.313 0.291 0.251 0.268 0.262 0.172

Hiragana 0.228 0.209 0.247 0.212 0.291 0.313 0.260 0.262 0.261 0.176Thai 0.212 0.194 0.261 0.210 0.251 0.260 0.368 0.230 0.214 0.180

Syllabics 0.221 0.198 0.223 0.199 0.268 0.262 0.230 0.259 0.241 0.162Han 0.210 0.192 0.210 0.188 0.262 0.261 0.214 0.241 0.259 0.155

Hieroglyphs 0.154 0.154 0.177 0.156 0.172 0.176 0.180 0.162 0.155 0.155

Table D.2: Similarity between the glyphs, compared by pixel values. Light cells have a higherlevel of similarity than darker cells.

Cop

tic

Run

ic

Heb

rew

Ara

bic

Han

gul

Hir

agan

a

Tha

i

Sylla

bics

Han

Hie

rogl

yphs

CopticRunic

HebrewArabicHangul

HiraganaThai

SyllabicsHan

Hieroglyphs

Page 109: Classifying Glyphs - NTNU

D-3

Table D.3: Comparison of Richard’s curve (BioSS) as a function of pixel value, with 128 astime of maximum growth. This was the function used during learning (Section 3.3).

Cop

tic

Run

ic

Heb

rew

Ara

bic

Han

gul

Hir

agan

a

Tha

i

Sylla

bics

Han

Hie

rogl

yphs

CopticRunic

HebrewArabicHangul

HiraganaThai

SyllabicsHan

Hieroglyphs

Table D.4: Comparison of Richard’s curve (BioSS) as a function of pixel value, with 200 astime of maximum growth.

Cop

tic

Run

ic

Heb

rew

Ara

bic

Han

gul

Hir

agan

a

Tha

i

Sylla

bics

Han

Hie

rogl

yphs

CopticRunic

HebrewArabicHangul

HiraganaThai

SyllabicsHan

Hieroglyphs

Table D.5: Comparison of Richard’s curve (BioSS) as a function of pixel value, with 250 astime of maximum growth.

Cop

tic

Run

ic

Heb

rew

Ara

bic

Han

gul

Hir

agan

a

Tha

i

Sylla

bics

Han

Hie

rogl

yphs

CopticRunic

HebrewArabicHangul

HiraganaThai

SyllabicsHan

Hieroglyphs

Page 110: Classifying Glyphs - NTNU

APPENDIX D. PIXEL COMPARISON D-4

Table D.6: Comparison where the pixels are considered similar if the difference between themis less than 40.

Cop

tic

Run

ic

Heb

rew

Ara

bic

Han

gul

Hir

agan

a

Tha

i

Sylla

bics

Han

Hie

rogl

yphs

CopticRunic

HebrewArabicHangul

HiraganaThai

SyllabicsHan

Hieroglyphs

Table D.7: Comparison where the pixels are considered similar if the difference between themis less than 20.

Cop

tic

Run

ic

Heb

rew

Ara

bic

Han

gul

Hir

agan

a

Tha

i

Sylla

bics

Han

Hie

rogl

yphs

CopticRunic

HebrewArabicHangul

HiraganaThai

SyllabicsHan

Hieroglyphs

Table D.8: Comparison where the pixels are considered similar if the difference between themis less than 10.

Cop

tic

Run

ic

Heb

rew

Ara

bic

Han

gul

Hir

agan

a

Tha

i

Sylla

bics

Han

Hie

rogl

yphs

CopticRunic

HebrewArabicHangul

HiraganaThai

SyllabicsHan

Hieroglyphs

Page 111: Classifying Glyphs - NTNU

D-5

Table D.9: Comparison where the pixels are considered similar if the difference between themis less than 5.

Cop

tic

Run

ic

Heb

rew

Ara

bic

Han

gul

Hir

agan

a

Tha

i

Sylla

bics

Han

Hie

rogl

yphs

CopticRunic

HebrewArabicHangul

HiraganaThai

SyllabicsHan

Hieroglyphs