Proceedings of the Seventeenth SIGMORPHON Workshop on Computational Researchin Phonetics, Phonology, and Morphology, pages 162–170

Online, July 10, 2020. c©2020 Association for Computational Linguisticshttps://doi.org/10.18653/v1/P17

162

Linguist vs. Machine: Rapid Development of Finite-State MorphologicalGrammars

Sarah Beemer and Zak Boston and April Bukoski and Daniel Chen andPrincess Dickens and Andrew Gerlach and Torin Hopkins and

Parth Anand Jawale and Chris Koski and Akanksha Malhotra and Piyush Mishra andSaliha Muradoglu� and Lan Sang and Tyler Short and Sagarika Shreevastava and

Elizabeth Spaulding and Tetsumichi Umada and Beilei Xiang and Changbing Yang andMans Hulden

University of ColoradoThe Australian National University�

Abstract

Sequence-to-sequence models have proven tobe highly successful in learning morphologicalinflection from examples as the series of SIG-MORPHON/CoNLL shared tasks have shown.It is usually assumed, however, that a linguistworking with inflectional examples could inprinciple develop a gold standard-level mor-phological analyzer and generator that wouldsurpass a trained neural network model in ac-curacy of predictions, but that it may requiresignificant amounts of human labor. In this pa-per, we discuss an experiment where a groupof people with some linguistic training de-velop 25+ grammars as part of the shared taskand weigh the cost/benefit ratio of develop-ing grammars by hand. We also present toolsthat can help linguists triage difficult complexmorphophonological phenomena within a lan-guage and hypothesize inflectional class mem-bership. We conclude that a significant devel-opment effort by trained linguists to analyzeand model morphophonological patterns arerequired in order to surpass the accuracy ofneural models.

1 Introduction

Hand-written grammars for modeling derivationaland inflectional morphology have long been seenas the gold standard for incorporating a word in-flection aware component into NLP systems. How-ever, the recent successes of sequence-to-sequence(seq2seq) models in learning morphological pat-terns, as seen in multiple shared tasks that ad-dress the topic (Cotterell et al., 2016, 2017, 2018;McCarthy et al., 2019), have raised the questionwhether there is any advantage in developing hand-written grammars for performance reasons. Thisquestion has special relevance with regard to low-resource languages when there is a desire to quicklydevelop fundamental NLP resources such as a mor-phological analyzer and generator with minimal

bus;N;PL sheep;N;PL

Lexicon (lexc) Guesser

blarg;N;PL

Morphophonological FST cascade

bus+s sheep blarg+s

buses sheep blargs

Figure 1: Basic FST grammar design used in thisproject which combines a lexicon-based model with aguesser to handle unseen lemmas.

resource expenditure (Maxwell and Hughes, 2006).It is clear that there is a need for hand-written

morphological grammars, even if neural networkmodels approach the performance of carefullyhand-crafted morphologies. Normative and pre-scriptive language models, such as those neededby language academies in many countries—e.g.RAE in Spain, Academie Francaise in France, orthe Council of the Cherokee Nation in the U.S.—would need to rely on explicitly designed modelsfor providing guidance in word inflection, spellingrules, and orthography if they were to be imple-mented computationally. Currently, neural modelstrained on examples provide no verifiable guaran-tees that certain prescriptive phenomena have beenlearned by a trained model and can be reliably used.

In this paper1 we document an experiment wherea number of morphological grammars were hand-written by a group of 19 students enrolled in theclass “LING 7565—Computational Phonology &Morphology” at the University of Colorado, each

1All tools and grammars developed are available onhttps://github.com/mhulden/7565tools.

163

Tagalog inflectional strategies

Agent AGFOC

Ptv IPFV LGSPEC1um Rtum R I

hag hag R mas R II

hang hang R Mang R IIIna na R ma R II

Thaha haha R maka Rnag many R many R VInan han R man R VIIPatient PFOC 2also hah

PFV IPFV LGSPEC1 Cepenthetich

in R in R sin IIin an in R an R an I

in R in R I

ni ni R i R IIJhaha an hahaha ah

ni an ni R an R an VIi in i R in i R VIin R in i r VIIIan R in an R ah IIIni ni R R in Fni ni R R Xi

Figure 2: Old-school pencil-and-paper Linguistics: hy-pothesizing the possible inflectional patterns for Taga-log Actor Focus and Object Focus verb forms (fromproject notes). The symbol R represents reduplicationof the first CV(V) in the stem.

student having training in either computer sci-ence or linguistics, and some previous training inwriting finite-state morphological grammars. Thelanguages were chosen from the 2020 SIGMOR-PHON shared task 0 (Vylomova et al., 2020), andthe grammars were designed so as to be able toinflect unseen forms. The design was also such thatthe grammars were able to function as “guessers”and inflect lexemes never seen in the training data.

2 Finite-State Grammars

Finite-state Transducer (FST) solutions have longbeen the foremost paradigm in which to developlinguistically informed large-scale morphologicalgrammars (Koskenniemi, 1983; Beesley and Kart-tunen, 2003; Hulden, 2009). The availability of avariety of tools (Hulden (2009); Riley et al. (2009);Beesley (2012) inter alia) has also supported thismode of development, and by now hundreds of lan-

guages have grammars developed by linguists inthis paradigm.

The usual approach to developing morphologicalanalyzers is to model the mapping from a lemma(citation form) and a morphosyntactic description(MSD) into an inflected form (target form) as a two-step process. The first step maps the lemma+MSDinto an intermediate form that represents a combina-tion of canonical morpheme representations, whilethe second step employs a cascade of transducerswhich handle morphophonological alternations. Itis customary to handle inflectional classes by ex-plicitly dividing lemmas into groups in the firststep so that correct morphemes are chosen for eachlemma. Analyzers built in such a way generally arenot capable of inflecting lemmas that are not explic-itly encoded in a lexicon. However, it is commonto integrate an additional “guesser” component thatcan handle any valid lemma in a language, and passit through the relevant morphophonological com-ponent only (Beesley and Karttunen, 2003). Basicfinite-state calculus is then used to construct a sin-gle FST that “overrides” outputs from the guesserwhenever a known lexeme is inflected, so conflict-ing outputs are avoided. The basic design is illus-trated in Figure 1.

3 Approach

All of the grammars were built with the foma finite-state tool (Hulden, 2009). Before grammar writingcommenced, the participants were urged to spendroughly 1 hour in groups of 3 to quickly analyzeall the languages in the development and surprisegroups as follows:

• Triage: the training sets for all languagesin the shared task were rapidly analyzed fordifficulty, and possible complex inflectionalclasses. Following this, a selection of lan-guages were chosen by the participants tomodel. This was done once for the devel-opment languages, and through an additionalround of triage for the surprise languages.

• Each language was scored for difficultybased on familiarity with the writing system,paradigm size, complexity, and the apparentnumber of inflectional classes; naturally theactual number was not known, and this repre-sented an educated guess. Participants wereasked to informally rate the difficulty of a lan-guage on a 1(easy)–5(very difficult) before

164

choosing languages to work on. The partici-pants were not explicitly instructed to pick aneasy language, but rather, to choose one thatwould provide an interesting experience andwould be feasible to complete.2

• Computational tools (discussed below) wereused to reconstruct the partial paradigms givenin the training data, to extract the alphabetsused in the languages, to canonicalize the Uni-Morph tag order (Kirov et al., 2018) used inthe data, and to provide a rapid developmentenvironment that could give instant feedbackon accuracy on the training and dev sets aftercompilation of FSTs.

• A template grammar was used as a startingpoint; it provided both the possibility of de-veloping a morphophonology-only grammar,or a grammar where all lemmas needed to bedivided into inflectional classes.

Through the above process, a number of lan-guages were selected as the primary targets, anddevelopment was launched for some 40 languagesin total—roughly 20 for the development languagesand a similar number for the surprise languages, asthey were published. In the end, the output of 25languages was submitted to the shared task. Thecriterion for actually submitting a language wasthat the grammar was mature enough, judged byexamining whether accuracy on the developmentset was within 5% of the neural baseline models(Wu et al., 2020) provided by the organizers.

4 Tools

As mentioned above, a number of tools for the sup-port of rapid grammar writing were also developed.These included the tools to reconstruct the partialinflection tables from the data and various analysistools for accuracy and error reporting.

Apart from that, a separate tool for inflectiontable clustering and a non-neural tool for hypoth-esizing forms for missing slots in paradigms werealso developed. This latter tools’ output was alsosubmitted as a second system (CU-7565-02) to theshared task for nearly all languages. These twotools were more involved and are discussed in de-tail below.

2On average, the surprise languages were deemed consid-erably more difficult, largely because of paradigm size.

4.1 Inflection Table ClusteringCrucial in the development of a grammar from raw,partial inflection table data is the ability to hypothe-size if lexemes fall into different inflectional classesquickly, and if so, how. This is non-trivial to de-termine, especially with large amounts of lexemesrepresented in the various data sets. It is also es-sential to disentangle phonological regularity frominflectional classes which may be significant redherrings in the analysis of a language. For example,while cat in English pluralizes as cats, bus plural-izes as buses—by an epenthetic e inserted betweensibilants. A naive analysis would postulate thatthe two lexemes behave differently and place themin separate inflectional classes, although a prop-erly designed phonological component could avoidthis unnecessary complexity in the morphologicalcomponent.

4.1.1 Lexeme similarity measureTo facilitate providing a linguist with a quickoverview, we developed a model to perform rapidhierarchical clustering of all lexemes in a lan-guage’s data set. To this end, we developed a metricfor lexeme similarity with respect to inflectionalbehavior. This metric is calculated by a two-stepprocess. First, all pairs of word forms for a lexeme(within a paradigm) are aligned using an out-of-the-box Monte Carlo aligner (Cotterell et al., 2016)written by the last author. This is shown in figure3 (a). Following this alignment procedure, we au-tomatically produce a crude approximation of thestring transformation implied by the alignment asa regular expression, which is then compiled intoan FST.

In the conversion process, matching input se-quences in the alignment are modeled by ?+ (re-peat one or more symbols3) and non-matching sym-bols are replaced by the symbol-pair found in thealignment: i:o. For example, the aligned pair runs↔ ran in Figure 3 (b) is converted into the regularexpression

?+ u:a ?+ s:0 (1)

which can be compiled into a transducer in Figure3 (c). This transducer generalizes over the matchedelements in the input-output pair and can be ap-plied to other third-person present forms, such asoutruns to produce outran. Obviously, this exam-ple transformation only applies to this particular

3We use foma regular expression notation.

165

inflectional class and will give incorrect transfor-mations such as pulls→ pall for words that do nothave the same inflectional behavior. The purposeof calculating all-known-pairs mappings for eachlexeme is to provide a similarity measure betweenlexemes. In particular, we use the following mea-sure for two lexemes l1 and l2, which compares theoverlap of all transformation rules found betweenthe forms in l1 with the transformation rules in l2:

sim(l1, l2) =2×#shared(l1, l2)

#shared(l1, l1) + #shared(l2, l2)(2)

Here, #shared(l1, l2) is the simple count reflect-ing how many of the slot-to-slot transformationrules in l1 are identical for l2.

We subsequently convert this similarity scoreinto a distance for the purposes of clustering:

distance(l1, l2) = 1− sim(l1, l2) (3)

Note that the denominator in the similarity cal-culation in effect expresses the maximum possiblesimilarity scores for l1 and l2 by calculating the sim-ilarity with themselves, resulting in a range of [0, 1]for the overall similarity and distance measures.Since many given paradigms contain missing formsand are therefore missing pair-transformations aswell, this maximum score will vary from lexeme tolexeme.

With this similarity in hand between all lexemes,we can perform a (single-link) agglomerative hi-erarchical clustering of all lexemes in the trainingdata of a language.

Example results of the clustering are shown inFigure 4 for Ingrian (the full training set which con-tained partial inflectional tables for 50 lexemes),and English (a small subset). Included in the In-grian clustering are our final linguist-hypothesizedinflectional class numbers for each lexeme for com-parison.

4.2 Inflection with transformation FSTsAs a byproduct of the clustering distance measurethat uses slot-to-slot transformation FSTs, we canalso address the shared task itself. Since the de-velopment and test sets largely contain unknowninflections from lexemes where some forms havebeen seen, we can make use of the learned trans-formation rules from other lexemes that target anunknown form asked for in the development or

run

ran

running runs

?+ 0:s

?+ u:a ?+ s:0

?+ {ning}:s

?+ 0:{ning}

?+ u:a ?+ {ning}:0 ?+ u:a ?+

run ran

run0000 running

run0 runs

running runs000

running ran0000

runs ran0

(a)

(b)

0 1@ s a u

@ s a u

2<u:a> 3@ s a u

@ s a u

4<s:0>

(c)

Figure 3: Generating transformation rules for each pair-wise slot for a lexeme: (a) we perform alignment of allpairs, (b) a regular expression is issued to model thetransformation which is compiled into an FST (c).

test sets. To this end, we collect all known source→ target transformation rules from all other tableswhere the target form is the desired slot (MSD). Wethen apply all of these transformations, generatingpotentially hundreds of inflection candidates forthe missing target slot of a lexeme. From amongthe candidates, we perform a majority vote. Forall languages, we experimented with weightingthe majority vote so that transformation rules thatcome from paradigms that share many transforma-tion rules with the target lexeme’s paradigm get amultiplier for the vote using the similarity measurein (2). This strategy produced slightly superior re-sults throughout, as analyzed by performance onthe development set, and was hence used in thefinal submission for our system CU-7565-02.

5 Results

The results for the hand-written grammars (CU-7565-01) and the non-neural paradigm completionmodel (CU-7565-02) are given in Table 1. We notethat we were able to match or surpass the strongestneural participant in the task on 13 languages withthe hand-written grammars. Several of these, how-

166

Ingrian English

Figure 4: Hierarchical clustering of lexemes by apparent inflectional behavior based on string transformationsbetween inflectional slots for Ingrian (left) and English (right). The numbers in parentheses in Ingrian refer to theLinguist-derived inflectional class number after developing a grammar. The Ingrian data is the output from the fulltraining data while the English is a small selection of verbs to illustrate clustering behavior.

ever, were relatively “easy” languages and oftendid not contain any significant morphophonologyat all. On two languages, Ingrian (izh) and Taga-log (tgl), we were able to significantly improveupon the other models participating in the task.These languages had a fairly large number of in-flectional classes and very complex morphophonol-ogy. Ingrian features a large variety of consonantgradation patterns common in Uralic languages,and Tagalog features intricate reduplication pat-terns (see Figure 2).

We include results for train, dev, and test as weused tools to continuously evaluate our progressduring development on the training set. It is worthnoting that the linguist-driven development processdoes not seem to be prone to overfitting—accuracyfor several languages on the test set was actuallyhigher than on the training set.

The non-neural paradigm completion model(CU-7565-02), which was submitted for nearlyall 90 languages performed reasonably well, andis to our knowledge the best-performing non-neural model available for morphological inflection.Never outperforming the strongest neural models;it nevertheless represents a strong improvementover the baseline non-neural model provided by theorganizers. Additionally, it provides another toolto quickly see reasonable hypotheses for missingforms in inflection tables.

6 Discussion

6.1 Earlier work

To our knowledge, no extensive comparison be-tween well-designed manual grammars and neural

Language trn1 dev1 tst1 tst2

aka 100.0 100.0 100.0 89.8ceb 85.2 86.2 86.5 84.7crh 97.5 97.0 96.4 97.7czn 79.0 76.0 72.5 76.1dje 100.0 100.0 100.0 100.0gaa 100.0 100.0 100.0 100.0izh 93.4 91.1 92.9 77.2kon 100.0 100.0 98.7 97.4lin 100.0 100.0 100.0 100.0mao 85.5 85.7 66.7 57.1mlg 100.0 100.0 100.0 -nya 100.0 100.0 100.0 100.0ood 81.0 87.5 71.0 62.4orm 99.6 100.0 99.0 93.6ote 91.2 93.5 90.9 91.3san 88.5 89.7 89.0 88.3sna 100.0 100.0 100.0 99.3sot 100.0 100.0 100.0 99.0swa 100.0 100.0 100.0 100.0syc 89.3 87.3 88.3 89.1tgk 100.0 100.0 93.8 93.8tgl 77.9 75.0 77.8 -xty 81.1 80.0 81.7 70.3zpv 84.3 77.9 78.9 81.1zul 82.9 88.1 83.3 88.5

Table 1: Results for the train, dev, and test sets with ourhandwritten grammars (1) and our non-neural learner(2). The non-neural model also participated in addi-tional languages not shown here. Languages with ac-curacies on par with or exceeding the best shared taskparticipants are shown in boldface.

167

network models for morphology have been pro-posed. Pirinen (2019) reports on a small exper-iment that compares an earlier SIGMORPHONshared task winner’s results to a Finnish hand-written morphological analyzer (Pirinen, 2015),with the seq2seq-based participant’s model yield-ing higher precision than the rule-based FST ana-lyzer. In another related experiment, Moeller et al.(2018) train neural seq2seq models from an exist-ing hand-designed transducer acting as an oracleand note that the seq2seq model begins to convergeto the FST with around 30,000 examples in a verycomplex language, Arapaho (arp).

The non-neural inflection model (CU-7565-02)builds upon paradigm generalization work by Fors-berg and Hulden (2016), which in turn is anextension of Hulden et al. (2014) and Ahlberget al. (2015). An earlier non-neural model forparadigm generalization is found in Dreyer andEisner (2011).

6.2 Human ResourcesWe did not record the exact amounts of time spenton the project individually for each participant.However, we can estimate this based on previousyears’ class surveys in the same course (LING7565—Computational Phonology and Morphol-ogy) as regards the number of hours per week stu-dents spend working on course projects. Each stu-dent on average in the course spends 6.6 hours perweek; as the project ran for 5 weeks with 19 partic-ipants, we roughly estimate a total of 627 person-hours spent on the task of developing grammars.As reflected in the results, we considered 13–15languages to have largely completed grammars, orvery nearly completed. The remainder of the 25languages submitted were known to require furtherwork, but very little work to reach accuracies be-yond or at the best-performing neural models forthe task. These estimates do not include studenttraining in morphology, finite-state machines, andgrammar writing. Likewise, some languages withvery large number of forms per lexeme—such asErzya (myv) with 1,597 forms and Meadow Mari(mhr) with 1,597 forms—were deemed outside therealm of realistic analysis and linguist-driven gram-mar writing within a scope of 5 weeks that wereallotted to the work.

6.3 Neural or Human?Given the above estimates, we can provide a conser-vative estimate of at least 40 person-hours of work

on average—not counting infrastructure develop-ment and strategizing—to develop a hand-writtenmorphological analyzer and generator that is onpar with a model learned by state-of-the-art neuralapproaches. There is large variance around thisfigure, however, as some very regular languagesonly required 30 minutes of work and a dozen-or-so lines of code to produce a model that captures allthe morphology and morphophonology involved.Others required a much greater and more intenseeffort in analyzing the partial inflection tables givenin the training data, classifying lemmas into inflec-tional classes and modeling morphophonologicalrules as FSTs. Additionally, we note that all theparticipants had already been trained in this kindof analysis and grammar writing, a factor that ourestimate does not take into account.

6.4 Language Notes

In the course of the development of the grammars,we observed that many languages had a skewedselection of data, or inconsistencies that would notbe fruitful to model in a hand-written grammar.This also meant that in such cases it was unlikelythat the hand-written grammar would ever attain theperformance of a neural model, which can betterhandle the inconsistencies described below. Wehope to be able to clean up the data as the test datais released to re-evaluate our grammars for theselanguages, without this additional noise.

Maori (mao) is an example of a language wherethe given data set provides a hard ceiling on howmuch can be inferred either by a linguist or amachine learning model. The data provided con-tains only maximally two forms for each verb—the active and the passive. Some examples ofactive-passive alternations include: neke ∼ neke-hia, nehu ∼ nehua, kati ∼ katia. In this dataset, the passive form is utterly unpredictable fromthe active form (but not vice versa). The standardphonological analysis of the data (Kiparsky, 1982;Harlow, 2007)—familiar to many from phonologytextbooks—is that the underlying stem containsa consonant which is removed by a phonologicalrule that deletes word-final consonants in the lan-guage. The traditional phonological analysis is thatthe lemma listed as neke, for example, is underly-ingly /nekeh/, and the passive suffix is regularly-ia, while the active suffix is the zero morpheme-0. The consonant-deletion rule applies to the ac-tive form, which surfaces as neke, but not to the

168

MacGyvering abominating rendering V.PTCP;PRS ? abominated rendered V.PTCP;PST - - - V;NFIN MacGyvers abominates renders V;SG;3;PRS

MacGyver abominate render

Candidates for ?: [MacGyvered, MacGyverd, MacGyvered, MacGyvered]

1

1

2

2 3 4

3

4

Figure 5: Generating candidate inflections for V.PTCP.PST for the verb “to MacGyver”. We use all the can-didates generated by known transformation rules from all other tables (only 2 other tables shown here). A list ofcandidate inflections is produced, where the final inflection is decided by majority vote.

passive form nekehia, where the added suffix pre-vents the consonant from deleting. There is also anadditional hiatus-avoiding rule—deleting a vowel—seen in e.g. /nehu/+/ia/→ nehua. Obviously, theconsonant which is not seen in the active formgiven in the training data can not be used to pre-dict the passive form. The best one can do is toguess the most likely consonant in the language asbeing present in the underlying stem. Had the train-ing data contained a third form which maintainsthe consonant—e.g. the Maori gerundive suffix/-aNa/—the missing consonant of the passive couldbe predicted from the gerundive and vice versa.4

Hiligaynon (hil) contained several lemmas listedwith multiple alternate forms, such as:

bati/batian/pamatian ginpamantian V;PROG;PST

It is very challenging to account for the occasionallemma being listed in two or three parts in a stan-dard FST design, and so this kind of transformationwas not attempted.

Syriac, Sanskrit, Oromo, Tohono O’odham(syc,san,orm,ood) contained multiple lines wherethe lemma and MSD were identical, but the out-put was not. In some languages this was pervasiveenough to cause us to exclude them (ctp,pei) fromour selection of attempted languages.

Chichicapan Zapotec (zpv) contained severalinflected forms where the target form actuallycontained two alternatives separated by a slash.Predicting and modeling when this happens wasdeemed to be irregular and was not attempted.

4“If we wanted an A on our [phonology] exam, we wouldof course say the underlying forms are [the ones with theconsonant] . . . If someone were to say that the underlyingforms are [consonantless] he’d flunk.” (Kiparsky, 1982)

Zenzontepec Chatino (czn) contained a mixtureof hyphens (-) and en-dashes (–) where presumablyonly one of them should have been used. Again,this was deemed hard to predict manually and noobvious pattern was found.

7 Conclusion

We have done a preliminary investigation in pit-ting neural inflection models against more tradi-tional hand-written grammars, designed by non-naive grammar developers with some training inthe field of linguistics and computational modeling.The results point to two main directions.

First, it is very difficult in many cases to outper-form a state-of-the-art neural network model with-out significant development effort and attentionto nuanced morphophonological patterns. Indeed,some data sets in the task were very simple, andin such cases, it is quite trivial to develop a high-accuracy grammar. This advantage is somewhatnullified by the apparent ability of neural seq2seqmodels to also model such morphologies with highaccuracy, despite little data.

The second observation is the following: for lan-guages where the group was able to significantlyoutperform neural models (such as Tagalog and In-grian), success did not come cheaply. We estimatethat for any language with high morphophonologi-cal complexity and a variety of inflectional classes,possibly hundreds of hours of development effortis required even by a trained linguist to surpass theperformance of a current state-of-the-art seq2seqmodel. But it is also precisely in this latter caseof high-complexity languages where linguists canstill prevail with a margin.

169

ReferencesMalin Ahlberg, Markus Forsberg, and Mans Hulden.

2015. Paradigm classification in supervised learn-ing of morphology. In Proceedings of the 2015 Con-ference of the North American Chapter of the Asso-ciation for Computational Linguistics: Human Lan-guage Technologies, pages 1024–1029, Denver, Col-orado. Association for Computational Linguistics.

Kenneth R. Beesley. 2012. Kleene, a free and open-source language for finite-state programming. InProceedings of the 10th International Workshop onFinite State Methods and Natural Language Process-ing, pages 50–54, Donostia–San Sebastian. Associa-tion for Computational Linguistics.

Kenneth R. Beesley and Lauri Karttunen. 2003. Fi-nite State Morphology. CSLI Publications, Stanford,CA.

Ryan Cotterell, Christo Kirov, John Sylak-Glassman,Geraldine Walther, Ekaterina Vylomova, Arya D.McCarthy, Katharina Kann, Sebastian Mielke, Gar-rett Nicolai, Miikka Silfverberg, David Yarowsky,Jason Eisner, and Mans Hulden. 2018. The CoNLL–SIGMORPHON 2018 shared task: Universal mor-phological reinflection. In Proceedings of theCoNLL–SIGMORPHON 2018 Shared Task: Univer-sal Morphological Reinflection, pages 1–27, Brus-sels. Association for Computational Linguistics.

Ryan Cotterell, Christo Kirov, John Sylak-Glassman,Geraldine Walther, Ekaterina Vylomova, PatrickXia, Manaal Faruqui, Sandra Kubler, DavidYarowsky, Jason Eisner, and Mans Hulden. 2017.CoNLL-SIGMORPHON 2017 shared task: Univer-sal morphological reinflection in 52 languages. InProceedings of the CoNLL SIGMORPHON 2017Shared Task: Universal Morphological Reinflection,pages 1–30, Vancouver. Association for Computa-tional Linguistics.

Ryan Cotterell, Christo Kirov, John Sylak-Glassman,David Yarowsky, Jason Eisner, and Mans Hulden.2016. The SIGMORPHON 2016 shared Task—Morphological reinflection. In Proceedings of the14th SIGMORPHON Workshop on ComputationalResearch in Phonetics, Phonology, and Morphol-ogy, pages 10–22, Berlin, Germany. Association forComputational Linguistics.

Markus Dreyer and Jason Eisner. 2011. Discoveringmorphological paradigms from plain text using aDirichlet process mixture model. In Proceedingsof the 2011 Conference on Empirical Methods inNatural Language Processing, pages 616–627, Edin-burgh, Scotland, UK. Association for ComputationalLinguistics.

Markus Forsberg and Mans Hulden. 2016. Learn-ing transducer models for morphological analysisfrom example inflections. In Proceedings of theSIGFSM Workshop on Statistical NLP and WeightedAutomata, pages 42–50, Berlin, Germany. Associa-tion for Computational Linguistics.

Ray Harlow. 2007. Maori: A Linguistic Introduction.Cambridge University Press.

Mans Hulden. 2009. Foma: a finite-state compiler andlibrary. In Proceedings of the Demonstrations Ses-sion at EACL 2009, pages 29–32, Athens, Greece.Association for Computational Linguistics.

Mans Hulden, Markus Forsberg, and Malin Ahlberg.2014. Semi-supervised learning of morphologicalparadigms and lexicons. In Proceedings of the 14thConference of the European Chapter of the Asso-ciation for Computational Linguistics, pages 569–578, Gothenburg, Sweden. Association for Compu-tational Linguistics.

Paul Kiparsky. 1982. Explanation in Phonology, vol-ume 4. Walter de Gruyter.

Christo Kirov, Ryan Cotterell, John Sylak-Glassman,Geraldine Walther, Ekaterina Vylomova, PatrickXia, Manaal Faruqui, Sebastian Mielke, Arya Mc-Carthy, Sandra Kubler, David Yarowsky, Jason Eis-ner, and Mans Hulden. 2018. UniMorph 2.0: Uni-versal morphology. In Proceedings of the EleventhInternational Conference on Language Resourcesand Evaluation (LREC-2018), Miyazaki, Japan. Eu-ropean Languages Resources Association (ELRA).

Kimmo Koskenniemi. 1983. Two-level morphology: Ageneral computational model for word-form recog-nition and production. Publication 11, Univer-sity of Helsinki, Department of General Linguistics,Helsinki.

Mike Maxwell and Baden Hughes. 2006. Frontiers inlinguistic annotation for lower-density languages. InProceedings of the Workshop on Frontiers in Lin-guistically Annotated Corpora 2006, pages 29–37,Sydney, Australia. Association for ComputationalLinguistics.

Arya D. McCarthy, Ekaterina Vylomova, Shijie Wu,Chaitanya Malaviya, Lawrence Wolf-Sonkin, Gar-rett Nicolai, Christo Kirov, Miikka Silfverberg, Se-bastian J. Mielke, Jeffrey Heinz, Ryan Cotterell, andMans Hulden. 2019. The SIGMORPHON 2019shared task: Morphological analysis in context andcross-lingual transfer for inflection. In Proceedingsof the 16th Workshop on Computational Research inPhonetics, Phonology, and Morphology, pages 229–244, Florence, Italy. Association for ComputationalLinguistics.

Sarah Moeller, Ghazaleh Kazeminejad, Andrew Cow-ell, and Mans Hulden. 2018. A neural morphologi-cal analyzer for Arapaho verbs learned from a finitestate transducer. In Proceedings of the Workshopon Computational Modeling of Polysynthetic Lan-guages, pages 12–20, Santa Fe, New Mexico, USA.Association for Computational Linguistics.

Tommi A Pirinen. 2015. Omorfi — free and opensource morphological lexical database for Finnish.In Proceedings of the 20th Nordic Conference

170

of Computational Linguistics (NODALIDA 2015),pages 313–315, Vilnius, Lithuania. Linkoping Uni-versity Electronic Press, Sweden.

Tommi A Pirinen. 2019. Neural and rule-based FinnishNLP models—expectations, experiments and expe-riences. In Proceedings of the Fifth InternationalWorkshop on Computational Linguistics for UralicLanguages, pages 104–114, Tartu, Estonia. Associa-tion for Computational Linguistics.

Michael Riley, Cyril Allauzen, and Martin Jansche.2009. OpenFst: An open-source, weighted finite-state transducer library and its applications to speechand language. In Proceedings of Human LanguageTechnologies: The 2009 Annual Conference of theNorth American Chapter of the Association for Com-putational Linguistics, Companion Volume: TutorialAbstracts, pages 9–10, Boulder, Colorado. Associa-tion for Computational Linguistics.

Ekaterina Vylomova, Jennifer White, ElizabethSalesky, Sabrina J. Mielke, Shijie Wu, EdoardoPonti, Rowan Hall Maudslay, Ran Zmigrod, JosephValvoda, Svetlana Toldova, Francis Tyers, ElenaKlyachko, Ilya Yegorov, Natalia Krizhanovsky,Paula Czarnowska, Irene Nikkarinen, AndrejKrizhanovsky, Tiago Pimentel, Lucas TorrobaHennigen, Christo Kirov, Garrett Nicolai, AdinaWilliams, Antonios Anastasopoulos, Hilaria Cruz,Eleanor Chodroff, Ryan Cotterell, Miikka Silfver-berg, and Mans Hulden. 2020. The SIGMORPHON2020 Shared Task 0: Typologically diverse mor-phological inflection. In Proceedings of the 17thWorkshop on Computational Research in Phonetics,Phonology, and Morphology.

Shijie Wu, Ryan Cotterell, and Mans Hulden. 2020.Applying the transformer to character-level transduc-tion. arXiv:2005.10213 [cs.CL].