Proceedings of Recent Advances in Natural Language Processing, pages 711–719,Varna, Bulgaria, Sep 2–4, 2019.

https://doi.org/10.26615/978-954-452-056-4_083

711

Inforex — a Collaborative Systemfor Text Corpora Annotation and Analysis Goes Open

Michał Marcinczuk and Marcin OleksyG4.19 Research Group

Department of Computational IntelligenceFaculty of Computer Science and Management

Wrocław University of Science and Technology, Wrocław, Poland{michal.marcinczuk,marcin.oleksy}@pwr.edu.pl

Abstract

In the paper we present the latest changesintroduce to Inforex — a web-basedsystem for qualitative and collaborativetext corpora annotation and analy-sis. One of the most important newsis the release of source codes. Nowthe system is available on the GitHubrepository (https://github.com/CLARIN-PL/Inforex) as an opensource project. The system can be easilysetup and run in a Docker containerwhat simplifies the installation process.The major improvements include: semi-automatic text annotation, multilingualtext preprocessing using CLARIN-PL webservices, morphological tagging of XMLdocuments, improved editor for annota-tion attribute, batch annotation attributeeditor, morphological disambiguation,extended word sense annotation. Thispaper contains a brief description ofthe mentioned improvements. We alsopresent two use cases in which variousInforex features were used and tested inreal-life projects.

1 Introduction

Development and evaluation of tools for variousnatural language processing task (named entityrecognition, sentiment analysis, cyberbully detec-tion and many other) require dedicated resourcesin a form of manually or semi-automatically an-notated corpora. Corpus-based studies in the do-main of Digital Humanities also require a supportin the form of specialized tools and system. Bothcreate a demand on development of tools and sys-tems qualitative text corpora management, anno-tation, analysis and visualization.

Inforex is one of several web-based systemsfor text corpora annotation which is being devel-oped as an open source project. The other well-known systems include, but are not limited to, We-bAnno 3.0 (de Castilho et al., 2016), Brat (Stene-torp et al., 2012) and Anafora (Chen and Styler,2013). Comparing to the other systems Inforexhas some distinct features, including: support foruntokenized and tokenized documents, support forboth plain text and XML documents (XML tagsare used to format the document layout) and in-tegration with CLARIN-PL web services (utilizeson-demand morphological tagging).

In Section 2 we present the basic characteristicof the Inforex system. In Section the 3 we presentthe recents improvements and new features imple-mented in the system. In the Section 4 we presenttwo projects in which the latest features were uti-lized.

2 Inforex Overview

Inforex is a web-based system for text corporamanagement, annotation and analysis. Since 2018it is available as an open source project on theGitHub repository and is a part of the PolishCLARIN infrastructure1 — it is integrated withthe official repository for language resources inPolish CLARIN2. From the user perspective In-forex requires only a modern web browser to usethe system.

Inforex offers several features for collabora-tive work on a single corpus, including concur-rent access to data stored in the central database,role-based access to different modules, flag-basedmechanism to track the process of various typesof tasks. It support text cleanup, mention anno-tation, relations between annotations, morpholog-

1https://inforex.clarin-pl.eu2https://clarin-pl.eu/dspace/

712

ical tagging, annotation attributes, metadata andmany others. A more comprehensive list of func-tions can be found in (Marcinczuk et al., 2017).

3 Recent Changes and Improvements

3.1 Open Source Project

After 10 years of development the project has beenfinally released as an open source project. Thesource codes are available under the LGPL licenseand can be obtained from https://github.com/CLARIN-PL/Inforex.

3.2 Easy Installation

The installation process was simplified by convert-ing the system and all required components to runwithing a set of Docker containers3 defined in aCompose file. The Compose4 file defines fourcontainers: (1) www — web server running theInforex application with background services (seeSection 3.3), (2) db — MySQL database server,(3) liquibase — Liquibase database schema con-trol and (4) phpmyadmin — web-based accessto the database (for development and maintenancepurposes).

A new installation of Inforex boils down to run-ning the following two lines of code:

sudo apt-get install composer \docker docker-compose

./docker-dev-up.sh

3.3 Background Processes

Time consuming tasks, like corpus export or mor-phological tagging, are handled by processes run-ning in the background. That’s how we avoidthe web server timeouts and handle task queuing.The background processes have been added to theDocker container with web server and they are au-tomatically run on the container startup.

3.4 Multilingual Morphological Tagging

Inforex uses CLARIN-PL Web Service API5

(Walkowiak, 2018) to facilitate the on-demandmorphological document tagging. CLARIN-PLWS API provides access to morphological taggersfor 11 languages. Seven of them are available fromInforex, i.e. Polish, English, German, Russian,Hebrew, Czech and Bulgarian (see Figure 1). In-forex automatically choose the language specific

3https://www.docker.com/4https://docs.docker.com/compose/5http://ws.clarin-pl.eu/tagerml.shtml

tagger based on the document language set in themetadata.

3.5 Extended Annotation Attribute EditorWe have extended the annotation attribute editorto handle dictionary-based attributes with a largenumber of possible values (see Figure 2). The im-provements include the following:

• Filtering of the list of values,

• Feature to add a new element to the dictio-nary directly from the value picker level.

• Suggestions based on values assigned toother annotations. We have implemented twoheuristic of generating the candidates withdifferent levels of certainty, i.e.:

– values for other annotations matched bythe text form with the Soundex algo-rithm6. The list of candidates is sortedby their frequency,

– attribute values matched by the annota-tion text (full or partial matching).

Figure 2: Extended annotation attribute editor

3.6 Batch Annotation Attribute EditorUp to now the modification of annotation at-tributes was available only from the Annotatorperspective using the annotation editor (see Fig-ure 2). When an user had to modify an attribute foreach annotation the only way was to go through allthe annotations one by one. This process was time

6https://www.archives.gov/research/census/soundex.html

713

Figure 1: Document tokenization perspective

consuming and error-prone because it was easy tomiss some annotations. To overcame these prob-lems we have created a page for batch annotationattribute modification. The page consists of twomain components, i.e. a document content withannotation preview and a table with annotationswith their attribute (see Figure 3).

3.7 Document Auto Annotation

This feature was designed to reduce user effort inannotating repeatable phrases in and across doc-uments. Auto annotation works by annotating ingiven documents all phrases that were already an-notated in other documents. This feature works forboth untokenized and tokenized documents, how-ever we advise to use it on tokenized documentsas the phrases are aligned with token boundariesand we avoid matching of incomplete words. Af-ter running auto annotation the new annotationsare presented to the user for verification. User candecide whether given annotation is correct, incor-rect or the annotation type needs a change (seeFigure 4). The discarded annotation are stored inthe system for future run of auto annotation. Dur-ing the next use of auto annotation the new an-notations which were previously discarded are ig-nored.

3.8 Lemma and Attribute Auto Fill

These features were designed to reduce user effortin setting annotation lemmas and attribute values.They both works in a similar way — for each an-notation in the document the lemma or attributevalue is set based on other annotations in the cor-pus. For lemma we collect annotations with the

same text form and category. For attribute valuewe collect annotations with the same text form orlemma and category. In case of ambiguity, i.e.there are more than one possible value of lemmaor attribute, the value remain empty and the userhas to fill it manually. The lemma auto fill featureis available in the Annotation lemmas perspectiveand the attribute auto fill feature is available in theAnnotation attributes perspective.

3.9 Tokenization of XML documents

Inforex allows to store documents in one of thetwo formats: plain text or XML. The XML formatis used to encode document structure, like in theKPWr (Broda et al., 2012) and PCSN (Marcinczuket al., 2011) corpora. During tagging the XMLtags should be ignored and only the content shouldbe processed. Thus, we made the tokenization pro-cess to be aware of the document format (see Fig-ure 1). For XML format the document content iscleaned from XML tags, than the content is pro-cessed by the tagging service and at the end thetokenization is aligned with the original XML doc-ument.

3.10 Annotation Attribute Browser

The attribute value browser (see Figure 5) allowsto browse corpus annotations by given attributevalue. The page consists of three elements:

• View configuration — provides a set of fil-ters, including: shared attribute, documentlanguage and subcorpus,

• Attribute values assigned to annotations —list of values their frequency,

714

Figure 3: Batch annotation attribute editor

Figure 4: Auto annotation and candidate verification perspective

• Annotations with the selected value.

3.11 Export of Morphological Tagging andAnnotations Agreements

For tokenized document Inforex can store up tothree layers of morphological tags:

• Tagger — tags produced by a tool,

• Agreement — tags entered by a user in theagreement mode,

• Final — tags approved by the super user.

During export it is possible to define whichlayer of tags should be exported. It is possible tochoose one of the following options (see Figure 6):

• Final or tagger (if final not present) — exportthe final tags. For tokens which does not havethe final tag a tagger tag is taken.

• Final — export only the final tags. If thereare tokens without final tags than the missingtags are reported as errors.

• User (agreement) — export tags created byselected user. For tokens which does not con-tain user agreement tags the tagger tags aretaken.

• Tagger — export tagger tags.

3.12 Improved Support for Word SenseAnnotation

The existing mechanism for word sense annota-tion was limited to a single set of words and theirsenses (Marcinczuk et al., 2012). We have re-moved the limitation and allow to define and useany number of sets of word senses in the WSD

715

Figure 5: Annotation attribute browser

Figure 6: Export configuration dialog window

perspective. We also added the option to anno-tate the word senses in an agreement mode for fur-ther agreement (see Figure 7). Finally, we haveimported all lexical units with their senses fromSłowosiec 3.2 (Piasecki et al., 2016) as an annota-tion set to Inforex.

3.13 Morphological Agreement

The last feature is support for morphological dis-ambiguation agreement. Inforex provides a pagewith morphological tag agreement across a givenset of documents (see Figure 8). The agreement ispresented in a numerical form for each documentsin the set and after a specific document a list ofdisagreements is presented. This feature is com-plemented by a document perspective for compar-ing and choosing the final morphological tags (seeFigure 9).

4 Case Studies

In this section we present two uses cases in whichvarious features of Inforex were used in real-lifeprojects.

4.1 BSNLP 2019 Shared TaskInforex was used to create the training and testingdatasets for the need of 2nd Edition of the SharedTask on Multilingual Named Entity Recognitionfor Slavic languages7. The task aims at recogniz-ing mentions of named entities in news articles inSlavic languages, their lemmatization, and cross-language matching.

More than 10 people were involved in the an-notation process for four languages, i.e. Polish,Czech, Russian and Bulgarian. There were 1–3annotators per language. The annotation processconsists of four main steps:

1. Selection of relevant documents — the doc-ument were automatically crawled and up-loaded to Inforex, therefore some of themwere duplicates or text not relevant to the sub-corpus topic. The selected documents weremarked with a flag Valid content.

2. Annotation of named entity mentions — thesame set of five annotatation types was usedto annotated all the selected documents. ForPolish and Czech the annotators utilized theauto annotate feature described in Section 3.7and we were able to evaluate the usability of

7http://bsnlp.cs.helsinki.fi/shared_task.html

716

Figure 7: The extended perspective for word sense annotation

the auto annotation feature. Table 1 containsevaluation of the automatically recognizedand added annotations for two languages andtwo subcorpora (each on a different topic).The auto annotation feature yielded very highprecision of 97-99% with relatively high re-call of 66-82%. This means that in case ofPolish and Czech subcorpora 10k out of 14kannotations were added automatically. It wasa significant facilitation of the work.

3. Assignment of annotation lemmas — to as-sign lemmas the annotators utilized batchlemma editor and lemma auto fill. For thecorrectly added annotations the lemmas werealso automatically assigned.

4. Assignment of cross-lingual identifier — thegoal was to assign the same identifier for eachmention across all languages referring to thesame real-world entity. There were morethan 4k identifiers. The annotators utilizedthe auto fill features described in Section 3.8.The attribute browser was used to validate theentity mentions.

4.2 Polish Translation of the NTUMultilingual Corpus

An ongoing project which goal is to provide Pol-ish translation of the NTU Multilingual Corpus(Tan and Bond, 2011) which consists of two sto-ries from the Sherlock Holmes Canon (The Ad-venture of the Speckled Band and The Adventureof the Dancing Men. The Adventure of the Speck-led Band is the first one translated and prepared for

Language Polish CzechSubcorpus A B A BTotal 5139 2440 4183 2504Final 5032 2386 4128 2502Discarded 107 53 55 2Add by user 1015 648 696 829Precision [%] 97.4 97.0 98.4 99.9Recall [%] 79.9 72.8 83.1 66.9

Table 1: Evaluation of the auto annotation featureon the BSNLP 2019 Shared Task dataset

manual annotation. The text was divided into 31samples (txt files) of a similar size and importeddirectly into Inforex system. Then automatic mor-phological tagging was performed using WCRFTmorpho-syntactic tagger for Polish (Radziszewski,2013). The tagger provided morphological dis-ambiguation on the basis of its context but alsoother possible forms for this particular word werelisted. The result of the automatic annotation wasthen verified by two linguists independently. Theywere able to see morphological analysis of eachtoken and the decision of the tagger (see Fig. 9).It could be accepted or discarded by the humanannotator. It was also possible to add and as-sign an interpretation which was not identified bythe tagger (e.g. in the case of unknown words).Inter-annotator agreement was calculated and itslevel was high enough (0,97) to perform further.Then, after completion the manual verification ofmorphological tagging by both linguists team co-ordinator proceeded with inconsistencies analysis.The decision was made for every token differently

717

Figure 8: Summary of morphological disambiguation agreement

annotated. All tags verified by the team leader ob-tained the status of final annotations. They wereadded to the version published within CLARIN-PL infrastructure (Błaszczak et al., 2019).

Figure 10: Morphological information providedfor human annotators

After the Inforex functionality was developed

and primarily used for creation of The Adventureof the Speckled Band corpus, it was successfullyapplied to prepare Corpus of the colloquial Pol-ish language (Oleksy, 2019) in another project.This corpus has been designed to address the prob-lem of morphological tagging of user-generatedcontent (UGC) as part of the project ”SentiCog-nitiveServices — next generation service for au-tomating voice of customer and social media sup-port based on artificial intelligence methods”8.The whole corpus (approximately 400000 tokens)is manually annotated with morphological infor-mation and furthermore the sample of 100 docu-ments was prepared as a result of 2+1 annotation.

5 Summary

The last two years have been productive in thedevelopment of the Inforex system. Many newfeatures and extensions were implemented duringthat time and the most important were presented inthis paper. Majority of the features and improve-ments were dedicated by users. The most impor-tant news is the that Inforex has been finally re-leased as an open source project.

Work financed as part of the investment in theCLARIN-PL research infrastructure funded by the

8https://sentione.com/knowledge/eu-research-project

718

Figure 9: The perspective for morphological disambiguation agreement

Polish Ministry of Science and Higher Education.

ReferencesMarta Błaszczak, Kacper Paszke, Ewa Rudnicka,

Marcin Oleksy, Jan Wieczorek, Wioleta Kobylinska,Dominika Fikus, and Dagmara Kałkus. 2019.The adventure of the speckled band 1.0 (man-ually tagged). CLARIN-PL digital repository.http://hdl.handle.net/11321/667.

Bartosz Broda, Michał Marcinczuk, Marek Maziarz,Adam Radziszewski, and Adam Wardynski. 2012.KPWr: Towards a Free Corpus of Polish. In Nico-letta Calzolari, Khalid Choukri, Thierry Declerck,Mehmet Ugur Dogan, Bente Maegaard, Joseph Mar-iani, Jan Odijk, and Stelios Piperidis, editors, Pro-ceedings of LREC’12. ELRA, Istanbul, Turkey.

Wei-Te Chen and Will Styler. 2013. Anafora:A web-based general purpose annotationtool. In Proceedings of the 2013 NAACLHLT Demonstration Session. Associationfor Computational Linguistics, pages 14–19.http://www.aclweb.org/anthology/N13-3004.

Richard Eckart de Castilho, Eva Mujdricza-Maydt,Seid Muhie Yimam, Silvana Hartmann, IrynaGurevych, Anette Frank, and Chris Biemann. 2016.A web-based tool for the integrated annotation ofsemantic and syntactic structures. In Proceedingsof the workshop on Language Technology Resources

and Tools for Digital Humanities (LT4DH) at COL-ING 2016. pages 76–84. http://tubiblio.ulb.tu-darmstadt.de/97939/.

Michał Marcinczuk, Marcin Oleksy, and Jan Kocon.2017. Inforex - a collaborative system for text cor-pora annotation and analysis. In Ruslan Mitkovand Galia Angelova, editors, Proceedings of theInternational Conference Recent Advances in Nat-ural Language Processing, RANLP 2017, Varna,Bulgaria, September 2 - 8, 2017. INCOMA Ltd.,pages 473–482. https://doi.org/10.26615/978-954-452-049-6 063.

Michał Marcinczuk, Monika Zasko-Zielinska, and Ma-ciej Piasecki. 2011. Structure annotation in the pol-ish corpus of suicide notes. In Ivan Habernal andVaclav Matousek, editors, Text, Speech and Dia-logue, Springer Berlin Heidelberg, volume 6836 ofLecture Notes in Computer Science, pages 419–426.

Michał Marcinczuk, Jan Kocon, and Bartosz Broda.2012. Inforex – a web-based tool for text corpusmanagement and semantic annotation. In Nico-letta Calzolari (Conference Chair), Khalid Choukri,Thierry Declerck, Mehmet Ugur Dogan, BenteMaegaard, Joseph Mariani, Asuncion Moreno, JanOdijk, and Stelios Piperidis, editors, Proceedingsof the Eight International Conference on LanguageResources and Evaluation (LREC’12). EuropeanLanguage Resources Association (ELRA), Istanbul,Turkey.

719

Marcin Oleksy. 2019. Corpus of the colloquialpolish language. CLARIN-PL digital repository.http://hdl.handle.net/11321/637.

Maciej Piasecki, Stan Szpakowicz, Marek Maziarz,and Ewa Rudnicka. 2016. PlWordNet 3.0 – Al-most There. In Verginica Barbu Mititelu, CorinaForascu, Christiane Fellbaum, and Piek Vossen, edi-tors, Proceedings of the 8th Global Wordnet Confer-ence, Bucharest, 27-30 January 2016. Global Word-net Association, pages 290–299.

Adam Radziszewski. 2013. A tiered crf tagger for pol-ish. In Intelligent Tools for Building a Scientific In-formation Platform.

Pontus Stenetorp, Sampo Pyysalo, Goran Topic,Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsu-jii. 2012. brat: a web-based tool for NLP-assistedtext annotation. In Proceedings of the Demonstra-tions Session at EACL 2012. Association for Com-putational Linguistics, Avignon, France.

Liling Tan and Francis Bond. 2011. Buildingand annotating the linguistically diverse NTU-MC (NTU-multilingual corpus). In Proceed-ings of the 25th Pacific Asia Conference onLanguage, Information and Computation. Insti-tute of Digital Enhancement of Cognitive Process-ing, Waseda University, Singapore, pages 362–371.https://www.aclweb.org/anthology/Y11-1038.

Tomasz Walkowiak. 2018. Language processing mod-elling notation – orchestration of nlp microser-vices. In Wojciech Zamojski, Jacek Mazurkiewicz,Jarosław Sugier, Tomasz Walkowiak, and JanuszKacprzyk, editors, Advances in Dependability Engi-neering of Complex Systems. Springer InternationalPublishing, Cham, pages 464–473.