Extracting Clinical Findingsfrom Swedish Health Record Text

Maria Skeppstedt

Doctoral ThesisDepartment of Computer and Systems SciencesStockholm UniversityDecember, 2014

Stockholm UniversityReport seriesc© 2014 Maria Skeppstedt

Typeset by the author using LATEXPrinted in Sweden by US-AB

i

Abstract

Information contained in the free text of health records is useful for the immediatecare of patients as well as for medical knowledge creation. Advances in clinicallanguage processing have made it possible to automatically extract this informa-tion, but most research has, until recently, been conducted on clinical text writtenin English. In this thesis, however, information extraction from Swedish clinicalcorpora is explored, particularly focusing on the extraction of clinical findings.Unlike most previous studies, Clinical Finding was divided into the two moregranular sub-categories Finding (symptom/result of a medical examination) andDisorder (condition with an underlying pathological process).

For detecting clinical findings mentioned in Swedish health record text, a ma-chine learning model, trained on a corpus of manually annotated text, achievedresults in line with the obtained inter-annotator agreement figures. The machinelearning approach clearly outperformed an approach based on vocabulary map-ping, showing that Swedish medical vocabularies are not extensive enough for thepurpose of high-quality information extraction from clinical text. A rule and cuevocabulary-based approach was, however, successful for negation and uncertaintyclassification of detected clinical findings.

Methods for facilitating expansion of medical vocabulary resources are partic-ularly important for Swedish and other languages with less extensive vocabularyresources. The possibility of using distributional semantics, in the form of Randomindexing, for semi-automatic vocabulary expansion of medical vocabularies was,therefore, evaluated. Distributional semantics does not require that terms or abbre-viations are explicitly defined in the text, and it is, thereby, a method suitable forclinical corpora. Random indexing was shown useful for extending vocabularieswith medical terms, as well as for extracting medical synonyms and abbreviationdictionaries.

ii

Sammanfattning

Den information som dokumenteras i patientjournaler är viktig i vården avpatienter, men informationen kan även användas för att utvinna ny medicinsk kun-skap. Genom att anpassa språkteknologiska verktyg till journaltext är det ävenmöjligt att automatiskt extrahera den information som dokumenteras i löpandetext. Hittills har emellertid språkteknologisk forskning på journaltext mest be-drivits på text skriven på engelska. Ämnet för den här avhandlingen är istäl-let extraktion av information från journaltext skriven på svenska, och framförallt undersöks automatisk extraktion av kliniska fynd. Till skillnad från tidigarestudier, delades Kliniskt Fynd upp i två delkategorier, Fynd (symtom/resultat frånen medicinsk undersökning) och Sjukdom (tillstånd med en underliggande pato-logisk process).

För att hitta kliniska fynd i texten användes maskinlärning, och en modell trä-nad på manuellt annoterad journaltext uppnådde resultat som var fullt jämförbaramed den uppmätta samstämmigheten mellan annoterarna. Att istället användatermlistor från medicinska lexikala resurser för att hitta kliniska fynd i journal-texten var dock mindre framgångsrikt, vilket visar att svenska lexikala resurserbehöver utökas för att kunna användas för att extrahera information ur patient-journaler. Ett system byggt på regler och lexikala resurser lyckades däremot bramed att detektera vilka av de hittade kliniska fynden som var negerade och vilkasom uttrycktes med osäkerhet.

För svenska, och andra språk med begränsade lexikala resurser, är metoder somgör det lätt att utöka befintliga resurser extra viktiga. I denna avhandling utvärd-erades Random indexing, vilket är en metod som bygger på distributionell seman-tik och därför är lämpad för termextraktion från journaltext, där använda termersällan definieras. Random indexing visade sig vara ett användbart stöd för atthalv-automatiskt utöka lexikala resurser med medicinska termer och för att extra-hera medicinska synonymer och förkortningsordlistor.

iii

Acknowledgements

To some extent, it is possible to gain new knowledge on your own by absorbing thecontent of relevant books and articles, and by processing this content in solitude.To gain enough knowledge to earn a doctoral degree, however, this is not enough(at least not for me): you need other people who can help you by guiding you andby sharing their knowledge. I have been very fortunate during my doctoral studiesto be able to do research and to interact with a large number of very intelligentpeople, who have shared their knowledge and, thereby, contributed to the contentsof this doctoral thesis.

I would first like to thank my three supervisors: Hercules for believing in me byaccepting me as his doctoral student and by spurring and advising me to do betterresearch than I thought we would be able to; Gunnar for invaluable advice on howto plan and structure my research and for always being able to increase its qualityby providing a final touch; and Mia, who has formally been my supervisor for onlyjust over a year, but, in reality, has been an informal supervisor from the first weekshe joined our research group, and without whose support and knowledge, muchof the work in this doctoral thesis would have been impossible.

Apart from my supervisors, there are two co-authors of the papers includedin this thesis whom I would like to especially thank: Aron, whose rare abilityof combining intelligent, critical questions with encouraging comments has beena constant source of inspiration and has spurred many of the research ideas ofthis thesis; and Sumithra who as a senior doctoral student and later post doctoralresearcher has given me much valuable advice and support as if she were my forthsupervisor.

I would also like to thank the other co-authors of the papers included in this the-sis: Martin, who has not only shared his knowledge as a senior researcher but alsoas a senior teacher; Hans and Vidas for exchange of knowledge and ideas in theHEXAnord network; and Wendy, Brian and Danielle, not only for their importantcontributions to the paper included in this thesis, but also for their great kindnessto Aron and me at our month long research visit in their group at the Universityof California San Diego. For the same reason, I would like to thank Mike, who isalso a part of the San Diego research group, as well as Araki-sensei, Rafal, Shihoand Jonas, but, in their case for their great kindness during my research visits atHokkaido University.

There are two additional persons whom I am very indebted to for the content ofthis thesis: Panos and Isak, who I had expected would give me many intelligentcomments on my thesis draft as pre-doctoral seminar opponents, but who exceeded

iv

my expectations by far. Many thanks also to the management of Karolinska Uni-versity Hospital for giving us access to the health record texts, and to the technicalstaff at SLL IT and DSV for help with technical issues regarding the data.

The above mentioned research visits, as well as the many conferences, networkmeetings and courses I have participated in, were not only great learning oppor-tunities, but they will also form the fondest memories from my doctoral studies.I would like to thank the persons with whom I shared those experiences, makingthem such great memories: again, the research groups I visited (and Sumithra formaking the San Diego research visit possible); the members of the HEXAnordnetwork and the Dadel project; Martin, Aron and Hideyuki for AIME in Slovenia;my parents Birgitta and Staffan for IJCAI in Barcelona; Hercules and Alyaa forLREC in Istanbul; Eriks and Aron for winter school in Switzerland; and Claudiafor SMBM in Portugal, just to mention a small number. A thanks also to my fellowdoctoral students and colleagues at DSV, for your kind smiles when meeting in thecorridors: they have brightened up the everyday workdays.

I would also like to thank my parents as well as my brother Martin and mysister-in-law Hanna for your support and endless kindness, and also for help withthe content of my research and my teaching. In addition, there are a number ofgreat people whom I would like to call my friends, but whom I have neglectedthese past years, being buried in research or being away travelling. As you arevery important to me, I hope at least some of you have not forgotten my existence.

Finally, there is one paper co-author and travel companion whom I have yetfailed to mention, Magnus, whom I would like to thank not only for sharing in-teresting research and memorable conference experiences and research visits, butalso for sharing my life. Thank you.

v

Table of contents

1 Introduction and general aims 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Problems and general aims . . . . . . . . . . . . . . . . . . . . . 2

2 Included studies, research questions and study alignment 52.1 Included studies . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2 Study alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3 Background 113.1 Research area . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.2 Machine learning versus hand-crafted rule-based methods . . . . . 123.3 Recognising entities in clinical corpora . . . . . . . . . . . . . . . 12

3.3.1 Annotation . . . . . . . . . . . . . . . . . . . . . . . . . 133.3.2 Named entity recognition . . . . . . . . . . . . . . . . . . 133.3.3 Conditional random fields . . . . . . . . . . . . . . . . . 153.3.4 Mapping named entities to specific vocabulary concepts . 20

3.4 Expanding vocabulary and abbreviations lists . . . . . . . . . . . 213.4.1 Expanding vocabularies with new terms . . . . . . . . . . 213.4.2 Expanding vocabularies with abbreviations . . . . . . . . 233.4.3 Random indexing . . . . . . . . . . . . . . . . . . . . . . 23

3.5 Clinical negation and uncertainty detection . . . . . . . . . . . . 273.5.1 Negation and scope detection . . . . . . . . . . . . . . . 283.5.2 Modifiers other than negations . . . . . . . . . . . . . . . 29

4 Scientific method 314.1 Overall research approach . . . . . . . . . . . . . . . . . . . . . 314.2 Research strategy for the evaluation stage . . . . . . . . . . . . . 324.3 Alternative research approach . . . . . . . . . . . . . . . . . . . 33

5 Used and created corpora 355.1 Used corpora . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

5.1.1 The Stockholm Electronic Patient Record Corpus . . . . 355.1.2 Additional corpora . . . . . . . . . . . . . . . . . . . . . 37

5.2 Corpora created by annotation . . . . . . . . . . . . . . . . . . . 375.2.1 The Stockholm EPR Clinical Entity Corpus . . . . . . . . 37

vi

5.2.2 The Stockholm EPR Negated Findings Corpus . . . . . . 415.3 Ethics when using health record corpora . . . . . . . . . . . . . . 43

6 Extracting mentioned clinical findings 456.1 Study I: Rule-based entity recognition and coverage of SNOMED

CT in Swedish clinical text . . . . . . . . . . . . . . . . . . . . . 456.1.1 Design and development of the artefact . . . . . . . . . . 466.1.2 Evaluation of the artefact . . . . . . . . . . . . . . . . . . 476.1.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

6.2 Study II: Vocabulary expansion by semantic extraction of medicalterms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 486.2.1 Design and development of the artefact . . . . . . . . . . 486.2.2 Evaluation of the artefact . . . . . . . . . . . . . . . . . . 496.2.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

6.3 Study III: Synonym extraction and abbreviation expansion withensembles of semantic spaces . . . . . . . . . . . . . . . . . . . . 526.3.1 Design and development of the artefact . . . . . . . . . . 536.3.2 Evaluation of the artefact . . . . . . . . . . . . . . . . . . 556.3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

6.4 Study IV: Automatic recognition of disorders, findings, pharma-ceuticals and body structures from clinical text . . . . . . . . . . 596.4.1 Design and development of the artefact . . . . . . . . . . 606.4.2 Evaluation of the artefact . . . . . . . . . . . . . . . . . . 626.4.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

7 Negation and uncertainty detection 677.1 Study V: Negation detection in Swedish clinical text . . . . . . . . 67

7.1.1 Design and development of the artefact . . . . . . . . . . 687.1.2 Evaluation of the artefact . . . . . . . . . . . . . . . . . . 687.1.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

7.2 Study VI: Cue-based assertion classification for Swedish clinicaltext . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 707.2.1 Design and development of the artefact . . . . . . . . . . 707.2.2 Evaluation of the artefact . . . . . . . . . . . . . . . . . . 717.2.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

8 Discussion and conclusions 758.1 Recognising mentioned clinical findings . . . . . . . . . . . . . . 75

vii

8.1.1 Study I . . . . . . . . . . . . . . . . . . . . . . . . . . . 758.1.2 Study II . . . . . . . . . . . . . . . . . . . . . . . . . . . 768.1.3 Study III . . . . . . . . . . . . . . . . . . . . . . . . . . 788.1.4 Study IV . . . . . . . . . . . . . . . . . . . . . . . . . . 80

8.2 Negation and uncertainty detection . . . . . . . . . . . . . . . . . 858.2.1 Study V . . . . . . . . . . . . . . . . . . . . . . . . . . . 858.2.2 Study VI . . . . . . . . . . . . . . . . . . . . . . . . . . 86

8.3 General discussion and conclusions . . . . . . . . . . . . . . . . 888.3.1 Exploring techniques previously used for English . . . . . 888.3.2 Evaluation and expansion of clinical vocabulary resources 898.3.3 Distinguishing between the semantic categories Disorder

and Finding . . . . . . . . . . . . . . . . . . . . . . . . . 918.4 Main contributions and final conclusions . . . . . . . . . . . . . . 92

9 Future directions 959.1 Further exploring vocabulary expansion . . . . . . . . . . . . . . 95

9.1.1 Comparison to other approaches and further tuning of theRandom indexing models . . . . . . . . . . . . . . . . . 95

9.1.2 Clustering context vectors . . . . . . . . . . . . . . . . . 969.1.3 Addressing multiword terms . . . . . . . . . . . . . . . . 969.1.4 Identifying connections between entities of the types Body

Structure and Clinical Finding . . . . . . . . . . . . . . . 979.1.5 Further expanding the uncertainty cue lexicon . . . . . . . 97

9.2 Applying developed vocabulary . . . . . . . . . . . . . . . . . . 979.2.1 Evaluating the usefulness of created vocabularies for

named entity recognition . . . . . . . . . . . . . . . . . . 989.2.2 Evaluating the usefulness of created vocabularies for con-

cept matching . . . . . . . . . . . . . . . . . . . . . . . . 989.2.3 Evaluating the usefulness of created vocabularies for other

applications . . . . . . . . . . . . . . . . . . . . . . . . . 999.3 Exploring resource efficient methods . . . . . . . . . . . . . . . . 99

9.3.1 Semi-supervised methods . . . . . . . . . . . . . . . . . 1009.3.2 Resource efficiency in annotation . . . . . . . . . . . . . 100

9.4 Applying the developed artefacts . . . . . . . . . . . . . . . . . . 101

References 103

viii

Appendices 114

A Authors’ contributions 117A.1 Study I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117A.2 Study II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117A.3 Study III . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117A.4 Study IV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118A.5 Study VI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

B Used evaluation methods 121B.1 Measuring the performance of an artefact . . . . . . . . . . . . . 121B.2 Statistical tests . . . . . . . . . . . . . . . . . . . . . . . . . . . 122B.3 Evaluating the quality of the reference standard . . . . . . . . . . 123

C Used vocabularies 125

Chapter 1

Introduction and general aims

1.1 Introduction

When patients are under care, their medical status and their treatment are system-atically documented in health records (Nilsson, 2007, pp. 11–12). A health recordtypically consists of structured data and of narrative text, and its content is a criticalsource of information for those involved in the immediate care of a patient.

Health records have traditionally been kept on paper (Nilsson, 2007, p. 150), butthe digitalisation of the health care documentation offers new possibilities for auto-matic processing of recorded information. Advances in clinical language process-ing have made it possible to automatically extract information from the narrativehealth record text. This enables new types of tools for presenting and documentinghealth record information. In addition, by aggregating extracted information froma large database of health records, it is possible to use clinical language processingfor creating new medical knowledge (Meystre et al., 2008).

The medical status of the patient is typically documented in the health recordin form of clinical findings. This information entity is defined by the InternationalHealth Terminology Standards Development Organisation, IHTSDO (2008b) asobservations made when examining patients (the category Finding) and as diag-nostic assessments of patients (the category Disorder). Some of these clinical find-ings are stored in a structured format in the health record, for example as diagnosiscodes. This structured data does, however, not cover the full medical status of thepatient (Petersson et al., 2001). Therefore, to make full use of the information on

2 Chapter 1.

clinical findings contained in health records, automatic extraction of informationfrom free text is required (Friedman, 2005, p. 425).

Such an automatic extraction of clinical findings from health record text canform the basis of a number of text mining systems, e.g. systems for syndromicsurveillance (Chapman et al., 2005), automatic detection of adverse drug reactions(Melton and Hripcsak, 2005; Eriksson et al., 2013), comorbidity studies (Roqueet al., 2011) and studies of disorder-finding relations (Cao et al., 2005). Auto-matically extracted clinical findings can also be used in tools for presenting healthrecord content. Examples of such tools are automatically generated text summari-sations (Hallett et al., 2006; Aramaki et al., 2009; Kvist et al., 2011), problemlists (Meystre and Haug, 2006), visualisations (Plaisant et al., 1998) and markupsof clinical findings (Shablinsky et al., 2000). An additional application is toolsfor facilitating the documentation of patient information, e.g. automatically gener-ated drafts for discharge summaries and problem lists, as well as tools that suggestdiagnosis codes (Henriksson and Hassel, 2011).

1.2 Problems and general aims

There are a number of studies in which established information extraction tech-niques have been applied for extracting the more general entity category ClinicalFinding (Chapman and Dowling, 2006; Roberts et al., 2009; Wang, 2009; Uzuneret al., 2011) or the more granular sub-category Disorder (Ogren et al., 2008) fromdifferent types of English clinical corpora, often using the large resources of med-ical vocabularies that are available for English.

There is research on the extraction of clinical findings, as well as on related top-ics, using health record text written in languages other than English (Boytchevaet al., 2005; Hiissa et al., 2007; Deléger and Grouin, 2012; Santiso et al., 2014;Aramaki et al., 2014). For clinical text in most languages, however, it is not yetexplored to what extent information extraction techniques used for English healthrecord text can be successfully applied. The first aim of this thesis was, therefore,to explore how the techniques that have been used for extracting clinical findingsfrom English text perform when applied to another, relatively closely related, lan-guage.

Medical vocabulary resources are often less extensive for other languages thanfor English, which might affect the performance of tools for extracting clinicalfindings. The manual creation of vocabularies is, however, expensive and time-consuming, which makes methods for facilitating this process valuable. Previous

Introduction and general aims 3

studies of medical vocabulary expansion have, however, mainly focused on the ex-traction of medical terms and abbreviations using techniques that are not optimalfor clinical corpora (McCrae and Collier, 2008; Neelakantan and Collins, 2014;Dannélls, 2006). The second aim was, therefore, to explore methods for facilitat-ing the expansion and creation of vocabularies relevant for information extractionfrom the clinical sublanguage, as well as to study the usefulness of a smaller vo-cabulary resource for the task of extracting clinical findings.

Previous studies of the extraction of entities in clinical text have typically ex-tracted a general category, equivalent to the more general category Clinical Find-ing (Chapman and Dowling, 2006; Roberts et al., 2009; Wang, 2009; Uzuner etal., 2011), or focused only on an entity category equivalent to the more granularsub-category Disorder (Ogren et al., 2008). Although this might not be enough forsome text mining applications (Cao et al., 2005), there is not much work in whichthe feasibility of a more granular categorisation is studied (Albright et al., 2013).The third aim was therefore to study to what extent it is possible to extract the moregranular sub-categories Disorder and Finding from health record text (Figure 2.1).

The three aims are summarised in Table 1.1.

1) Apply techniques, previously used for extracting Clinical Findings fromEnglish text, on a related language.2) Study the usefulness of a smaller vocabulary resource when extractingClinical Findings, and explore methods for expansion and creation ofvocabularies relevant for clinical information extraction.3) Study to what extent it is possible to divide the entity categoryClinical Finding into the more granular categories Disorder and Finding.

Table 1.1: A summary of the aims of the thesis.

4

Chapter 2

Included studies, researchquestions and study alignment

2.1 Included studies

The following studies are included in the thesis:

• Study I: Maria Skeppstedt, Maria Kvist, and Hercules Dalianis. 2012.Rule-based entity recognition and coverage of SNOMED CT in Swedishclinical text. In Proceedings of the Eighth International Conference on Lan-guage Resources and Evaluation (LREC’12), pages 1250–1257, Istanbul,Turkey, May 2012.

• Study II: Maria Skeppstedt, Magnus Ahltorp, and Aron Henriksson. 2013.Vocabulary expansion by semantic extraction of medical terms. In Proceed-ings of Languages in Biology and Medicine (LBM), Tokyo, Japan, December2013.

• Study III: Aron Henriksson, Hans Moen, Maria Skeppstedt, Vidas Daudar-avicius, and Martin Duneld. 2014. Synonym extraction and abbreviationexpansion with ensembles of semantic spaces. Journal of Biomedical Se-mantics, 5(1):6.

• Study IV: Maria Skeppstedt, Maria Kvist, Gunnar H Nilsson, and HerculesDalianis. 2014. Automatic recognition of disorders, findings, pharmaceu-

6 Chapter 2.

ticals and body structures from clinical text: An annotation and machinelearning study. Journal of Biomedical Informatics, 49:148–58.

• Study V: Maria Skeppstedt. 2011. Negation detection in Swedish clinicaltext: An adaption of NegEx to Swedish. Journal of Biomedical Semantics,2(Suppl 3):S3.

• Study VI: Sumithra Velupillai, Maria Skeppstedt, Maria Kvist, DanielleMowery, Brian E Chapman, Hercules Dalianis, and Wendy W Chapman.2014. Cue-based assertion classification for Swedish clinical text – de-veloping a lexicon for PyContextSwe. Artificial Intelligence in Medicine,61(3):137–44.

The thesis describes and discusses the results that are relevant for the introducedaims as well as the methods used for obtaining these results.

2.2 Study alignment

Clinical findings are sometimes mentioned in the health record text as somethingthat the patient might have or does not have, or as a something that is not observed(Friedman, 2005, p. 426). This has the effect that the task of extracting clinicalfindings needs to be divided into two sub-tasks: a) locating (or recognising) men-tioned entities of the type Clinical Finding and b) determining if the recognisedentities are mentioned as affirmed, negated and/or with uncertainty.

The thesis is structured according to these two identified sub-tasks, and for eachof the two tasks, a number of research questions were posed (Table 1.1 and Figure2.2):

a) Recognising mentioned clinical findings

The three aims were first studied in relation to the task of recognising mentionedentities belonging to the category Clinical Finding. Corpora and medical vocab-ularies in Swedish, which like English is a Germanic language, were used in allstudies. The following research questions were posed:

• Study I: With what precision and recall can Findings and Disorders, men-tioned in Swedish health record text, be recognised by using hand-craftedrules for mapping to existing Swedish vocabularies?

Included studies, research questions and study alignment 7

• Study II: What is the recall, among a limited set of candidate terms, for re-trieving known terms denoting Clinical Findings and Pharmaceutical Drugsby using similarity to seed terms in a Random indexing space built on med-ical text?

• Study III: What is the recall, among a limited set of candidate terms, forretrieving known synonyms and abbreviations by using term similarity in aRandom indexing space built on medical text?

• Study IV: With what precision and recall can Findings and Disorders, men-tioned in Swedish health record text, be recognised by using conditionalrandom fields trained on a limited-sized corpus of manually annotated clin-ical text?

How is the performance affected when the machine learning methods thathave been successful for English Clinical Findings recognition are appliedto the task of recognising Clinical Findings in Swedish health record text?

To what extent is it possible to separate the more general entity categoryClinical Finding into the two more granular entity categories Disorder andFinding?

The main aim, i.e. applying techniques previously used for English clinical find-ings extraction to a related language, was addressed in Studies I and IV. The sec-ondary aim, i.e. studying the usefulness and expansion of a smaller medical vocab-ulary resource, was addressed in Studies I, II and III. The final third aim, separatingClinical Findings into Disorders and Findings, was addressed in Studies I and IV.

b) Negation and uncertainty detection

Two of the aims were studied in relation to the task of automatically determiningwhich of the recognised clinical findings are negated or expressed with uncertainty.Also here, Swedish was the language used for the studies. The following researchquestions were asked:

• Study V: With what precision and recall can an English hand-crafted rule-based system detect negation when adapted to Swedish clinical text?

What is the coverage of a Swedish vocabulary of cue terms for negationobtained from translating English negation cues?

8 Chapter 2.

• Study VI: With what precision and recall can an English hand-crafted rule-based system detect negation and uncertainty when adapted to Swedish clin-ical text?

What is the coverage of a Swedish vocabulary of cue terms for uncertaintyobtained from translating English uncertainty cues?

In both study V and study VI, how English techniques for clinical findings ex-traction perform when applied to a related language was explored, as well as howvocabularies relevant for clinical information extraction can be created.

Included studies, research questions and study alignment 9

Disorders:

1) Necessarily abnormal. 2) Temporal persistence, with the (at least theoretical) possibility of their manifestations being treated, in remission, or quiescent. 3) Have an underlying pathological process.

Findings:

1) May be normal (but not necessarily); no disorders may. 2) May exist only at a single point in time (e.g. a serum sodium level); no disorders may. 3) Cannot be temporally separate from the observing of them (you can’t observe them and say they are absent, nor can you have the finding present when it is not capable of being observed). 4) Cannot be defined in terms of an underlying pathological process that is present even when the observation itself is not present.

Clinical Findings

Figure 2.1: The most important semantic categories of this the-sis. Based on the definition by International Health TerminologyStandards Development Organisation, IHTSDO (2008b).

10 Chapter 2.

Sub-task A: Recognising Clinical Findings.I: With what precision and recall can Findings and Disorders, mentioned in Swedish health record text, be recognised by using hand-crafted rules for mapping to existing Swedish vocabularies? (Aim 2, 3)Expand vocabularies Train a

machine learning model for recognition

II: What is the recall, among a limited set of candidate terms, for retrieving known terms denoting Clinical Findings and Pharmaceutical Drugs by using similarity to seed terms in a Random indexing space built on medical text? (Aim 2)

IV: With what precision and recall can Findings and Disorders, mentioned in Swedish health record text, be recognised by using conditional random fields trained on a limited-sized corpus of manually annotated clinical text? (Aim 1, 2)

How is the performance affected when the machine learning methods that have been successful for English Clinical Findings recognition are applied on the task of recognising Clinical Findings in Swedish health record text? (Aim 1)

To what extent is it possible to separate the more general entity category Clinical Finding into the two more granular entity categories Disorder and Finding? (Aim 3)

III: What is the recall, among a limited set of candidate terms, for retrieving known synonyms and abbreviations by using term similarity in a Random indexing space built on medical text? (Aim 2)

Sub-task B: Classifying recognised Clinical Findings as affirmed, negated and/or uncertain.V: With what precision and recall can an English hand-crafted rule-based system detect negation when adapted to Swedish clinical text? (Aim 1)

What is the coverage of a Swedish vocabulary of cue terms for negation obtained from translating English negation cues? (Aim 2)Add detection of uncertainty

VI: With what precision and recall can an English hand-crafted rule-based system detect negation and uncertainty when adapted to Swedish clinical text? (Aim 1)

What is the coverage of a Swedish vocabulary of cue terms for uncertainty obtained from translating English uncertainty cues? (Aim 2)

Overall task: Information extraction of Clinical Findings.

Figure 2.2: Research questions. Addressed aims are shown inparenthesis.

Chapter 3

Background

This chapter gives an overview of previous research related to the thesis as well asthe theoretical background of the methods used.

3.1 Research area

The research area for this thesis is the sub-domain of natural language process-ing (NLP) that is known as information extraction, more specifically informationextraction from Swedish health record texts. Information extraction is the task ofautomatically extracting specific, predefined types of information from unstruc-tured data, such as free text. (Meystre et al., 2008).

The focus of this thesis lies in the extraction of the clinical findings that arerelated to a patient. Detecting mentioned occurrences of findings, disorders andother medical entities from free text is a kind of named entity recognition (NER)(Meystre et al., 2008), which is an important task within information extraction.This task consists of automatically detecting spans of text referring to entitiesof certain semantic categories (Jurafsky and Martin, 2008, pp. 759–768). Deter-mining if a recognised entity is expressed with uncertainty or as a negated entity(Friedman, 2005, p. 426) is another task within information extraction that is veryimportant for extracting clinical findings related to a patient. This negation anduncertainty detection task is also explored in the thesis.

12 Chapter 3.

3.2 Machine learning versus hand-crafted rule-basedmethods

Methods for constructing information extraction and other NLP systems can bedivided into hand-crafted rule-based methods and machine learning (or statisticallearning) methods. For a rule-based system, rules are manually constructed toperform a required task (Alpaydin, 2010, p. 1) while a machine learning systemuses observed examples to automatically learn to perform a task (Alpaydin, 2010,pp. 2–14). When the precise method for performing a certain task is very complexor not known, machine learning is typically preferred (Alpaydin, 2010, p. 1) whilerule-based methods are preferred when labelled data is difficult or expensive toobtain. Labelled examples within NLP are often obtained by manual annotation ofcorpora, i.e. of collections of naturally occurring language data, such as texts. Themanual annotation typically consists of manual classification of text structure orcontent, for instance by labelling a token according to its part of speech or semanticcategory (Ogren, 2006).

In this thesis, rule-based methods were used for performing vocabulary match-ing to recognise clinical findings as well as for performing vocabulary-based un-certainty and negation detection. For the task of recognising clinical findings, ma-chine learning methods (in the form of Conditional random fields) were, however,also used.

When feeding textual data into machine learning methods, the data can, for in-stance, be represented as sequential data, where each word in a text is one observa-tion in a sequence of data (Bishop, 2006, p. 605). More high-level representationsof words can also be created, for instance by deriving an approximate represen-tation of their meaning based on the idea that words occurring in similar textualcontexts are likely to also have a similar meaning (Landauer, 2007, p. X). Here,random indexing was used as a method to create such high-level representations(Kanerva et al., 2000).

3.3 Recognising entities in clinical corpora

Common methods for named entity recognition are matching to vocabulary lists,machine learning approaches as well as combinations of the two (Mikheev et al.,1999). Corpora annotated for named entities are used to train machine learningmodels and to evaluate the performance of named entity recognition systems. An-

Background 13

notation of text, therefore, often forms a component in the process of constructinga named entity recognition system.

3.3.1 Annotation

In the clinical domain, as well as in many other specialised domains, domain ex-perts are typically required for semantic annotation tasks. This is often the casefor the task of annotating clinical named entities, i.e. to manually label a clini-cal text span according to its semantic category. These expert annotators can bemore expensive than annotators without the required specialised knowledge. It isalso difficult to use crowdsourcing approaches, e.g. to hire online annotators withthe required knowledge (Xia and Yetisgen-Yildiz, 2012). A further challenge isposed by the content of the clinical data, which is often sensitive and should onlybe accessed by a limited number of people. Research community annotation is,consequently, another option that is not always open to annotation projects in theclinical domain, even if there exist examples of such community annotations forclinical texts (Uzuner et al., 2010b).

Despite the difficulties of annotating clinical texts, a number of smaller andlarger projects including annotation of clinical entities have been carried out, manyof them on English text, e.g. by Chapman et al. (2008), Roberts et al. (2009),Wang (2009), Ogren et al. (2008) and Uzuner et al. (2010b). In some clinicalannotation studies, pre-annotation was applied to facilitate the manual annotationwork (Uzuner et al., 2010a,b; Albright et al., 2013). Pre-annotation consists ofautomatically annotating a text by an existing system, and the annotator eithercorrects the mistakes made by this existing system (Chou et al., 2006) or choosesbetween different annotations provided by the system (Brants and Plaehn, 2000).

In most of these clinical annotation studies, the quality of the annotations wasmeasured by calculating inter-annotator agreement between different annotators,among which one or several had a medical education. The inter-annotator agree-ment for the most successful annotations was typically an F-score between 0.70and 0.90, depending on annotation class and study.

3.3.2 Named entity recognition

Many of the above mentioned annotated corpora have been used as training andevaluation data for machine learning-based NER systems. The most frequentlyused machine learning algorithms are SVM (support vector machine) and CRFs(Conditional random fields). An SVM was trained on a subset of the above de-scribed corpus by Roberts et al. (2008). On the corpus created by Wang, two

14 Chapter 3.

studies have been performed: one using CRFs (Wang, 2009) and one combiningoutput from a CRFs model with an SVM model and an ME (maximum entropy)classifier (Wang and Patrick, 2009). In the i2b2/VA challenge on concepts, asser-tions, and relations (Uzuner et al., 2011; i2b2/VA, 2010), all but one of the bestperforming systems used CRFs for concept recognition. The best performing sys-tem (de Bruijn et al., 2011) used semi-Markov HMM instead. The participantswith second best system (Jiang et al., 2011) found that CRFs outperformed SVMand that results could be improved with a rule-based post-processing module.

In the i2b2 medication challenge, on the other hand, which included the identi-fication of medication names, a majority of the ten top-ranked systems were rule-based (Uzuner et al., 2010a). The best performing system (Patrick and Li, 2010)did, however, use CRFs while the second best (Doan et al., 2010) was built onvocabulary matching and a spell checker developed for drug names. This rule-based system was later employed by Doan et al. (2012) in an ensemble classifiertogether with an SVM and a CRFs system. On the corpus by Ogren et al. (2008),a rule-based vocabulary matching method that recognises disorders was evaluated(Kipper-Schuler et al., 2008; Savova et al., 2010). There is also a previous study ona vocabulary-based system that uses the MeSH vocabulary to recognise diseasesand drugs in Swedish discharge summaries (Kokkinakis and Thurin, 2007).

Vocabulary-based named entity recognition matches words that occur in thetext to words in available medical vocabulary resources, often by using some typeof pre-processing, for instance grammatical normalisation such as lemmatisation(Uzuner et al., 2010a), permutations of words (Kipper-Schuler et al., 2008), sub-string matching (Patrick et al., 2007) or manipulation of characters by, for in-stance, applying Levenshtein distance (ul Muntaha et al., 2012) or spelling correc-tion (Doan et al., 2010).

For machine learning models, it must be decided which features are importantfor the intended classification task. Typical features used for training the modelsdescribed above were the tokens (sometimes in a stemmed form); orthographics(e.g. number, word, capitalisation); prefixes and suffixes; part-of-speech informa-tion as well as output from vocabulary matching, which had a large positive effectin many studies (Wang, 2009; Wang and Patrick, 2009). Most studies used featuresextracted from the current and the two preceding and two following tokens whileRoberts et al. (2008) used a window size of +/-1. The best performing system inthe i2b2/VA concepts challenge used a very large feature set with a window size of+/-4, also including character n-grams, word bi/tri/quad-grams and skip-n-gramsas well as sentence, subsection and document features (e.g. sentence and documentlength and section headings). In addition, features from semi-supervised learning

Background 15

Table 3.1: NER results for previous studies. n is the number oftraining instances.

Corpus Entity category n Precision RecallClinical E- Roberts et al. (2008); SVM (10-fold cross-validation)Science Condition 739 0.82 0.65Framework Drug or Device 272 0.83 0.59

Locus 490 0.80 0.62Wang Wang (2009); CRFs (10-fold cross-validation)

Finding 4,741 0.83 0.79Substance 2,249 0.92 0.89Body 735 0.72 0.64

Wang and Patrick (2009);Combining CRFs, SVM and ME (10-fold cross-validation)

Finding 4,741 0.84 0.82Substance 2,249 0.92 0.88Body 735 0.76 0.66

i2b2/VA Jiang et al. (2011); Combining 4 CRFs modelschallenge (Separate evaluation set)on concepts... Medical Problem 11,968 0.87 0.84

de Bruijn et al. (2011); Semi-Markov HMM(Separate evaluation set)

Medical concepts (including Problem) 27,837 0.87 0.84i2b2 Patrick and Li (2010); CRFsmedication Medication Names - 0.91 0.86challenge Doan et al. (2012); vocabulary matching

Medication Names - 0.85 0.87Doan et al. (2010); CRFs, SVM, vocabulary

Medication Names - 0.94 0.90Previous Kokkinakis and Thurin (2007); vocabulary matchingSwedish study Disease - 0.98 0.87

Drug - 0.95 0.93Ogren et al. Savova et al. (2010); vocabulary matching

Disorder - 0.80 0.65

methods were incorporated in the form of hierarchical word clusters constructedon unlabelled data (de Bruijn et al., 2011).

3.3.3 Conditional random fields

Similar to many of the described related studies, Conditional random fields wereused as the machine learning method for named entity recognition in this thesis.Conditional random fields (CRF or CRFs), introduced by Lafferty et al. (2001), isa machine learning method suitable for segmenting and labelling sequential data.In contrast to many other types of data, observed data points for sequential data,such as text, are dependent on other observed data points. Constructing models forwhich there is a dependence between all observed data points would, however, be

16 Chapter 3.

intractable. Therefore, given a certain data point, models of sequential data typ-ically assume independence of most other data points, except for a small subsetof the points in the data set (Bishop, 2006, pp. 606–607). These dependences andindependences between data points are practical to describe within the frameworkof graphical models (Bishop, 2006, p. 359) to which CRFs belong (Sutton and Mc-Callum, 2006, p. 1). Graphical models are represented by nodes, which representrandom variables or groups of random variables, and links between nodes, whichrepresent relations between variables. CRFs are described by undirected graphicalmodels, in which the links have the following meaning. If there is no path betweenthe nodes A and B that does not pass through C, then A and B are conditionallyindependent, given C (Marsland, 2009, p. 345).

CRFs are closely related to Hidden Markov Models, which are also typicallydescribed as graphical models (see Table 3.2). A difference, however, is that Hid-den Markov Models belong to the class of generative models, whereas CRFs areconditional (or discriminative) models (Sutton and McCallum, 2006, p. 1). In gen-erative models, the joint distribution between input variables and the variables thatare to be predicted is modelled (Bishop, 2006, p. 43). Modelling the input vari-ables might, however, make it impossible to use a large number of features for theinput variables since there are often dependencies between the different features.Including these dependences in the model can lead to intractable models, whereasnot modelling them (as for Naive Bayes) might lead to a lower performance ofthe classifier. CRFs and other conditional models directly model the conditionaldistribution instead (Sutton and McCallum, 2006, p. 1), i.e. the probability of theclasses that are to be predicted, given the observed input variables. Thereby, de-pendencies between input variables do not need to be explicitly modelled. If ydenotes the output variables (i.e. the NER classes that are to be predicted) and xdenotes the observed input variables (i.e. the features of the text such as the tokensor their part-of-speech), then p(y|x) is modelled when training CRFs.

Independent data Sequential data General graphsGenerative Naive Bayes HMM Generative directed modelsConditional Logistic regression Linear-chain CRFs General CRFs

Table 3.2: CRFs compared to other ML approaches. From Suttonand McCallum (2006).

In the most basic case of CRFs, linear chain CRFs (Table 3.2 and Figure 3.1),which is often used for named entity recognition, output variables are linked in a

Background 17

y (output)

x (input)

Figure 3.1: Dependencies. Inspired from Sutton and McCallum(2006).

chain. Apart from being dependent on the input variables, each output variable isthen conditionally independent on all other output variables, except on the previousand following output variable, given these two neighbouring output variables.

For named entity recognition, IOB-encoding is typically used for encoding theoutput variables (y). Tokens not annotated as an entity are then encoded withthe label O, whereas labels for annotated tokens are prefixed either with a B, ifit is the first token in the annotated chunk, or otherwise with an I (Jurafsky andMartin, 2008, pp. 763–764). This is exemplified in Figure 3.2, for the two entityclasses Disorder and Finding. In this case, the model thus learns to classify in 4+1different classes: B-Disorder, I-Disorder, B-Finding, I-Finding and O.

The dependencies are defined by a number of K (typically) binary feature func-tions of input and output variables (x denotes input variables while yt denotes theoutput variable in the current position in the sequence and yt−1 denotes the outputvariable in the previous position):

fk(t,yt−1,yt ,x)

E.g. are all of the following true?

• Output: The output at position t is I-Disorder

18 Chapter 3.

Token Category

DVT B-Disorder

patient O

with O

problems B-Finding

to I-Finding

breathe I-Finding

Figure 3.2: IOB-encoding for Disorder and Finding.

• Output: The output at position position t−1 is B-Disorder

• Input: The token at position t−1 is experiences

• Input: The token at position t−2 is patient

The formula for the conditional distribution p(y|x) can be written as a log linearmodel as follows using the feature functions (Tsuruoka et al., 2009):

p(y|x) = 1Z(x)

expT

∑t=1

K

∑k=1

λk fk(t,yt−1,yt ,x)

Where Z(x) is a normalisation function:

Z(x) = ∑y

expT

∑t=1

K

∑k=1

λk fk(t,yt−1,yt ,x)

A linear combination can be both positive and negative, but using the exponen-tial ensures that the expression is always positive. The Z(x) further ensures thatthe result is between 0 and 1 and, thereby, a valid probability (Elkan, 2008, p. 10).

The model is trained through setting the weights, i.e. estimating the parameters(θ = {λk}) for the feature functions. This is done by a penalised maximum like-lihood estimation. The following is the formula for the conditional log-likelihoodfor which θ are found so that the function is maximised for the observed outputlabels given the corresponding inputs in the training data. If there are N sequences(for the NER task, typically sentences) in the training data, then the following is

Background 19

the training data: {x(i),y(i)}Ni=1, for which {x(i)1 ,x(i)2 , ...x(i)T } is the input sequence

and {y(i)1 ,y(i)2 , ...y(i)T } is the observed output sequence. Then, this maximization canbe written as follows (Sutton and McCallum, 2006, p. 11):

θ∗ = argmax

θ

N

∑i=1

log p(y(i)|x(i))−R(θ)

Penalised means that regularisation is used, and regularisation is performed byadding the penalty term R, which prevents the weights from reaching values thatare too large and, thereby, prevents over-fitting (Bishop, 2006, p. 10). The L1-norm (Bishop, 2006, p. 146) and L2-norm (Bishop, 2006, p. 10) are often used forregularisation (Tsuruoka et al., 2009), and a variable C governs the strength of theregularisation. Using the L1-norm also results in that if C is large enough, some ofthe weights will be driven to zero, resulting in a sparse model and, thereby, the fea-ture functions that those weights control will not play any role in the model. Thismeans that, when using regularisation, initially complex models can be trained ondata sets with a limited size, without being severely over-fitted, since the complex-ity of the actual model is automatically reduced. However, a suitable value of Cmust still be determined (Bishop, 2006, p. 145).

The L1-norm (Lasso): R(λ ) =C2

K

∑k=1|λk|

The L2-norm (Ridge regression): R(λ ) =C2

K

∑k=1

λ2k

The CRFs implementation used in this thesis was the CRF++ package (Kudo,2012), which automatically generates feature functions from user-defined tem-plates, specifying what features are of interest to include in the feature functions.When using CRF++ as linear chain CRFs, the CRF++ system generates one bi-nary feature function for each combination of output class, previous output classand unique string in the training data that is expanded by a template. This meansthat L ∗ L ∗M feature functions are generated for each template, where L is thenumber of output classes and M is the number of unique expanded strings. Thismeans that if only the current token were to be used as a feature for detectingthe two entities in the previous example, the number of feature functions wouldbe 5 ∗ 5 ∗ |the number of unique tokens in the corpus|. In practice, a lot of otherfeatures are also used and, thereby, many more feature functions are generated.

20 Chapter 3.

The maximisation of the regularised function cannot normally be solved ana-lytically, so numerical methods must be applied. However, since the regularisedfunction is strictly concave, it has exactly one global optimum, which is guaran-teed to be found (Sutton and McCallum, 2006, p. 12). A class of methods calledquasi-Newton methods are typically used.

The most probable classification sequence for a sequence of unseen data is then:

y∗ = argmaxy

p(y|x)

This can be calculated using dynamic-programming algorithms, which are alsoused for, e.g. Hidden Markov Models (Sutton and McCallum, 2006, p. 13). CRF++uses a combination of forward Viterbi and backward A* search for finding the mostprobable classification (Kudo, 2012).

3.3.4 Mapping named entities to specific vocabulary concepts

In some vocabularies, each entry is associated with a unique concept identifier,which represents its meaning and which makes it possible to map words or textsegments to a specific concept (Savova et al., 2010) rather than to a semantic cate-gory, which is the case for named entity recognition. Each concept can have severallexical instantiations (e.g. synonyms), which enable a mapping from the clinicaltext to the concept, regardless of which of the included lexical instantiation thatis used. There are a number of such studies, in which clinical text segments aremapped to specific vocabulary concepts.

Aronson (2001) parses text with a shallow parser to filter out noun phrases forwhich inflections and spelling variants are generated and mapped to the UnifiedMedical Language System (UMLS) metathesaurus as well as to a synonym andabbreviation lexicon. Instead of parsing the text, Zou et al. (2003) generate allpossible permutations of short text chunks and map these to UMLS. A UMLSmapping system by Friedman et al. (2004) also performs normalising of extractedtext segments to determine that, e.g. enlarged spleen and spleen was enlarged referto the same concept.

The precision of vocabulary mapping has been shown to improve when auto-matically restricting the used vocabulary to combinations of subsets of UMLS(Huang et al., 2003). There are e.g. mapping studies restricted to using theSNOMED CT vocabulary that also apply different techniques for capturing abbre-viations, misspellings, inflections and word order differences (Long, 2005; Patricket al., 2007).

Background 21

In addition, there are shared tasks for extracting disorders from clinical corpora,which have not been limited to named entity recognition of mentioned disordersbut which have also included the mapping of extracted disorders to vocabularyconcepts. Examples are the CLEF eHealth shared task (Suominen et al., 2013),in which disorders were mapped to an UMLS CUI (Concept Unique Identifier)as well as the NTCIR MedNLP shared task, in which disorders were mapped toICD-10 codes (Aramaki et al., 2014).

3.4 Expanding vocabulary and abbreviations lists

To enable medical vocabulary concept mapping and to use vocabularies for medi-cal named entity recognition, extensive vocabularies are required, covering a largenumber of concepts and including synonyms for each concept, e.g. the UMLSresource referred to in the previous section. Medical resources are, however, gen-erally less extensive for languages other than English, and methods to facilitatethe expansion of such resources are therefore valuable. For such an expansionof vocabularies and abbreviations lists, there are a number of previously exploredsemi-automatic approaches.

3.4.1 Expanding vocabularies with new terms

The aim of methods for facilitating the creation or expansion of vocabularies is typ-ically to find hyponyms to selected terms (Hearst, 1992), i.e. categorising wordsinto semantic categories (Thelen and Riloff, 2002), or to find synonyms (Blondelet al., 2004; Henriksson et al., 2013a). Materials used could be existing vocabular-ies, e.g. when finding synonym candidates by comparing term descriptions (Blon-del et al., 2004) or when translating a vocabulary resource from a foreign language(Perez-de Viñaspre and Oronoz, 2014). An alternative is to extract new vocabu-lary terms from text corpora. This can, for instance, be accomplished by findingtext patterns which explicitly state that two words are synonyms, e.g. “term1 alsoknown as term2” (Yu and Agichtein, 2003), or that a word belongs to a certain se-mantic category, e.g. “term1 such as term2” (Hearst, 1992). These patterns can bemanually crafted or found by automatic methods. For instance, Yu and Agichtein(2003) and Cohen et al. (2005) have applied these methods to extract synonyms togene and protein names. Another example of synonym extraction in the biomedi-cal domain is provided by McCrae and Collier (2008). They extracted synonymsto terms belonging to medically relevant semantic categories including diseasesand symptoms. Patterns for expressions of synonymy were automatically gener-

22 Chapter 3.

ated from a biomedical corpus using seed synonym pairs, and, thereafter, occur-rences of term pairs in these patterns were used as features for training a classifierto determine whether a term pair was a synonym pair or not. The approach wasevaluated on synonym pairs that had been manually extracted from the biomedicalcorpus, and the best classifier achieved a precision of 0.73 and a recall of 0.30when automatically classifying the term pairs. Neelakantan and Collins (2014)did not target synonyms, but aimed instead at expanding medical vocabularieswith terms of certain semantic categories, including the category Disease. All sen-tences containing the word disease were extracted from a large biomedical corpusand a classifier was thereafter trained to extract terms belonging to this semanticcategory using a set of known disease terms to train the SVM classifier. When ap-plying the obtained disease list for vocabulary-based NER, annotated entities wererecognised with a precision of 0.38 and a recall of 0.45. The approach of searchingfor language patterns expressing synonymous relations between terms is, however,not suitable for clinical corpora, as used terms are rarely defined or explained inthis sub-language. The extraction of terms of a certain semantic category withtext pattern-based approaches similar to the one used by Neelakantan and Collins(2014) is also not optimal for the clinical text genre. This method requires thatterms occur in conjunction with the name of the semantic category, which is likelyto occur less frequently in clinical texts than in biomedical texts in general.

A different approach for extracting terms, which is more suitable for clini-cal corpora, is to find contexts in which words typically occur (Biemann, 2005).This approach is not dependent on synonymy or hyper/hyponymy being explicitlystated in the text, but is based on the distributional hypothesis, which states thatwords which often occur in similar contexts also are likely to have a similar mean-ing (Sahlgren, 2008). If dermatitis and eczema often occur in similar contexts,e.g. “Patient complains of itching dermatitis” and “Patient complains of itchingeczema”, it is likely that dermatitis and eczema have a similar meaning. Modelsfor representing this idea are typically divided into probabilistic and spatial models(Cohen and Widdows, 2009). Brown clustering (Brown et al., 1992) is an exampleof a probabilistic method in which agglomerative hierarchical clusters are createdfor each term in a corpus based on the statistical similarity of their immediateneighbours. For the spatial models on the other hand, word co-occurrence infor-mation is given a geometric representation in the form of a semantic (word) space,in which semantic similarity is represented by geometrical proximity. This rep-resentation has been used both for creating clusters of semantically related words(Song et al., 2007) and for determining whether unknown words belong to pre-defined semantic categories (Widdows, 2003; Curran, 2005). These methods have

Background 23

also been applied in the biomedical domain for literature-based knowledge discov-ery and information retrieval (Cohen and Widdows, 2009).

3.4.2 Expanding vocabularies with abbreviations

There are a number of techniques for abbreviation dictionary construction thatautomatically extract abbreviations and their corresponding expansions, throughlocating text segments in corpora in which abbreviations are explicitly defined.Candidates for abbreviation-expansion pairs can, for example, be extracted byassuming that either the long form or the abbreviation is written in parentheses(Schwartz and Hearst, 2003) or by the use of other rule-based pattern matchingtechniques (Ao and Takagi, 2005). Extracted abbreviation-expansion pairs can befurther filtered by either rule-based (Ao and Takagi, 2005) or machine learning(Chang et al., 2002; Movshovitz-Attias and Cohen, 2012) methods. Most medicalabbreviation extraction studies have been conducted for English, but there are alsostudies in other language, e.g. Swedish (Dannélls, 2006; Isenius et al., 2012; Kvistand Velupillai, 2014). With these methods, it is possible to automatically constructdictionaries of high quality. A dictionary constructed using Swedish biomedicalcorpora achieved, for instance, a precision of 0.95 and a recall of 0.97 when eval-uated against manually annotated abbreviations and expansions (Dannélls, 2006).Yu et al. (2002) have, however, found that around 75% of all abbreviations presentin biomedical articles are never defined. Also, similar to text pattern-based ap-proaches for finding synonyms, these pattern matching methods are unsuitable forclinical texts, since used abbreviations are not defined (Broumana and Edvinsson,2014).

Automatic extraction of medical abbreviations that are explicitly defined in atext is, thus, well studied, but there are fewer studies on how to find abbreviation-expansion pairs that are not defined.

3.4.3 Random indexing

All methods used in this thesis for vocabulary expansion were based on extract-ing distributionally similar words from word space models built using Randomindexing. Random indexing is one version of the word space model, and, as isthe case for all word space models, it is a method for representing distributionalsemantics. The random indexing method was originally devised by Kanerva et al.(2000) to deal with performance problems (in terms of memory and computationtime) that were associated with LSA/LSI implementations at that time. Due to itscomputational efficiency, random indexing remains popular for building distribu-

24 Chapter 3.

tional semantics models on very large corpora, e.g. large web corpora (Sahlgrenand Karlgren, 2009) or Medline abstracts (Jonnalagadda et al., 2012).

A simple way of representing word co-occurrence information is to construct aterm-by-term co-occurrence matrix, i.e. a matrix of dimensionality w×w in whichw is the number of terms (unique semantic units, e.g. words) in the corpus. Eachterm is assigned a vector of dimensionality w, and each position in this vector (thecontext vector) represents a term in the corpus, containing the number of times thisterm occurs in the context of the term to which the context vector is assigned. Thecontext of a term is defined as n preceding and m following words in the corpus,denoted as a context window of (n+m) in this thesis. In the example “Patientcomplains of itching dermatitis”, using a context window of (1+ 1) would meanthat the words Patient and of are positioned in the context window of complains.

The context vectors of two terms can be compared as a measure of semanticsimilarity between them. A frequently used measure for this similarity is the cosineof the angle between the two vectors, the cosine similarity. This is computed asfollows (Anton and Rorres, 1994, pp. 127–134):

cos(θ) =u ·v‖u‖‖v‖

=

n∑

i=1ui · vi√

n∑

i=1(ui)2 ·

√n∑

i=1(vi)2

The large dimension of a term-by-term matrix can, however, lead to scalabil-ity problems for very large corpora, and the typical solution to this is to applydimensionality reduction on the matrix. In a semantic space created by latent se-mantic analysis, for instance, dimensionality reduction is performed by applyingSingular value decomposition (Landauer and Dutnais, 1997). Random indexing isanother solution, in which a matrix with a smaller dimension is created from thestart, using the following method, which was originally suggested by Kanerva etal. (2000) and which has been evaluated and further developed by, e.g. Sahlgrenet al. (2008). Each term in the corpus is assigned a unique representation, calledan index vector, with the dimensionality d (where d� w). Most of the elementsof the index vectors are set to 0, but a few randomly selected elements are set toeither +1 or -1 (Figure 3.3), often around 1–2% of the elements.

Each term in the data is also assigned a context vector, also of dimensionalityd. Initially, all elements in the context vectors are set to 0 (Figure 3.4). Thecontext vector of each term is then updated by, for every occurrence of the term inthe corpus, adding the index vectors of the neighbouring terms within the context

Background 25

1 2 3 ... d

... [0 0 1 ... 0]

complain: [0 0 0 ... 1]

itch: [0 1 1 ... 0]

patient: [-1 0 0 ... 0]

... [... ... ... ... ..]

word w [0 0 -1 ... 0]

Figure 3.3: Index vectors.

Context vectors at start (All zero)

1 2 3 ... d.... [0 0 0 ... 0]

complain: [0 0 0 ... 0]

itching: [0 0 0 ... 0]

patient: [0 0 0 ... 0]

... [0 0 0 ... 0]

Figure 3.4: The initial context vectors.

window (Figure 3.5). The size of the context window can have a large impacton the results (Sahlgren et al., 2008), and, for detecting paradigmatic relations (i.e.terms that occur in similar contexts rather than terms that occur together), a narrowcontext window (around 2+2) has been shown to be most effective.

The resulting context vectors of the random indexing space form a matrix ofdimension w× d. A term-by-term matrix can be seen as a matrix consisting ofcontext vectors, which have been accumulated through adding w-dimensional in-dex vectors, which have a single 1 in a different position for each term to whichthe index vector belongs. These index vectors of the term-by-term matrix are,therefore, orthogonal to each other, while the index vectors in the random index-ing space are nearly orthogonal. The resulting random indexing space is, thereby,an approximation of the term-by-term matrix, and the same similarity measuresbetween the context vectors can be applied (Figure 3.6).

26 Chapter 3.

Index vectors (never change)1 2 3 ... d

...

itching: [0 1 1 ... 0]

patient: [-1 0 0 ... 0]

...

___________________________________________________________

Context vectors 1 2 3 ... d

...complain: [-1 1 1 ... 0]

...

Figure 3.5: The updated context vectors.

There is also a version of Random indexing, called Random permutation, inwhich the position in the context window of a context term is also taken into ac-count. When adding an index vector to the context vector, the index vector iseither used as-is or rotated one step before it is added, depending on if the contextterm appears to the right or to the left of the target term. These vectors are calledDirection vectors. Alternatively, the index vectors of the surrounding terms canbe rotated additional steps, based on how far from the target term the surroundingterm is positioned. These vectors are called Order vectors, and vectors both to theright and to the left of the target term are rotated, but they are rotated in oppositedirections.

Both with and without Random permutation, the distance to the target term canbe modelled by giving the index vectors close to the target term a higher weightthan the more distant terms, before adding them to the context vector of the tar-get term. Apart from determining the weighting scheme and window size, thedimension of the vector space, as well as the number of non-zero elements, mustbe determined. In addition, it needs to be decided what pre-processing to applyto the text, e.g. whether the corpus is to be lemmatised and whether stop wordsare to be removed. To obtain semantic units, e.g. terms, on which to build theword space model, the text is typically tokenised, often by white space tokenisa-tion when it is possible (Ahltorp et al., 2014a). Semantic spaces can, however,also be constructed with, e.g. multiword terms as a semantic unit (Henriksson etal., 2013b).

One possible usage of a word space is to query the word space for a specificterm, with the aim of retrieving other terms that are close in the semantic space

Background 27

00

θ

Figure 3.6: Context vectors for terms in a hypothetical two-dimensional word space. The distance between to terms can forinstance be measured by cosine(θ).

(Henriksson et al., 2013a). Another possibility is to do additional processing of thevectors of the semantic space (Evangelopoulos et al., 2012), such as categorizationor clustering. The clusters can, for instance, be used for evaluating the quality ofthe word space model (Rosell et al., 2009) as well as for features for training, e.g.named entity recognition models (Pyysalo et al., 2013).

3.5 Clinical negation and uncertainty detection

Negations, as well as other types of factuality modifiers such as expressions of un-certainty, are frequent in the clinical text genre (Skeppstedt et al., 2011; Velupillaiet al., 2011), which makes negation and uncertainty detection an important com-ponent in clinical information extraction.

28 Chapter 3.

3.5.1 Negation and scope detection

The NegEx system (Chapman et al., 2001), which has been used in a number ofclinical information extraction studies, is a rule-based system that detects whethera mentioned clinical finding is negated. The system is built on lists of pre- and post-negation cues, and a clinical finding that follows or precedes a cue is classifiedas negated. The first version of NegEx classifies a clinical finding as negatedif it is positioned in the range of one to six tokens from a post- or pre-negationcue. This version achieved a recall of 0.824 and a precision of 0.845 on the taskof detecting negations in sentences containing a negation cue. Later versions ofNegEx, as well as some other negation detection systems (Elkin et al., 2005),use a list of termination terms instead, e.g. conjunctions, to limit the scope of acue. The NegFinder system is also built on negation cues and termination terms,which are classified into different groups and used as building blocks in a manuallyconstructed negation detection grammar (Mutalik et al., 2001).

Syntactic parsers have also been used for scope detection, many for detectionof scope in the BioScope corpus (Vincze et al., 2008), which is annotated for cuesand the scope of text that the cues affect. Huang and Lowe (2007) used the outputfrom a phrase structure parser, and Ballesteros et al. (2012) used the output from adependency parser for constructing grammars defining negation scope, while Zhuet al. (2010) and Velldal et al. (2012) trained machine learning models to detectscope given the parser output. For scientific texts in the BioScope corpus, Velldalet al. (2012) achieved a substantial improvement over the baseline, which used theentire sentence as the scope. For clinical texts in the BioScope corpus, however,the baseline method was slightly better, partly owing to parser errors in the clinicaldomain.

In addition to these parser-based machine learning models, there are a numberof other negation detection systems that use machine learning. Using the BioScopecorpus, Morante and Daelemans (2009) trained an IGTree model to detect negationcues and an ensemble classifier consisting of k-nearest neighbour, SVM and CRFsto detect their scope. Rokach et al. (2008) used an annotated clinical corpus forderiving text patterns that consisted of negation cues and the number of allowedwords between these cues and a diagnosis, as well as for deriving patterns of ex-pressions for diagnoses occurring in an affirmed context. These patterns were thenused as features for a cascade of decision trees. There is also a machine learning-based extension of NegEx, which uses decision trees and a Naive Bayes classifierto determine when the cue not indicates a negation (Goldin and Chapman, 2003).

Background 29

3.5.2 Modifiers other than negations

Apart from negation, there are other factuality modifiers that could be associatedwith a clinical finding mentioned in a health record text. The author could beuncertain of the existence of the clinical finding, e.g. since the used diagnosismethod has a low precision, or the clinical finding could refer to something thatwas observed in the past or in someone other than the patient (e.g. a relative).

To cover some of these cases, the extended NegEx has been further developedinto the Context system, which apart from detecting negations, also detects histor-ical and hypothetical clinical conditions and whether a condition is experienced bysomeone other than the patient (Chapman et al., 2007; Harkema et al., 2009).

The previously mentioned 2010 i2b2/VA challenge on extracting concepts, as-sertions and relations also included the task of assessing if a clinical finding was:

[...] present, absent, or possible in the patient, conditionallypresent in the patient under certain circumstances, hypotheticallypresent in the patient at some future point, and mentioned in thepatient report but associated with someone other than the patient(Uzuner et al., 2011).

Most of the top-ranked systems in the challenge used SVM for classifying en-tities into these assertion levels. Included in used features were either the outputof rule-based systems (as those described above) or vocabulary matching to listsof cue phrases (Uzuner et al., 2011). The best system (de Bruijn et al., 2011) wasmainly based on different combinations of SVMs, first using an ensemble clas-sifier for predicting the assertion level of each token in an identified entity and,thereafter, using a multiclass-SVM for determining the assertion level of the entireidentified entity. The results for the individual assertion classes were not given,but their best result, macro-averaged for all classes, was an F-score of 0.77. Thesecond best system used CRFs and a maximum entropy classifier combined witha rule-based system for identifying cues as well as their scope and assertion class(Clark et al., 2011). They were able to detect the assertion category Present with aprecision of 0.94 and a recall of 0.98, the category Absent with a precision of 0.95and a recall of 0.92 and the category Possible with a precision of 0.77 and recallof 0.53.

There are also a number of Swedish studies on clinical uncertainty. A CRFsmodel was trained on the Swedish clinical corpus Stockholm EPR Diagnosis Un-certainty Corpus1 to classify diagnoses into six factuality levels (Velupillai, 2011).

1See section 5.1.1.

30 Chapter 3.

For detection of the combined class Probably and Possibly Negative, the precisionwas 0.58 and the recall was 0.55, while the precision was 0.79 and the recall was0.60 for the class Certainly Negative. For the combined class Probably and Pos-sibly, the precision was 0.83 and the recall 0.72, while the precision was 0.83 andthe recall was 0.82 for the class Certainly Positive.

Chapter 4

Scientific method

This chapter describes the scientific research approach and research strategy thatwere used in the studies of this thesis.

4.1 Overall research approach

Design science is a research approach that is concerned with the study of howartefacts are developed and used, and how they can help solve practical problems(Johannesson and Perjons, 2012). As the research of this thesis aims at exploringhow artefacts can solve practical problems regarding the extraction of informationfrom clinical text, design science was assessed to be suitable as the overall researchapproach. The design science approach can be divided into five stages.

1) The Explicate Problem stage concerns the study of a practical problem, whichmight be solved by an artefact. There is, as mentioned in the introduction, valuablepatient information contained in the free text of a health record. This informationis difficult to access in a structured manner, which is a problem for systems thatpresent patient information to care givers or that mine for information. Previousresearch has mainly focused on the construction of artefacts for solving this prob-lem for English clinical text. The problem of valuable patient information beinghidden in the free text is, however, not limited to clinical texts written in English,and the artefacts of this thesis aim at solving this problem for a language closelyrelated to English. The nature and existence of this problem was derived fromprevious research.

32 Chapter 4.

2) In the Outline Artefact and Define Requirements stage, it is studied how thepractical problem can be solved with an artefact and what the requirements wouldbe for such an artefact. Studying what kind of information that should be extracted,and possibly also how it could be made available to stakeholders such as medicalresearchers and care givers, could be relevant activities for this stage. In this thesis,it was, however, assumed that the same type of information that has been consid-ered important to extract from English clinical text, should be equally relevant toextract from clinical text in other languages.

3) The Design and Develop Artefact stage includes designing the functionalityof the artefact that was outlined in the previous stage, as well as carrying out thepractical work of constructing the artefact. The artefacts constructed in the studiesof this thesis focused solely on the extraction of relevant information from the freetext, as opposed to also including components for making the extracted informa-tion available to the stakeholders. The constructed artefacts could, therefore, becharacterised as forming potential sub-components of an artefact that solves theexplicated problem.

4–5) In the Demonstrate Artefact stage, one case in which the developed arte-fact is used to solve the explicated problem is demonstrated, and, in the EvaluateArtefact stage, a more thorough evaluation is carried out. No separate artefactdemonstration was performed, but the focus was placed on an experimental eval-uation. As the artefacts constructed in this thesis are possible sub-components ofan artefact that could solve the targeted practical problem, they were accordinglyevaluated for their ability to solve these sub-problems.

Different types of design science studies put the main focus on different stages,and here the focus was put on development and evaluation. Each study was, there-fore, divided into two phases; a development phase and an evaluation phase.

4.2 Research strategy for the evaluation stage

The employed research strategy in the evaluation stage was experimental research,in the form of automatic evaluations against reference standards. This is a fre-quently used strategy for experimental evaluation of NLP components, before theyare mature enough for user-centred evaluations (Hirschman and Mani, 2003).

A drawback of using the experimental research strategy is that the external va-lidity is low, since the context in which the artefact is to be used might be differentfrom the context of the applied experimental setting (Johannesson and Perjons,2012). An alternative evaluation could, therefore, consist of including the devel-

Scientific method 33

oped components in larger artefacts aimed at solving a specific problem and eval-uating to what extent these artefacts are able to solve this problem in the contextthey are intended for. However, such an evaluation would: a) require large manualresources as it would most likely have to be a manual evaluation with potentialusers and b), due to a larger complexity, not be useful for drawing conclusionson implementation choices of included sub-components. Such a strategy would,therefore, be more appropriate for a later phase, when implementation choices forsub-components have been thoroughly studied by controlled and automated exper-iments (Friedman and Hripcsak, 1998).

The structure of the two design science phases applied here, i.e. first construct-ing and thereafter evaluating an artefact, is similar to the structure of the exper-imental Cranfield evaluation paradigm (Voorhees, 2002), which has been usedwithin information retrieval since the 1960s. In this paradigm, one or severalartefacts are first constructed and their performance is thereafter evaluated on con-structed test collections of documents. This approach makes it possible to evalu-ate different parameters of constructed artefacts, without requiring expensive userstudies for each evaluated parameter.

4.3 Alternative research approach

Standard empirical research is concerned with studying the world as it is, oftenwith the aim of explaining or predicting it (Johannesson and Perjons, 2012). Sci-ence involving the construction and study of artefacts would, therefore, be difficultto even describe as science with this standard view of empirical research, unlessthe constructed artefacts were used for studying the world as it is. Therefore, usingstandard empirical research would have required a change of focus for the studies,and more emphasis would then have been put on what already exists in the world,instead of on the constructed artefacts. Such a focus shift could, for instance, be:1) to more closely study the ability of human annotators to extract clinical find-ings from texts compared to the ability of automatic systems to perform this task;2) to study what effect the constructed artefacts would have on users accessingpatient information or 3) to study to what extent the information mined using theconstructed artefacts, e.g. relations between findings and disorders, can be verifiedon patients using clinical trials. The argument stated above against using realisticuser studies for evaluating constructed artefacts is, however, also a valid argumentagainst adopting any of these three foci. These foci are only relevant in a laterphase when suitable parameters and methods for constructing the artefacts already

34 Chapter 4.

have been determined and the performance of the artefacts has been assessed in anexperimental setting.

Chapter 5

Used and created corpora

This chapter describes existing corpora that were used in the studies of this thesis,as well as annotated corpora that were created as a part of the studies.

5.1 Used corpora

The Stockholm Electronic Patient Record Corpus is the main material used in thisthesis, but a few other corpora were also used.

5.1.1 The Stockholm Electronic Patient Record Corpus

The Stockholm Electronic Patient Record Corpus (Stockholm EPR Corpus), withclinical text written in the years 2006–2008 (Dalianis et al., 2009),1 contains healthrecords for more than 600 000 patients from over 900 health units in the Stockholmarea. The used and created annotated subsets in this thesis were all extracted fromfields with the sub-heading Assessment (Bedömning in Swedish). These fields areparticularly suited for studies of information extraction of clinical findings, as theycontain reasoning about findings as well as diagnostic speculations and, thereby,many instances of the entity classes Disorder and Finding. Assessment fields arealso interesting in that they form a prototypical example of the difficulties asso-ciated with conducting text processing on clinical texts, difficulties that form amotivation for studying information extraction from clinical texts separate frominformation extraction in general. Clinical texts are, for instance, often less com-

1There are also more recent versions of this corpus (Dalianis et al., 2012), which are not used here.

36 Chapter 5.

pliant with formal grammatical rules than are other types of texts (e.g. shown inFigure 5.1). The texts are written in a highly telegraphic language, and sentencesoften lack a subject, a verb and/or function words (Skeppstedt, 2013a; Smith et al.,2014) and contain many non-standard and ambiguous abbreviations and acronymsas well as technical words (Allvin et al., 2011; Smith et al., 2014). There can alsobe a considerable variation in how the same word is written, especially a word witha complex spelling. For instance, about 60 different versions of the word Nora-drenalin were found in a relatively small corpus of Swedish clinical text (Allvinet al., 2011). These differences have the effect that processing clinical texts canbe challenging for natural language processing tools that have been developed forother text types (Skeppstedt, 2013a).

Cirk och resp stabil, pulm ausk något nedsatt a-ljud bilat, cor RR HF 72, sat 91% på 4 l O2. Följer Miktionslissta. I samråd med <title> bakjour <First name> <Second name>, som bedömmer pat som komplicerad sjukdomsbild, så följer vi vitala parametrar, samt svara han ej på smärtlindring, så går vi vidare med CT BÖS.

[Circ and resp stable, pulm ausc somewhat weak resp sound bilat, cor RR HF 72, sat 91% on 4 l O2. Following list for micturation. Consulting <title> senior dr on call <First name> <Second name>, who aseses pat as complicated condition, so we follow vital parameters, and anwers he not to pain-relief, so we go on to CT ABD.]

Figure 5.1: An example from Grigonyte et al. (2014) of Swedishclinical text that includes misspellings (underlined), abbreviations(bold), and words of foreign origin (italics).

From the Stockholm EPR Corpus, several smaller sub-corpora have been ex-tracted for the purpose of creating annotated corpora. Two such sub-corpora wereused here:

1) The Stockholm EPR Sentence Uncertainty Corpus (Dalianis and Velupil-lai, 2010) consists of 6 740 randomly extracted sentences from assessment fields inthe Stockholm EPR Corpus. Three annotators labelled cue words for speculationand negation, as well as classified sentences (or clauses) as certain, uncertain orundefined. From this corpus, only lists created from extracting the annotated cueswere used.

2) The Stockholm EPR Diagnosis Uncertainty Corpus (Velupillai et al.,2011) consists of 1 297 assessment fields extracted from the Stockholm EPR Cor-pus, for which a list of 337 diagnosis terms were used to automatically identify

Used and created corpora 37

diagnoses in the text. Identified diagnoses had been manually classified by twosenior physicians into six classes: Certainly Positive, Probably Positive, Possi-bly Positive, Certainly Negative, Probably Negative and Possibly Negative. Theoverall agreement, measured by Cohen’s κ , was 0.73 (intra-annotator) and 0.60(inter-annotator).

5.1.2 Additional corpora

Apart from the Stockholm EPR Corpus, three additional corpora were used:1) A subset of Läkartidningen (1996–2005), the Journal of the Swedish Medical

Association (Kokkinakis, 2012), which has been made available for research. Thisweekly journal is written in Swedish and contains articles discussing new scientificfindings in medicine, pharmaceutical studies, health economic evaluations, work-related issues, etc. Due to copyright reasons, the sentences in the freely availablesubset appear in a randomised order.

2) The BioScope Corpus (Vincze et al., 2008), which contains clinical radiol-ogy reports and other English biomedical texts that have been annotated by twostudents for negation and speculation cues as well as their scope. The annotatedcues were extracted and translated from English to Swedish.

3) Swedish Parole (Gellerstam et al., 2000), which is a corpus of standardSwedish compiled from text types including newspapers, fiction, etc. This cor-pus was used for compiling a list of words occurring in a non-medical Swedishtext.

5.2 Corpora created by annotation

Two annotated corpora were created in the studies carried out for this thesis: theStockholm EPR Clinical Entity Corpus (Study I and IV) and the Stockholm EPRNegated Findings Corpus (Study V).

5.2.1 The Stockholm EPR Clinical Entity Corpus

As previously mentioned, clinical NER studies have typically used a general cate-gory, similar to the category Clinical Finding (Wang and Patrick, 2009; de Bruijn etal., 2011; Jiang et al., 2011), or have focused on an entity category similar to whatis here called Disorder (Kokkinakis and Thurin, 2007; Savova et al., 2010). Thereare, thus, few studies in which the categories Disorder and Finding are annotatedas two separate categories (Albright et al., 2013). For the Stockholm EPR Clini-

38 Chapter 5.

cal Entity Corpus, however, the general category Clinical Finding was divided intothe two more granular categories Disorder and Finding, following the semanticcategories of SNOMED CT (International Health Terminology Standards Devel-opment Organisation, IHTSDO, 2008b). Also, the category Body Structure wasannotated to capture clinical findings that include a part of the body, for instance,“pain in left knee”. The category Body Structure is defined in SNOMED CT as aphysical anatomical entity (International Health Terminology Standards Develop-ment Organisation, IHTSDO, 2008a). In addition, the entity category Pharmaceu-tical Drug was annotated.

Texts for annotation were compiled from the Stockholm EPR Corpus by ran-domly extracting 1 148 Assessment fields from an internal medicine emergencyunit at Karolinska University Hospital. The used definition of the annotated entitycategories can be summarised from Figure 2.1 in the introduction as follows:

1) A Disorder is a disease or abnormal condition that is not momentary and thathas an underlying pathological process.

2) A Finding is a symptom reported by the patient, an observation made by thephysician or the result of a medical examination. This includes non-pathologicalfindings with medical relevance.

3) A Pharmaceutical Drug is a medical drug that is either mentioned with ageneric name or trade name or with other expressions denoting drugs, e.g. drugsexpressed by their effect, with words such as painkiller or sleeping pill. Narcoticdrugs used outside of medical care were excluded.

4) A Body Structure is an anatomically defined body part, excluding body fluidsand expressions for positions on the body.

There are other entities that would also be useful to be able to extract fromclinical texts and which, thereby, would be relevant to include in the annotationscheme, e.g. the entity categories Procedure or Occupation. The categories Disor-der and Finding were, however, chosen since they are the most important entitiesfor describing the medical status of a patient, and the category Body Structure waschosen since entities of this category are sometimes important for specifying thelocation of a disorder or finding. The category Pharmaceutical Drug was chosen tobe able to associate clinical findings with pharmaceuticals, e.g. for the purpose ofdetecting adverse drug reactions. Other previously mentioned applications whenextracting these entities are comorbidity studies and syndromic surveillance. Thechosen entity categories are also among the most important in the construction oftools for patient history summaries and overviews.

Annotation guidelines were developed by a senior physician (PH1) and a com-putational linguist (CL). A test annotation of 664 Assessment fields (not included

Used and created corpora 39

in the 1 148 fields compiled for the final corpus) was performed by PH1, for train-ing as well as for development of the annotation guidelines. The final version ofthe guidelines was reviewed by a second physician (PH2).

The following are the most important points of the guidelines.2 The shortestpossible expression that still fully describes an identified entity was annotated.Modifiers that, for example, describe severity were, therefore, excluded, whilemodifiers describing the type of an entity were included. In the example “The pa-tient experiences a strong stabbing pain in left knee”,3 the words strong and leftwere, therefore, not annotated, whereas stabbing pain was annotated as a Findingand knee as a Body Structure. All mentions of any of the four selected classeswere annotated, regardless of whether, for example, a Disorder was referred towith an abbreviation or acronym or in a negated or a speculative context, or if theperson experiencing the Finding was someone other than the patient. The guide-lines also included rules for handling the frequent occurrence of compound words.Compound words were not split up into substrings, and, therefore, e.g., diabetesin diabetesclinic4 was not annotated as a Disorder, whereas the word heartdis-ease5 which is a compound denoting a Disorder, was annotated as such. A com-pound including a treatment with a pharmaceutical drug was, however, classifiedas belonging to the entity category Pharmaceutical Drug. The definition of thecategory Finding was broader than the definition used in other annotation studies,e.g. i2b2 shared task (i2b2/VA, 2010), and more closely followed the definitionin SNOMED CT, as also non-pathological, medically relevant findings were in-cluded. The guidelines did, however, agree with the i2b2 guidelines (i2b2/VA,2010) in that findings that were explicitly stated (e.g. high blood pressure) wereincluded, while test measures (e.g. blood pressure 145/95) were not.

PH1 had the role as the main annotator, annotating all notes included in thestudy. A subset of the notes was independently annotated by PH2, and yet anothersubset was independently annotated by CL (Table 5.1). To become familiar withthe annotation task, PH2 and CL carried out a test annotation on 50 notes. Neitherthese test annotations, nor the texts annotated by PH1 in the guideline developmentphase, were included in the constructed corpus. The annotation tool Knowtator(Ogren, 2006), a plug-in to Protégé, was used for all annotations. The doublyannotated notes were used for measuring inter-annotator agreement, as well as forconstructing a reference standard to use in the final evaluation. Inter-annotator

2The complete guidelines are available at http://dsv.su.se/health/guidelines/3In Swedish: “Patienten känner en kraftigt huggande smärta i vänster knä”4In Swedish: diabetesklinik5In Swedish: hjärtsjukdom

40 Chapter 5.

Table 5.1: Data used for measuring inter-annotator agreement be-tween the annotators PH1, PH2 and CL.

Annotator: PH1 PH2 CLEntity category Annotated (Entity Annotated (Entity Annotated (Entity

entities types) entities types) entities types)Disorder 766 (354) 329 (174) 355 (214)Finding 1 327 (715) 631 (466) 686 (461)Pharmaceuticals 636 (249) 282 (119) 262 (143)Body Structure 275 (112) 117 (67) 101 (65)

agreement was measured in terms of F-score, and disagreements were: entitiesannotated by only one of the annotators, differences in choice of entity category,and differences in the length of annotated text spans.

Out of a large subset (25 370 tokens) of the doubly annotated notes shown inTable 5.1, a sub-corpus (Stockholm EPR Clinical Entity Corpus Final Evaluationsubset) was created to use as a reference standard. This sub-corpus was compiledby PH1, who resolved each conflicting annotation in the doubly annotated data. Aprogram for presenting and resolving annotations was developed, which presentedpairs of conflicting annotations on a sentence level without revealing who had pro-duced which annotation. PH1 could, thus, select one of the presented annotationswithout knowing who had produced it, thereby minimising bias (Figure 5.2). Therest of the annotated corpus is called the Stockholm EPR Clinical Entity CorpusDevelopment subset (Table 5.2).

Figure 5.2: A simple program for choosing between two alterna-tive annotations, showing a constructed example in English.

Inter-annotator agreement was calculated, using the evaluation script from theCoNLL shared task (Conll, 2000). Agreement between the physicians (PH1 andPH2), between the main physician annotator and the computational linguist (PH1and CL) and the average results are shown in Table 5.3 for the four categories, as

Used and created corpora 41

Table 5.2: Number of annotated entities in Development subsetand Final Evaluation subset.

Data set: Development Final EvaluationEntity category Annotated entities (Entity types) Annotated entitiesDisorder 1 317 (607) 681Finding 2 540 (1 353) 1 282Pharmaceuticals 959 (350) 580Body Structure 497 (197) 253Tokens in corpus 45 482 25 370

well as for Disorder and Finding merged into one class. Annotations for the entitycategory Pharmaceutical Drug had the highest agreement for both pairs of anno-tators, and there was also a relatively high agreement for annotations of entitiesbelonging to the categories Body Structure and Disorder, while the agreement forthe category Finding was lower. Finding was also the only category for which therewas a large difference between the two pairs of annotators, with higher agreementbetween PH1 and CL than between PH1 and PH2. When automatically remov-ing all annotated Findings containing a numerical value for PH2 and CL (to makethem better comply with the guidelines), the agreement between the two physi-cians increased from 0.58 to 0.61, but it still remained lower than the agreementbetween PH1 and CL.

In the Development subset, there were a total of 27 unique entity types some-times annotated as a Disorder and sometimes as a Finding: this was 3% of the totalnumber of unique entity types among annotated Disorders and Findings. Amongthese, some were equally frequently annotated as a Disorder and as a Finding (e.g.stress and muscle pain), whereas some were more often classified as a Finding(e.g. dizziness/vertigo and tachycardia). In accordance with the annotation guide-lines, these entities were annotated as a Finding when they were symptoms ofanother disorder, whereas they were annotated as a Disorder when they were themain described medical problem.

5.2.2 The Stockholm EPR Negated Findings Corpus

The concept of negation is defined in SNOMED CT (IHTSDO, 2008) through thefollowing examples:

42 Chapter 5.

Table 5.3: Inter-annotator agreement scores. PH2 shows the inter-annotator agreement between PH1 and PH2. CL shows the inter-annotator agreement between PH1 and CL. Disorder + Findingshows the results of merging the two classes Disorder and Find-ing into one class, and -num. val. is the agreement when whenannotations with numerical values are removed for the class Find-ing. The average results were also measured.

PH2 CL AverageF-Score F-score F-score

Disorder 0.77 0.80 0.79Finding 0.58 0.73 0.66Pharmaceutical Drug 0.88 0.92 0.90Body Structure 0.80 0.80 0.80Disorder + Finding 0.72 0.84 0.78Finding -num. val. 0.61 0.73 0.67Disorder + Finding -num. val. 0.75 0.84 0.80

The meaning of some concepts in SNOMED CT depends conceptuallyon negation (e.g. “absence of X”, “lack of X”, “unable to do X”etc). (IHTSDO, 2008)

The Stockholm EPR Negated Findings Corpus consists of 900 sentences ex-tracted from the annotated sentences in the Stockholm EPR Uncertainty Corpus aswell as from surrounding sentences. The extracted sentences all contain a clinicalfinding that matches a textual description in ICD-10. The sentences were dividedinto two groups: 1) those containing known negation cues and 2) those not con-taining known negation cues. Each clinical finding was manually classified intoNegated, Affirmed or Uncertain, and the two groups Affirmed and Uncertain werethen collapsed into the group Not Negated. All clinical findings in sentences witha negation cue were classified by two persons (one of them a senior physician),while only a subset of the other clinical findings was classified by two persons(Table 5.4).

In group 1, the inter-annotator agreement with respect to the two categoriesNegated and Not negated was 0.874% (Cohen’s κ 0.745), and, for the 95 sentencesin group 2 that were categorised by two annotators, there was an inter-annotatoragreement of 100%.

Used and created corpora 43

Negated Not negated TotalGroup 1: Sentences with negation cues 269 289 558Group 2: Sentences without negation cues 12 330 342

Table 5.4: Sentences classified as negated and not negated (un-certain and affirmed).

5.3 Ethics when using health record corpora

The content of health records is often very sensitive, which makes it extremely im-portant that it is read by as few people as possible and that the identity of patientsis not unnecessarily revealed to researchers. Therefore, the health record datain Stockholm EPR Corpus has been provided for research in a format in whichstructured information that can identify patients, such as patient names and socialsecurity numbers, has been removed. Also, when extracting data for annotation,small extracts from health records from many patients have been randomly cho-sen, as opposed to using the entire available medical history of only one patient,thereby reducing the risk of involuntarily identifying patients from context. Thestudies using the Stockholm EPR corpus were conducted after approval from theRegional Ethical Review Board in Stockholm, permission number 2009/1742-31/5and permission number 2012/834-31/5.

44

Chapter 6

Extracting mentioned clinicalfindings

This chapter describes the studies that address the task of recognising mentionedClinical Findings in Swedish clinical text, as well as the task of building vocab-ulary resources. In each study, an artefact is first designed and developed andthereafter evaluated.

6.1 Study I: Rule-based entity recognition and coverageof SNOMED CT in Swedish clinical text

The first attempted approach for extracting clinical findings was to use existingSwedish vocabularies for searching for entities in the text. As this requires a mini-mum of manual annotation resources (only a reference standard for the evaluation),it is a suitable first step. The experiments of this study, therefore, aim at answeringthe following research question:

• With what precision and recall can Findings and Disorders, mentioned inSwedish health record text, be recognised by using hand-crafted rules formapping to existing Swedish vocabularies?

46 Chapter 6.

6.1.1 Design and development of the artefact

Existing Swedish medical vocabulary terms were matched to the clinical text todetect entities of the three semantic types, Disorder, Finding and Body Structure,corresponding to the annotated entity types in the Stockholm EPR Clinical En-tity Corpus. The text was first divided into tokens and sentences using Granska(Knutsson et al., 2003). An attempt to match the entire sentence to a vocabularyterm was then carried out, before dividing the sentence into increasingly shortertoken sequences and matching these sequences to the vocabulary. Each matchedtoken in a sequence was assigned the semantic category or categories to whichthe matched vocabulary term belonged, and when a token matched several seman-tic categories, its final semantic category was determined by priority rules corre-sponding to those of the annotation task (in which Body Structure had priorityover Finding and Disorder, and Disorder had priority over Finding). The vocabu-lary was stored in a hash table, which made it possible to quickly search for a largenumber of substrings.

The vocabularies employed for matching were SNOMED CT, ICD-10,Wikipedia: Projekt medicin (a Wikipedia list of names of diseases in Swedish,Wikipedia, 2012), Medical abbreviations and acronyms collected by Cederblom(2005) and the freely available version of MeSH.1 The lemmatised, as well as un-lemmatised, version of the text was matched to the vocabularies. SNOMED CTterms belonging to the semantic category Body Structure were also pre-processedby stop word filtering, since these terms contain re-occurring sub-phrases such asstructure of, which are not used when body parts are referred to in health records.The stop word list was assembled by collecting frequent tokens in SNOMED CTterms belonging to the category Body Structure. A match to SNOMED CT termsbelonging to the semantic categories Qualifier and Person was also performed andgiven priority over the other categories. This was motivated by the fact that quali-fiers were not included in an annotated entity and by that the set of SNOMED CTterms belonging to the category Body Structure includes terms that describe per-sons.

Attempts were also made to 1) match text chunks with a Levenshtein distance(e.g. Jurafsky and Martin, p. 697, 2008) of one from the original text chunk, 2)match all possible permutations of token sequences (only performed on token se-quences of a minimum length of two tokens and a maximum of five) and 3) applya compound splitter (Sjöbergh and Kann, 2004) developed for standard Swedishto all words that contained at least ten letters. If a word was split into smaller parts

1See appendix for more on used vocabularies.

Extracting mentioned clinical findings 47

by the compound splitter, each of these smaller parts was matched to the vocab-ularies and if any of them matched a term, it was assigned the category of thatterm. Versions of the word constituents with a Levensthein distance of one werealso matched. These three approaches, however, either had no effect or led to de-creased precision without any large positive effects on recall, and were, therefore,not used in the final version.

6.1.2 Evaluation of the artefact

The system was applied on a subset of the Stockholm EPR Clinical Entity Corpus,containing 26 011 tokens, in which annotator PH1 had annotated 2 342 entities(759 Disorder, 1 319 Finding and 264 Body Structure). Precision, recall and F-score were measured with the CoNLL 2000 shared task evaluation script (Conll,2000).

6.1.3 Results

For the category Finding, lemmatisation improved recall, while adding a match toSNOMED CT terms for qualifiers and persons slightly improved precision. Stopword filtering improved recall considerably for the recognition of entities belong-ing to the category Body Structure, while a match to SNOMED CT terms forqualifiers and persons further improved precision. For the category Disorder, onthe other hand, pre-processing had no or little effect, perhaps since disorders to alarger extent than findings are mentioned in the lemma form in the text. This could,however, also be an indication that lemmatisation needs to be adapted to the med-ical vocabulary, as findings might be described with standard Swedish vocabularyto a larger extent than disorders.

Although pre-processing improved results for two of the categories, disordersand findings were recognised with a relatively low F-score also after the pre-processing (Table 6.1). This shows that additional methods are needed for entityrecognition in Swedish clinical text.

Nr. Precision (95% CI) Recall (95% CI) F-ScoreDisorder 0.75 (± 0.04) 0.55 (± 0.04) 0.63Finding 0.57 (± 0.04) 0.30 (± 0.02) 0.39Body Structure 0.74 (± 0.05) 0.80 (± 0.05) 0.77

Table 6.1: Results for the rule-based recognition of entities.

48 Chapter 6.

6.2 Study II: Vocabulary expansion by semantic extrac-tion of medical terms

The relatively low recall achieved in Study I shows that a more extensive Swedishmedical vocabulary is needed to support vocabulary based named entity recog-nition as well as for making concept matching possible. To manually develop acontrolled vocabulary is, however, expensive and time-consuming, and methodsthat can support this process are therefore valuable. What specifically is requiredfor recognising the annotated entities is a larger vocabulary list of terms belongingto the category Clinical Findings, i.e. an expansion of the vocabulary resource withmore hyponyms to the concept Clinical Findings. It would be useful to achieve thiswith as little manual effort as possible. The experiments of Study II explore thepossibility of using Random indexing for decreasing the manual effort requiredfor vocabulary expansion. In addition to terms belonging to the semantic cate-gory Clinical Finding, terms belonging to the category Pharmaceutical Drug werestudied, with the following research question:

• What is the recall, among a limited set of candidate terms, for retrievingknown terms denoting Clinical Findings and Pharmaceutical Drugs by usingsimilarity to seed terms in a Random indexing space built on medical text?

6.2.1 Design and development of the artefact

A possible use-case for a semi-automatic system for vocabulary expansion wouldbe to let the system present a list of candidate terms to a user for possible inclusionin a vocabulary. The user (e.g. a medical lexicographer) then has to decide whichof the presented terms belong to the semantic category for which the user aims toadd new terms.

The general idea for extracting candidate terms belonging to the semantic cate-gories Clinical Finding and Pharmaceutical Drug was to use a set of known termsas seed terms, and to extract unknown terms occurring in contexts similar to theseseed terms.

The method for extracting such unknown terms used a Random indexing spacebuilt on the Läkartidningen Corpus. A list of candidate terms was generated fora semantic category by ranking the unknown terms in the corpus according totheir summed similarity to the seed terms of that semantic category. The summedsimilarity was calculated by summing the cosine of the angle (θu,s) between the

Extracting mentioned clinical findings 49

context vector of the unknown term (u) and the context vector of each term (s) inthe set of seed terms (S) of the semantic category in question.

summed similarity(u) = ∑s∈S

cos(θu,s)

The unknown terms were sorted according to their summed similarity score, andterms with the top n largest summed similarity score were retrieved as candidatesfor inclusion in the list of terms belonging to the semantic category.

Apart from the method using summed similarity, a method was attempted inwhich all occurrences the seed terms of a semantic category were replaced by acommon string denoting that category, before the corpus was used for creatingthe Random indexing space. Only results from the more successful of the twomethods, the summed similarity method, are, however, presented here.

The Läkartidningen Corpus was pre-processed by white-space tokenising andlower-casing the text, and a document break was inserted between sentences to en-sure that co-occurrence information was not collected across sentences. The cor-pus was not lemmatised, as inflected forms of medical terms may also be relevantcandidates for vocabulary expansion. Four different Random indexing spaces withdifferent context window sizes were created. The used window sizes were 1+1,2+2, 4+4 and 50+50, i.e. index vectors for the immediate preceding and the imme-diate following word were added to context vectors in the first Random indexingspace, and index vectors for the two immediate preceding and the two immediatefollowing words were used in the second created Random indexing space and soon. For all spaces, a vector dimension of 1 000 with ten non-zero elements (i.e.,1%) were assigned to the index vectors. When populating the context vectors, in-creasingly less weight was assigned to index vectors as the distance from the targetterm increased. As the contexts were not allowed to cross a sentence boundary, the50+50 window size was, in effect, a sentence-level context definition.

6.2.2 Evaluation of the artefact

As seed terms and as reference standard for the evaluation, terms from SwedishMeSH were used. For the semantic category Clinical Finding, terms belonging tothe Swedish MeSH category Disease or Syndrome and the MeSH category Signor Symptom were used, while the MeSH category Pharmacologic Substance wasused for Pharmaceutical Drug. MeSH terms occurring fewer than 50 times wereexcluded as seed terms, and as reference standard terms, since a certain numberof observations of a term is required for its context vector to be accurately posi-tioned in a Random indexing semantic space. The Random indexing spaces were

50 Chapter 6.

0

0.2

0.4

0.6

0.8

1

0 200 400 600 800 1000 1200 1400

Rec

all

Cut off (number of retrieved terms)

Drug CosAddFinding CosAdd

Figure 6.1: Recall values for different cut offs in form of retrievedterms.

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

Prec

isio

n

Recall

Drug CosAddFinding CosAdd

Figure 6.2: Precision (partially based on manual classification) vs.recall (automatically measured against the reference standard), cutoff 50–500.

Extracting mentioned clinical findings 51

constructed to only handle unigrams, and multiword terms were therefore also ex-cluded from used vocabularies. When rare and multiword terms had been removed,309 Clinical Finding terms and 181 Pharmaceutical Drug terms remained. Tomake it easier to compare the two semantic categories, 181 Clinical Finding terms(identical to the number of Pharmaceutical Drug terms) were randomly selected.These terms were divided into two stratified, equally large groups, a seed set andan evaluation set, in which the strata consisted of terms with similar frequencies inthe corpus. The seed terms represent terms that, in a real-world scenario, would beknown and already included in the vocabulary and, therefore, used as seed termsin the described method, while the terms in the evaluation set represent terms thatshould be added to the vocabulary.

Different window sizes were evaluated by using 10-fold cross-validation on theseed terms and measuring recall averaged for different retrieved list size cut-offs(from 50 to 1000, with a step size of 50). The best recall was obtained for thecategory Clinical Finding when using the Random indexing model with a contextwindow of 2+2 and for Pharmaceutical Drug when using a model with a windowsize of 1+1. These results decided what spaces to use in the evaluation. The entireset of seed terms were then used as query terms, and two ranked lists of corpusterms were generated with the above described summed similarity method: one listconsisting of candidates for Clinical Finding and one list consisting of candidatesfor Pharmaceutical Drug.

The primary evaluation was a fully automatic calculation of recall for the twoproduced ranked lists, by comparing them against the two evaluation sets, one forthe category Clinical Finding and one for the category Pharmaceutical Drug. Thissimulates the case in which a medical lexicographer would manually go throughtop x of the retrieved terms in search for new terms to add to the vocabulary. Todetermine to what extent retrieved terms belong to the expected semantic cate-gory, despite not being present in the evaluation set, a semi-automatic evaluationof precision among the 500 top-ranked terms was also performed. Retrieved termsclassified as Clinical Finding or Pharmaceutical Drug in MeSH or FASS2 wereautomatically classified as correct or incorrect (assuming that a known ClinicalFinding can never be a Pharmaceutical Drug and vice versa). The remaining termswere manually classified by a single annotator as belonging to the category or not.

2See appendix for more on used vocabularies.

52 Chapter 6.

6.2.3 Results

The ability of the methods to retrieve the terms in the evaluation set is shown inFigure 6.1, for cut-offs ranging from the top 50 retrieved terms to the top 1500.Recall against existing vocabulary terms was consistently higher for Pharmaceu-tical Drug than for Clinical Finding. Among the top 1000 retrieved terms, forinstance, a recall of 0.88 was obtained for Pharmaceutical Drug and a recall of0.53 for Clinical Finding. Precision was, however, higher for Clinical Findingthan for Pharmaceutical Drug (0.80 vs. 0.64 for the top 50 and 0.68 vs. 0.47 forthe top 100, Figure 6.2). The results for precision also indicate that many terms ofthe correct category were retrieved that were not among the vocabulary terms usedfor measuring recall. Although precision steadily decreases, for Clinical Finding itis still close to 0.4 for the top 500. To manually investigate 500 candidates wouldthus generate almost 200 new terms for Clinical Finding, in the hypothetical casethat the used seed terms had made up the entire available vocabulary of terms forthis category.

6.3 Study III: Synonym extraction and abbreviation ex-pansion with ensembles of semantic spaces

The other vocabulary expansion study did not, as Study II, aim at expanding vo-cabularies with more instances of terms belonging to a certain semantic category,but aimed at extracting synonyms to specific terms in an existing vocabulary. Thisis a more challenging task, but the resulting expansion is even more useful to a textmining application, as it makes it possible to map the extracted entities to a spe-cific concept in a controlled vocabulary. With an extensive resource of synonyms,the text mining application has the possibility to treat, e.g. allergy3 and hypersen-sitivity4 as the same concept or as two similar concepts, for instance when miningfor disorder-finding relations. The expansion of vocabularies also included ex-tracting abbreviations of terms (Exp→Abbr), as well as extracting the expandedforms of abbreviations (Abbr→Exp), which are as useful as synonyms for conceptmapping. The experiments of this study can, therefore, be used for answering thefollowing research question:

3allergi in Swedish4överkänslighet in Swedish

Extracting mentioned clinical findings 53

• What is the recall, among a limited set of candidate terms, for retrievingknown synonyms and abbreviations by using term similarity in a Randomindexing space built on medical text?

6.3.1 Design and development of the artefact

A possible use-case for a semi-automatic system for synonym vocabulary expan-sion would be to let the system present an existing term in the vocabulary, togetherwith a limited number of synonym and/or abbreviation candidates for this term.The users of the system, e.g. medical lexicographers, could then decide whetherthe candidates should be included in the controlled vocabulary as a synonym to anexisting term.

The method used for generating candidates for possible inclusion was, as inStudy II, Random indexing. A number of different Random indexing semanticspaces were created with different parameters in terms of window size, stop wordfiltering and whether Random permutation was applied or not. In addition, com-binations of Random indexing spaces were created on two different corpora: onemedical (from Läkartidningen) and one clinical (from the Stockholm EPR corpus).The output from different created semantic spaces was then combined to make useof the fact that different model types (regarding the use of Random indexing orRandom permutation, and regarding window size) capture slightly different as-pects of semantics, as well as to make use of the fact that models created fromcorpora of different genres capture differences in language use.

Simple post-processing rules were also devised for filtering out suitable abbre-viation/expansion candidates. These rules were built on the idea that abbreviationsare shorter than expansions and that the characters that make up an abbreviationare normally a subset of the characters in the corresponding expansion.5

The suggested synonym candidates for possible inclusion in a controlled vocab-ulary were obtained by retrieving the ten terms that were ranked as most similar toa given query term in the Random indexing space combinations. (Exemplified inFigure 6.3.) The choice of letting the limited number of candidates be representedby ten was based on an estimation of the number of candidate terms that would bereasonable to present to a lexicographer for manual inspection. If more candidateswere to be presented, recall might be improved, but this would also probably resultin decreased usability.

5Post-processing rules were also attempted on the synonyms, built on thresholds of similarity, butas these were unsuccessful, they will not be further discussed.

54 Chapter 6.

bihåleinflammation (”nasal-sinus- inflammation”)

Output terms Cosine similarity

lunginflammation 0.73 (”lung-inflammation”)

urinvägsinfektion 0.71 (urinary tract infection)

halsfluss 0.68 (tonsillitis)

öroninflammation 0.67 (otitis media)

halsinfektion 0.65 (throat infection)

pneumoni 0.61 (pheumonia)

sinuit 0.61 (sinusitis)

borreliainfektion 0.60 (borrelia infection)

öroninfektion 0.60 (otitis media)

influensa 0.59 (Influenza)

Regular random indexing space

Clinical corpus

Random permutation space

Regular random indexing space

Medical corpus

Random permutation space

Combination

Figure 6.3: Retrieving the ten most similar terms from a combi-nation of four semantic spaces.

Ten semantic spaces were created from the clinical corpus and another ten fromthe medical corpus. As a comparison, the two corpora were conjoined into onecorpus, and ten semantic spaces were also created from this conjoint corpus. Foreach of the three corpora, four regular Random indexing (RI) spaces were cre-ated, with the four context window size configurations: 1+1, 2+2, 4+4 and 10+10.Moreover, for each of the three corpora, six spaces, for which Random permuta-tion (RP) was applied, were created with the window size configurations: 1+1, 2+2and 4+4. Each configuration of the Random permutation space was created in twoversions: one in which stop words (sw) were retained and one in which they wereremoved. For all spaces, a dimensionality of 1 000 was used, with eight non-zeroelements in the index vectors: four 1s and four -1s. For the regular Random in-dexing models, the index vectors were given less weight with increasing distancefrom the target term, while no weighting was performed for Random permutation,and order vectors were used.

The output of the regular Random indexing space was combined with the out-put of the Random permutation space with the same window size. In addition, aRandom indexing semantic space with a large window size and a Random per-mutation space with a small window size were combined. The combination ofthe two spaces was performed by summing the cosine similarity scores from the

Extracting mentioned clinical findings 55

two spaces for each candidate term, and retrieving the ten terms with the highestsummed cosine similarity (Figure 6.3). An overview of created semantic spaces isshown in Table 6.2.

As a final step, the combination was taken one step further by including twospaces from the clinical corpus and two from the medical corpus. The spaces werecombined by summing the cosine similarity to the query term for each term in thecorpus according to each of the four created spaces and retrieving the top ten termswith the highest summed cosine similarity.6

For each of the 3 corpora, 10 semantic spaces were created:RI Spaces RI_20 RI_2 RI_4 RI_8RP Spaces RP_2 RP_2_sw RP_4 RP_4_sw RP_8 RP_8_sw

The created semantic spaces were combined in 10 different combinations:Identical window size RI_2+RP_2 RI_4+RP_4 RI_8+RP_8Identical window size, RI_2+RP_2_sw RI_4+RP_4_sw RI_8+RP_8_swstop wordsLarge window size RI_20+RP_2 RI_20+RP_4Large window size, RI_20+RP_2_sw RI_20+RP_4_swstop words

Table 6.2: Overview of experiments conducted with a single cor-pus. The configurations are described according to the followingpattern: model_windowSize. sw means that stop words are re-tained in the semantic space. For instance, model_20 means awindow size of 10+10 was used.

6.3.2 Evaluation of the artefact

The evaluation was mainly performed by measuring recall against synonyms andabbreviation-expansion pairs in existing vocabularies. One of the members in asynonym or abbreviation-expansion pair was given as the query term, with thehope of retrieving the other member of the pair among the candidates suggestedby the semantic space combination. The query term represented a term that, in areal world scenario, would already be included in the vocabulary, while the othermember of the pair represented a term that ought to be added as a synonym or anabbreviation/expansion. Recall for including the other member of a pair among

6Other methods for combining spaces were also attempted, but they will not be discussed here.Also results for the four-space combination for retrieving abbreviations are omitted, due to the smallevaluation set available for this task.

56 Chapter 6.

the top ten suggested candidates was measured, thereby simulating the case ofhaving a lexicographer manually examine ten new terms for possible inclusion inthe controlled vocabulary. The evaluation could be described as a version of a two-fold cross validation; one of the members of a pair first functioned as a query term,and the other as the expected reference standard term, and, thereafter, the reverseset-up was applied by using the expected term as the query term. There were,however, also cases when there were several synonyms/abbreviations/expansionsto a term, and, in these cases, all of them functioned as query terms in turn, and allof the others had to be included among the top ten candidates to achieve full recall.

The used vocabulary for synonyms was derived from the freely available partof the Swedish version of MeSH, as well as from the Swedish extension, whilethe vocabulary for abbreviations was derived from the medical abbreviations bookby Cederblom (2005). As the semantic spaces were constructed only to modelunigrams, all multiword expressions were removed from the reference standards,as were also hypernym/hyponym and other non-synonym pairs found in the UMLSversion of MeSH. Terms occurring only rarely in a corpus are often incorrectlymodelled by Random indexing spaces. Therefore, only terms that occurred atleast 50 times in the corpus, from which the semantic space was created, wereincluded in the reference standard. This resulted in a very small reference standardfor abbreviations/expansions for the experiment that included the medical corpus,and, therefore, only evaluations for extracting abbreviations/expansions from theclinical corpus will be included here.

The vocabularies were divided into a development and an evaluation set, withall members of a (synonym or abbreviation-expansion) pair/group in either the de-velopment or the evaluation set. Statistics for the used vocabularies are shown inTable 6.3. The development sets were used for measuring the recall of the combi-nation strategies shown in Table 6.2, as well as for designing post-processing rulesfor abbreviations/expansions. The semantic spaces that performed best in combi-nation for a single corpus for the synonym task (evaluated on the development setsfor the Clinical corpus and for the Medical corpus) would also be used in the sub-sequent combinations from multiple corpora (evaluated on the Clinical + Medicalcorpus). The combination strategies that resulted in the highest recall were thenevaluated on the evaluation sets.

There is a possibility that, among the ten suggested candidates, there are un-known synonyms (in the sense of not yet included in the vocabulary) or other se-mantically related words that also should be included in the vocabulary. In order toget an idea of how often this was the case, a small manual evaluation was also car-ried out on the synonym task. This was done by randomly selecting 30 query terms

Extracting mentioned clinical findings 57

Reference standards for the Clinical corpus Size 2 Cor 3 CorAbbr→Exp (Devel) 117 9.4% 0.0%Abbr→Exp (Eval) 98 3.1% 0.0%Exp→Abbr (Devel) 110 8.2% 1.8%Exp→Abbr (Eval) 98 7.1% 0.0%Syn (Devel) 334 9.0% 1.2%

Reference standards for the Medical corpus Size 2 Cor 3 CorSyn (Devel) 266 11% 3.0%

Reference standards for the Clinical + Medical corpus Size 2 Cor 3 CorSyn (Devel) 122 4.9% 0%Syn (Eval) 135 11% 0%

Table 6.3: Reference standards statistics. Size shows the numberof queries, 2 cor shows the proportion of queries with two cor-rect answers and 3 cor the proportion of queries with three (ormore) correct answers. The remaining queries have one correctanswer. Syn (Devel) for the Clinical corpus and Syn (Devel) forthe Medical corpus were used for finding the best combinationsbetween two semantic spaces created using a single corpus. Syn(Devel) for the Clinical + Medical corpus was used for finding thebest combinations of semantic spaces created from two corpora,while Syn (Eval) was used for determining the performance of thiscombination on unseen data. Only terms that occurred at least 50times in the corpus were included, and for the Clinical + Medicalreference standard, they had to occur 50 times in both corpora.

from the Clinical + Medical evaluation reference standard and manually inspectingthe ten candidate terms that were suggested by the best combination of semanticspaces (according to recall against the development reference standard). The can-didates were classified as a synonym, an antonym, a hypernym, a hyponym, analternative spelling or none of these.

6.3.3 Results

The results for the extraction of abbreviations and their expansions from the clin-ical corpus are shown in Table 6.4. For both tasks, a window size of 2+2 was the

58 Chapter 6.

most successful among the combination strategies.7 For the task of finding ab-breviations for an expanded form, however, using the output only from a Randompermutation space resulted in a higher recall than using the combined spaces. Incontrast, the post-processing was successful for both abbreviation tasks. The rulesfor post-processing, derived from the properties of the development set, are shownin Eq. 6.1 and 6.2.

Exp→ Abbr ={

True, if (Len < 5)∧ (Subout = True)False, Otherwise

(6.1)

Abbr→ Exp ={

True, if (Len > 4)∧ (Subin = True)False, Otherwise (6.2)

Subout : Whether each letter in the candidate term is present in the query term, in the same order

and with identical initial letters.

Subin: Whether each letter in the query term is present in the candidate term, in the same order and

with identical initial letters.

Len: The length of the candidate term.

For synonyms, the best results on the development sets were achieved whencombining the output of semantic spaces created using a small or a very smallwindow size for the Random permutation space and a very large window size forthe Random indexing space, when using the corpora separately. When buildingcombined spaces on the conjoint corpus, a medium sized window for both theRandom permutation and the Random indexing space was instead most successful.These combinations were, therefore, applied on the evaluation set, with resultsas shown in Table 6.5. The method of combining different spaces was clearlysuccessful for the synonym task, with a recall of 30% when building one spaceusing the corpus constructed by conjoining the clinical and the medical corpus anda recall of 47% when combining four different semantic spaces.

In the random sample of 30 query terms, for which the output from the DisjointCorpus Ensemble was manually classified, there were a total of 33 synonyms inthe reference standard and ten of these were included among the top ten candi-dates suggested by the Disjoint Corpus Ensemble. In the manual classification ofthe suggested candidates, however, 29 additional terms not occurring in the ref-erence standard were classified as synonyms. There were also 15 hyponyms, 14hypernyms, three spelling variants and three antonyms.

7A window size of 4+4 gave the same recall results for the task of finding expansions to abbrevia-tions.

Extracting mentioned clinical findings 59

Recall, Abbr→Exp Recall, Exp→AbbrRI Baseline 0.22 0.19RP Baseline 0.23 0.24Clinical Ensemble (RI_4+RP_4_sw) 0.31 (RI_4+RP_4_sw) 0.20+Post-Processing 0.42 0.33

Table 6.4: Recall (top 10) on the clinical evaluation set for thetask of finding expansions to abbreviations and abbreviations toexpanded forms. The best strategy for the Clinical Ensemble isshown in parentheses.

Recall, SynonymsBest results, one semantic spaceClinical space 0.29Medical space 0.18Conjoint corpus space 0.30Best results, two semantic spacesClinical ensemble (RI_20+RP_4_sw) 0.34Medical ensemble (RI_20+RP_2_sw) 0.33Conjoint corpus ensemble (RI_8+RP_8_sw) 0.40Best results, four semantic spacesDisjoint corpora ensemble(RI_20clinical+RP_4_swclinical + RI_20medical+RP_2_swmedical ) 0.47

Table 6.5: Results on the clinical+medical evaluation set, whenusing the combination strategies that were most successful on thedevelopment sets. Synonyms were extracted from spaces createdusing the clinical and the medical corpora as well as from combi-nations of spaces from these two corpora.

6.4 Study IV: Automatic recognition of disorders, find-ings, pharmaceuticals and body structures fromclinical text: An annotation and machine learningstudy

Since the vocabulary matching approach in Study I resulted in relatively low re-sults for the recognition of entities of the category Clinical Finding, the approachof training a machine learning model on annotated data was studied, and the re-sults were also compared to previous studies on English clinical text. As manuallyannotating text is resource demanding, and the large, manually annotated corpora

60 Chapter 6.

that are available for Clinical English cannot always be obtained for a smaller lan-guage, the performance of a model trained on a corpus of limited size was studied.In addition, the effect of separating Clinical Finding into two more granular cate-gories was more closely investigated than in Study I.

The experiments of this study resulted in an artefact in the form of a machinelearning model, which was used for answering the following three research ques-tions:

• With what precision and recall can Findings and Disorders, mentioned inSwedish health record text, be recognised by using conditional random fieldstrained on a limited-sized corpus of manually annotated clinical text?

• How is the performance affected when the machine learning methods thathave been successful for English Clinical Findings recognition are appliedon the task of recognising Clinical Findings in Swedish health record text?

• To what extent is it possible to separate the entity category Clinical Findinginto the two more granular entity categories Disorder and Finding?

6.4.1 Design and development of the artefact

The choice of machine learning system was the CRF implementation CRF++(Kudo, 2012), which has been used in a number of previous studies on detec-tion of entities in English clinical text. Also feature types that have been usedin previous studies for English were applied, including the output in form of thevocabulary-matching system developed in Study I. The used vocabularies were,however, extended by also including FASS, the Swedish extension of MeSH, aswell as a vocabulary list of general Swedish, compiled from the Swedish ParoleCorpus. Also, the used semantic categories were not restricted to those corre-sponding to the four annotated entity categories. Instead, all semantic classes inSNOMED CT and MeSH were used as feature values, and tokens matching theFASS vocabulary were given the feature value Pharmaceutical Drug (ul Muntahaet al., 2012). Tokens matching the descriptions of ICD-10 codes in chapter 1–17and 19 (except codes T36–T62.9, which list substances) were assigned the featurevalue ICD-10-disorder and tokens matching the descriptions of codes in chapter 18(listing symptoms and clinical findings) were assigned the value ICD-10-finding.Tokens not found in any of the medical resources, but in the vocabulary from Pa-role, were assigned the feature value Parole. As in Study I, a token matchingseveral semantic categories was assigned a feature value according to the priority

Extracting mentioned clinical findings 61

stated in the annotation guidelines, and the list of SNOMED CT terms belongingto the semantic category Body Structure was stop word filtered.

Another difference from the system used in Study I, was that a compoundsplitting which searched for constituents among the used vocabularies was im-plemented. Compound splitting was attempted for all words that were not foundin any of the used vocabularies. This was carried out by dividing words into amaximum of two word constituents, if at least one of the constituents was foundin one of the vocabulary lists and the other constituent was found when applyingfuzzy matching. The fuzzy matching, which used a Levenshtein string distanceof one, was motivated by the fact that Swedish compounds are sometimes con-structed with an “s” binding the two constituents, and sometimes constructed bythe removal or change of the last vowel in the first constituent. The compoundsplitting favoured two equally long constituents over one short and one long con-stituent. The minimum length of a constituent was set to four letters. If it waspossible to divide a token, the constituents as well as the semantic classes foundby the vocabulary matching for the constituents were used as features. Tokens thatwere neither found in a vocabulary, nor were compound words, were given thevocabulary feature value Unknown.

Granska (Carlberger and Kann, 1999) was also used for obtaining the lemmaform and part-of-speech of a token as features. A simple method for selectingfeatures was used, adding potentially useful features one by one. The followingfeatures were added, in the following order (Figure 6.4.):

1) The current token.2) The lemma form of the current token.3) The lemma form of surrounding tokens.4) The part-of-speech of the current token and surrounding tokens.5) The output of the vocabulary-based system for the current token and sur-

rounding tokens.6) Compound splitting for the current token.7) Orthographic features, i.e. initial upper case, all upper case or no upper case

for the current token.The use of an increasingly larger window size, as well as fuzzy vocabulary

matching using a Levenshtein distance of one, was evaluated for features (2)–(5). Features and window sizes that lead to an improved result were retained,whereas the others were not used. An improved result was a result for which anincreased macro-averaged F-score over the four entity categories was obtained.For all experiments, L2-regularisation was used, and 30-fold cross validation onthe Development subset was used for selecting features. To use 30-fold cross is

62 Chapter 6.

more time-consuming than to use standard 10-fold cross validation, but since alarger proportion of the Development subset is used as training data in each fold,it more closely resembles the situation that is being optimised for, i.e. that theentire Development subset is available as training data. As the final step in thedevelopment, the features giving the best results were used for training the finalCRF model on the entire Development subset of the corpus, producing the artefactof the study.

6.4.2 Evaluation of the artefact

The main evaluation consisted of evaluating the model on the previously unusedFinal Evaluation subset. Precision, recall and F-score were measured for the taskof recognising the four annotated entity types. As in Study I, the CoNLL 2000shared task evaluation script was used (Conll, 2000).

As the features were added incrementally, with each feature improving the re-sults being retained, figures from this evaluation could not determine how mucheach individual feature contributed to the results. To be able to study this, modelswere also constructed, in which one feature type at a time had been removed whileretaining all other features. 30-fold cross validation on the Development subsetwas used also for the evaluation of these models, which were constructed with thebest features, except the feature type whose contribution was to be evaluated.

An error analysis was carried out on classifications obtained when applying 30-fold cross-validation with the best features on the Development subset. A manualanalysis was carried out by PH1 for the classes Disorder, Drug and Body Structure,while only errors due to incorrect span and to confusion between classes weremeasured for Finding (due to the large number of instances for this class).

6.4.3 Results

The following features and window sizes gave the best average results in the fea-ture selection process:

1) Lemma forms for the current token and the previous token.2) The part-of-speech for the current token, the following token and the two

previous tokens.3) Vocabulary match for the current token and the previous token.4) The compound splitting features for the current token.5) Orthographic features for the current token.

Extracting mentioned clinical findings 63

Token Lemma POS Terminology Compound Orthographics Category

DVT dvt noun disorder - - all upper B-Disorder

patient patient noun person - - - O

with with preposition parole - - - O

chestpain chestpain noun finding chest pain - B-Finding Current

and and conjunction parole - - - O

problems problem noun finding - - - B-Finding

to to preposition finding - - - I-Finding

breathe breathe verb finding - - - I-Finding

Figure 6.4: Used features in a constructed example. Each rowcontains one token and each column feature values correspondingto that token. Data in boldface is used for predicting the label forchestpain (shown as a compound as in Swedish).

These features are illustrated with boldface in Figure 6.4. Fuzzy vocabularymatching, as well as larger window sizes for lemma, part-of-speech and vocabularymatch, gave slightly lower results.

As can be seen in Table 6.6, results were high above the baseline (i.e. the rule-based vocabulary matching) for all categories, except for Pharmaceutical Drug,which had the best baseline. Table 6.7 shows the extent to which each individualfeature type contributed to the best average results, by presenting the results whenone feature type at a time was removed, while all other features were retained. Thecurrent lemma was the most important feature for all categories, while vocabularymatching was the second most important. All other features played a minor role,except for lemmatisation, which was the second most important feature for theentity category Finding. All results are presented for a CRF++ hyper-parameterof 6, for which the best results were achieved when giving the parameter integervalues between 1 and 11 (with the best average F-score of 0.794 at 6 and the lowestF-score of 0.779 at 1).

The error analysis (Table 6.8) showed that confusions between entity categories(especially between Disorder and Finding), manual annotation errors, an incorrectspan and borderline cases (i.e. when it was not evident whether the CRF classi-fier or the human annotator was correct) were occurring error types among falsepositives as well as among false negatives. Abbreviations were also a source of

64 Chapter 6.

Table 6.6: The results for the four categories with the CRF modelusing 30-fold cross validation on the Development set, comparedto the results obtained by the vocabulary matching baseline.

Entity category and evaluated system Precision Recall F-ScoreDisorder: Baseline, i.e. vocabulary match 0.677 0.574 0.621Disorder: Best average CRF settings 0.853 0.762 0.805Finding: Baseline, i.e. vocabulary match 0.574 0.312 0.404Finding: Best average CRF settings 0.737 0.636 0.683Pharmaceutical Drug: Baseline, i.e. vocabulary match 0.918 0.702 0.795Pharmaceutical Drug: Best average CRF settings 0.900 0.832 0.865Body Structure: Baseline, i.e. vocabulary match 0.618 0.739 0.673Body Structure: Best average CRF settings 0.836 0.808 0.822

Table 6.7: The average results for all four categories, compared tothe average results for vocabulary match (macro average over thefour classes Disorder, Finding, Pharmaceutical Drug and BodyStructure). Results when removing one feature type at a time,while retaining all other features, are also shown, as well as theresults when using the inflected tokens instead of the lemmas.

Evaluated configuration Precision Recall F-ScoreBaseline: Vocabulary match 0.700 0.582 0.623Best average CRF settings 0.832 0.759 0.794

Without lemmatisation 0.823 0.741 0.780Best settings - current lemma 0.758 0.633 0.688Best settings - previous lemma 0.826 0.758 0.790Best settings - part-of-speech tagging 0.830 0.751 0.788Best settings - vocabulary match 0.848 0.688 0.759Best settings - compound splitting 0.837 0.742 0.786Best settings - orthographics 0.832 0.755 0.791

both false positives and negatives, as were compound words. A compound wordthat was incorrectly classified as belonging to an entity category had either, as oneof its constituents, a word belonging to that category (e.g. lyme-disease-test8 andliver-values9), or had, as one of its constituents, a word that frequently occurredin compound words of this category (e.g. COPD-treatment10, which was misclas-sified as a Drug because of the constituent treatment). Compound words werealso frequent among false negatives, both compounds of full-length words and

8In Swedish: borreliaprov.9In Swedish: levervärden.

10In Swedish: KOL-behandling.

Extracting mentioned clinical findings 65

Table 6.8: Results from error analysis. A relevant stem meansa stem that belongs to the category or that is frequent in wordsbelonging to that category. Location is a physical location, suchas the abbreviated word infection for department of infection. nmstands for not measured.

Classified by the model as:False positives, error types Disorder Finding Drug Body StructureAnnotated as disorder - 23 % 5 % 14 %Annotated as finding 48 % - 7 % 15 %Annotated as drug 1 % 3 % - 1 %Annotated as body structure 1 % 1 % 0 % -Manual annotation error 8 % nm 49 % 56 %Span error 12 % 13 % 0 % 1 %Borderline case (model and/or annotator is correct) 2 % nm 3 % 3 %Incorrect: Abbreviation 4 % nm 11 % 0 %Incorrect: Compound word with one relevant stem 11 % nm 8 % 7 %Incorrect: Location 5 % nm 0 % 0 %Incorrect: No other reason 9 % nm 17 % 1 %Total number of false positives(classification instances) 138 389 75 71

Annotated as:False negatives, error types Disorder Finding Drug Body StructureIncorrectly classified as disorder - 9 % 4 % 2 %Incorrectly classified as finding 31 % - 8 % 9 %Incorrectly classified as drug 1 % 1 % - 0 %Incorrectly classified as body structure 3 % 1 % 1 % -Manual annotation error 4 % nm 0 % 2 %Span error 10 % 18 % 5 % 5 %Borderline case (model and/or annotator is correct) 3 % nm 3 % 0 %Incorrect: Abbreviation 5 % nm 7 % 2 %Incorrect: Compound word 7 % nm 35 % 27 %Incorrect: Compound word with an abbreviation 7 % nm 8 % 0 %Incorrect: Lemmatisation failed 6 % nm 2 % 6 %Incorrect: Misspelling or spelling variant 5 % nm 2 % 2 %Incorrect: Jargon (a non-standard term version) 8 % nm 3 % 1 %Incorrect: No other reason 9 % nm 22 % 43 %Total number of false negatives(annotation instances) 292 808 155 93

66 Chapter 6.

Table 6.9: The final results of the NER model. The modelwas trained on the Development subset, using the best features,and thereafter evaluated on the Final Evaluation subset. Find-ing+Disorder shows the result for the two categories Finding andDisorder merged into one category. A 95% confidence interval iscalculated (Campbell et al., 2007, pp. 91–92, 94–96).

Precision Recall F-scoreDisorder 0.80 (± 0.03) 0.82 (± 0.03) 0.81Finding 0.72 (± 0.03) 0.65 (± 0.03) 0.69Drug 0.95 (± 0.02) 0.83 (± 0.03) 0.88Body Structure 0.88 (± 0.04) 0.82 (± 0.05) 0.85Disorder+Finding 0.80 (± 0.02) 0.76 (± 0.02) 0.78

compounds with an abbreviation as one of its constituents. Other false negativeswere inflected words, which the lemmatiser had failed to lemmatise, misspellingsor spelling variants and entities expressed with jargon or non-standard language.Among the false negatives for the category Drug grouped into Incorrect: No otherreason were expressions for groups of medicines, for instance Cortisone.

Training a model on the entire Development subset using the best settings, andevaluating it on the held-out Final Evaluation subset resulted in the figures shownin Table 6.9. All categories were recognised with an F-score that is in line with theaverage inter-annotator agreement scores, and the final results were also very closeto those achieved on the Development subset during feature selection, showing thatresults obtained during development generalise well on unseen data.

Chapter 7

Negation and uncertaintydetection

This chapter describes the two studies that address the task of determining whichof the extracted clinical findings are negated or expressed with uncertainty. Ineach study, an artefact is first designed and developed and thereafter evaluated.

7.1 Study V: Negation detection in Swedish clinicaltext: An adaption of NegEx to Swedish

As previously described, there are a number of different approaches for negationdetection in clinical text. Despite its simple approach, the English rule-based nega-tion detection system NegEx has shown relatively good results compared to ap-proaches built on large corpora of annotated text. Therefore, when implementingclinical negation detection for similar languages, an adaption of this system is anatural first step, as evaluation data is the only annotated text required. Experi-ments were, therefore, carried out to answer the following research questions:

• With what precision and recall can an English hand-crafted rule-based sys-tem detect negation when adapted to Swedish clinical text?

• What is the coverage of a Swedish vocabulary of cue terms for negationobtained from translating English negation cues?

68 Chapter 7.

7.1.1 Design and development of the artefact

The English NegEx system (Chapman et al., 2001) uses regular expressions andthree lists of cue phrases: 1) the pre-negation list consists of cue phrases that cannegate a following clinical finding, for example no signs of ; 2) the post-negationlist consists of cue phrases that can negate a preceding clinical finding, for exampleunlikely; and 3) the pseudo-negation list consists of phrases that contain negationcues, but which should not trigger a negation, for example not only. The clinicalfinding is classified as negated if it occurs in the same sentence as a pre- or post-negation cue and is positioned in the range of one to six words after/before thecue.

The adaption to Swedish was carried out by translating English negation cues,through the use of Google translate and a dictionary, and by generating inflectionsfor the translations using the Granska inflector. A total of 148 cue expressionsfrom NegEx version 2 (NegEx, 2009) were translated. To make the adaption morecomparable to the previously evaluated English version, which only contained alimited number of cues, only the 42 translated cue expressions that were mostfrequent in a subset of the Stockholm EPR Corpus were included in the Swedishversion of NegEx. This was six more cues than included in the English version, tocompensate for the fact that Swedish is more inflective.

7.1.2 Evaluation of the artefact

The adapted system was evaluated using the Stockholm EPR Negated FindingsCorpus as the reference standard, using the two categories Negated and NotNegated (where Not Negated was a combination of the two annotated categoriesAffirmed and Uncertain). This corpus was also used for evaluating the complete-ness of the 42 used negation cues, together with a comparison to manually anno-tated negation cues in the Stockholm EPR Uncertainty Corpus.

7.1.3 Results

The recall for the Swedish adaption of NegEx was very close to the recall for theEnglish system, while the precision was lower (Table 7.1). The lower precisioncan partly be attributed to the fact that two of the translated negation cues, icke1

and utan2, were not entirely suitable as negation cues for Swedish, since utan isalso a conjunction meaning but, and icke mostly occurs in disease names such

1non-, not in English.2without in English.

Negation and uncertainty detection 69

as icke allergisk astma.3 When removing the cue icke and including rules fordisambiguating the cue utan, the precision was increased to 0.779.

That negation is expressed in Swedish, as in English, with a limited set of cuephrases was indicated by the fact that 76% of the negated clinical findings, thatwere correctly identified, were negated by the four most frequently occurring cuephrases. In the used reference standard, there were no frequent negation cues, apartfrom cues included in the cue phrase list. The only re-occurring cues that were notincluded in any of the three lists were two other forms of the phrase rule out thanthe form included in the cue list. Nor did the list of manually annotated cues de-rived from the Stockholm EPR Uncertainty Corpus contain any frequent negationcues that were not obtained through the translation of English cues. However, twoless common re-occurring manually annotated cues, both meaning nothing, werenot included. There were, however, also two useful negation cues, förnekar4 andavsaknad av5, that were obtained through the translation of English cues, but notthrough the manual annotation of Swedish clinical text.

A performed error analysis showed that to automatically distinguish uncertaintyfrom negation was the most difficult aspect in the detection of negation. This issupported by the fact that 20% of the sentences that were incorrectly classified byNegEx as negated were rated as uncertain by the annotator and, thereby, groupedas not negated. These sentences all contained phrases expressing uncertainty, suchas no evident, not certain or no real noticeable.

Sentences with negation cues English Swedish (95% CI)Recall 0.824 0.819 (0.773 - 0.864)Specificity 0.825 0.747 (0.696 - 0.797)Precision 0.845 0.752 (0.702 - 0.801)Negative predictive value 0.802 0.814 (0.767 - 0.861)

Sentences without negation cues English Swedish (95% CI)Negative predictive value 0.970 0.965 (0.945 - 0.986)

Table 7.1: Results for the Swedish adaption of NegEx. Resultsfor English from Chapman et al. (2001).

3non-allergic asthma in English.4denies in English5absence of in English

70 Chapter 7.

7.2 Study VI: Cue-based assertion classification forSwedish clinical text – developing a lexicon for Py-ContextSwe

As the error analysis of Study V showed that distinguishing uncertainty from nega-tion was the most difficult aspect of the rule-based negation detection, the next stepwas to also explore rule-based uncertainty detection. There is an English extensionof NegEx, called pyContextNLP (Chapman et al., 2011), in which uncertainty isalso included. In Study VI, this system was adapted to Swedish and evaluated;thereby, experiments were carried out to answer the following research questions:

• With what precision and recall can an English hand-crafted rule-based sys-tem detect negation and uncertainty when adapted to Swedish clinical text?

• What is the coverage of a Swedish vocabulary of cue terms for uncertaintyobtained from translating English uncertainty cues?

7.2.1 Design and development of the artefact

The general approach of pyContextNLP is very similar to that of NegEx, but in-stead of using the distance to the cue for determining its scope, a list of cue ter-mination phrases are used. The scope of a pre-cue is, then, the entire part of thesentence following the cue, and the scope of a post-cue is the entire part precedingthe cue, unless there is a termination phrase ending its scope.

The Swedish adaption of the system aimed at classifying clinical findings intofour different factuality categories: Definite Existence, Probable Existence, Prob-able Negated Existence and Definite Negated Existence. Swedish cues for thesecategories were obtained from seven different sources:

1) Translated cues from the English version of pyConTextNLP.2) Translated cues from the clinical part of the English BioScope Corpus.3) Manually annotated cues from the Stockholm EPR Sentence Uncertainty Cor-

pus.4) Cues derived from Swedish SNOMED CT terms expressing negation and

uncertainty.5) Inflections generated with the Granska inflector from cues obtained through

sources 1–4.6) Automatically generated cue combinations from cues obtained through

sources 1–5.

Negation and uncertainty detection 71

7) Cues obtained from error analysis on annotated data.To limit the lexicon to useful cues, cues not included in an extracted subset of

the Stockholm EPR Corpus (consisting of 48 512 sentences containing an auto-matically matched clinical finding) were excluded from the lexicon. With the helpof the same extracted corpus subset, the remaining cues were used for generatingcue combinations. This was carried out by first turning all cues into one-wordtokens by splitting multi-token cues into separate tokens, and then generating allpossible permutations of all tokens (allowing for a maximum sequence of five to-kens). The generated cue combinations that occurred at least seven times in theextracted corpus subset were added to the cue lexicon.

Collected cues were manually classified for assertion level and directionalityby two persons, and a third person resolved disagreements. A cue could also beclassified as a pseudo negation or a termination phrase.

7.2.2 Evaluation of the artefact

Two subsets of the Stockholm EPR Diagnosis Uncertainty Corpus were created touse as a Development set and an Evaluation set. In both of these two subsets, thesix originally annotated assertion classes were mapped to the four classes targetedin this study (Table 7.2).

First, the performance of the system only using cues translated from pyCon-textNLP (source 1) was measured on the Development set. Thereafter, the De-velopment set was used for measuring the performance of the system when cuesfrom other external resources and from the inflection generation were also added(sources 1–5). Cues from the generation of combinations were then added to thecue lexicon, and, as a result of an error analysis on the output given when run-ning the system on the Development set, more cues were added, and existing cueswere modified. This final cue lexicon (sources 1–7) was then evaluated on thepreviously unused Evaluation set.

7.2.3 Results

The results show that when including uncertainty, it is not enough to translatecues from the English PyContextNLP system to obtain a high-coverage cue lex-icon. Cues from other external resources, as well as from generated inflectionsand combinations, improve the result, as does correcting and adding to the cue setthrough an error analysis of results from annotated data (Table 7.3).

An error analysis showed that missing cues and borderline cues, in particularfor the distinction between Definite Negated Existence and Probable Negated Ex-

72 Chapter 7.

Gold Devel Eval Mapped Devel Evalcertainly positive 1291 780 definite existence 1291 780probably positive 442 261 probable existence 767 468possibly positive 261 170possibly negative 64 37probably negative 240 130 probable negated existence 240 130certainly negative 276 147 definite negated existence 276 147total 2574 1525 2574 1525

Table 7.2: Number of classified clinical findings in the used datasets (original annotation classes and mapped output values).

istence, were the two most common causes of misclassifications. It was the dis-tinction between the two classes Definite Negated and Probable Negated that wasmost problematic for the adapted system, and results for these two classes are muchlower than for the combined class Uncertainty_no (as can be seen when compar-ing Tables 7.3 and 7.4). Additional error sources were incorrect scope detections,annotation mistakes and misspelled cues.

Experiments were also conducted when only the most frequently occurring cuesof each category were used, showing that results for the negated classes startedto stabilise already when using the ten most frequent cues, and only changedmarginally after adding more than the 30 most frequent. For Probable Existence onthe other hand, results continued to improve when more cues were added, whichcan probably only be partly explained by the fact that there were more sentencesin the data belonging to the category Probable Existence.

Negation and uncertainty detection 73

Class Cue set Cues Precision (95% CI) Recall (95% CI) F-scoredef_existence baseline (on Devel) 5 (None) 67.83 ±1.8 90.01 ±1.16 77.36

extended (on Devel) 30 (12) 84.83 ±1.39 84.9 ±1.38 84.86final (on Devel) 33 (15) 91.5 ±1.08 90.09 ±1.15 90.79

final (on Eval) 33 (14) 88.8 ±1.58 87.44 ±1.66 88.11prob_existence baseline (on Devel) 87 (24) 62.15 ±1.87 25.68 ±1.69 36.34

extended (on Devel) 224 (92) 71.68 ±1.74 69.62 ±1.78 70.63final (on Devel) 351 (162) 82.0 ±1.48 81.36 ±1.5 81.68

final (on Eval) 351 (106) 81.6 ±1.94 80.56 ±1.99 81.08prob_negated baseline (on Devel) 91 (10) 39.29 ±1.89 9.17 ±1.11 14.87

extended (on Devel) 57 (18) 56.12 ±1.92 22.92 ±1.62 32.55final (on Devel) 140 (62) 55.81 ±1.92 60.0 ±1.89 57.83

final (on Eval) 140 (27) 54.48 ±2.5 56.15 ±2.49 55.3def_negated baseline (on Devel) 98 (20) 48.98 ±1.93 86.59 ±1.32 62.57

extended (on Devel) 59 (24) 53.08 ±1.93 84.42 ±1.4 65.18final (on Devel) 100 (45) 70.07 ±1.77 72.1 ±1.73 71.07

final (on Eval) 100 (31) 60.25 ±2.46 65.99 ±2.38 62.99

Table 7.3: baseline = corrected, translated cues from PyCon-TextNLP, extended = added, modified cues from external re-sources. final = added, modified cues from error analysis andgenerated combinations. Cues = cues in lexicon (number of cuespresent in the data). 95% CI = 95% Confidence Intervals.

Class Cue set Precision (95% CI) Recall (95% CI) F-scoreexistence_yes extended (on Devel) 96.12 ±0.75 95.14 ±0.83 95.63

final (on Eval) 97.72 ±0.75 96.31 ±0.95 97.01existence_no extended (on Devel) 81.38 ±1.5 84.69 ±1.39 83.0

final (on Eval) 84.41 ±1.82 89.89 ±1.51 87.06uncertainty_yes extended (on Devel) 76.75 ±1.63 64.25 ±1.85 69.95

final (on Eval) 78.52 ±2.06 78.26 ±2.07 78.39uncertainty_no extended (on Devel) 79.2 ±1.57 87.49 ±1.28 83.14

final (on Eval) 86.01 ±1.74 86.19 ±1.73 86.1

Table 7.4: Results when using two binary classes: existenceyes/no and uncertainty yes/no. extended = added, modified cuesfrom external resources. final = added, modified cues from erroranalysis. 95% CI = 95% Confidence Intervals.

74

Chapter 8

Discussion and conclusions

This chapter discusses the results in relation to the posed research questions andthe overall aims, as well as the main contributions of the thesis.

8.1 Recognising mentioned clinical findings

For the task of recognising mentioned entities belonging to the category ClinicalFindings, the application of techniques previously used for English clinical find-ings extraction was studied. Also, the usefulness of medical vocabulary resourcesfor this task was studied, as well as the possibilities for expanding these resources.Another topic included in the task of extracting mentioned clinical findings was tostudy the two more granular entity categories Disorder and Finding.

8.1.1 Study I

The first study investigated the precision and recall with which Findings and Dis-orders, mentioned in Swedish clinical text, can be recognised by using mapping toexisting Swedish vocabularies.

A number of pre-processing strategies were applied for automatically mappingthe content of Swedish assessment field text to existing Swedish medical vocabu-laries. This resulted in the entities of the category Disorder being recognised witha precision of 75% and a recall of 55% and entities of the category Finding be-ing recognised with a precision of 57% and a recall of 30%. It can, therefore, be

76 Chapter 8.

concluded that mapping to existing vocabularies lead to relatively low recognitionresults, in particular for the entity category Finding and especially for recall.

Savova et al. (2010) achieved higher precision as well as higher recall (Figure8.1), when recognising the entity category Disorder using mapping to English vo-cabularies. This is an expected difference, since English vocabulary resources aremore extensive than medical vocabulary resources available for Swedish. Thereis, however, also a previous Swedish vocabulary mapping study (Kokkinakis andThurin, 2007), which not only outperforms Savova et al. (2010) for recognisingentities of the category Disorder, but also outperforms machine learning models,trained on English corpora, for recognising entities of the category PharmaceuticalDrug. This previous Swedish study was, however, purely conducted on dischargesummaries, in which a more formal language is used, and in which terms that con-form to the terms present in Swedish vocabulary resources might have been used toa larger extent. The lower results achieved in Study I, compared to in the previousSwedish study, might, therefore, be due to that recognising entities in assessmentfields is a more challenging task than recognising entities in discharge summaries.

The achieved results clearly show that existing Swedish medical vocabular-ies are not sufficient for recognising clinical findings in assessment fields, evenwhen applying pre-processing rules. This conclusion has the implications that a)Swedish medical vocabularies need to be expanded to enable vocabulary basednamed entity recognition with high recall, as well as to enable mapping to vocab-ulary concepts, and that b) mapping to existing vocabularies does not automati-cally result in a high precision named entity recognition, and, therefore, additionalmethods for recognising clinical findings in Swedish health record text need to beexplored, even if larger vocabulary resources would be available.

8.1.2 Study II

Here, the first implication of Study I was addressed by the investigation of semi-automatic expansion of vocabularies with terms belonging to certain semantic cat-egories. A set of known terms belonging to the categories Clinical Finding andPharmaceutical Drug was used as reference standard, and with what recall it waspossible to retrieve these terms, among a limited set of candidate terms, was in-vestigated. The candidate terms were retrieved by using similarity to a set of seedterms in a Random indexing space built on a medical corpus.

It could be concluded that more than half (recall 53%) of the 90 expected termsfor the category Clinical Finding were found among the top 1 000 best candi-date terms, retrieved by using similarity to the seed terms in the Random indexing

Discussion and conclusions 77

space. When using the same method for retrieving terms of the entity categoryPharmaceutical Drug, the recall for retrieving the 90 expected terms was 88%.

Comparing the exact achieved recall figures to other studies is difficult, as theachieved results, to a large extent, are dependent on what reference standards, seedterms and corpora are used. This is clearly illustrated by the two categories com-pared here. The relatively high recall results achieved for the category Pharmaceu-tical Drug can probably be explained by the fact that the used reference standard ismore complete for this category and that the relevant terms in the candidate list forPharmaceutical Drug, therefore, more often are included in the reference standard.For the category Clinical Finding, on the other hand, there are more relevant termsnot included in the used reference standard, and, thus, more terms among the re-trieved candidates that belong to the semantic category Clinical Finding withoutcontributing positively to the recall. This was shown by the manual inspection ofthe candidate terms for Clinical Finding, as there were many terms among the sug-gested ones that belonged to the correct semantic category, without being amongthe expected ones used for measuring recall (resulting in a precision of 80% forthe top 50 and 68% for the top 100, for Clinical Finding, but in lower figures forPharmaceutical Drug). Precision after manual inspection could be a better evalua-tion metric when comparing different studies, as it is not dependent on the qualityof the used reference standard, but the drawback is, instead, a potential bias intro-duced by the manual inspection (and used corpus and seed terms still affect theresults).

The study by Neelakantan and Collins (2014) had an objective similar to StudyII, i.e. to expand a vocabulary of terms belonging to certain semantic categoriesrelevant for medical information extraction (including the category Disease). Theretrieved candidate terms were evaluated through their usefulness for named entityrecognition in biomedical text, by using them as the vocabulary in a system similarto the one used in the first study of this thesis. Neelakantan and Collins obtaineda precision of 38% and a recall of 45% for recognising the entity category Dis-ease, but the fact that they used a more application oriented evaluation method,instead of evaluating their approach against existing vocabularies, introduces yetanother obstacle for comparing their results to those achieved here. Their obtainedresults are, however, useful for showing that fully automatic generation of vocabu-laries for recognition of medical named entities is a difficult task, and that currentapproaches result in low precision as well as low recall.

As mentioned in the background, the approach by Neelakantan and Collins(2014) is not optimal for clinical corpora, as it requires that the semantic cate-gory of a term is explicitly mentioned in conjunction with the term. Therefore,

78 Chapter 8.

more generally applicable methods, such as the Random indexing method appliedhere, need to be explored.

The manual evaluation of the used Random indexing approach was carried outfor the top 500 candidate terms. Among the top 500 candidates for Clinical Find-ing, almost 200 relevant terms were found, which is a higher frequency of uniquerelevant terms denoting Clinical Findings than what was found when annotatingthe clinical text for the studies of this thesis. Therefore, at least for these top 500candidates, the study showed that for the task of finding new terms for inclusionin medical vocabularies, the used Random indexing approach is more efficientwhen it comes to manual resources than is manual corpora annotation. The studydoes not, however, aim to show that this application of Random indexing is moresuitable for the task than any other method based on distributional semantics. In-stead, the results show at least with what recall distributional semantics can beused for retrieving terms of semantic categories relevant for clinical informationextraction. Using other types of distributional semantics methods, other configu-rations of Random indexing or other types of processing of the Random indexingspace, e.g. clustering the context vectors, are all examples of methods, which havepotential for further improving the results.

8.1.3 Study III

The next step was to explore the possibility of using distributional semantics (in theform of Random indexing) for the task of semi-automatically extracting synonymsand abbreviations. That is, instead of finding hyponyms to a certain hypernym, asin Study II, different instantiations of the same semantic concept were searchedfor. In addition to using a medical journal corpus, as in Study II, the methodswere also applied on a clinical corpus, and different configurations of the Randomindexing space were evaluated. Similar to Study II, existing vocabulary resourceswere used as reference standards; here in the form of MeSH synonym pairs andknown abbreviation-expansion pairs. Using these existing vocabulary resources,it was investigated with what recall known synonyms and abbreviations could befound in a limited set of candidate terms. These candidate terms were retrievedby using combined term similarity from Random indexing spaces built on clinicaland medical corpora.

A list of 10 candidate terms was obtained for each query word by retrievingthe 10 most similar words when combining different semantic spaces. When fourmodels were combined (two types of models, built on two types of corpora: aclinical corpus and a medical corpus), 47% of the expected synonyms were found.

Discussion and conclusions 79

For abbreviations, using a combination of two models constructed from the clin-ical corpus, 42% of expected expansions were found when querying the modelswith abbreviations, and 33% of the expected abbreviations were found when usingexpansions as query terms.

Similar to Study II, it is difficult to compare the recall figures achieved here tothose of previous studies, as results are dependent on used materials and evalua-tion strategies. For instance, the figures obtained by McCrae and Collier (2008)for medical synonym extraction (a precision of 73% and a recall of 30%) cannotbe directly compared to the figures obtained in Study III, since they measure theresults for classifying whether a pair of two terms are synonyms or not, as opposedto the approach used here of measuring recall by evaluating the ability to retrieve aknown synonym among 10 candidate terms. The results achieved by McCrae andCollier (2008) can only function as an illustration of the difficulty of extractingsynonyms and show that the approach used here of presenting a limited number ofcandidate terms might be necessary to increase recall.

As previously discussed, the strategy applied by McCrae and Collier (2008) isnot suitable for clinical corpora, since it relies on text patterns that explain termsto the reader, text patterns that are rarely present in clinical text. Extracting syn-onyms from clinical text, to an even larger extent than extracting words of a certainsemantic category, instead requires methods based on distributional semantics, e.g.the Random indexing word space model explored here or other word space rep-resentations (Pyysalo et al., 2013). Text pattern based methods, e.g. of the formLong-Form (Short-Form) (Chang et al., 2002; Schwartz and Hearst, 2003; Ao andTakagi, 2005; Dannélls, 2006; Movshovitz-Attias and Cohen, 2012), are also un-suitable for automatic construction of abbreviation and acronym dictionaries fromclinical text, since abbreviations are not explicitly defined. The results of StudyIII show that distributional semantics methods, instead, offer a possibility for au-tomatically constructing abbreviation and acronym dictionaries from clinical text.Extracting dictionaries for abbreviations and acronyms not defined in the text is,however, a much more difficult task. The results achieved here can, therefore, notbe directly compared to, e.g. a precision of 95% and a recall of 97%, which wasachieved by Dannélls (2006) for pattern based extraction of abbreviations definedin Swedish medical journal text.

Whether other distributional semantics methods (Brown et al., 1992; Pyysalo etal., 2013) are equally or more suitable for the vocabulary expansion task was notevaluated. Different Random indexing configurations were, however, evaluated,and it was shown that results for synonym extraction could be improved by tun-ing the output of the word space models, e.g. by applying different combination

80 Chapter 8.

strategies of models and corpora, and that the results for extracting abbreviationdictionaries could be improved by applying post-processing based on the proper-ties of abbreviations and their expansions.

8.1.4 Study IV

Study IV addressed the second implication of the results of Study I, i.e. that othermethods than vocabulary matching need to be explored for recognising clinicalfindings in Swedish health record text. In the study, a limited-sized corpus ofmanually annotated clinical text was used for training a conditional random fieldsmodel. Using this model, it was investigated with what precision and recall find-ings and disorders that are mentioned in Swedish health record text can be auto-matically recognised.

The model was trained on clinical text, in which 1 317 entities of the categoryDisorder and 2 540 entities of the category Finding had been manually annotated.When evaluating this model, it could be concluded that it recognised the entity cat-egory Disorder with a precision of 80% and a recall of 82% and the entity categoryFinding with a precision of 72% and a recall of 65%. For the combined categoryClinical Finding, a precision of 80% and a recall of 76% were obtained. The de-velopment results were also compared to results from the vocabulary matchingapproach. This showed that training a conditional random fields model was moresuccessful, with a difference in F-score of 18 percentage points for Disorder and28 percentage points for Finding.

The obtained inter-annotator agreement scores cannot be used as an absoluteupper ceiling for the performance of the named entity recognition model, since themodel mimics the behaviour of one single annotator, which might be easier thanagreeing on how to annotate given the annotation guidelines. Another reason fornot treating the agreement scores as an absolute upper ceiling is that the agree-ment could probably have been improved with more extensive annotator training,more detailed guidelines and if more time had been spent by the annotators onreviewing and improving their own annotations. It can, however, be noted thatthe results achieved for the named entity recognition models are slightly betterthan the average inter-annotator agreement figures. The named entity recogni-tion F-score for the category Disorder was two percentage points higher than theaverage inter-annotator agreement for this category, and the named entity recogni-tion F-score for Finding was three percentage points higher than the average inter-annotator agreement for Finding. Despite the caveats given for the inter-annotatoragreement figures, this shows that a conditional random fields model trained on a

Discussion and conclusions 81

limited-sized corpus can achieve results that are at least comparable to the resultsof human annotators.

Comparison to English studies

0.7

0.8

0.9

0 2000 4000 6000 12000

Body StructurePharmaceutical DrugDisorder + Finding

Training instances

Study IV of this thesis

Jiang et al.

Wang and Patrick

Roberts et al.

F-score

Traininginstancesunknown

Rule-based

Doanet al.

Doanet al.

Patrick & Li

Kokkinakis&Thulin

Disorder

Savovaet al.

Figure 8.1: Comparison to other named entity recognition studies

Features similar to those used for entity recognition in English clinical text wereused when training the machine learning model on Swedish text, with the additionof compound splitting features. The results for the model trained on Swedishwere then compared to the results of previous named entity recognition studies

82 Chapter 8.

(Figure 8.1). The F-score for the Swedish model for recognising the categoryClinical Finding was, for instance, around seven percentage points lower than theresults achieved on English clinical text by Jiang et al. (2011). The difference can,however, for the most part, probably be explained by a) the fact that this previousEnglish study was performed on a considerably larger set of annotated entities; b)the fact that the English corpus consisted to a large extent of discharge summaries,which (as already discussed) are the opposite in terms of style to the assessmentfields used here and c) the fact that a wider definition of the category ClinicalFinding was used here, as non-pathological findings were also included.

The two English named entity recognition studies that are most similar to theexperiments performed in this thesis, in terms of the number of available annotatedentities and in terms of clinical text types, are probably those conducted by Wangand Patrick (2009) and by Roberts et al. (2008). Wang and Patrick used intensivecare service progress notes and a somewhat larger set of annotated entities andalso achieved better results than presented here, while Roberts et al. used a num-ber of different clinical text types and a smaller set of annotated data and achievedlower results. That lower results were achieved in Study IV than in the study byWang and Patrick could indicate that there are additional difficulties associatedwith named entity recognition in Swedish clinical text. This is also indicated bythe error analysis, which showed that some of the identified errors belong to errortypes that are less likely to occur in English clinical text. These error types werefalse negatives caused by entities being expressed with jargon, spelling variantsand abbreviations, which might have been avoided with the more extensive vocab-ulary resources available for English. Another frequent cause for false negatives,more likely to occur for Swedish, was errors caused by the lemmatiser failing tolemmatise inflected words. Compound words were also problematic, as they werefrequent among false positives as well as among false negatives, which includedcompounds with an abbreviation as one of its constituents.

For achieving results in line with previous studies on English texts, Swedishlemmatisation and compound splitting might need to be adapted to clinical text.Existing Swedish medical vocabulary resources were useful, since the F-score wasimproved by 3.5 percentage points when they were used as features. Even moreextensive vocabulary resources might, however, also be required. Such resourcescould, for instance, be constructed with the vocabulary expansion techniques ex-plored in the studies of thesis.

Discussion and conclusions 83

Dividing the entity category Clinical Finding into Disorder and Finding

The more general entity category Clinical Finding was separated into the two moregranular categories Disorder and Finding, for annotation as well as for namedentity recognition.

It could be concluded from the named entity recognition results that it was moredifficult to differentiate between the two entity categories Disorder and Findingthan between those two categories and the two other investigated entity categories,Body Structure and Pharmaceutical Drug. The error analysis of the named entityrecognition model showed that, in many of the cases for which the model failedto detect a Disorder, the model had instead classified the entity as a Finding (31%of the false negatives for Disorder) and vice versa (9% of the false negatives forFinding).

The inter-annotator agreement scores also showed that the division into Disorderand Finding was difficult. The agreement for the category Finding was low for bothpairs of annotators, 0.58 for PH1–PH2 and 0.73 for PH1–CL, while the agreementfor Disorder was higher with an F-score of 0.77 for PH1–PH2 and 0.80 for PH1–CL. Merging the two classes Disorder and Finding, resulted in higher agreementbetween the two physicians (F-score 0.72) than for a weighted average of the twoclasses (F-score 0.65). For the physician and the computational linguist, the resultof merging the classes had an even larger effect, with an F-score of 0.84. Thesefigures show that disagreements between annotators were often regarding which ofthese two categories to use. However, in the Development subset, annotated by themain annotator, only 3% of the unique entity types among annotated disorders andfindings were annotated as belonging to the category Disorder in some contextsand as belonging to the category Finding in other contexts. This supports themeaningfulness of this more granular division.

Separating into Disorder and Finding also provides an opportunity to comparethe two categories in terms of the difficulty of annotating and automatically recog-nising them. Inter-annotator agreement results, as well as named entity recognitionresults, were considerably lower for Finding than for Disorder. One reason for thisdifference is probably that the category Disorder is better represented in used vo-cabularies. This is indicated by the results for the vocabulary based named entityrecognition. In addition, when removing the vocabulary matching feature for Dis-order, the results decreased by five percentage points, while the same decrease forFinding was 1.5 percentage points. This could perhaps partly be due to that en-tities of the category Finding, to a larger extent than for Disorder entities, mightbe expressed with the non-professional language used by patients and, therefore,

84 Chapter 8.

be less likely to be included in medical vocabulary resources. It might also be thecase that findings are denoted with more complex expressions than Disorders, forinstance more multi-word expressions, which also makes them more difficult tolearn.

There is not much previous work on the more granular division of Clinical Find-ings that was explored here. Chapman and Dowling (2006), Roberts et al. (2009),Wang (2009) and Uzuner et al. (2011) all describe annotation studies of clinicaltext, in which a more general category corresponding to Clinical Finding was used,while Ogren et al. (2008) and Kokkinakis and Thurin (2007) only use a categorycorresponding to Disorder. The only other widely known corpus in which the twomore granular categories Disorder and Finding are both annotated is the MiPACQcorpus (Albright et al., 2013). Albright et al. (2013), who do not investigate howthe inter-annotator agreement results are affected by this more granular division,motivate their choice of these more granular categories by their usefulness for phe-notype extraction and clinical question answering.

A division into the categories Disorder and Finding could, for instance, also berelevant when mining for Disorder-Finding co-occurrences (Cao et al., 2005) orwhen presenting a patient overview in a clinical setting. Which of these possi-ble applications are targeted is likely to have an effect on whether the achievedrecognition results for the two separate categories are high enough to make this di-vision meaningful. An automatically generated high-level patient summary might,for instance, place high demands on recall of the category Disorder. This meansthat recognised entities classified as belonging to the category Finding (or enti-ties for which the model is uncertain whether to classify as Findings or Disorders)also ought to be included in the summary, thereby boosting recall for includedDisorders. The results achieved are, however, likely to be high enough for it tobe meaningful to make a distinction between these categories when mining fornew medical knowledge, e.g. since known co-morbidities have been successfullyextracted from structured data that contains inaccuracies (Socialstyrelsen, 2006;Tanushi et al., 2011). Having access to a named entity recognition system thatdistinguishes between disorders and findings makes it possible to separately minefor co-morbidity relations and disorder-finding relations. Such a system would,however, also greatly benefit from concept mapping, to be able to link synonymsand abbreviations to the same concept. This could, for instance, be achieved byusing vocabulary constructed from the methods for semi-automatic synonym andabbreviation extraction that have been explored in this thesis.

Discussion and conclusions 85

8.2 Negation and uncertainty detection

For the negation and uncertainty detection task, it was explored how English tech-niques for this task perform when applied to a related language, as well as howvocabularies for this task can be created.

8.2.1 Study V

First, how an English hand-crafted rule-based negation detection system performswhen adapted to Swedish text was studied. This was carried out by translatingcues from the English system into Swedish, and using the 42 translated cues thatwere most frequent in Swedish clinical text.

Evaluating this translation on Swedish clinical text resulted in a recall (82%)for recognising negated clinical findings that was very close to the recall for theoriginal English system (Chapman et al., 2001), while the precision was lower(75%, which is around nine percentage points lower than what was achieved forthe English version). The lower precision can be partly attributed to two of thetranslated cues, but also without the problems caused by these cues, the precisionwas lower (78%).

Similar to the results for the above described entity recognition, the lower resultsachieved for Swedish could, perhaps, be partly explained by the English versionbeing evaluated on discharge summaries, while the Swedish version was evaluatedon assessment fields, which contain more reasoning and, therefore, more instancesof uncertain expressions. The Swedish classifier, therefore, had to face the problemof distinguishing between negated and uncertain clinical findings more often thanthe English classifier.

The coverage of the Swedish vocabulary of cue terms for negation that was ob-tained from the translation of English negation cues was also evaluated. 76% of thenegated clinical findings, which were correctly identified by the adapted Englishsystem, were negated by the four most frequently occurring cue phrases. This in-dicates that negation is expressed in Swedish, as in English, with a limited set ofcue phrases. It could, therefore, be concluded that a translated list of English nega-tion cues covers a large part of the expressions used for negating Swedish clinicalfindings. This would, however, probably also be true for a list created throughlinguistic introspection performed by a speaker of Swedish. Other useful nega-tion cues, apart from the four most frequent ones, were, however, also obtained bytranslating English cues. In addition, no moderately frequent negation cues werefound in the manually annotated resources that were not in the list of translated

86 Chapter 8.

ones, showing that translating English cues from an English negation detectionsystem gives rise to a high coverage vocabulary list of Swedish negation cues. Thecoverage was, however, not complete, since a few additional infrequently occur-ring, cues were found in the manually annotated resources. On the other hand, thetime-consuming manual annotation did not give rise to a complete list of negationcues either, as it did not contain all the cues in list obtained by translation. Trans-lating cues from English is, therefore, a resource efficient method for obtaininga Swedish list covering frequently and moderately frequently occurring negationcues.

8.2.2 Study VI

As a result of the problem of separating uncertainty and negation for the rule-based system, uncertainty was also included in the next study. How well a hand-crafted English rule-based negation and uncertainty detection system performswhen adapted to Swedish text was, therefore, studied. Four classes were used:Definite Existence, Probable Existence, Probable Negated Existence and DefiniteNegated Existence. It could, however, be concluded that when involving uncer-tainties, lower results for the class Definite Negated Existence were achieved thanfor the corresponding class Negated in the previous study (a precision of 60% anda recall of 66% compared to a precision of 75% and a recall of 82% in the previ-ous study). The results are, however, not directly comparable, as the results in theprevious study show the performance on sentences containing a known negationcue, while the results when uncertainty is also included show the performance onrandomly extracted sentences containing a clinical finding. In addition, the def-inition for Negated used in Study V was slightly wider than the definition of theclass Definite Negated Existence. For the class Probable Negated Existence, lowresults were also obtained (a precision of 54% and a recall of 56%), while resultsfor combining these two classes into the one class Negated were much better (aprecision of 84% and a recall of 90%). These results show that the more granulardivision of Negated into Definite Negation and Probable Negation is difficult tomake. The exact interpretation of cues for Definite Negated Existence or ProbableNegated Existence might be more dependent on the medical context in which theyoccur than is the case for cues for the other classes, for which better results wereobtained. For Definite Existence, a precision of 89% and a recall of 87% wereachieved, while the precision was 82% and the recall 81% for Probable Existence.

The more granular classes for uncertainty are grouped into larger classes slightlydifferently here than in previous research using the Stockholm EPR Diagnosis Un-

Discussion and conclusions 87

certainty Corpus (Velupillai, 2011). It is, however, still possible to approximatelycompare the results. Although the previous approach used a larger subset of theStockholm EPR Diagnosis Uncertainty Corpus for training a machine learningmodel, than was used here for correcting and expanding the cue sets, similar re-sults were achieved. F-scores for Velupillai (2011) compared to results achievedhere are as follows: Definite Existence 0.83 vs. 0.88, Probable Existence 0.77 vs.0.81, Probable Negated Existence 0.56 vs. 0.55 and Definite Negated Existence0.69 vs. 0.63.

For comparing to English, results from the 2010 i2b2/VA challenge could beused (i2b2/VA, 2010). The class Possible in the i2b2/VA challenge is similar tothe combined class Uncertainty_yes, evaluated in Study VI. For this class, Clarket al. (2011) achieved an F-score of 0.63, while an F-score of 0.78 was achievedhere. The 2010 i2b2/VA challenge did, however, also include the related classHypothetical, which the classifier sometimes confused with Possible. For the classcorresponding to Definite Existence, an F-score of 0.96 was achieved by Clark etal. (2011) and for Definite Negated Existence an F-score of 0.94, thus better thanthe results achieved in Study VI. This shows that approaches involving machinelearning on large data sets can achieve better results than those obtained in thestudies of this thesis.

The coverage of the obtained Swedish vocabulary of uncertainty cues was alsoevaluated. This showed that the recall for Probable Existence was low when onlycues translated from the English system were used, but that the recall improvedwhen uncertainty cues from other sources were added. It could, therefore, be con-cluded that only relying on cues translated from the English system results in a lowcoverage vocabulary. Uncertainty is, however, expressed with a large set of cuesin Swedish clinical text. In contrast to the other classes, the results for ProbableExistence continued to improve substantially also after the 10 most frequently oc-curring cues were used for detecting the class. It is, consequently, more difficult toobtain a high coverage vocabulary for uncertainty cues, which could explain whytranslating cues is less successful for uncertainty than for negation

As the result for Probable Existence continues to improve with additional un-certainty cues, it is likely that the final version of the vocabulary of moderatelyfrequently occurring uncertainty cues is not complete. There is, therefore, a pos-sibility to further improve uncertainty detection by expanding the uncertainty cuephrase vocabulary, for instance by the explored semi-automatic vocabulary expan-sion methods.

88 Chapter 8.

8.3 General discussion and conclusions

All three aims of the thesis dealt with exploring the possibilities of enabling infor-mation extraction from clinical text. The first two aims dealt with enabling clinicalinformation extraction in languages other than English, and the third aim dealtwith enabling information extraction of entities on a more granular level.

8.3.1 Exploring techniques previously used for English

The first aim consisted of exploring techniques previously used for English clinicalfindings extraction on a related language. In an initial attempt, this aim was ad-dressed by mapping Swedish clinical assessment field text to existing medical vo-cabularies. As this resulted in low recall and relatively low precision, a conditionalrandom fields model was trained to perform this task. Despite being trained ona relatively small corpus (1 317 Disorder mentions and 2 540 Finding mentions),the model achieved results comparable to the inter-annotator agreement results ofhuman annotators. The model achieved a precision of 80% and a recall of 82% forDisorder, a precision of 72% and a recall of 65% for Finding and a precision of80% and a recall of 76% for the combined entity category Clinical Finding. Fornegation and uncertainty detection, on the other hand, the approach of using rulesand vocabulary mapping was more successful. When adapting an English system,and obtaining cues from this system as well as from other sources, results in linewith a previously evaluated machine learning model (Velupillai, 2011) were ob-tained. For the combined class Negated, a precision of 84% and a recall of 90%were achieved, for Definite Existence a precision of 89% and a recall of 87% wereachieved, and for Probable Existence, a precision of 82% and a recall of 81% wereachieved.

It can, therefore, be concluded that, for negation and uncertainty detection ina language related to English, leveraging English rule-based resources can be aresource efficient approach for obtaining relatively good results. In contrast, cur-rently available vocabularies are not extensive enough to enable a rule-based vo-cabulary mapping approach for high quality named entity recognition in Swedishassessment fields.

The Swedish negation and uncertainty detection system achieved, however,slightly lower results than English systems trained on a large corpus of annotateddata (Clark et al., 2011). A more resource demanding machine learning basedapproach might, therefore, be necessary to further improve Swedish negation anduncertainty detection. This is also supported by the fact that using more than the

Discussion and conclusions 89

30 most frequent cues for the negated classes only had a marginally positive ef-fect on the results, which shows that the system cannot be further improved bysolely extending the cue lexicon for these classes, but that other methods might berequired.

Although relatively good results were achieved for the named entity recognitionmodel trained on a limited-sized corpus, similar to the negation and uncertaintydetection system, the achieved results were lower than results for English modelstrained on much larger corpora (Jiang et al., 2011; de Bruijn et al., 2011). Thelower results could, perhaps, be partly attributed to factors other than a smallertraining set, e.g. to differences between different genres within the clinical domainand differences between how the entities were defined. The general trend for En-glish is, however, that clinical named entity recognition models trained on a largerdata set perform better (Patrick et al., 2007; Roberts et al., 2008; Jiang et al., 2011;de Bruijn et al., 2011). Therefore, to obtain results for Swedish that are in line withthe best English systems, a larger annotated corpus is probably required. There is,however, also room for improvement by means other than an increased trainingcorpus, since errors caused by an incorrect lemmatisation and by incorrect vocab-ulary mapping of compound words were identified, errors that are more likely tooccur in Swedish than in English clinical text. Among identified errors was alsoan inability to recognise jargon and abbreviations, an error type, which might havebeen less frequent if more extensive vocabularies had been available.

8.3.2 Evaluation and expansion of clinical vocabulary resources

The conclusion above regarding existing vocabulary resources is related to thesecond aim of the thesis, the aim of studying the usefulness of a smaller vocabu-lary resource for the task of extracting clinical findings and of exploring methodsfor facilitating the expansion and creation of clinical vocabularies. From the firstattempt at recognising clinical findings, an attempt at mapping to existing vocab-ularies that resulted in low recall and relatively low precision, it could be con-cluded that existing Swedish medical vocabularies are insufficient for the purposeof high-recall named entity recognition. This further implies that existing vocabu-laries are equally insufficient for the purpose of high-quality concept mapping ofclinical text, which is an important component in clinical informations extraction,e.g. as shown by its inclusion in shared tasks (Suominen et al., 2013; Aramaki etal., 2014). In turn, this shows the importance, for Swedish as well as for otherlanguages with smaller medical vocabulary resources, to have access to methodsthat facilitate vocabulary expansion. To enable an expansion of vocabularies with

90 Chapter 8.

terms that are relevant for information extraction from clinical text, it is particu-larly important to explore vocabulary expansion using methods suitable for clinicalcorpora. The Random indexing based methods explored in this thesis are examplesof such methods, since they, in contrast to many other explored medical vocabularyexpansion methods (Dannélls, 2006; McCrae and Collier, 2008; Neelakantan andCollins, 2014), do not require terms and abbreviations to be defined or explainedin the text.

One of the vocabulary expansion approaches explored in this thesis created listsof candidates for terms belonging to certain semantic categories, i.e. producingvocabularies that have the potential to improve named entity recognition. Amongthe top 1 000 candidate terms for the category Clinical Finding, 53% of the 90reference standard terms were retrieved, while the same figure for PharmaceuticalDrug was 88%. Although it is likely that results could be improved, e.g. by furthertuning the used application of Random indexing, the results obtained show thepotential of using distributional semantics as a practical aid when constructingvocabulary resources relevant for clinical information extraction.

The other vocabulary expansion method explored in the thesis aimed at extract-ing synonyms as well as abbreviations and their expansions, thereby creating vo-cabulary resources useful also for concept mapping. Similar to the other vocab-ulary extraction study, Random indexing was used, but combinations of modelsand post-processing of the results were evaluated. For retrieving known synonymsamong 10 candidate terms, a maximum recall of 47% was achieved, while a max-imum recall of 42% was achieved for retrieving abbreviation expansions. Theseresults also indicate the potential of using distributional semantics for facilitatingthe construction of clinically relevant vocabularies. In addition, for both Randomindexing studies, a manual inspection of retrieved candidates showed that gener-ated candidates were often relevant terms, despite not being included in the usedreference standards. For instance, among the top 100 candidates for terms be-longing to the category Clinical Finding, 68% were classified as belonging to thecorrect semantic category. This indicates that Random indexing based vocabularyexpansion might be even more useful than what is shown by the obtained recallfigures.

For negation and uncertainty detection, the used semi-automatic method for vo-cabulary expansion was to translate English negation and uncertainty cues. In con-trast to, i.e. terms denoting clinical findings, the cue words are not specific to themedical or clinical language, and translations could be found using Google Trans-late. The translation was successful for negation cues, for which a large part ofthe expressions used for negating Swedish clinical findings were obtained. For un-

Discussion and conclusions 91

certainty cues, however, the lexicon obtained by translation had a lower coverage,probably due to the fact that a much larger set of cues is used to express uncertaintythan to express negation. In contrast to the detection of negated classes, there is,therefore, a possibility to improve the detection of uncertainty by further expand-ing the cue lexicon with uncertainty cues. Also for uncertainty, however, manyuseful cues were achieved by the semi-automatic vocabulary expansion method oftranslating English uncertainty cues.

8.3.3 Distinguishing between the semantic categories Disorder and Finding

The third aim regarded the possibility to distinguish between the semantic cat-egories Disorder and Finding. This distinction has previously been describedas difficult to define when giving instructions for vocabulary construction (In-ternational Health Terminology Standards Development Organisation, IHTSDO,2008b). That the vocabulary mapping required priority rules, prioritising matchesagainst vocabulary terms of the category Disorder over matches against vocabu-lary terms of the category Finding, shows that among terms in the used corpora,there were such terms, which potentially belong to both semantic categories. Thiswas also a reason for not separating Clinical Finding into more granular categoriesfor the vocabulary expansion study using this semantic category.

In the Development corpus, annotated by the main annotator, however, only 3%of unique entity types among annotated disorders and findings were annotated asbelonging to the category Disorder in some contexts and as belonging to the cate-gory Findings in other contexts. This shows that, at least when using the definitionof one experienced annotator with medical education, the actual use of terms aseither belonging to the category Finding or the category Disorder might be moreclear cut than what might appear to be the case in medical vocabularies. There-fore, if excluding terms that are categorised as both a Finding and a Disorder incurrent vocabularies from the list of seed terms, it might be possible to apply thesemi-automatic vocabulary expansion technique explored here to extract terms ofthe two more granular semantic categories Disorder and Finding.

To agree among annotators on which entities should be classified as Disordersand which should be classified as Findings, proved, however, to be more difficult.Especially when comparing the main annotator and the annotator without medicaleducation, the results were substantially improved when collapsing the two cat-egories into the one category Clinical Finding (Disorder had an F-score of 0.80,Finding an F-score 0.73 and the combined class Clinical Finding an F-score of0.84). This is not surprising since a medical education is likely to increase the

92 Chapter 8.

ability to, e.g. determine which mentioned entity types have an “underlying patho-logical process”, which is one the distinguishing features. The two categorieswere, however, not clear cut between the two physicians either, as the agreementimproved when using the combined category Clinical Finding instead of the twoseparate categories.

These granular categories caused problems also for the named entity recogni-tion. In many of the cases for which the system failed to detect a Disorder, ithad instead classified the entity as a Finding and vice versa. A more extensivevocabulary, separating the categories Disorder and Finding, might be required todecrease these misclassifications. Since it might not be enough using only fea-tures from the most closely neighbouring words to distinguish between these twoentity categories, a larger annotated corpus might also be required to enable theuse of a larger feature context window. As previously mentioned, however, knownco-morbidities have been successfully extracted from structured data that containsinaccuracies (Socialstyrelsen, 2006; Tanushi et al., 2011). This makes it likelythat co-morbidities, as well as finding-disorder relations, could be extracted froma clinical text in which disorders are recognised with a precision of 80% and find-ings with a precision of 72%.

8.4 Main contributions and final conclusions

• The main contribution of this thesis is the exploration of information extrac-tion from non-English clinical corpora, since studies of clinical English untilrecently have dominated clinical NLP research.

For named entity recognition of clinical findings, a machine learning model,trained on a limited-sized corpus of annotated text, achieved results in linewith inter-annotator agreement figures from manual annotation. The ma-chine learning model clearly outperformed the vocabulary-based approachfor recognising clinical findings, showing that Swedish medical vocabular-ies are not extensive enough for the purpose of high-quality information ex-traction from clinical text. For negation and uncertainty detection of recog-nised clinical findings, on the other hand, an adaption of an English rule-and vocabulary-based system achieved results in line with previous machinelearning approaches.

• The second most important contribution is the exploration of distributionalsemantics for semi-automatic medical vocabulary expansion. This approach

Discussion and conclusions 93

does not require that terms or abbreviations are explicitly defined in the typeof text used for vocabulary extraction, and is, thereby, more suitable for clin-ical corpora than are many of the previously explored methods for biomed-ical vocabulary extraction. Methods for facilitating vocabulary expansionare particularly important for non-English languages, for which existing vo-cabulary resources are often not extensive enough to enable high qualityinformation extraction.

Methods based on Random indexing were used for semi-automatically ex-tracting terms of the semantic categories Clinical Finding and Pharmaceuti-cal Drug, as well as for extracting synonyms and abbreviation dictionaries.The ability of the evaluated approaches to retrieve known terms showed thepotential in using distributional semantics for facilitating the construction ofvocabularies relevant for the clinical sublanguage.

• Another important contribution of the thesis is the training and evaluationof a machine learning model that divides Clinical Finding into the two moregranular categories Finding and Disorder. There is little previous researchon this division, despite its importance for many clinical text-mining appli-cations.

The more granular division into Finding and Disorder decreased agreementbetween the annotators and caused misclassifications for the trained ma-chine learning model. There were, however, very few unique entity typesthat were annotated as Disorders in some contexts and as Findings in othercontexts by the main annotator, which supports the meaningfulness of thismore granular division. In addition, the results achieved for recognising thetwo separate categories are likely to be high enough for some clinical infor-mation extraction applications, such as the extraction of co-morbidities anddisorder-finding relations.

• In the process of exploring the extraction of Clinical Findings from Swedishhealth record corpora, a number of artefacts were developed for this Swedishsublanguage, in the form of annotation guidelines, annotated corpora andinformation extraction components (Table 8.1).

Developed information extraction components have been useful and willhopefully continue to be useful for future studies on the extraction of in-formation from Swedish clinical corpora.

94 Chapter 8.

• Finally, the combination of a number of relatively resource efficient ap-proaches for enabling information extraction in a non-English language,which are described in this thesis, has inspired the construction of clini-cal information extraction components for a non-Indo-European language(Ahltorp et al., 2014b). This combination of approaches can, hopefully, alsoinspire the research community to further information extraction studies, us-ing clinical corpora from a number of different languages.

GuidelinesSwedish clinical entities annotation guidelines

http://dsv.su.se/health/guidelines/Corpora

Stockholm EPR Clinical Entity CorpusNot publicly available (sensitive content)

Stockholm EPR Negated Findings CorpusNot publicly available (sensitive content)

VocabulariesNegation cue list to use together with NegEx

http://people.dsv.su.se/~mariask/resources/triggers.txtNegation and uncertainty cue lists to use together with PyContextSwe

http://dsv.su.se/health/dictionariesSystems

Rule-based vocabulary matchinghttp://people.dsv.su.se/~mariask/resources/snomedMatchning_conllformat.zip

PyContextSweAvailable upon request

ModelsNER model trained on Stockholm EPR Clinical Entity Corpus

Not publicly available (trained on sensitive content)Random Indexing models built on subset of Stockholm EPR Corpus

Not publicly available (built on sensitive content)Random Indexing models built on Läkartidningen

http://people.dsv.su.se/~mariask/random_indexing/random_indexing.html

Table 8.1: Availability of developed artefacts.

Chapter 9

Future directions

This chapter discusses future directions for the work of this thesis. Some possibil-ities for future studies have been briefly indicated in the discussion, but here theideas will be developed.

9.1 Further exploring vocabulary expansion

There are a number of possible future directions for developing, evaluating andapplying the explored vocabulary expansion techniques.

9.1.1 Comparison to other approaches and further tuning of the Randomindexing models

As mentioned in the discussion, it is difficult to compare two vocabulary expansionapproaches for which different reference standards, seed terms and corpora havebeen used. Therefore, to be able to determine the performance of Random indexingcompared to other distributional semantics methods, e.g. to newer versions of spa-tial methods (Mikolov et al., 2013), the different methods must be compared usingthe exact same resources and evaluation framework. Since combinations of differ-ent semantic spaces and the use of post-processing techniques were successful forimproving the results of the synonym and abbreviation extraction task, it is likelythat there is a possibility to improve results by further developing these strategies.Comparison to other distributional semantics methods as well as further tuning andpost-processing of the used Random indexing methods are therefore two possible

96 Chapter 9.

future directions for improving and evaluating the explored vocabulary expansionapproaches.

9.1.2 Clustering context vectors

Another possible post-processing technique is clustering of the context vectors.For instance, when extracting terms belonging to the categories Clinical Findingand Pharmaceutical Drug, the used seed terms could be described as modelledas one cluster in the created semantic space, since unknown terms were rankedhigher in the candidate list, the smaller the summed distance to the seed terms.This is most likely an over-simplification, as there might be a number of sub-clusters within each of the two evaluated semantic categories, sub-clusters that arepositioned at large distances from each other in the semantic space. Terms not partof these sub-clusters, but close to two or more clusters, receive a high ranking withthe used approach, although they should be ranked lower than terms close to thecentroids of the sub-clusters. Exploring clustering of seed terms and measuring thedistance of unknown terms to these sub-clusters might, therefore, be a possibilityfor further improving the word space based vocabulary expansion techniques.

9.1.3 Addressing multiword terms

In an extension of Study III, which is not included in this thesis, the task of extract-ing multi-word terms was also addressed (Henriksson et al., 2013b). Multi-wordterms are frequent within the medical domain, especially for the two semantic cat-egories that have been the focus of this thesis. In the sub-corpus used in Study I forinstance, the average length of an annotated Disorder was 1.28 tokens, while theaverage length of an annotated Finding was 1.61 tokens. Not all of these annotatedtoken combinations are suitable to include in vocabularies, but these statistics stillshow that a medical vocabulary expansion approach that does not include multi-word terms will not be able to produce a high-coverage vocabulary. This reasoningis also supported by the fact that existing medical vocabularies consist of a largenumber of multi-word terms, which were excluded from the sets of seed and ref-erence standard terms used in the vocabulary extraction studies of this thesis. Inthe study by Henriksson et al. (2013b), the corpus was pre-processed by joiningneighbouring tokens into one term, if they were likely to be part of a multi-wordterm according to the C-value method for multi-word term extraction. This methodcould be further developed, e.g. by using part-of-speech information for filteringthe multi-word terms achieved by the C-value method, and compositional distri-butional semantics methods (Socher et al., 2012) could also be attempted.

Future directions 97

9.1.4 Identifying connections between entities of the types Body Structureand Clinical Finding

Related to the topic of multi-word terms is the relation between the Finding andDisorder categories and the category Body Structure. An annotation of the seman-tic category Body Structure had priority over the semantic categories Finding andDisorder. Therefore, in the text chunk “pain in arm”, arm was annotated as a BodyStructure and pain as a Finding. These guidelines allowed for more flexibility and,thereby, fewer annotations of longer sequences, such as “pain in arm and head”,but they also have the effect that a further processing of the text is required todetermine whether the recognised Body Structure entities constitute the site for aClinical Finding. Such post-processing has been explored by Forsgren and Gredin(2013), who used the machine learning model of Study IV on assessment fields,and thereafter applied rule-based chunking on sentences that contained at least oneentity of the category Body Structure as well as at least one entity of the categoryClinical Finding, in order to determine whether the Body Structure was the site forthe Clinical Finding. This resulted in a high precision (84%) but a relatively lowrecall (44%), and there is, therefore, room for further development of this approachor for the exploration of alternative methods.

9.1.5 Further expanding the uncertainty cue lexicon

For expanding negation and uncertainty detection vocabularies, the main approachused in this thesis was to translate English cues. This resulted in a high cover-age vocabulary for negation, while the results indicated that uncertainty detectioncould be improved by further expanding the cue lexicon with uncertainty cues.The Random indexing based approach evaluated in Study II could, for instance,be applied to this task, using the list of uncertainty cues developed in Study VI asseed terms. Also for expansion of uncertainty cues, the existence of multi-wordterms needs to be taken into account. It is also possible to further improve resultsof the cue-based approach by improving the methods for scope detection.

9.2 Applying developed vocabulary

Evaluating the usefulness of the vocabulary expansion techniques for named entityrecognition and concept mapping is another important future direction.

98 Chapter 9.

9.2.1 Evaluating the usefulness of created vocabularies for named entityrecognition

In Study IV, it was shown that existing vocabularies were useful as a feature forthe machine learning model and that abbreviations and jargon expressions gaverise to false negatives for the trained model. With a more extensive vocabulary ofthe studied entity types, this feature might have been even more useful, and someof the false negatives caused by the use of abbreviations and jargon might havebeen avoided. Study II consisted of an exploration of methods for semi-automaticcreation of a more extensive vocabulary, i.e. a vocabulary that could be used forimproving named entity recognition. It remains, however, to be investigated howuseful the output from Study II would be for named entity recognition. Neelakan-tan and Collins (2014) evaluated their created vocabulary by using it in a rule- andvocabulary based named entity recognition approach, similar to the approach ofStudy I, as well as by using it for generating features when training a machinelearning model on annotated text. Both these two evaluation metrics would beinteresting to apply to vocabulary generated by the methods of Study II. It mightalso be interesting to create a vocabulary list of the manually annotated entitiesin the Stockholm EPR Clinical Entity Corpus to use as a reference standard forevaluating the term candidates produced by the methods of Study II.

9.2.2 Evaluating the usefulness of created vocabularies for concept matching

An implication of the low results in Study I is that Swedish medical vocabulariesare not extensive enough to enable high-quality concept mapping. This was amotivation for the exploration of synonym and abbreviation dictionary expansionin Study III. Similar to the created vocabularies of Study II, it remains, however, tobe evaluated how useful an expanded vocabulary of synonyms and abbreviationswould be for concept mapping of the four evaluated entity types.

In Study III, the vocabulary expansion approach was evaluated for terms forwhich there exists at least one known synonym, while in a realistic setting, it isnot known for which terms there are synonyms. Also, when adding new terms toa vocabulary, it cannot be taken for granted that a new term is a synonym to anexisting term. Instead, it might be the case that the new term should be added as anew independent concept to the vocabulary, and the task is to determine to whichhypernym the new concept should be attached. The approaches of Study II andStudy III could, however, be combined to create vocabularies suitable for conceptmapping. First, a set of unknown terms of a given semantic category could beretrieved using the method of Study II. Thereafter, each unknown term could be

Future directions 99

positioned in the existing vocabulary using the method of Study III, by associatingthe unknown term to the vocabulary, either as a synonym or as a hyponym to anexisting vocabulary concept.

9.2.3 Evaluating the usefulness of created vocabularies for other applica-tions

Another piece of conducted research that was not included in the thesis is med-ical text simplification by automatic synonym replacement. In a study by Abra-hamsson et al. (2014), current Swedish medical synonym resources were used forsimplifying medical text by replacing terms that were infrequent in a corpus ofstandard Swedish with more frequently occurring synonyms. A conclusion of thisstudy was, however, that many terms in the specialist medical language lackedsynonyms in the used resources. This could probably be partly explained by thefact that there are medical terms, for which there are no synonyms in the everydaylanguage. It could, however, also be an indication that current medical synonymresources should be expanded to be able to provide patients with text that is easierto understand. The methods explored in Study III could be used for expanding syn-onym and abbreviation resources, with the aim of developing vocabulary resourcesthat can be used for making medical texts more accessible to patients.

9.3 Exploring resource efficient methods

The importance of resource efficiency has been briefly discussed in the thesis,mainly referring to resources in the form of manual work effort required for an-notation or for manually correcting term candidates. Resource efficiency has beenthe motivation for evaluating performance for vocabulary extraction using a lim-ited number of generated candidate words, as well as for evaluating a machinelearning model for named entity recognition on a limited-sized corpus of manuallyannotated clinical text. Resource efficiency is particularly important for languages,for which it is unlikely that large corpora, such as the i2b2/VA Challenge Corpus(Uzuner et al., 2011) or the MiPACQ Clinical Corpus (Albright et al., 2013), willbe developed. There are, however, a number of standard methods for increasingresource efficiency, and, while these methods have not been explored in the thesis,they constitute important future work.

100 Chapter 9.

9.3.1 Semi-supervised methods

There are previous approaches for incorporating semi-supervised methods intonamed entity recognition of entities in clinical text. In the study by de Bruijnet al. (2011), for instance, features in the form of hierarchical word clusters con-structed by Brown clustering on unlabelled data were incorporated when traininga machine learning model on annotated text. There are also approaches in whichfeatures from spatial models have been used when training medical named entityrecognition models (Pyysalo et al., 2013), including features obtained from Ran-dom indexing (Jonnalagadda et al., 2012; Henriksson et al., 2014). As there arevocabulary resources for the entities explored in this thesis, a possible method ofusing Random indexing would be to create clusters around context vectors for ex-isting vocabulary terms and to use cluster membership in these clusters as a featurefor training the machine learning models (Skeppstedt, 2014).

9.3.2 Resource efficiency in annotation

Other possibilities for resource efficiency (which is orthogonal to the use of semi-supervised methods) are methods for facilitating annotation and methods for min-imising the data required for training machine learning models. These approachesare particularly important for clinical text, since, as previously mentioned, crowd-sourcing approaches cannot always be used. There are examples of studies inwhich pre-annotation has been applied on clinical text (Henriksson et al.; Southet al., 2014), and Olsson (2008) suggests that pre-annotation could be combinedwith active learning.

Pre-annotation and active learning might, however, appear to be an incompatiblecombination, i.e. to provide the human annotator with pre-annotated data that hasbeen selected for active learning. The reason for this incompatibility is that dataselected for annotation when using active learning, is the data for which the pre-annotator is most uncertain and, therefore, the data which would be least suitablefor pre-annotation (since a very low-performing pre-annotation is not helpful).

Skeppstedt (2013b) suggests a method for solving this incompatibility by alwayspresenting two alternative annotations to the annotator, and letting the annotatorselect the correct version among these two alternatives. To minimise the instancesin which none of the presented pre-annotations are correct, the texts presentedto the annotator are actively selected from the pool of unlabelled text, with theselection criterion that the third most probable annotation should have a muchlower probability than the two most probable ones, while the two most probableones have a fairly similar probability. The probability that one of the presented

Future directions 101

pre-annotations is correct is then high, making the pre-annotation useful to theannotator. In addition, the machine learner is uncertain which of the two mostprobable annotations is correct, thereby also making the selection of annotationinstances useful to the machine learner.

9.4 Applying the developed artefacts

An advantage of the developed rule-based methods is that they can be distributedfreely, as they are built on a few simple rules for determining scope and on a listsof cues. So far, the negation detection system in Study V has evaluated as a com-ponent in a system for suggesting ICD-10 codes (Henriksson and Hassel, 2011) aswell as in a system for detecting healthcare associated infections (Ehrentraut et al.,2014), and the cues for Study VI have been translated into several other languages(Chapman et al., 2013; Ahltorp et al., 2014b).

The machine learning model for named entity recognition on the other hand,which is trained on authentic clinical text, is only used by the limited number ofpeople who also have access to the corpus on which it is trained. The model has,however, been used successfully as a pre-annotation tool for annotations aimingat detecting adverse drug reactions in a study by Henriksson et al., in which it iscalled Clinical Entity Finder.

The Random indexing models that were built on clinical text also contain sen-sitive information and cannot be made freely available. In Study II, however, theRandom indexing models were not constructed on clinical text, but on a freelyavailable medical corpus and have, therefore, been made freely available.

The intended use of the components constructed in this thesis is as an infor-mation extraction pipeline, in which the named entity recognition model is firstapplied for recognising the four entity types Disorder, Finding, Body Structureand Pharmaceutical Drug. Thereafter, the Swedish adaption of PyContextNLP isapplied to detect if the recognised Disorders and Findings are affirmed, negated,and/or expressed with uncertainty. A third component in such a pipeline could bea system like the one constructed by Forsgren and Gredin (2013), in which entitiesof the categories Body Structure are associated with entities of the Clinical Findingcategories.

Current versions of developed components are not likely to be possible to usein a clinical setting, e.g. for generating patient summaries or problem lists, as suchusages would require near perfect performance. For information extraction, onthe other hand, e.g. for generating new medical knowledge, it is likely that the

102 Chapter 9.

developed components could be useful. Previous research has mined informationfrom the structured data of the health records of the Stockholm EPR Corpus, e.g.ICD-10 codes to mine for comorbidity (Tanushi et al., 2011) as well as ICD-10and ATC (Anatomical Therapeutic Chemical Classification System) codes (Zhaoet al., 2014) to mine for adverse drug reactions. Similar studies could be performedbut also include disorders, findings and pharmaceutical drugs extracted from theclinical text.

References 103

ReferencesEmil Abrahamsson, Timothy Forni, Maria Skeppstedt, and Maria Kvist. 2014. Medical text simpli-

fication using synonym replacement: Adapting assessment of word difficulty to a compoundinglanguage. In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability forTarget Reader Populations (PITR), pages 57–65, Gothenburg, Sweden, April 2014. Association forComputational Linguistics.

Magnus Ahltorp, Maria Skeppstedt, Shiho Kitajima, Rafal Rzepka, and Kenji Araki. 2014a. Medi-cal vocabulary mining using distributional semantics on Japanese patient blogs. In Proceedings ofSMBM 2014.

Magnus Ahltorp, Hideyuki Tanushi, Shiho Kitajima, Maria Skeppstedt, Rafal Rzepka, and Kenji Araki.2014b. MedNLP 2: Automatic extraction of medical complaints from Japanese health records usingmachine learning and rule-based methods. Submitted to the proceedings of NTCIR-11.

Daniel Albright, Arrick Lanfranchi, Anwen Fredriksen, William F Styler, 4th, Colin Warner, Jena DHwang, Jinho D Choi, Dmitriy Dligach, Rodney D Nielsen, James Martin, Wayne Ward, MarthaPalmer, and Guergana K Savova. 2013. Towards comprehensive syntactic and semantic annotationsof the clinical narrative. J Am Med Inform Assoc.

Helen Allvin, Elin Carlsson, Hercules Dalianis, Riitta Danielsson-Ojala, Vidas Daudaravicius, Mar-tin Hassel, Dimitrios Kokkinakis, Helja Lundgren-Laine, Gunnar H Nilsson, Oystein Nytro, SannaSalantera, Maria Skeppstedt, Hanna Suominen, and Sumithra Velupillai. 2011. Characteristics ofFinnish and Swedish intensive care nursing narratives: a comparative analysis to support the devel-opment of clinical language technologies. J Biomed Semantics, 2 Suppl 3:S1. ISSN 2041-1480(Electronic).

Ethem Alpaydin. 2010. Introduction to Machine Learning, 2. ed. edition. MIT Press, Cambridge, MA.ISBN 978-0-262-01243-0 (hardcover : alk. paper).

Douglas G Altman and J Martin Bland. 1994. Statistics notes: Diagnostic tests 2: predictive values.BMJ, 308(896).

Göran Andersson. 1968. Repetitorium i statistik. D.1, Teori och formler (In Swedish).

Howard Anton and Chris Rorres. 1994. Elementary linear algebra : applications version, 7., [ex-tended] ed. edition. Wiley, New York. ISBN 0-471-58741-9 ;.

Hiroko Ao and Toshihisa Takagi. 2005. ALICE: An Algorithm to Extract Abbreviations from MED-LINE. Journal of the American Medical Informatics Association, 12(5):576–586. ISSN 1067-5027.

Eiji Aramaki, Yasuhide Miura, Masatsugu Tonoike, Tomoko Ohkuma, Hiroshi Mashuichi, andKazuhiko Ohe. 2009. Text2table: Medical text summarization system based on named entity recog-nition and modality identification. In Proceedings of the BioNLP 2009 Workshop, pages 185–192,Boulder, Colorado, June 2009. Association for Computational Linguistics.

Eiji Aramaki, Mizuki Morita, Yoshinobu Kano, and Tomoko Ohkuma. 2014. Overview of the NTCIR-11 MedNLP-2 Task. In Proceedings of NTCIR-11 (To be published).

Alan R Aronson. 2001. Effective mapping of biomedical text to the UMLS Metathesaurus: theMetaMap program. In Proceedings of AMIA Annual Symposium, pages 17–21.

Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Comput.Linguist., 34(4):555–596.

104 References

Miguel Ballesteros, Alberto Díaz, Virginia Francisco, Pablo Gervás, Jorge Carrillo de Albornoz, andLaura Plaza. 2012. Ucm-2: a rule-based approach to infer the scope of negation via dependencyparsing. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics –Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings ofthe Sixth International Workshop on Semantic Evaluation, pages 288–293, Montréal, Canada, 7-8June 2012. Association for Computational Linguistics.

Chris Biemann. 2005. Ontology learning from text: A survey of methods. In Alexander Mehler, editor,Themenschwerpunkt Korpuslinguistik, GLDV-Journal for Computational Linguistics and LanguageTechnology. Gesellschaft für Linguistische Datenverarbeitung e. V. (GLDV).

Christopher M. Bishop. 2006. Pattern recognition and machine learning. Springer, New York, NY.

Gunnar Blom. 1989. Sannolikhetsteori och statistikteori med tillämpningar (In Swedish), 4. uppl.edition. Studentlitteratur, Lund. ISBN 91-44-03594-2 ;.

Vincent D. Blondel, Anahí Gajardo, Maureen Heymans, Pierre Senellart, and Paul Van Dooren. 2004.A measure of similarity between graph vertices: Applications to synonym extraction and web search-ing. SIAM Review, 46(4):pp. 647–666. ISSN 00361445.

Svetla Boytcheva, Albena Strupchanska, Elena Paskaleva, Dimitar Tcharaktchiev, and Dame GruevStr. 2005. Some aspects of negation processing in electronic health records. In Proceedings ofInternational Workshop Language and Speech Infrastructure for Information Access in the BalkanCountries, pages 1–8, Borovets, Bulgaria.

Thorsten Brants and Oliver Plaehn. 2000. Interactive corpus annotation. In LREC. European LanguageResources Association. ISBN 2-9517408-6-7.

Kristan-Brent Broumana and Mia Edvinsson. 2014. Automatic extraction of abbreviations from aSwedish medical journal. Bachelor thesis, Stockholm University.

Peter F. Brown, Peter V. deSouza, Robert L. Mercer, Vincent J. Della Pietra, and Jenifer C. Lai. 1992.Class-based n-gram models of natural language. Computational Linguistics, 18:467–479.

Michael J. Campbell, David Machin, and Stephen J. Walters. 2007. Medical statistics : A textbook forthe health sciences, 4. ed. edition. Wiley, Chichester. ISBN 978-0-470-02519-2.

Hui Cao, Marianthi Markatou, Genevieve B Melton, Michael F Chiang, and George Hripcsak. 2005.Mining a clinical data warehouse to discover disease-finding associations using co-occurrence statis-tics. AMIA Annu Symp Proc, pages 106–110. ISSN 1942-597X (Electronic); 1559-4076 (Linking).

Johan Carlberger and Viggo Kann. 1999. Implementing an efficient part-of-speech tagger. Software–Practice and Experience, 29:815–832.

Staffan Cederblom. 2005. Medicinska förkortningar och akronymer (In Swedish). Studentlitteratur,Lund. ISBN 91-44-03393-1.

Jeffrey T. Chang, Hinrich Schütze Ph. D, Novation Biosciences, Russ B. Altman, and Ph. D. 2002.Creating an online dictionary of abbreviations from medline. Journal of the American MedicalInformatics Association, 9:612–620.

Brian E Chapman, Sean Lee, Hyunseok Peter Kang, and Wendy W Chapman. 2011. Document-levelclassification of CT pulmonary angiography reports based on an extension of the ConText algorithm.J Biomed Inform, 44(5):728–37.

Wendy W. Chapman, Will Bridewell, Paul Hanbury, Gregory F. Cooper, and Bruce G. Buchanan. 2001.A simple algorithm for identifying negated findings and diseases in discharge summaries. J BiomedInform, 34(5):301–310. ISSN 1532-0464 (Print).

References 105

Wendy W Chapman, Lee M Christensen, Michael M Wagner, Peter J Haug, Oleg Ivanov, John N Dowl-ing, and Robert T Olszewski. 2005. Classifying free-text triage chief complaints into syndromic cat-egories with natural language processing. Artif Intell Med, 33(1):31–40. ISSN 0933-3657 (Print);0933-3657 (Linking).

Wendy W Chapman, John Dowling, and David Chu. 2007. Context: An algorithm for identifyingcontextual features from clinical text. In Biological, translational, and clinical language processing,pages 81–88, Prague, Czech Republic, June 2007. Association for Computational Linguistics.

Wendy W Chapman and John N Dowling. 2006. Inductive creation of an annotation schema for man-ually indexing clinical conditions from emergency department reports. Journal of Biomedical Infor-matics, 39(2):196–208.

Wendy W Chapman, John N Dowling, and George Hripcsak. 2008. Evaluation of training with anannotation schema for manual annotation of clinical conditions from emergency department reports.Int J Med Inform, Epub 2007 Feb 20, 77(2):107–113.

Wendy W Chapman, Dieter Hillert, Sumithra Velupillai, Maria Kvist, Maria Skeppstedt, Brian E Chap-man, Mike Conway, Melissa Tharp, Danielle L Mowery, and Louise Deleger. 2013. Extending theNegEx lexicon for multiple languages. Stud Health Technol Inform, 192:677–81.

Wen-chi Chou, Richard Tzong-han Tsai, and Ying-shan Su. 2006. A semi-automatic method for anno-tating a biomedical proposition bank. In FLAC’06. ACL.

Cheryl Clark, John Aberdeen, Matt Coarr, David Tresner-Kirsch, Ben Wellner, Alexander Yeh, andLynette Hirschman. 2011. Mitre system for clinical assertion status classification. J Am Med InformAssoc, 18(5):563–7.

Aaron Cohen, William R. Hersh, Christopher Dubay, and Kent Spackman. 2005. Using co-occurrencenetwork structure to extract synonymous gene and protein names from medline abstracts. BMCBioinformatics, 6(1):103. ISSN 1471-2105.

Trevor Cohen and Dominic Widdows. 2009. Empirical distributional semantics: Methods and biomed-ical applications. Journal of Biomedical Informatics, 42(2):390 – 405. ISSN 1532-0464.

Conll. 2000. CoNLL-2000. http://www.cnts.ua.ac.be/conll2000/chunking/.

James R. Curran. 2005. Supersense tagging of unknown nouns using semantic similarity. In Pro-ceedings of the 43rd Annual Meeting on Association for Computational Linguistics, ACL ’05, pages26–33, Stroudsburg, PA, USA. Association for Computational Linguistics.

Hercules Dalianis, Martin Hassel, Aron Henriksson, and Maria Skeppstedt. 2012. Stockholm EPRCorpus: A Clinical Database Used to Improve Health Care. In Proceedings of SLTC 2012 - FourthSwedish Language Technology Conference, Lund, Sweden, October 25-26 2012.

Hercules Dalianis, Martin Hassel, and Sumithra Velupillai. 2009. The Stockholm EPR Corpus - Char-acteristics and Some Initial Findings. In Proceedings of ISHIMR 2009, Evaluation and implemen-tation of e-health and health information initiatives: international perspectives. 14th InternationalSymposium for Health Information Management Research, Kalmar, Sweden, pages 243–249.

Hercules Dalianis and Sumithra Velupillai. 2010. How certain are clinical assessments? AnnotatingSwedish clinical text for (un)certainties, speculations and negations. In Proceedings of the Sev-enth conference on International Language Resources and Evaluation (LREC’10) 19-21 May 2010;Valletta, Malta. European Language Resources Association (ELRA).

Dana Dannélls. 2006. Automatic acronym recognition. In Proceedings of the 11th conference onEuropean chapter of the Association for Computational Linguistics (EACL).

106 References

Berry de Bruijn, Colin Cherry, Svetlana Kiritchenko, Joel D. Martin, and Xiaodan Zhu. 2011. Machine-learned solutions for three stages of clinical information extraction: the state of the art at i2b2 2010.J Am Med Inform Assoc, 18(5):557–562.

Louise Deléger and Cyril Grouin. 2012. Detecting negation of medical problems in french clinicalnotes. In Proceedings of the 2Nd ACM SIGHIT International Health Informatics Symposium, IHI’12, pages 697–702, New York, NY, USA. ACM. ISBN 978-1-4503-0781-9.

Son Doan, Lisa Bastarache, Sergio Klimkowski, Joshua C Denny, and Hua Xu. 2010. Integratingexisting natural language processing tools for medication extraction from discharge summaries. JAm Med Inform Assoc, 17(5):528–531. ISSN 1527-974X (Electronic); 1067-5027 (Linking).

Son Doan, Nigel Collier, Hua Xu, Hoang Duy Pham, and Minh Phuong Tu. 2012. Recognition ofmedication information from discharge summaries using ensembles of classifiers. BMC Med InformDecis Mak, 12:36. ISSN 1472-6947 (Electronic); 1472-6947 (Linking).

Claudia Ehrentraut, Maria Kvist, Elda Sparrelid, and Hercules Dalianis. 2014. Detecting healthcare-associated infections in electronic health records - evaluation of machine learning and preprocessingtechniques. In Proceedings of the 6th International Symposium on Semantic Mining in Biomedicine(SMBM 2014).

Charles Elkan. 2008. Log-linear models and conditional random fields. Notes for a tutorial atCIKM’08.

Peter L Elkin, Steven H Brown, Brent A Bauer, Casey S Husser, William Carruth, Larry R Bergstrom,and Dietlind L Wahner-Roedler. 2005. A controlled trial of automated classification of negationfrom clinical notes. BMC Med Inform Decis Mak, 5:13+. ISSN 1472-6947 (Electronic).

Robert Eriksson, Peter Bjødstrup Jensen, Sune Frankild, Lars Juhl Jensen, and Søren Brunak. 2013.Dictionary construction and identification of possible adverse drug events in danish clinical narrativetext. J Am Med Inform Assoc.

Nicholas Evangelopoulos, Xiaoni Zhang, and Victor R. Prybutok. 2012. Latent semantic analysis: fivemethodological recommendations. EJIS.

FASS. 2012. Fass.se. www.fass.se, Accessed 2012-08-27 08-27.

Erik Forsgren and Robin Gredin. 2013. Semantisk parsning av kliniska fynd i svenska kliniska texter(In Swedish. English translation: Semantic parsing of clinical findings in Swedish health recordtext). Bachelor thesis, Stockholm University.

Carol Friedman. 2005. Semantic text parsing for patient records. In Hsinchun Chen, Sherrilynne S.Fuller, Carol Friedman, and William Hersh, editors, Medical informatics : knowledge managementand data mining in biomedicine. Springer Science+Business Media, Inc., Boston, MA. ISBN 978-0-387-25739-6.

Carol Friedman and George Hripcsak. 1998. Evaluating natural language in processors in the clinicaldomain. Methods of Information in Medicine, 37:334–344.

Carol Friedman, Lyuda Shagina, Yves Lussier, and George Hripcsak. 2004. Automated encoding ofclinical documents based on natural language processing. J Am Med Inform Assoc, 11(5).

Martin Gellerstam, Yvonne Cederholm, and Torgny Rasmark. 2000. The bank of Swedish. In LREC2000. The 2nd International Conference on Language Resources and Evaluation, pages 329–333,Athens, Greece.

References 107

Ilya M. Goldin and Wendy W. Chapman. 2003. Learning to detect negation with ‘not’ in medical texts.In Workshop at the 26th ACM SIGIR Conference.

Cyril Goutte and Eric Gaussier. 2005. A probabilistic interpretation of precision, recall and f-score,with implication for evaluation. In David Losada and Juan Fernández-Luna, editors, Advances inInformation Retrieval, volume 3408 of Lecture Notes in Computer Science, pages 345–359. SpringerBerlin / Heidelberg. ISBN 978-3-540-25295-5.

Gintare Grigonyte, Maria Kvist, Sumithra Velupillai, and Mats Wirén. 2014. Improving readability ofSwedish electronic health records through lexical simplification: First results. In Proceedings of the3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations – PITR,pages 74–83, Gothenburg, Sweden, April 2014. Association for Computational Linguistics.

Catalina Hallett, Richard Power, and Donia Scott. 2006. Summarisation and visualisation of e-healthdata repositories. In UK E-Science All-Hands Meeting, Nottingham, UK.

Henk Harkema, John N Dowling, Tyler Thornblade, and Wendy W Chapman. 2009. Context: analgorithm for determining negation, experiencer, and temporal status from clinical reports. J BiomedInform, 42(5):839–851. ISSN 1532-0480 (Electronic); 1532-0464 (Linking).

Marti Hearst. 1992. Automatic acquisition of hyponyms from large text corpora. In Proceedings ofCOLING 1992, pages 539–545.

Aron Henriksson, Mike Conway, Martin Duneld, and Wendy W. Chapman. 2013a. Identifying syn-onymy between SNOMED clinical terms of varying length using distributional analysis of electronichealth records. In Proceedings of the Annual Symposium of the American Medical Informatics As-sociation (AMIA 2013), Washington DC, USA.

Aron Henriksson, Hercules Dalianis, and Stewart Kowalski. 2014. Generating features for named en-tity recognition by learning prototypes in semantic space: The case of de-identifying health records.In IEEE International Conference on Bioinformatics and Biomedicine (BIBM).

Aron Henriksson and Martin Hassel. 2011. Exploiting Structured Data, Negation Detection andSNOMED CT Terms in a Random Indexing Approach to Clinical Coding. In Proceedings of Work-shop on Biomedical Natural Language Processing, Hissar, Bulgaria, September 15 2011.

Aron Henriksson, Maria Kvist, Hercules Dalianis, and Martin Duneld. Identifying named entities andtheir relationships in clinical notes for pharmacovigilance. To be submitted.

Aron Henriksson, Maria Skeppstedt, Maria Kvist, Martin Duneld, and Mike Conway. 2013b. Corpus-driven terminology development: Populating Swedish SNOMED CT with synonyms extracted fromelectronic health records. In Proceedings of the 2013 Workshop on Biomedical Natural LanguageProcessing, pages 36–44, Sofia, Bulgaria, August 2013. Association for Computational Linguistics.

Marketta Hiissa, Tapio Pahikkala, Hanna Suominen, Tuija Lehtikunnas, Barbro Back, Helena Karsten,Sanna Salanterä, and Tapio Salakoski. 2007. Towards automated classification of intensive carenursing narratives. Int J Med Inform, 76 Suppl 3:S362–8.

Lynette Hirschman and Inderjeet Mani. 2003. Evaluation. In Ruslan Mitkov, editor, The Oxford Hand-book of Computational Linguistics, pages 414–429. Oxford University Press, Oxford.

George Hripcsak and Adam S. Rothschild. 2005. Technical brief: Agreement, the f-measure, andreliability in information retrieval. J Am Med Inform Assoc, 12(3):296–298.

George Hripcsak and Adam Wilcox. 2002. Reference standards, judges, and comparison subjects:roles for experts in evaluating system performance. J Am Med Inform Assoc, pages 1–15.

108 References

Yang Huang and Henry J Lowe. 2007. A novel hybrid approach to automated negation detection inclinical radiology reports. J Am Med Inform Assoc, 14(3):304–311. ISSN 1067-5027 (Print).

Yang Huang, Henry J. Lowe, and William R. Hersh. 2003. A pilot study of contextual UMLS index-ing to improve the precision of concept-based representation in XML-structured clinical radiologyreports. J Am Med Inform Assoc, 10(6):580 – 587. ISSN 1067-5027.

i2b2/VA. 2010. 2010 i2b2/VA challenge evaluation assertion annotation guidelines.https://www.i2b2.org/NLP/Relations/assets/Assertion

i2b2/VA. 2010. 2010 i2b2/VA challenge evaluation, concept annotation guidelines.https://www.i2b2.org/NLP/Relations/ assets/Concept%20Annotation %20Guideline.pdf.

IHTSDO. 2008. International Health Terminology Standards Development Organisation: SNOMEDClinical Terms User Guide, July 2008 International Release. http://www.ihtsdo.org. Accessed 2011-01-24.

International Health Terminology Standards Development Organisation, IHTSDO. 2008a. SNOMEDCT style guide: Body structures – anatomy. http://www.ihtsdo.org. Accessed 2012-04-13.

International Health Terminology Standards Development Organisation, IHTSDO. 2008b. SNOMEDCT Style Guide: Clinical Findings. http://www.ihtsdo.org, Accessed 2011-01-24.

Niklas Isenius, Sumithra Velupillai, and Maria Kvist. 2012. Initial results in the development of scan:a swedish clinical abbreviation normalizer. In Proceedings of the CLEF 2012 Workshop on Cross-Language Evaluation of Methods, Applications, and Resources for eHealth Document Analysis -CLEFeHealth2012, Rome, Italy, September 2012. CLEF.

Min Jiang, Yukun Chen, Mei Liu, S Trent Rosenbloom, Subramani Mani, Joshua C Denny, and Hua Xu.2011. A study of machine-learning-based approaches to extract clinical entities and their assertionsfrom discharge summaries. J Am Med Inform Assoc.

Paul Johannesson and Erik Perjons. 2012. A Design Science Primer.

Siddhartha Jonnalagadda, Trevor Cohen, Stephen Wu, and Graciela Gonzalez. 2012. Enhancing clini-cal concept extraction with distributional semantics. J Biomed Inform, 45(1):129–40.

Daniel Jurafsky and James H. Martin. 2008. Speech and Language Processing: An Introduction toNatural Language Processing, Computational Linguistics and Speech Recognition, second edition.Prentice Hall. ISBN 013122798X.

Pentti Kanerva, Jan Kristoferson, and Anders Holst. 2000. Random indexing of text samples for latentsemantic analysis. In L. R. Gleitman and A. K. Joshi, editors, Proceedings of the 22nd AnnualConference of the Cognitive Science Society, Mahwah, NJ.

Karolinska Institutet. 2012. Hur man använder den svenska MeSHen (In Swedish, translated as: Howto use the Swedish MeSH). http://mesh.kib.ki.se/swemesh/manual_se.html. Accessed 2012-03-10.

Karin Kipper-Schuler, Vinod Kaggal, James J. Masanz, Philip V. Ogren, and Guergana K. Savova.2008. System evaluation on a named entity corpus from clinical notes. In Proceedings of the SixthInternational Conference on Language Resources and Evaluation (LREC’08), pages 3007–3011,Marrakech, Morocco, May 2008.

Ola Knutsson, Johnny Bigert, and Viggo Kann. 2003. A robust shallow parser for swedish. In Pro-ceedings of Nodalida.

References 109

Dimitrios Kokkinakis. 2012. The journal of the Swedish medical association - a corpus resource forbiomedical text mining in Swedish. In The Third Workshop on Building and Evaluating Resourcesfor Biomedical Text Mining (BioTxtM), an LREC Workshop. Turkey.

Dimitrios Kokkinakis and Anders Thurin. 2007. Identification of entity references in hospital dischargeletters. In Proceedings of the 16th Nordic Conference of Computational Linguistics (NODALIDA),pages 329–332, Estonia.

Taku Kudo. 2012. CRF++: Yet Another CRF toolkit. http://crfpp.sourceforge.net/. Accessed 2012-06-15.

Maria Kvist, Maria Skeppstedt, Sumithra Velupillai, and Hercules Dalianis. 2011. Modeling humancomprehension of Swedish medical records for intelligent access and summarization systems - futurevision, a physician’s perspective. In Proceedings of SHI 2011, Scandinavian Health Informaticsmeeting, pages 31–35.

Maria Kvist and Sumithra Velupillai. 2014. SCAN: A Swedish Clinical Abbreviation Normalizer.In Information Access Evaluation. Multilinguality, Multimodality, and Interaction, volume 8685 ofLecture Notes in Computer Science, pages 62–73. Springer International Publishing. ISBN 978-3-319-11381-4.

John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilis-tic models for segmenting and labeling sequence data. In Proc. 18th International Conf. on MachineLearning, pages 282–289. Morgan Kaufmann, San Francisco, CA.

Thomas K. Landauer. 2007. Handbook of latent semantic analysis. In Handbook of latent semanticanalysis, Mahwah, N.J. Lawrence Erlbaum Associates.

Thomas K Landauer and Susan T. Dutnais. 1997. A solution to Plato’s problem: The latent semanticanalysis theory of acquisition, induction, and representation of knowledge. Psychological review,pages 211–240.

William Long. 2005. Extracting diagnoses from discharge summaries. In AMIA Annual Symp Proc,pages 470–474.

Stephen Marsland. 2009. Machine learning : an algorithmic perspective. Chapman & Hall/CRC,Boca Raton, FL.

John McCrae and Nigel Collier. 2008. Synonym set extraction from the biomedical literature by lexicalpattern discovery. BMC Bioinformatics, 9:159.

Genevieve B Melton and George Hripcsak. 2005. Automated detection of adverse events using naturallanguage processing of discharge summaries. J Am Med Inform Assoc, 12(4):448–457. ISSN 1067-5027 (Print); 1067-5027 (Linking).

Stephane Meystre and Peter J Haug. 2006. Natural language processing to extract medical problemsfrom electronic clinical documents: performance evaluation. J Biomed Inform, 39(6):589–599.ISSN 1532-0480 (Electronic); 1532-0464 (Linking).

Stephane M. Meystre, Guergana K. Savova, Karin C. Kipper-Schuler, and John F. Hurdle. 2008. Ex-tracting information from textual documents in the electronic health record: a review of recent re-search. Yearb Med Inform, pages 128–144. ISSN 0943-4747 (Print); 0943-4747 (Linking).

Andrei Mikheev, Marc Moens, and Claire Grover. 1999. Named entity recognition without gazetteers.In Proceedings of the ninth conference on European chapter of the Association for ComputationalLinguistics, EACL ’99, pages 1–8, Stroudsburg, PA, USA. Association for Computational Linguis-tics.

110 References

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word repre-sentations in vector space. CoRR, abs/1301.3781.

Roser Morante and Walter Daelemans. 2009. A metalearning approach to processing the scope of nega-tion. In CoNLL ’09: Proceedings of the Thirteenth Conference on Computational Natural LanguageLearning, pages 21–29, Morristown, NJ, USA. Association for Computational Linguistics.

Dana Movshovitz-Attias and William W. Cohen. 2012. Alignment-HMM-based Extraction of Ab-breviations from Biomedical Text. In Proceedings of the 2012 Workshop on Biomedical NaturalLanguage Processing (BioNLP 2012), pages 47–55, Montréal, Canada, June 2012. Association forComputational Linguistics.

Pradeep G Mutalik, Aniruddha Deshpande, and Prakash M Nadkarni. 2001. Use of general-purposenegation detection to augment concept indexing of medical documents: a quantitative study usingthe umls. J Am Med Inform Assoc, 8(6):598–609. ISSN 1067-5027 (Print); 1067-5027 (Linking).

Arvind Neelakantan and Michael Collins. 2014. Learning dictionaries for named entity recognitionusing minimal supervision. In Gosse Bouma and Yannick Parmentier 0001, editors, Proceedingsof the 14th Conference of the European Chapter of the Association for Computational Linguistics,EACL 2014, April 26-30, 2014, Gothenburg, Sweden, pages 452–461. The Association for ComputerLinguistics. ISBN 978-1-937284-78-7.

NegEx. 2009. Negex version 2. http://www.dbmi.pitt.edu/chapman/negex.html. Accessed 2009-12-01.

Paul Newbold, William L. Carlson, and Betty Thorne. 2003. Statistics for business and economics, 5.ed. edition. Prentice-Hall, Upper Saddle River, N. J. ISBN 0-13-029320-2.

Inga Nilsson. 2007. Medicinsk dokumentation genom tiderna : En studie av den svenska patientjour-nalens utveckling under 1700-talet, 1800-talet och 1900-talet. Enheten för medicinens historia,Medicinska fakulteten, Lunds universitet, Lund. ISBN 978-91-633-1987-7.

Philip Ogren, Guergana Savova, and Christopher Chute. 2008. Constructing evaluation corpora forautomated clinical named entity recognition. In Proceedings of the Sixth International LanguageResources and Evaluation (LREC’08), pages 3143–3149, Marrakech, Morocco, May 2008. Euro-pean Language Resources Association (ELRA). ISBN 2-9517408-4-0.

Philip V. Ogren. 2006. Knowtator: A protégé plug-in for annotated corpus construction. In Proceed-ings of the 2006 Conference of the North American Chapter of the Association for ComputationalLinguistics on Human Language Technology, pages 273–275, Morristown, NJ, USA. Associationfor Computational Linguistics.

Fredrik Olsson. 2008. Bootstrapping Named Entity Annotation by Means of Active Machine Learning.PhD thesis, University of Gothenburg. Faculty of Arts.

Jon Patrick and Min Li. 2010. High accuracy information extraction of medication information fromclinical notes: 2009 i2b2 medication extraction challenge. J Am Med Inform Assoc, 17(5):524–527.ISSN 1527-974X (Electronic); 1067-5027 (Linking).

Jon Patrick, Yefeng Wang, and Peter Budd. 2007. An automated system for conversion of clinical notesinto SNOMED clinical terminology. In Proceedings of the fifth Australasian symposium on ACSWfrontiers - Volume 68, ACSW ’07, pages 219–226, Darlinghurst, Australia, Australia. AustralianComputer Society, Inc. ISBN 1-920-68285-X.

References 111

Olatz Perez-de Viñaspre and Maite Oronoz. 2014. Translating SNOMED CT terminology into a minorlanguage. In Proceedings of the 5th International Workshop on Health Text Mining and InformationAnalysis (Louhi), pages 38–45, Gothenburg, Sweden, April 2014. Association for ComputationalLinguistics.

Håkan Petersson, Gunnar Nilsson, Lars-Erik Strender, and Hans Åhlfeldt. 2001. The connection be-tween terms used in medical records and coding system: A study on Swedish primary health caredata. Med Inform Internet Med, 26(2):87–99. ISSN 1463-9238 (Print).

Catherine Plaisant, Richard Mushlin, Aaron Snyder, Jia Li, Dan Heller, Ben Shneiderman, andKaiser Permanente Colorado. 1998. Lifelines: Using visualization to enhance navigation and ana-lysis of patient records. In Proceedings of the 1998 American Medical Informatic Association An-nual Fall Symposium, pages 76–80.

Sampo Pyysalo, Filip Ginter, Hans Moen, Tapio Salakoski, and Sophia Ananiadou. 2013. Distribu-tional semantics resources for biomedical text processing. In Proceedings of Languages in Biologyand Medicine.

Angus Roberts, Robert Gaizasukas, Mark Hepple, and Yikun Guo. 2008. Combining terminologyresources and statistical methods for entity recognition: an evaluation. In Proceedings of the SixthInternational Conference on Language Resources and Evaluation (LREC’08), pages 2974–2979,Marrakech, Morocco, may 2008. European Language Resources Association (ELRA). ISBN 2-9517408-4-0.

Angus Roberts, Robert Gaizauskas, Mark Hepple, George Demetriou, Yikun Guo, Ian Roberts, andAndrea Setzer. 2009. Building a semantically annotated corpus of clinical texts. J. of BiomedicalInformatics, 42:950–966. ISSN 1532-0464.

Lior Rokach, Roni Romano, and Oded Maimon. 2008. Negation recognition in medical narrativereports. Information Retrieval, 11(6):499–538.

Francisco S. Roque, Peter B. Jensen, Henriette Schmock, Marlene Dalgaard, Massimo Andreatta,Thomas Hansen, Karen Søeby, Søren Bredkjær, Anders Juul, Thomas Werge, Lars J. Jensen, andSøren Brunak. 2011. Using electronic patient records to discover disease correlations and stratifypatient cohorts. PLoS Comput Biol, 7(8).

Magnus Rosell, Martin Hassel, and Viggo Kann. 2009. Global evaluation of random indexing throughSwedish word clustering compared to the people’s dictionary of synonyms. In Proceedings of RecentAdvances in Natural Language Processing, Borovets, Bulgaria, September 14-16 2009.

Magnus Sahlgren. 2008. The distributional hypothesis. Italian Journal of Linguistics, 20(1):33–54.

Magnus Sahlgren, Anders Holst, and Pentti Kanerva. 2008. Permutations as a means to encode orderin word space. In Proceedings of the 30th Annual Meeting of the Cognitive Science Society, pages1300–1305.

Magnus Sahlgren and Jussi Karlgren. 2009. Terminology mining in social media. In Proceedings ofthe 18th ACM conference on Information and knowledge management, CIKM ’09.

Sara Santiso, Arantza Casillas, Alicia Perez, Maite Oronoz, and Koldo Gojenola. 2014. Adverse drugevent prediction combining shallow analysis and machine learning. In Proceedings of the 5th Inter-national Workshop on Health Text Mining and Information Analysis (Louhi), pages 85–89, Gothen-burg, Sweden, April 2014. Association for Computational Linguistics.

112 References

Guergana K Savova, James J Masanz, Philip V Ogren, Jiaping Zheng, Sunghwan Sohn, Karin CKipper-Schuler, and Christopher G Chute. 2010. Mayo clinical text analysis and knowledge ex-traction system (cTAKES): architecture, component evaluation and applications. J Am Med InformAssoc, 17(5):507–513. ISSN 1527-974X (Electronic); 1067-5027 (Linking).

Ariel S. Schwartz and Marti A. Hearst. 2003. A simple algorithm for identifying abbreviation defini-tions in biomedical text. Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing,pages 451–462. ISSN 1793-5091.

Irina Shablinsky, Justin Starren, and Carol Friedman. 2000. Knowledge-driven highlighting of clinicaltexts: does it help or distract. In Proc AMIA Symp, page 1134.

Jonas Sjöbergh and Viggo Kann. 2004. Finding the correct interpretation of Swedish compounds astatistical approach. In Proceedings of LREC-2004, pages 899–902, Lisbon, Portugal.

Maria Skeppstedt. 2013a. Adapting a parser to clinical text by simple pre-processing rules. In Pro-ceedings of the 2013 Workshop on Biomedical Natural Language Processing, pages 98–101, Sofia,Bulgaria. Association for Computational Linguistics.

Maria Skeppstedt. 2013b. Annotating named entities in clinical text by combining pre-annotation andactive learning. In Proceedings of the Student Research Workshop at ACL, pages 74–80.

Maria Skeppstedt. 2014. Enhancing medical named entity recognition with features derived fromunsupervised methods. In in Proceedings of the Student Research Workshop at EACL.

Maria Skeppstedt, Hercules Dalianis, and Gunnar Nilsson. 2011. Retrieving disorders and findings:Results using SNOMED CT and NegEx adapted for Swedish. In Proceedings of the LOUHI 2011,Third International Workshop on Health Document Text Mining and Information Analysis. CEUR-WS.

Kelly Smith, Beata Megyesi, Sumithra Velupillai, and Maria Kvist. 2014. Professional language inswedish clinical text: Linguistic characterization and comparative studies. In Nordic Journal ofLinguistics.

Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng. 2012. Semantic Com-positionality Through Recursive Matrix-Vector Spaces. In Proceedings of the 2012 Conference onEmpirical Methods in Natural Language Processing (EMNLP).

Socialstyrelsen. 2006. Diagnosgranskningar utförda i Sverige 1997-2005 samt rad inför granskning.

Socialstyrelsen. 2011. Språkliga riktlinjer för översättningen av Snomed CT finns nu att beställa, (InSwedish, translated as: Linguistic guidelines for the translation of SNOMED CT can now be or-dered). http://www.socialstyrelsen.se/nyheter/2011mars/.

Socialstyrelsen. 2012. Diagnoskoder (ICD-10). http://www.socialstyrelsen.se/ klassifi-ceringochkoder/diagnoskoder. Accessed 2012-03-31.

Dawei Song, Guihong Cao, Peter D. Bruza, and Raymond Lau. 2007. Concept induction via fuzzyc-means clustering in a high-dimensional semantic space. In Advances in Fuzzy Clustering and itsApplications, pages 393–403. John Wiley & Sons, Chichester.

Brett R South, Danielle Mowery, Ying Suo, Jianwei Leng, Óscar Ferrández, Stephane M Meystre,and Wendy W Chapman. 2014. Evaluating the effects of machine pre-annotation and an interactiveannotation interface on manual de-identification of clinical text. J Biomed Inform, 50:162–72.

References 113

H Suominen, S Salantera, S Velupillai, WW Chapman, G Savova, N Elhadad, S Pradhan, BR South,DL Mowery, GJF Jones, J Leveling, L Kelly, L Goeuriot, D Martinez, and G Zuccon. 2013.Overview of the ShARe/CLEF eHealth Evaluation Lab 2013. In Pamela Forner, Henning Muller,Roberto Paredes, Paolo Rosso, and Benno Stein, editors, Information Access Evaluation. Multilin-guality, Multimodality, and Visualization. 4th International Conference of the CLEF Initiative, CLEF2013, Valencia, Spain, September 23-26, 2013. Proceedings, volume 8138 2013 of Lecture Notes inComputer Science, pages 212–231. Springer. ISBN 978-3-642-40801-4.

Charles. Sutton and Andrew McCallum. 2006. An introduction to conditional random fields for re-lational learning. In Lise Getoor and Ben Taskar, editors, Introduction to Statistical RelationalLearning. MIT Press.

Hideyuki Tanushi, Hercules Dalianis, and Gunnar Nilsson. 2011. Calculating prevalence of comor-bidity and comorbidity combinations with diabetes in hospital care in Sweden using a health carerecord database. In LOUHI, Third International Workshop on Health Document Text Mining andInformation Analysis, pages 59–65.

Michael Thelen and Ellen Riloff. 2002. A bootstrapping method for learning semantic lexicons usingextraction pattern contexts. In In Proc. 2002 Conf. Empirical Methods in NLP (EMNLP.

Yoshimasa Tsuruoka, Jun’ichi Tsujii, and Sophia Ananiadou. 2009. Fast full parsing by linear-chainconditional random fields. In Proceedings of the 12th Conference of the European Chapter of theAssociation for Computational Linguistics, EACL ’09, pages 790–798, Stroudsburg, PA, USA. As-sociation for Computational Linguistics.

Sidrat ul Muntaha, Maria Skeppstedt, Maria Kvist, and Hercules Dalianis. 2012. Entity recognition ofpharmaceutical drugs in Swedish clinical text. In Proceedings of SLTC 2012 The Fourth SwedishLanguage Technology Conference, pages 77–78.

U.S. National Library of Medicine. 2014. MeSH (Medical Subject Headings).http://www.ncbi.nlm.nih.gov/mesh.

Özlem Uzuner, Imre Solti, and Eithon Cadag. 2010a. Extracting medication information from clinicaltext. J Am Med Inform Assoc, 17(5):514–518. ISSN 1527-974X (Electronic); 1067-5027 (Linking).

Özlem Uzuner, Imre Solti, Fei Xia, and Eithon Cadag. 2010b. Community annotation experiment forground truth generation for the i2b2 medication challenge. J Am Med Inform Assoc, 17(5):519–523.

Özlem. Uzuner, Brett R. South, Shuying Shen, and Scott L. DuVall. 2011. 2010 i2b2/va challenge onconcepts, assertions, and relations in clinical text. J Am Med Inform Assoc, 18(5):552–556.

Erik Velldal, Lilja Øvrelid, Jonathon Read, and Stephan Oepen. 2012. Speculation and negation: Rules,rankers, and the role of syntax. Comput. Linguist., 38(2):369–410.

Sumithra Velupillai. 2011. Automatic Classification of Factuality Levels – A Case Study on SwedishDiagnoses and the Impact of Local Context. In Proc. The Fourth International Symposium on Lan-guages in Biology and Medicine – LBM 2011, Singapore, December 2011.

Sumithra Velupillai, Hercules Dalianis, and Maria Kvist. 2011. Factuality Levels of Diagnoses inSwedish Clinical Text. In A. Moen, S. K. Andersen, J. Aarts, and P. Hurlen, editors, Proc. XXIIIInternational Conference of the European Federation for Medical Informatics (User Centred Net-worked Health Care), pages 559 – 563, Oslo, August 2011. IOS Press.

Veronika Vincze, György Szarvas, Richárd Farkas, György Móra1, and János Csirik. 2008. The Bio-Scope Corpus: Biomedical texts annotated for uncertainty, negation and their scopes. BMC Bioin-formatics, 9(Suppl 11):S9.

114 References

Ellen M. Voorhees. 2002. The philosophy of information retrieval evaluation. In Revised Papers fromthe Second Workshop of the Cross-Language Evaluation Forum on Evaluation of Cross-LanguageInformation Retrieval Systems, CLEF ’01, pages 355–370, London, UK. Springer-Verlag. ISBN3-540-44042-9.

Yefeng Wang. 2009. Annotating and recognising named entities in clinical notes. In Proceedings ofthe ACL-IJCNLP Student Research Workshop, pages 18–26, Singapore.

Yefeng Wang and Jon Patrick. 2009. Cascading classifiers for named entity recognition in clinicalnotes. In Proceedings of the Workshop on Biomedical Information Extraction, pages 42–49.

WHO. 2012. WHO international classification of diseases (ICD).http://www.who.int/classifications/icd/en/. Accessed 2012-02-04.

Dominic Widdows. 2003. Unsupervised methods for developing taxonomies by combining syntacticand statistical information. In Proceedings of the 2003 Conference of the North American Chapter ofthe Association for Computational Linguistics on Human Language Technology - Volume 1, NAACL’03, pages 197–204, Stroudsburg, PA, USA. Association for Computational Linguistics.

Wikipedia. 2012. Projekt medicin/lista över sjukdomar (In Swedish, translated asProject drugs/ listing of deseases in Swedish. http://sv.wikipedia.org/w/index.php?title=Wikipedia:Projekt_medicin/Lista_över_sjukdomar&oldid =15872672. Accessed 2012-02-17,13:14.

Fei Xia and Meliha Yetisgen-Yildiz. 2012. Clinical corpus annotation: Challenges and strategies. InThe Third Workshop on Building and Evaluating Resources for Biomedical Text Mining (BioTxtM),an LREC Workshop. Turkey.

H. Yu, G. Hripcsak, and C. Friedman. 2002. Mapping abbreviations to full forms in biomedical articles.Journal of the American Medical Informatics Association : JAMIA, 9(3):262–272. ISSN 1067-5027.

Hong Yu and Eugene Agichtein. 2003. Extracting synonymous gene and protein terms from biologicalliterature. Bioinformatics, 1(19):340–349.

Jing Zhao, Aron Henriksson, and Henrik Boström. 2014. Detecting adverse drug events using concepthierarchies of clinical codes. In IEEE International Conference on Healthcare Informatics (ICHI).

Qiaoming Zhu, Junhui Li, Hongling Wang, and Guodong Zhou. 2010. A unified framework for scopelearning via simplified shallow semantic parsing. In EMNLP, pages 714–724.

Qinghua Zou, Wesley W. Chu, Craig Morioka, Gregory H. Leazer, and Hooshang Kangarloo. 2003.Indexfinder: A method of extracting key concepts from clinical texts for indexing. In Proceedingsof AMIA Annual Symposium, pages 763–767.

Appendices

116

Appendix A

Authors’ contributions

A.1 Study IRule-based Entity Recognition and Coverage of SNOMED CT in Swedish Clinical Text byMaria Skeppstedt, Maria Kvist, and Hercules Dalianis.

MS initiated and coordinated the study and was responsible for constructing and eval-uating the rule- and vocabulary-based system. MK provided domain expertise of clinicallanguage to all parts of the study and was responsible for the annotation part, which con-sisted of construction of annotation guidelines and of performing the annotation. The paperwas drafted jointly by MS and MK and HD provided feedback on several versions of thepaper.

A.2 Study IIVocabulary Expansion by Semantic Extraction of Medical Terms by Maria Skeppstedt, Mag-nus Ahltorp, and Aron Henriksson.

MS initiated and coordinated the study and designed and carried out the evaluation. MSand MA designed and carried out the CosAdd experiments, while AH designed and car-ried out the TermRep experiments. Introduction, background and discussion sections weredrafted by MS and MA, and the other sections by MS and AH.

A.3 Study IIISynonym Extraction and Abbreviation Expansion with Ensembles of Semantic Spaces byAron Henriksson, Hans Moen, Maria Skeppstedt, Vidas Daudaravicius, and Martin Duneld.

118 Appendix A.

AH was responsible for coordinating the study and was thus involved in all parts of it. AHwas responsible for the overall design of the study and for carrying out the experiments. AHinitiated the idea of combining semantic spaces induced from different corpora and imple-mented the evaluation and post-processing modules. AH also had the main responsibilityfor the manuscript and drafted parts of the background and results description. HM andMS contributed equally to the study. HM initiated the idea of combining semantic modelstrained differently (Random Indexing and Random Permutation) and was responsible fordesigning and implementing strategies for combining the output of multiple semantic mod-els. HM also drafted parts of the method description in the manuscript and surveyed relevantliterature. MS initiated the idea of applying the method to abbreviation-expansion extrac-tion and to different types of corpora. MS was responsible for designing the evaluation partof the study, as well as for preparing the reference standards. MS also drafted parts of thebackground and method description in the manuscript. VD, together with MS, was respon-sible for designing the post-processing filtering of candidate terms. MD provided feedbackon the design of the study and drafted parts of the background and method description inthe manuscript. MD also carried out the manual evaluation, and the analysis thereof. AH,HM, MS and MD analysed the results and drafted the discussion and conclusions in themanuscript. All authors read and approved the final manuscript.

A.4 Study IVAutomatic recognition of disorders, findings, pharmaceuticals and body structures fromclinical text: An annotation and machine learning study by Maria Skeppstedt, Maria Kvist,Gunnar H Nilsson, and Hercules Dalianis.

MS was responsible for the overall design of the study and for the named entity recogni-tion part, while MK was responsible for the annotation part. MK developed the annotationguidelines with assistance from MS, and GN revised them. MK was the main annota-tor, while MS and GN annotated subsets of the data. MK also provided domain expertiseof clinical language and carried out the error analysis. MS designed and carried out thenamed entity recognition experiments, with feedback from HD. HD drafted parts of thebackground, while MS drafted the rest of the manuscript with feedback from the other au-thors. All authors read and approved the final manuscript.

A.5 Study VICue-based assertion classification for Swedish clinical text – Developing a lexicon forpyConTextSwe by Sumithra Velupillai, Maria Skeppstedt, Maria Kvist, Danielle Mowery,Brian E Chapman, Hercules Dalianis, and Wendy W Chapman.

All authors planned and designed the study during a joint workshop. SV coordinated thestudy, designed and carried out the evaluation, managed the evaluation data and drafted the

Authors’ contributions 119

results section. MS designed and carried out the automatic generation of cue inflections,cue combinations as well as the frequency filtering of cues, and drafted the materials andmethods section. MK provided domain expertise of clinical language and was the mainperson responsible for translating and classifying the English cues, with assistance fromSV as well as from from MS. MK also performed the error analysis and drafted the erroranalysis section of the manuscript. DM drafted the introduction and background with as-sistance from HD, who also made the illustration. BC adapted pyContextNLP for Swedish,with additions by MS and SV, and also drafted the manuscript part describing the system.WC, together with MK, supervised all parts of the study and administrated the cue lists.WC, together with SV, MS, MK and DM, also determined what modality classes to use. Allauthors contributed to the manuscript discussion and conclusion. SV and MS finalised themanuscript which was read and approved by all authors.

120

Appendix B

Used evaluation methods

NLP systems are often evaluated by experimental methods. These evaluations are typicallybased on a comparison of the output of the artefact to a set of known expected outputs, i.e. areference standard (Friedman and Hripcsak, 1998). This could be manually annotated out-puts (Ogren, 2006), or an external resource, such as an existing vocabulary (Biemann, 2005;McCrae and Collier, 2008). There are a number of standard measures used for evaluatingNLP systems, as well as for evaluating the quality of the used reference standards.

B.1 Measuring the performance of an artefactThe most common measures (Alpaydin, 2010, pp. 489–493) (Campbell et al., 2007, p. 50),in e.g. information retrieval, for comparing the output of an artefact to the reference standardare precision, recall and F-score.

Precision (positive predictive value) =t p

t p+ f pRecall (sensitivity) =

t pt p+ f n

t p = true positives, the number of classification instances for which the predicted classmatches the actual class.

f p = false positives, the number of classification instances for which the classifier incor-rectly predicts the class.

tn = true negatives, the number classification instances for which the classifier correctlydoes not predict the class.

f n = false negatives, the number of classification instances for which the classifier failsto predict the actual class.

Precision for named entity recognition is thus the proportion of the total number of la-belled chunks that are correctly labelled, whereas recall is the proportion of chunks actually

122 Appendix B.

present in the reference standard that are correctly identified by the system (Jurafsky andMartin, 2008, p. 489). F-score or F-measure, is a combination of precision and recall.When precision and recall are given equal weight it is defined as follows (Jurafsky andMartin, 2008, p. 489):

F = 2 · precision · recallprecision+ recall

Negative predictive value measures the proportion of instances with a negative test out-come for which this is a correct outcome (Altman and Bland, 1994), while specificity mea-sures how well a system detects negatives occurring in the reference standard (Alpaydin,2010, p. 493) (Campbell et al., 2007, p. 50). Both are infrequently used within NLP, butwere included in one of the studies of this thesis to be able to compare achieved results tothose of a previous study.

Negative predictive value =tn

tn+ f nSpecificity =

tntn+ f p

B.2 Statistical testsTo estimate in what range a true value lies (Blom, 1989, p. 222), a confidence interval canbe computed for the values obtained in the experiments.

If the result of a random trial either can take the value success or the value failure andnothing else, and p is the probability for success, then if n independent executions of thetrial are performed, the random variable for the total number of successes is binomiallydistributed, written as Bin(n, p) (Newbold et al., 2003, p. 147). The distribution of truepositives given the number of returned objects M(= t p+ f p) is binomial with parametersn = M and p = precision. Likewise, the distribution of true positives given the number ofrelevant objects N(= t p+ f n) is binomial with parameters n = N and p = recall (Goutteand Gaussier, 2005). Precision and recall are thus binomial proportions, which means thatstatistical methods for binomial proportions can be applied for estimating confidence inter-val, as well as for significance testing. A binomial distribution can be approximated withthe normal distribution if the number of observed instances are large enough, which can bejustified through the central limit theorem (Andersson, 1968, chapter 2.7). The confidenceinterval can then be computed as (Campbell et al., 2007, p. 92, 94):

p± z1−α/2

√p(1− p)

np is the estimated proportion (e.g. precision or recall), n is the total number of instances

that were used in the test (e.g. the total number of relevant entities or the total number ofextracted entities). The constant z1−α/2 depends on the size of the confidence interval; fora 95% confidence interval the constant z1−α/2 is 1.96 (Campbell et al., 2007, pp. 91–92,

Used evaluation methods 123

94–96). As a rule of thumb, both p ∗ n and (1− p) ∗ n should be larger than 10 in orderto approximate the binomial distribution with the normal distribution (Andersson, 1968,chapter 2.7). For all confidence interval calculations in the studies of this thesis, the abovedescribed formula was used.

When two artefacts are compared by evaluating their outputs against a certain referencestandard, a significance test between the results of the two artefacts can be carried out tofind out if it is likely that the achieved results hold true for the population in general. If thetwo artefacts are evaluated against the same reference standard, a comparison of two groupsof paired observations is carried out (Campbell et al., 2007, p. 119). The same instance inthe reference standard then leads to two observations, the first one when evaluated againstone of the artefacts and the other when evaluated against the other. When the artefacts areevaluated against two different reference standards, a comparison of independent groupsis carried out (Campbell et al., 2007, p. 119). A significance test results in a p-value thatindicates the strength of evidence, and a p-value less than 0.01 indicates strong evidence ofa difference between the two artefacts (Campbell et al., 2007, p. 107).

For comparison of two groups of paired observations, McNemar’s test (Campbell et al.,2007, p. 120) or a binomial sign test (Newbold et al., 2003, p. 532) were used in the studiesof this thesis. For comparison of independent groups, the χ2-test was used (Campbell et al.,2007, pp. 126, 132–136).

B.3 Evaluating the quality of the reference standardA reference standard is often created using aggregated responses from many experts (Hripc-sak and Wilcox, 2002). The output from each of the experts can be combined by a majoritydecision, by a final evaluator that combines each individual response or by a discussion ofthe responses which leads to a consensus.

The reliability of the reference standard, can be measured by the agreement between thedifferent experts (the inter-annotator agreement). Reliability is, however, only a conditionfor validity, that is to what extent the annotated data correspond to a hypothetical ”trueannotation”. It is possible to achieve a high reliability and low validity if the comparedannotators all make the same mistakes compared to this ”true annotation” (Artstein andPoesio, 2008). A frequently used measure for inter-annotator agreement within NLP thattakes the probability of random agreement into account is the Cohen’s κ coefficient (Art-stein and Poesio, 2008). The κ score can, however, not be used when the negative cases arenot well-defined but overlap and vary in length between annotators. This is the case for an-notations of named entities in text, for which inter-annotator agreement is instead typicallymeasured in terms of F-score (Hripcsak and Rothschild, 2005).

124

Appendix C

Used vocabularies

Allthough medical vocabulary resources are more extensive for English than for Swedish,there are a number of medical vocabularies available in Swedish. The following vocabular-ies were used in the studies of this thesis:

SNOMED CT (Systematized Nomenclature of Medicine – Clinical Terms) has the pur-pose of providing a standardised terminology for clinical information (IHTSDO, 2008).Since 2011, there is an official Swedish translation, which, unlike the English version, doesnot contain any synonyms (Socialstyrelsen, 2011).

Clinical FindingsDisorders

Figure C.1: The hierarchy in SNOMED CT

The categories Disorder and Finding used in this thesis, were based on those givenby SNOMED CT International Health Terminology Standards Development Organisation,IHTSDO (2008b), which were as follows:

Clinical findings may be simply defined as observations, judgments or assessmentsabout patients. The problem with the terms “finding” and “observation” is thatthey seem to refer to the judgment of the observer rather than to the actual state

126 Appendix C.

of the body. “Organism state” has been suggested as a more neutral name, butit would need to be delimited from a “course of disease.” Examples of clinicalfindings include: difficulty swallowing, nose bleed, diabetes, headache, and soforth. More precise and reproducible definitions of clinical findings, and the pre-cise boundaries between findings and events, between findings and observables,between findings and situations, and the distinction between “finding” and “disor-der”, remain ongoing challenges at the margins. The distinction between a disor-der and an observation has proven to be difficult to define in a reproducible manneracross the tens of thousands of concepts included under clinical findings. Never-theless, there are several reliable characteristics of each sub-category (disordersand findings):

1.1) Disorders1) Disorders necessarily are abnormal.2) They have temporal persistence, with the (at least theoretical) possibility of their

manifestations being treated, in remission, or quiescent even though the disorderitself still present.

3) They necessarily have an underlying pathological process.1.2 Findings1) Findings may be normal (but not necessarily); no disorders may.2) Some findings may exist only at a single point in time (e.g. a serum sodium level);

no disorders may.3) Findings cannot be temporally separate from the observing of them (you can’t ob-

serve them and say they are absent, nor can you have the finding present when it isnot capable of being observed).

4) They cannot be defined in terms of an underlying pathological process that is presenteven when the observation itself is not present.

Disorders may be present as a propensity for certain abnormal states to occur, evenwhen treatment mitigates or resolves those abnormal states. In some cases thedisease process is irrefutable, e.g. meningococcal meningitis. In others an under-lying disease process is assumed based on the temporal and causal association ofthe disorder and its manifestation, e.g. nystagmus disorder is different from thefinding/observation of nystagmus, which can be a normal physiological responseto rotation of the head. If you spin around and around and then have nystagmus(the finding) you still do not have nystagmus disorder. And someone can have anystagmus disorder without currently manifesting nystagmus. Similarly, deafnessdisorder is different from the symptom (observation) of reduced hearing, which canbe due to a number of temporary causes such as excessive ear wax. (InternationalHealth Terminology Standards Development Organisation, IHTSDO, 2008b)

Finding and Disorder are two separate semantic categories within SNOMED CT. TheSNOMED CT hierarchy is structured according to the figure C.1, in which Clinical Findingencompasses what is called Disorder and Finding in this thesis, and Disorder is a sub-category of Clinical Finding.

ICD-10 codes (International Statistical Classification of Diseases and Related HealthProblems), is an international standard diagnostic classification for general epidemiologi-

Used vocabularies 127

cal and clinical use (WHO, 2012). The Swedish translation is named ICD-10-SE (Social-styrelsen, 2012).

MeSH (Medical Subject Headings) is a controlled vocabulary created for the purposeof indexing medical literature. The Swedish version has been translated from English byKarolinska Institutet University Library (Karolinska Institutet, 2012), of which a subset isfreely available from U.S. National Library of Medicine (2014).

Wikipedia: Projekt medicin is a Wikipedia list of names of diseases in Swedish(Wikipedia, 2012).

Abbreviations and acronyms extracted from the book ’Medicinska förkortningar ochakronymer’ (Cederblom, 2005), which lists Swedish medical abbreviations and acronymsthat have been manually collected from Swedish health records, newspapers, scientific arti-cles, etc.

FASS (Farmaceutiska Specialiteter i Sverige) A listing of pharmaceutical drugs used inSweden (FASS, 2012).

128

DEPARTMENT OF COMPUTER AND SYSTEMS SCIENCES Stockholm University/KTH

www.dsv.su.se Ph.D. theses:

No 91-004 Olsson, Jan An Architecture for Diagnostic Reasoning Based on Causal Models No 93-008 Orci, Terttu Temporal Reasoning and Data Bases No 93-009 Eriksson, Lars-Henrik Finitary Partial Definitions and General Logic No 93-010 Johannesson, Paul Schema Integration Schema Translation, and Interoperability in Federated Information Systems No 93-018 Wangler, Benkt Contributions to Functional Requirements Modelling No 93-019 Boman, Magnus A Logical Specification for Federated Information Systems No 93-024 Rayner, Manny Abductive Equivalential Translation and its Application to Natural-Language Database Interfacing No 93-025 Idestam-Almquist, Peter Generalization of Clauses No 93-026 Aronsson, Martin GCLA: The Design, Use, and Implementation of a Program Development No 93-029 Boström, Henrik Explanation-Based Transformation of Logic programs No 94-001 Samuelsson, Christer Fast Natural Language Parsing Using Explanation-Based Learning No 94-003 Ekenberg, Love Decision Support in Numerically Imprecise Domains No 94-004 Kowalski, Stewart IT Insecurity: A Multi-disciplinary Inquiry No 94-007 Asker, Lars Partial Explanations as a Basis for Learning No 94-009 Kjellin, Harald A Method for Acquiring and Refining Knowledge in Weak Theory Domains No 94-011 Britts, Stefan Object Database Design No 94-014 Kilander, Fredrik Incremental Conceptual Clustering in an On-Line Application No 95-019 Song, Wei Schema Integration: - Principles, Methods and Applications No 95-050 Johansson, Anna-Lena Logic Program Synthesis Using Schema Instantiation in an Interactive Environment No 95-054 Stensmo, Magnus Adaptive Automated Diagnosis No 96-004 Wærn, Annika Recognising Human Plans: Issues for Plan Recognition in Human - Computer Interaction No 96-006 Orsvärn, Klas Knowledge Modelling with Libraries of Task Decomposition Methods No 96-008 Dalianis, Hercules Concise Natural Language Generation from Formal Specifications No 96-009 Holm, Peter On the Design and Usage of Information Technology and the Structuring of Communication and Work No 96-018 Höök, Kristina A Glass Box Approach to Adaptive Hypermedia No 96-021 Yngström, Louise A Systemic-Holistic Approach to Academic Programmes in IT Security

No 97-005 Wohed, Rolf A Language for Enterprise and Information System Modelling No 97-008 Gambäck, Björn Processing Swedish Sentences: A Unification-Based Grammar and Some Applications No 97-010 Kapidzic Cicovic, Nada Extended Certificate Management System: Design and Protocols No 97-011 Danielson, Mats Computational Decision Analysis No 97-012 Wijkman, Pierre Contributions to Evolutionary Computation No 97-017 Zhang, Ying Multi-Temporal Database Management with a Visual Query Interface No 98-001 Essler, Ulf Analyzing Groupware Adoption: A Framework and Three Case Studies in Lotus Notes Deployment No 98-008 Koistinen, Jari Contributions in Distributed Object Systems Engineering No 99-009 Hakkarainen, Sari Dynamic Aspects and Semantic Enrichment in Schema Comparison No 99-015 Magnusson, Christer Hedging Shareholder Value in an IT dependent Business society - the Framework BRITS No 00-004 Verhagen, Henricus Norm Autonomous Agents No 00-006 Wohed, Petia Schema Quality, Schema Enrichment, and Reuse in Information Systems Analysis No 01-001 Hökenhammar, Peter Integrerad Beställningsprocess vid Datasystemutveckling No 01-008 von Schéele, Fabian Controlling Time and Communication in Service Economy No 01-015 Kajko-Mattsson, Mira Corrective Maintenance Maturity Model: Problem Management No 01-019 Stirna, Janis The Influence of Intentional and Situational Factors on Enterprise Modelling Tool Acquisition in Organisations No 01-020 Persson, Anne Enterprise Modelling in Practice: Situational Factors and their Influence on Adopting a Participative Approach No 02-003 Sneiders, Eriks Automated Question Answering: Template-Based Approach No 02-005 Eineborg, Martin Inductive Logic Programming for Part-of-Speech Tagging No 02-006 Bider, Ilia State-Oriented Business Process Modelling: Principles, Theory and Practice No 02-007 Malmberg, Åke Notations Supporting Knowledge Acquisition from Multiple Sources No 02-012 Männikkö-Barbutiu, Sirkku SENIOR CYBORGS- About Appropriation of Personal Computers Among Some Swedish Elderly People No 02-028 Brash, Danny Reuse in Information Systems Development: A Qualitative Inquiry No 03-001 Svensson, Martin Designing, Defining and Evaluating Social Navigation No 03-002 Espinoza, Fredrik Individual Service Provisioning No 03-004 Eriksson-Granskog, Agneta General Metarules for Interactive Modular Construction of Natural Deduction Proofs No 03-005 De Zoysa, T. Nandika Kasun A Model of Security Architecture for Multi-Party Transactions No 03-008 Tholander, Jakob Constructing to Learn, Learning to Construct - Studies on Computational Tools for Learning

No 03-009 Karlgren, Klas Mastering the Use of Gobbledygook - Studies on the Development of Expertise Through Exposure to Experienced Practitioners' Deliberation on Authentic Problems No 03-014 Kjellman, Arne Constructive Systems Science - The Only Remaining Alternative? No 03-015 Rydberg Fåhræus, Eva A Triple Helix of Learning Processes - How to cultivate learning, communication and collaboration among distance-education learners No 03-016 Zemke, Stefan Data Mining for Prediction - Financial Series Case No 04-002 Hulth, Anette Combining Machine Learning and Natural Language Processing for Automatic Keyword Extraction No 04-011 Jayaweera, Prasad M. A Unified Framework for e-Commerce Systems Development: Business Process Patterns Perspective No 04-013 Söderström, Eva B2B Standards Implementation: Issues and Solutions No 04-014 Backlund, Per Development Process Knowledge Transfer through Method Adaptation, Implementation, and Use No 05-003 Davies, Guy Mapping and Integration of Schema Representations of Component Specifications No 05-004 Jansson, Eva Working Together when Being Apart – An Analysis of Distributed Collaborative Work through ICT from an Organizational and Psychosocial Perspective No 05-007 Cöster, Rickard Algorithms and Representations for Personalised Information Access No 05-009 Ciobanu Morogan, Matei Security System for Ad-hoc Wireless Networks based on Generic Secure Objects No 05-010 Björck, Fredrik Discovering Information Security Management No 05-012 Brouwers, Lisa Microsimulation Models for Disaster Policy Making No 05-014 Näckros, Kjell Visualising Security through Computer Games Investigating Game-Based Instruction in ICT Security: an Experimental approach No 05-015 Bylund, Markus A Design Rationale for Pervasive Computing No 05-016 Strand, Mattias External Data Incorporation into Data Warehouses No 05-020 Casmir, Respickius A Dynamic and Adaptive Information Security Awareness (DAISA) approach No 05-021 Svensson, Harald Developing Support for Agile and Plan-Driven Methods No 05-022 Rudström, Åsa Co-Construction of Hybrid Spaces No 06-005 Lindgren, Tony Methods of Solving Conflicts among Induced Rules No 06-009 Wrigstad, Tobias Owner-Based Alias Management No 06-011 Skoglund, Mats Curbing Dependencies in Software Evolution No 06-012 Zdravkovic, Jelena Process Integration for the Extended Enterprise No 06-013 Olsson Neve, Theresia Capturing and Analysing Emotions to Support Organisational Learning: The Affect Based Learning Matrix No 06-016 Chaula, Job Asheri A Socio-Technical Analysis of Information Systems Security Assurance A Case Study for Effective Assurance

No 06-017 Tarimo, Charles N. ICT Security Readiness Checklist for Developing Countries: A Social-Technical Approach No 06-020 Kifle Gelan, Mengistu A Theoretical Model for Telemedicine - Social and Value Outcomes in Sub-Saharan Africa No 07-001 Fernaeus, Ylva Let’s Make a Digital Patchwork Designing for Children’s Creative Play with Programming Materials No 07-003 Bakari, Jabiri Kuwe A Holistic Approach for Managing ICT Security in Non-Commercial Organisations A Case Study in a Developing Country No 07-004 Sundholm, Hillevi Spaces within Spaces: The Construction of a Collaborative Reality No 07-005 Hansson, Karin A Framework for Evaluation of Flood Management Strategies No 07-007 Aidemark, Jan Strategic Planning of Knowledge Management Systems - A Problem Exploration Approach No 07-009 Jonsson, Martin Sensing and Making Sense Designing Middleware for Context Aware Computing No 07-013 Kabilan, Vandana Ontology for Information Systems (O4IS) Design Methodology: Conceptualizing, Designing and Representing Domain Ontologies No 07-014 Mattsson, Johan Pointing, Placing, Touching - Physical Manipulation and Coordination Techniques for Interactive Meeting Spaces No 07-015 Kessler, Anna-Maria A Systemic Approach Framework for Operational Risk - SAFOR No 08-001 Laaksolahti, Jarmo Plot, Spectacle and Experience: Contributions to the design and evaluation of Interactive Storytelling No 08-002 Van Nguyen Hong Mobile Agent Approach to Congestion Control in Heterogeneous Networks No 08-003 Rose-Mharie Åhlfeldt Information Security in Distributed Healthcare - Exploring the Needs for Achieving Patient Safety and Patient Privacy No 08-004 Sara Ljungblad Beyond users: Grounding technology in experience No 08-005 Eva Sjöqvist Electronic Mail and its Possible Negative Aspects in Organizational Contexts No 08-006 Thomas Sandholm Statistical Methods for Computational Markets – Proportional Share Market Prediction and Admission Control No 08-007 Lena Aggestam IT-supported Knowledge Repositories: Increasing their Usefulness by Supporting Knowledge Capture No 08-008 Jaana Nyfjord Towards Integrating Agile Development and Risk Management No 08-009 Åsa Smedberg Online Communities and Learning for Health - The Use of Online Health Communities and Online Expertise for People with Established Bad Habits No 08-010 Martin Henkel Service-based Processes - Design for Business and Technology No 08-012 Jan Odelstad Many-Sorted Implicative Conceptual Systems No 09-001 Marcus Nohlberg Securing Information Assets - Understanding, Measuring and Protecting against Social Engineering Attacks

No 09-002 Maria Håkansson Playing with Context - Explicit and Implicit Interaction in Mobile Media Applications No 09-003 Petter Karlström Call of the Wild Using language technology in the second language classroom No 09-009 Ananda Edirisurya Design Support for e-Commerce Information Systems using Goal, Business and Process Modelling No 10-005 Moses Niwe Organizational Patters for Knowledge Capture in B2B Engagements No 10-007 Mats Wiklund Perception of Computer Games in Non-Gaming Contexts No 10-008 Petra Sundström Designing Affective Loop Experiences No 10-009 Tharaka Ilayperuma Improving E-Business Design through Business No 11-002 David Sundgren The Apparent Arbitrariness of Second-Order Probability Distributions No 11-003 Atelach Argaw Resource Lenient Approaches to Cross Language Information Retrieval using Amharic No 11-004 Erik Perjons Model-Driven Networks, Enterprise Goals, Services and IT Systems No 11-005 Lourino Chemane ICT Platform Integration – A MCDM Based Framework for the Establishment of Value Network Case Study: Mozambique Government Electronic Network (GovNet) No 11-010 Christofer Waldenström Supporting Dynamic Decision Making in Naval Search and Evasion Tasks No 11-012 Gustaf Juell-Skielse Improving Organizational Effectiveness through Standard Application Packages and IT Services No 12-001 Edephonce Ngemera Nfuka IT Governance in Tanzanian public sector organizations No 12-002 Sumithra Velupillai Shades of Certainty Annotation and Classification of Swedish Medical Records No 12-003 Arvid Engström Collaborative Video Production After Television No12-007 Fatima Jonsson Hanging out in the game café. Contextualizing co-located game play practices and experiences No 12-008 Mona Riabacke A Prescriptive Approach to Eliciting Decision Information No 12-010 Geoffrey Rwezaura Karokola A Framework for Securing e-Government Services The Case of Tanzania No 13-001 Evelyn Kigozi Kahiigi A Collaborative E-learning Approach Exploring a Peer Assignment Review Process at the University Level in Uganda No 13-002 Mattias Rost Mobility is the message: Explorations in mobile media sharing No 13-003 Rasika Dayarathna Discovering Constructs and Dimensions for Information Privacy Metrics. No 13-004 Magnus Johansson Do Non Player Characters dream of electric sheep? No 13-006 Baki Cakici The Informed Gaze: On the Implications of ICT-Based Surveillance No 13-008 Johan Eliasson Tools for Designing Mobile Interaction with the Physical Environment in Outdoor Lessons No 13-010 Andreas Nilsson Doing IT Project Alignment – Adapting the DELTA Model using Design Science No. 14-001 Ola Caster Quantitative methods to support drug benefit-risk assessment

No. 14-002 Catarina Dudas Learning from Multi-Objective Optimization of Production Systems - A method for analyzing solution sets from multi-objective optimization No. 14-003 Thashmee M. Karunaratne Learning predictive models from graph data using pattern mining No. 14-007 David Hallberg Lifelong learning - The social impact of digital villages as community resource centres on disadvantaged women No. 14-008 Constantinos Giannoulis Model-driven Alignment - Linking Business Strategy with Information Systems No. 14-014 Jalal Nouri Orchestrating Scaffolded Outdoor Mobile Learning Activities No. 14-016 Jamie Walters Distributed Immersive Participation – Realising Multi-Criteria Context-Centric Relationships on an Internet of Things No. 14-017 Peter Mozelius Education for All in Sri Lanka - ICT4D Hubs for Region‐wide Dissemination of Blended Learning