1

Applications of Natural Language

ProcessingCourse 7 – 05 April 2012

Diana Trandabățdtrandabat@info.uaic.ro

2

NLP in eLearning◦Generating test questions◦Keywords identification◦Extraction of definitions

Content

3

eLearning comprises all forms of electronically supported learning and teaching.

eLearning 2.0 - with the emergence of Web 2.0  Conventional e-learning systems were based on

instructional packets, which were delivered to students using assignments. Assignments were evaluated by the teacher.

In contrast, the new e-learning places increased emphasis on social learning and use of social software such as blogs, wikis, podcasts etc.

eLearning

4

NLP techniques in educational applications working with textual data:◦ intelligent tutoring systems◦ automatic generation of exercises◦ assessment of learner generated discourse◦ reading and writting assistance

These applications require an adaptation of NLP techniques to various types of discourse, e.g. tutoring dialogues, which are different from typical task-oriented spoken dialogue systems.

Moreover, educational applications place strong requirements on NLP systems, which have to be robust yet accurate.

NLP in eLearning

5

eLearning

Educational Natural Language Processing

NLP

Computer assisted learning/instruction

Analysis and use of language by machines

6

Definition:◦ Field of research exploring the use of NLP

techniques in educational contexts Why?

◦ Large text repositories with user generated discourse and user generated metadata are created

◦ These repositories need advanced information management and NLP to be efficiently accessed

◦ Using these repositories to create structured knowledge bases can improve NLP

Educational NLP

7

Definition: All forms of assessment delivered with the help of computers

also called Computer Assisted/Aided Assessment (CAA)

Adequate question types for CAA (McKenna & Bull, 1999):◦ Multiple choice questions (MCQs)◦ True/False questions◦ Matching questions◦ Ranking questions◦ Sequencing questions◦ etc.

Computer-based Testing

8

Generation of questions and exercises◦ Writing test questions, especially objective test

items, is an extremely difficult and time consuming task for teachers

◦ Use of NLP to automatically generate objective test items, esp. for language learning

Assessment and evaluation of answers to subjective test items◦ Use of NLP to automatically:

Diagnose errors in short-answer essays Grade essays

NLP for Computer Assisted Assessment

9

Source data◦ Corpora: texts should be chosen according to

the learner model (level, mastered vocabulary) the instructor model (target language, word category)

◦ Lexical semantic resources, e.g. WordNet Tools

◦ Tokeniser and sentence splitter◦ Lemmatiser◦ Conjugation and declension tools◦ POS tagger◦ Parser and chunker

Automatic Generation of Test Items

10

Choose the correct answer among a set of possible answers:◦ Who was voted the best international footballer

for 2004?

(a) Henry(b) Beckham(c) Ronaldinho(d) Ronaldo Usually 3 to 5 alternative answers

Multiple-Choice Questions

Question focus

Distractors

Correct answer / Key

11

Distractors (also distracters) are the incorrect answers presented as a choice in a multiple-choice test◦ Challenge: Generation of "good" distractors

Ensure that there is only one correct response for single response MCQ

The key should not always occur at the same position in the list of answers

Distractors should be grammatically parallel with each other and approximately equal in length

Distractors should be plausible and attractive However, distractors should not be too close to the

correct answer and risk confusing students

Distractors

12

1. Selection of the key Unknown words that appear in a reading Domain-specific terms2. Generation of the question focus Constrained patterns Transformation of source clauses to

question focuses.Transitive verbs require objects → Which kind

of verbs require objects?

Multiple-Choice Questions

13

3, Generation of the distractors WordNet concepts which are semantically close to the

key, e.g. hypernyms and co-hyponyms◦ "Which part of speech serves as the most central element in a

clause?"◦ Key: "verb", ◦ Distractors: "noun", "adjective", "preposition“

Same POS Similar frequency range For grammar questions, use a declension or a

conjugation tool to generate different forms of the key, e.g. change case, number, person, mode, tense, etc.

Common student errors in the given context Collocations: frequent co-occurrence with either the left

or the right context

Multiple-Choice Questions

14

Consists of a portion of text with certain words removed

The student is asked to "fill in the blanks“ Challenges:

◦ Phrase the question so that only one correct answer is possible (e.g. verb to be conjugated)

Fill-in-the-Blank Questions

15

1. Selection of an input corpus 2. POS tagging 3. Selection of the blanks in the input corpus

◦ Every "n-th" (e.g. fifth or eighth) word in the text◦ Words in specified frequency ranges, e.g. only high

frequency or low frequency words◦ Words belonging to a given grammatical category◦ Open-class words, given their POS◦ Machine learning, based on a pool of input questions used

as training data 4. Where needed, provide some information about

the word in the blank, e.g. verb lemma when the test targets verb conjugation

Fill-in-the-Blank Question Generation

16

Short answer assessment◦ Learner's response, one +

target responses, question, source reading passage

◦ Linguistic analysis: annotation, alignment, diagnosis

Essays Plagiarism detection Speech generation

Overview on assessment of learner generated data

17

Related techniques: summarisation and sentence compression

Syntactic simplification:◦ Removal or replacement of difficult syntactic

structures, using hand-built transformational rules applied to dependency and parse trees

Lexical simplification:◦ Replace difficult words with simpler ones◦ Difficult words are identified using the number of

syllables and/or frequency counts in a corpus◦ Choose the simplest synonym for difficult words in

WordNet

Automatic Text Simplification

18

Overall goal: support vocabulary acquisition during reading for:◦ children, who learn to read◦ foreign language learners, who read texts in a

foreign language Problem: a word's context may not provide

enough information about its meaning Solution: augment documents with

dynamically generated annotations about (problematic) words

Vocabulary Assistance for Reading

19

A grammar is created for the automatic identification of definitions in texts

Types of definitions “is_def” – “HTML este tot un protocol folosit de

World Wide Web.” (HTML is also a protocol used by World Wide Web).

“verb_def” – “Poşta electronică reprezintă transmisia mesajelor prin intermediul unor reţele electronice.” (Electronic mail represents sending messages through electronic networks).

“punct_def” – “Bit – prescurtarea pentru binary digit” (Bit – shortcut for binary digit)

Automatic detection of definitions

Types of definitions layout_def

“pron_def” – “…definirii conceptului de baze de date. Acesta descrie metode de ….” (…defining the database concept. It describes methods of ….)

“other_def” – “triunghi echilateral, adică cu toate laturile egale” (equilateral triangle i.e. having all sides equal).

Ro:

Organizarea datelor

Cel mai simplu mod de organizare este cel secvenţial.

En:

Data organizing The simplest method is the sequential one.

Distribution of the definitions

Type Manual % Automatic %

is_def 70 33.8 204 32.8

verb_def 116 56.0 272 43.8

punct_def 15 7.2 124 20.0

layout_def 2 1.0 21 3.4

pron_def 4 2.0 0 0.0

Total 207 621

Rules Simple grammar rules Composed grammar rules

“is_def” grammar rule:<rule name="may_be_term">

<seq> <query match="tok[@base='fi' and

substring(@ctag,1,5)='vmip3']"/>

<first> <ref name="UndefNominal" />

<ref name="DefNominal" />

</first></seq>

</rule>

Evaluation

Definition Type Resultis_def Sentence-level matching:

P: 0.5366, R: 1.0, F2: 0.7765 Token-level matching:P: 0.0648, R: 0.3328, F2: 0.14

verb_def Sentence-level matching P: 0.7561, R: 1.0, F2: 0.9029 Token-level matchingP: 0.0471, R: 0.1422, F2: 0.085

punct_def Sentence-level matching P: 0.1463, R: 1.0, F2: 0.3396 Token-level matching P: 0.0025, R: 0.1163, F2: 0.0072

layout_def Sentence-level matching P: 0.0488, R: 1.0, F2: 0.1333 Token-level matching P: 0.0007, R: 0.1020, F2: 0.0022

Lxtransduce (Tobin 2005) is used to match the grammar in files

Question Answering Accordingly to the answer type, we have

the following type of questions (Harabagiu, Moldovan 2007):

◦ Factoid – “Who discovered the oxygen?” or “When did Hawaii become a state?” or “What football team won the World Coup in 1992?”.

◦ List – “What countries export oil?” or “What are the regions preferred by the Americans for holidays?”.

◦ Definition – “What is quasar?” or “What is a question-answering system?”

QA – Example Question: Cine este Zeus? (Cine, zeus, PERSON)

Snippet: 0026#10014#1.0#Zeus#Zeus\zeus\NP este\fi\V3\ cel\cel\TSR\ mai\mai\R\ puternic\puternic\ASN\ dintre\dintre\S\ olimpieni\olimpieni\NPN\ ,\,\COMMA\ socotit\socoti\VP\ drept\drept\S\ stăpânul\stăpân\NSRY\ suprem\suprem\ASN\ al\al\TS\ oamenilor\om\NPOY\ şi\şi\CR\ al\al\TS\ zeilor\zeu\NPOY\ .\.\PERIOD\

Our pattern for “is_def” (\zeus\.*\NP .*\fi\V3\ (.*)) match the snippet

Keywords extraction Using a trening corpus of documents

annotated with keywords Measuring distribution of manually marked

keywords over documents

# of annotated documents

Average length (# of tokens)

Bulgarian 55 3980Czech 465 672Dutch 72 6912English 36 9707German 34 8201Polish 25 4432Portuguese 29 8438Romanian 41 3375

# of keywords Average # of keywords per doc.

Bulgarian 3236 77Czech 1640 3.5Dutch 1706 24English 1174 26German 1344 39.5Polish 1033 41Portuguese 997 34Romanian 2555 62

Reflection Did the human annotators annotate

keywords of domain terms? Was the task adequately contextualised?

Keyword extraction

Good keywords have a typical, non random distribution in and across documents

Keywords tend to appear more often at certain places in texts (headings etc.)

Keywords are often highlighted / emphasised by authors

Keywords express / represent the topic(s) of a text

Modelling Keywordiness Linguistic filtering of KW candidates, based

on part of speech and morphology Distributional measures are used to identify

unevenly distributed words◦TFIDF

Knowledge of text structure used to identify salient regions (e.g., headings)

Layout features of texts used to identify emphasised words and weight them higher

Finding chains of semantically related words

Challenges

Treating multi word keywords Assigning a combined weight which takes

into account all the aforementioned factors

Multilinguality: finding good settings for all languages, balancing language dependent and language independent features

Treatment of keyphrases Keyphrases have to be restricted wrt to

length (max 3 words) and frequency (min 2 occurrences)

Keyphrase patterns must be restricted wrt to linguistic categories (style of learning is acceptable; of learning styles is not)

KWE Evaluation Human annotators marked n keywords in

document d First n choices of KWE for document d

extracted Measure overlap between both sets measure also partial matches

35

Resources:◦ Lexical semantic resources, e.g. WordNet◦ Web 2.0 resources, e.g. Wikipedia, Wiktionary

Tools:◦ Tokeniser and sentence splitting◦ Morphological analysis◦ Part of speech tagging◦ Parsing and chunking◦ Word sense disambiguation◦ Summarisation◦ Keyword extraction

NLP has lots to offer

36

To assist instructors◦ Automatic generation of questions and exercises◦ Assessment of learner-generated discourse

To assist learners◦ Reading and writing assistance◦ Electronic career guidance◦ Educational question answering

For all users in the Web 2.0◦ NLP for wikis◦ Quality assessment of user generated contents

Tasks and applications

37

Computer-Assisted Language Learning Intelligent Tutoring Systems Information search for eLearning Educational blogging Annotations and social tagging Analysing collaborative learning processes

automatically Learners' corpora and resources eLearning standards, e.g. SCORM

A lot more research is done on:

38

1a) Extract definitions from a given Wikipedia page 1b) Generate questions such as “what is …" or “what is

the meaning of …" from the list above

2) Automatic generation of “fill the blanks” questions Dacă nu ai nimic planificat diseară, hai __ teatru. (a) la (b) de (c) pentru (d) null

◦ Input: a sentence and the key Dacă nu ai nimic planificat diseară, hai la teatru. Key: la

◦ Output: generate three distractors using different approaches: baseline: word frequencies Collocations "creative" method, devised by the students

Requirements (Team: max 2 persons, Deadline: 12 April)

Further reading Jill Burstein: Opportunities for Natural Language

Processing Research in Education ProceedingCICLing '09 Proceedings of the 10th International Conference on Computational Linguistics and Intelligent Text Processing, Springer-Verlag Berlin, Heidelberg, 2009 

Paola Monachesi, Eline Westerhout. What can NLP techniques do for eLearning? Presented at INFOS 2008, 27-29 March.

Adrian Iftene, Diana Trandabăţ, Ionuţ Pistol: Grammar-based Automatic Extraction of Definitions and Applications for Romanian. RANLP 2007 workshop: Natural Language Processing and Knowledge Representation for eLearning Environments. 

39

40

Educational Applications of NLP http://www.ets.org/research/topics/as_nlp/educational_applications

Links

41

Thanks!

42

(1) Plagiarism of authorship: the direct case of putting your own name to someone else’s work

(2) Word-for-word plagiarism: copying of phrases or passages from published text without quotation or acknowledgement.

(3) Paraphrasing plagiarism: words or syntax are changed (rewritten), but the source text can still be recognized.

(4) Plagiarism of the form of a source: the structure of an argument in a source is copied (verbatim or rewritten)

(5) Plagiarism of ideas: the reuse of an original thought from a source text without dependence on the words or form of the source

(6) Plagiarism of secondary sources: original sources are referenced or quoted, but obtained from a secondary source text without looking up the original.

Types of plagiarism