Using Short-Answer Using Short-Answer Format Questions for an Format Questions for an

English Grammar Tutoring English Grammar Tutoring

SystemSystem Conceptualization & Research PlanningConceptualization & Research Planning

Jonggun GimJonggun Gim

Research Objective in one Research Objective in one sentencesentence

►To propose a software designed to To propose a software designed to generate short answer format question generate short answer format question and to analyze the answer to assess a and to analyze the answer to assess a student’s English grammar knowledge student’s English grammar knowledge and plan for further tutoring. and plan for further tutoring.

Intelligent Tutoring SystemIntelligent Tutoring System

► software packages that either teach or help software packages that either teach or help human tutors to teach closed sets of human tutors to teach closed sets of knowledge to students knowledge to students

► Shortcomings: Less Interactive, Less Shortcomings: Less Interactive, Less adaptive, Less Funadaptive, Less Fun

► AdvantagesAdvantages it can remember exactly what each student it can remember exactly what each student

knows and what should be taught next, knows and what should be taught next, it can employ limitless variation of approach to it can employ limitless variation of approach to

teach same lessonteach same lesson CheaperCheaper

Automatic Question Automatic Question GenerationGeneration

► [Mitkov] What type of verb form has to come [Mitkov] What type of verb form has to come after the verb, ‘enjoy’?①gerund ②infinitiveafter the verb, ‘enjoy’?①gerund ②infinitive

► [Chen] Complete the sentence by choosing t[Chen] Complete the sentence by choosing the most appropriate verb form: “I intend ___he most appropriate verb form: “I intend ______ you that we cannot approve your applicat___ you that we cannot approve your application..”① telling ② to tell ③ told ④ tell ion..”① telling ② to tell ③ told ④ tell

Originality in this researchOriginality in this research

►Short Answer format Question GenerationShort Answer format Question Generation►Change the following active voice Change the following active voice

sentence into a passive one: “All students sentence into a passive one: “All students admire Principal Lee.”admire Principal Lee.”

►We believe this approach is particularly We believe this approach is particularly efficient in tutoring grammar because an efficient in tutoring grammar because an instruction like English grammar requires instruction like English grammar requires not only the knowledge but also fluency not only the knowledge but also fluency that can be acquired through practice that can be acquired through practice with varying usage.with varying usage.

ApproachApproach

►Five Steps for using short-answer format Five Steps for using short-answer format for English Grammar Tutoring are;for English Grammar Tutoring are; 1) defining the set of grammatical rules to 1) defining the set of grammatical rules to

be mastered, be mastered, 2) selecting a grammatical rule to be 2) selecting a grammatical rule to be

questioned/practiced, questioned/practiced, 3) selecting an example sentence, 3) selecting an example sentence, 4) generating a question, and finally, 4) generating a question, and finally, 5) evaluating the answers for generating 5) evaluating the answers for generating

appropriate advice. appropriate advice.

Approach – Grammar Rule Approach – Grammar Rule setset

R2R2 TenseTense a Simple Present tense changes to a Present Continuous tense with changing the a Simple Present tense changes to a Present Continuous tense with changing the verb into appropriate ‘be’ + the verb inflected to a present participleverb into appropriate ‘be’ + the verb inflected to a present participle

R3R3 Passive Passive VoiceVoice

an Active voice sentence changes to passive one with swapped S and O + ‘be’ + past an Active voice sentence changes to passive one with swapped S and O + ‘be’ + past participle + ’by’participle + ’by’

R4R4 InterrogationInterrogation a yes/no question can be constructed from a descriptive sentence with a ‘be’ verb by a yes/no question can be constructed from a descriptive sentence with a ‘be’ verb by swapping S and V and changing the period to a question mark.swapping S and V and changing the period to a question mark.

R5R5 IdiomIdiom the verb ‘enjoy’ is followed by gerundthe verb ‘enjoy’ is followed by gerund

► Criteria for defining Subset of English Criteria for defining Subset of English Grammar for the systemGrammar for the system Rules that can be easily questioned by short Rules that can be easily questioned by short

answer formatanswer format Rules to be mastered by students who learns Rules to be mastered by students who learns

English as a second languageEnglish as a second language

Approach – Question PatternsApproach – Question Patterns► Question are made of a Direction and a Target SentenceQuestion are made of a Direction and a Target Sentence► Directions are decided by the rule to be tested (one-to-many)Directions are decided by the rule to be tested (one-to-many)

R2-P2, P2’, R3-P3, R4-P4, R2-P2, P2’, R3-P3, R4-P4, ► Target sentences are selected from existing corpus or reliable Target sentences are selected from existing corpus or reliable

web documentsweb documents

DirectionDirection Target SentencesTarget Sentences

P2P2 Change the following sentence to present Change the following sentence to present continuous tense:continuous tense:

““I play with my dog.”I play with my dog.”

P3P3 Change the following active voice sentence to Change the following active voice sentence to a passive one:a passive one:

““All students admire Principal Lee.”All students admire Principal Lee.”

P4P4 Change the sentence to a yes/no question:Change the sentence to a yes/no question: ““She wore a black sweater yesterday.”She wore a black sweater yesterday.”

P5P5 Complete the sentence:Complete the sentence: ““I enjoy _____ as a new hobby.”I enjoy _____ as a new hobby.”

Approach – Analysis of Approach – Analysis of Student’s AnswersStudent’s Answers

►Students’ knowledge on a grammatical rule iStudents’ knowledge on a grammatical rule is remembered as a probability between 0 ans remembered as a probability between 0 and 1, not as a True/False decisiond 1, not as a True/False decision

►For a question aimed to test the knowledge For a question aimed to test the knowledge of rule A can be used also to test knowledge of rule A can be used also to test knowledge of rule B by analyzing the student’s answerof rule B by analyzing the student’s answer ““Principal Lee is Principal Lee is admirenadmiren by all students.” by all students.”

Functional RequirementsFunctional Requirements

► Tutors shall be able to define Grammatical Rules to Tutors shall be able to define Grammatical Rules to be tested and Prerequisite relationship among thembe tested and Prerequisite relationship among them

► Tutors shall be able to choose a rule and command Tutors shall be able to choose a rule and command the system to generate certain numbers of the system to generate certain numbers of questions based on the rule and accompanying questions based on the rule and accompanying answer (or answer pattern)answer (or answer pattern)

► The system shall be able to parse the students’ The system shall be able to parse the students’ answer to:answer to: Tell if the answer is correct or notTell if the answer is correct or not Update the knowledge database of the studentUpdate the knowledge database of the student To explain the grammatical rule that the student seems To explain the grammatical rule that the student seems

not to understandnot to understand

Design and ImplementationDesign and Implementation

►Grammatical Rule RepresentationGrammatical Rule Representation GraviaX grammar checker (?)GraviaX grammar checker (?)

►Target sentence retrieval Target sentence retrieval Part of Speech analysis – NLTK (Natural LanguPart of Speech analysis – NLTK (Natural Langu

age ToolKit)age ToolKit)►Answer parsingAnswer parsing

POS analysis – NLTKPOS analysis – NLTK English Semantic Dictionary - WordNet English Semantic Dictionary - WordNet

Evaluation of the approachEvaluation of the approach

►Things to be evaluated from the Things to be evaluated from the suggested approachsuggested approach 1) how much the generated questions are 1) how much the generated questions are

similar to the ones generated by human similar to the ones generated by human experts experts

2) how much the provided advices based on 2) how much the provided advices based on the student’s answers are relevant the student’s answers are relevant

3) whether the questions are more diverse 3) whether the questions are more diverse than the ones generated by human than the ones generated by human

4) whether questions are more adaptively 4) whether questions are more adaptively generated than the ones developed by generated than the ones developed by humanhuman

ResultsResults

0

20

40

60

80

100

120

0 20 40 60 80 100 120

Human Generated Test ScoreC

om

put

er G

ener

ated

Tes

t Sco

re

►Score Correlation (chScore Correlation (chart on the right)art on the right)

►Survey on the relevaSurvey on the relevance of the advices, ance of the advices, adaptiveness & diversidaptiveness & diversity of questions will bty of questions will be collectede collected

DiscussionDiscussion

►Factors that determines the quality of Factors that determines the quality of tests will be reevaluatedtests will be reevaluated

►Will try to find more factors that Will try to find more factors that determine the quality of tests determine the quality of tests

►Will try to devise better ways to Will try to devise better ways to measure the current factors measure the current factors

►Other Components of Learning English Other Components of Learning English Short answer format question based Short answer format question based

Reading Comprehension TestReading Comprehension Test

ReferencesReferences

► Mitkov, R. & Ha, L.A. (2003). Mitkov, R. & Ha, L.A. (2003). Computer-Aided GeneratiComputer-Aided Generation of Multiple-Choice Tests. In Proceedings of the HLT-NAon of Multiple-Choice Tests. In Proceedings of the HLT-NAACL 2003 Workshop on Building Educational Applications ACL 2003 Workshop on Building Educational Applications Using Natural Language Processing, Edmonton, Canada, Using Natural Language Processing, Edmonton, Canada, May, p. 17-22May, p. 17-22

► Chia-Yin Chen, Hsien-Chin Liou, Jason S. Chang, (2006), Chia-Yin Chen, Hsien-Chin Liou, Jason S. Chang, (2006), FAST – An Automatic Generation for Grammar TestsFAST – An Automatic Generation for Grammar Tests

► Steven A. Stahl, (1986), Three principals of effective vocaSteven A. Stahl, (1986), Three principals of effective vocabulary instruction. Journal of Reading, 29 bulary instruction. Journal of Reading, 29

► NLTK (Natural Language ToolKit) documentation, http://nlNLTK (Natural Language ToolKit) documentation, http://nltk.sourceforge.net/index.php/Booktk.sourceforge.net/index.php/Book