ECEL 2010 - Presentation
Enhanced Approach of Automatic Creation of Test Items to
foster Modern Learning Setting
Institute for Information Systems and Computer Media (IICM)Faculty of Informatics – Graz University of Technology, Austria
Christian Gütl, Klaus Lankmayr, and Joachim Weinhofer
Version October 24th, 2010
© 2010 - Christian Gütl19/04/23 2
Advanced Educational Media Technologies Group
• Remote & Collaborative Learning• Adaptive e-learning systems• Virtual learning environments
• E-Assessment• Computer-based assessment• Peer assessment
• Virtual Environments & Setups • Virtual 3D World Infrastructure• End devices (immersive … mobile)
• Multimedia IR and NLP• Access to learning repositories• Pre-processing of content
© 2010 - Christian Gütl19/04/23 3
Agenda
• Introduction
• Background & Related Work
• Requirements & Solution
• A First Qualitative Study
• Current & Future Work
Introduction
© 2010 - Christian Gütl19/04/23 4
• Knowledge-driven and globalized world requires continuous adaption (Gütl, 2010)
• New forms and settings of learning… informal learning, self-guided learning, …
• Assessment and feedback (Bransford et al., 2000)
• Time consuming, in particular for adaptive learning environments
• Almost impossible for self-directed and life-long / informal learning settings
Flexible e-assessment models & toolsAutomatic or semi-automatic text-item
creation from learning content
Background & Related Work
© 2010 - Christian Gütl19/04/23 5
• Not much has be done so far• Multiple choice test item (Coniam, 1997)
part of speech tagging for word candidatesword distractors by statistical methods from text corpus
• Multiple choice test item (Mitkov et a., 2003)test items by rule-bases approachdistractors’ estimation by using WordNet
• Multiple choice test item (Hoshino et al., 2005)machine learning + training set to determine blanksapplication domain: news article
• Factual question test items (Rus et al., 2007)applying patterns, templates and special markup language
• Limited-choice & completion items (Gütl, 2008)automatic summarization, part of speech tagging
No flexible support of different test items and learning settings
Objectives & Requirements
© 2010 - Christian Gütl19/04/23 6
• Goal for Automatic Question Creator is• … support (semi-)automatic test item creation• … use stand alone and integrate in LMS• … flexible design to adjust tool to specific setting
• High-level requirements• Support various file formats, local & Internet• Domain knowledge and document structure
independent• Extraction of concepts from learning content• Creation of test items and reference answers
based on identified concepts• Support of various test items types:
open ended, single/multiple choice, completion exercise
Conceptual Architecture
© 2010 - Christian Gütl19/04/23 7
• File format conversationWord, OD, PDF HTML
• Text cleaningHTML clean up
• Internal RepresentationXML structure (HTML XML)
• Analysis & Weightingtokenizing, POS, stemming, named entities, co-occurrence, semantic & lexical relations
• Concept Selectionnoun phrases concept
• Preprocessingrelevant concepts per chaptermost appropriate content
distractors & reference answer
• Item Creation & Export
Test Item Creation
© 2010 - Christian Gütl19/04/23 8
• Completion Exercises (Fill-in blank)• Find relevant sentence for concept (weighting)
and semantically important additional text • Remove concept (phrases)
• Multiple Choice• Find relevant sentence for concept• Find relevant distractors for concept
• Single Choice (true/false questions)• Find antonyms for single words of concepts• Create sentences with wrong meaning
• Open Ended Questions• Using pattern and rules for question creation• Creating reference answer from content by finding
most appropriate text block (text tiling)
Prototype 1/4
© 2010 - Christian Gütl19/04/23 9
Prototype 2/4
© 2010 - Christian Gütl19/04/23 10
Prototype 3/4
© 2010 - Christian Gütl19/04/23 11
Prototype 4/4
© 2010 - Christian Gütl19/04/23 12
User Study - Setup
© 2010 - Christian Gütl19/04/23 13
• Research Question:Performance of Tool (concepts, test items)
• First qualitative study• 5 subjects (1 college & 4 university students)
• Evaluation Procedure1. Read learning content carefully
“Project Management” (MIT courseware)2. Identify 5 most expressive concepts for each of
the 3 chapters3. Construct per chapter 2 test item for each type4. Rate quality of given concepts for each chapter5. Rate quality (multiple dimensions) of given test
items for each chapter
User Study - Quality of Concepts
© 2010 - Christian Gütl19/04/23 14
• For each chapter 10 highest ranked concepts
• Appropriateness rating (0 …. 100 %)
• Ratings above 50 % counted as relevant
User Study - Quality of Test Items
© 2010 - Christian Gütl19/04/23 15
• For each chapter 2 automatic and 2 manual crated test items provides for assessment
• Assessment according to following criteria:• Pertinence: relevance of question in given context• Level: Level of difficulty of a test item• Distractor: Quality of listed distractors (multiple choice)• Answer: quality of reference answer (open ended questions)
Conclusion & Future Work
© 2010 - Christian Gütl19/04/23 16
• Surprisingly good quality of created test items• Subject domain indepentend• High time complexity, processing time
• Further improvements• Multilinguality (German, Spanish, …) • Improvements of methods to identify concepts and
create questions (quality and performance)• Web service & integration in assessment framework
• Further Evaluation and practical use of the tool• Detailed evaluation with 30 subjects• Usage and evaluation for test item creation support on
medical content (Medical University Graz)• Usage and evaluation for self-directed learning
activities and complex learning experiences (part of EC funded project ALICE)
ReferencesBransford, J.D., Brown, A.L., & Cocking; R.R. (Eds.) (2000) “How People Learn: Brain, Mind, Experience, and
School. Expanded Edition. Washington DC: National Academies Press.
Coniam, D. (1997). A Preliminary Inquiry Into Using Corpus Word Frequency Data in the Automatic Generation of English Language Cloze Tests. Calico Journal, Vol 14, No. 2–4, pp 15-34.
Gütl, C. (2008). Automatic Limited-Choice and Completion Test Creation, Assessment and Feedback in modern Learning Processes. Paper read at LRN Conference 2008, Guatemala, February 12th – 16th.
Gütl, C. (2010) “The Support of Virtual 3D Worlds for enhancing Collaboration in Learning Settings. In Francesca Pozzi and Donatella Persico (Eds.) Techniques for Fostering Collaboration in Online Learning Communities: Theoretical and Practical Perspectives, IGI Global, 2010 (accepted)
Hoshino, A. and Nakagawa, H. (2005). A real-time multiple-choice question generation for language testing: a preliminary study”, Proceedings of the second workshop on Building Educational Applications Using NLP, pp 17-20.
Mitkov, R. and Ha, A. L. (2003). Computer-Aided Generation of Multiple-Choice Tests. Proceedings of the HLT-NAACL 2003 workshop on Building educational applications using natural language processing, pp 17-22.
Rus, V., Cai, Z. and Graesser, A. C. (2007). Experiments on Generating Questions About Facts. Computational Linguistics and Intelligent Text Processing, 8th International Conference, CICLing 2007, Mexiko City, Mexiko, February 18-24, 2007, Proceedings, pp 444–455.
© 2010 - Christian Gütl19/04/23 17
© 2010 - Christian Gütl19/04/23 18
Questions & Contact Information
Thank you for your Attention!Questions are welcome!
Further Information:
Christian Gütl
http://www.iicm.edu/guetl
Weighting Algorithm
© 2010 - Christian Gütl19/04/23 19
Test Item Example 1/4
© 2010 - Christian Gütl19/04/23 20
Fill in the blank 2-2question: Hence, management functions can be organized into a hierarchical structure
designed to improve operational efficiency. ______________ are performed by all managers, regardless of enterprise, activity or hierarchical levels. Finally, the development of a management philosophy results in helping the manager to establish relationships between human and material resources.
correct answer The basic management functions
criteria Pertinence Level Concept
value
Test Item Example 2/4
© 2010 - Christian Gütl19/04/23 21
Multiple Choice 2-2question: Hence, management functions can be organized into a hierarchical
structure designed to improve operational efficiency. ______________ are performed by all managers, regardless of enterprise, activity or hierarchical levels. Finally, the development of a management philosophy results in helping the manager to establish relationships between human and material resources.
distractors The basic documentation functions
The basic serviceablenes functions
The basic chores functions
The basic management functions
correct answer The basic management functionscriteria Pertinence Level Distractors
value
Test Item Example 3/4
© 2010 - Christian Gütl19/04/23 22
Single Choice 2-1question: The optimization or suboptimization is often achieved by the use of operations
research techniques, such as linear programming, quadratic programming, graph theory, queuing theory and Michael porter simulation.
correct answer falsecriteria Pertinence Levelvalue
Test Item Example 4/4
© 2010 - Christian Gütl19/04/23 23
Open Ended 2-3question: Which operation research techniques are used for optimization?
Answer region A topic of major interest in management science is the maximization of profit, or in the absence of a workable model for the operation of the entire system, the suboptimization of the operations of its components. The optimization or suboptimization is often achieved by the use of operations research techniques, such as linear programming, quadratic programming, graph theory, queuing theory and Monte Carlo simulation
criteria Pertinence Level Answervalue
AQC
© 2010 - Christian Gütl19/04/23 24
AQC
© 2010 - Christian Gütl19/04/23 25
AQC
© 2010 - Christian Gütl19/04/23 26
AQC
© 2010 - Christian Gütl19/04/23 27
AQC
© 2010 - Christian Gütl19/04/23 28
AQC
© 2010 - Christian Gütl19/04/23 29
AQC
© 2010 - Christian Gütl19/04/23 30
AQC
© 2010 - Christian Gütl19/04/23 31
AQC
© 2010 - Christian Gütl19/04/23 32
AQC
© 2010 - Christian Gütl19/04/23 33