introduction to opensemcq
DESCRIPTION
Semantic Web technologies have been increasingly used as a tool for generating, organizing and personalizing e-learning content. In this presentation we will discuss and demonstrate an innovative approach to automated generation of computer-assisted assessment (CAA) from Semantic Web–based domain ontologies. The primary application domain of this work is in the automated assessment, and in particular, the development of intelligent CAA systems and question banks, but the ideas can be further generalized in the context of ontology engineering and evaluation. Prototype is implemented and available online at http://www.opensemcq.orgTRANSCRIPT
Towards automatic generation of
e-assessment using semantic web technologies * †
1
*Presentation based on Cubric,M.&Tosic,M.(2010/2011) ‘Towards automatic generation of e- assessment using semantic web technologies; International Conference on CAA, University of Southampton, July 2010; Also to appear in International Journal of E-assessment 2011†University of Hertfordshire Business School research Seminar 2/11/2011
Marija Cubric, [email protected] Tosic, [email protected]
AbstractSemantic Web technologies have been increasingly used as a tool for generating, organizing and personalizing e-learning content. In this presentation we will discuss and demonstrate an innovative approach to automated generation of computer-assisted assessment (CAA) from Semantic Web–based domain ontologies. The primary application domain of this work is in the automated assessment, and in particular, the development of intelligent CAA systems and question banks, but the ideas can be further generalized in the context of ontology engineering and evaluation.
2
ContentTerminology ……….. Research Problem………... ……….. Related Work……….. Methodology ………..Our Contribution ……….. Implementation ………. Benefits and Limitations ……….. Future Work …..……….. Other research ……….. Q&A
3
E-learning…is the use of innovative technologies and learning models to transform the way individuals and organisations acquire new skills and access knowledge (Moeng, 2004)
4
.
… comprises all forms of electronically supported learning and teaching. The information and communication systems, whether networked learning or not, serve as specific media to implement the learning process (Tavangarian et al.,2004)
… a broad combination of processes, content, and infrastructure to use computers and networks to scale and/or improve one or more significant parts of a learning value chain, including management and delivery (Aldrich, 2004) Adrich, C. (2004) Simulations and the Future of Learning. San Francisco: Pfeiffer, p.240Moeng, B. (2004). IBM tackles learning in the workplace. IBM Management Development Solutions, Nov 8, 2004.Tavangarian D., Leypold M., Nölting K., Röser M.,(2004). Is e-learning the Solution for Individual Learning? Journal of e-learning
E-learning
5
.
Information Systems
Information Systems EducationEducation
This research
6
.
EducationEducationInformation
SystemsInformation
Systems
Information TechnologyInformation Technology
AssessmentAssessment
Objective tests and CAA
7
Objective tests require a user to choose or provide a response to a question whose correct answer is predetermined.
Such a question might require a student to :- select a solution from a set of choices (MCQ, true-false, matching)- identify an object or position (graphical hotspot) or- supply brief numeric or text responses (text input).
Because the correct answers to objective test questions are pre-determined, they are well suited to the many forms of CAA (Computer-assisted assessment) or e-assessment (CAA Centre Resources)
MotivationStudents’ needs
– Prompt and frequent feedback (NSS)– Constant simulation, fun, new stuff (Bean, 2010)
Universities’ agendas– Increase opportunities for flexible learning (“anytime anywhere”)– Cost reduction (“more for less”)
Good teaching practice (Chickering and Gamson, 1987) – Encourages active learning– Gives prompt feedback– Emphasises time on task– Respects diverse talents and ways of learning
Teachers’ needs– Questions are difficult and time-consuming to write– Professional writers plan one hour or more per item (Van Hoozer
quoted in Collins, 2006)
8
Bean M (2010) JISC 2010 Opening keynote: The learning journey: From informal to formalChickering A.W. and Gamson Z.F (1987) "Seven principles for good practice in undergraduate education. The American Association for Higher Education Bulletin, March 1987Van Hoozer H, et al. The teaching process: theory and practice in nursing. Norwalk, CT: Appleton-Century-Crofts, 1987:279-280
Research Problem
9
Develop algorithms and tools for automated generation of objective tests, leveraging on increasing body of domain ontologies (“knowledge conceptualisations”) and other advances in the area of semantic web.
.
Information Systems
Information Systems EducationEducation
Information TechnologyInformation Technology
AssessmentAssessment
Semantic Web Primer
Semantic Web - “The Web of data with meaning in the sense that a computer program can learn enough about what the data means to process it” ("Weaving the Web" by Tim Berners-Lee, 1999)
Ontology - A formal specification of a conceptualization of a knowledge domain in terms of classes, instances, relations, properties and annotations relevant to modeling of the domain (Gruber, 1993)
W3C standards define specific formalisms for encoding ontologies, such as RDF (Resource Description Framework) and OWL (Ontology Web Language)
10
http://www.w3.org/
Gruber T. R. (1993), A Translation Approach to Portable Ontology Specifications. Knowledge Acquisition, 5(2):199-220
11
An Example Ontology from Business Domain (e-commerce)
12
E-commerce ontology (class view)
13
classes classes
properties
properties
instances instances
annotations annotations
Related Work- Ontologies in LearningOntological formalization of learning content, instructional processes and learning designs for the purpose of automated discovery, composition and presentation of knowledge
– Holohan & Pahl , 2003; Sicilia & Barriocanal, 2005; Knight, Gaševic & Richards,2006
Related data mining techniques for the discovery of ontologies from arbitrary text corpora
– Montoyo, et al. 2005Ontologies for fostering collaborative learning (Allert, Markkanen, Richter 2006)
– Collaborative concept mapping– Using ontologies to organize or annotate shared artifacts– Using existing ontologies to carry out an activity– Collaborative ontology development as part of an overarching task
14
Chung, Niemi & Bewley (2003): assessment authoring support system for “aiding assessment authors to populate the assessment ontology with values specific to the users’ purposes”
Holohan, et al, (2005): semi-automatic generation of simple learning objects such as slide shows and objective tests, in the context of an adaptive learning environment
Holohan, et al, (2006) & Zualkernan et al, (2009) dynamic problem generation, using domain-specific algorithms: the examples considered was from the domain of relational databases and software engineering and the resulting problems were database queries and questions related to the UML models of software artifacts
Mitkov, R. et al, (2006): NLP techniques such as shallow parsing, term extraction, sentence transformation and computation of semantic distance, aided by WordNet
Related Work- Automated Generation of E-assessment
15
Papasalouros A., Kotis K., Kanaris K. (2008): defined 11 ontology-based strategies for automatic generation of ‘distractors’ for Multiple Choice Questions (MCQ), from arbitrary knowledge domains.
The generation is based on the basic meta-ontology relations between a ‘class’ and ‘individual’, as well as between two individuals (binary ‘role’)
Related Work- Automated Generation of E-assessment (cont)
16
A(a) and not A(b) R(a,b) and not R(c,b) A<B and not D<B
Our ContributionFocus on ontology-based automatic generation of assessment of an arbitrary knowledge domain
Developed Protégé-based plugin (Tosic&Cubric,2009)–Extended and optimized strategies introduced by Papasalouros et al (2008)
In Cubric & Tosic (2010) :Added annotations to the meta-ontology used for question
generation
Added question templates for generating question text
Added semantic interpretation to the question templates in terms of learning theory - different types of questions can be generated based on different semantic interpretation
Most recent: : Developed open source prototype for evaluation purposes (www.opensemcq.org ) 17
@code.google.com
18
opensemcq.org
Design as a MethodologyDesign can be thought of as a mapping from function space - a functional requirement constituting a point in this multidimensional space - to attribute space, where an artifact satisfying the mapping constitutes a point in that space (Takeda, et al, 1990). Design then, is knowledge in the form of techniques and methods for performing this mapping – the know-how for implementing an artifact that satisfies a set of functional requirements. (DESRIST.org)
Design Science Research – “learning through building - artifact construction” and “using design as a research method or technique” (DESRIST.org) Subset of Design Research (research into or about design) whereas DSR is research using design as a research method or technique
19Takeda, H., Veerkamp, P., Tomiyama, T., Yoshikawam, H. (1990). "Modeling Design Processes." AI MagazineWinter: 37-48.
Information SystemsInformation Systems
Information TechnologyInformation Technology
Design Science Research Method
Design science research is most effective when its practitioners shift between pragmaticand critical realist perspectives, guided by a pragmatic assessment of progress in the design cycle( Bunge,1984)
20
Bunge, M. (1984). Philosophical Inputs and Outputs of Technology. History and Philosophy of Technology. G. Bugliarello and D. Donner. Urbana, IL, University of Illinois Press: 263-281.
Knowledge gained from construction
Philosophical assumptions of Design Science Research
(source: DESRIST.org)21
Protégé Plugin(“suggestions”)
Features• Variety of questions generated
based on different pluggable ontologies (19 test ontologies)
• Optimized existing 11 (Papasalouros et al, 2008) and added 4 new strategies for generating distracters
• Adjustable number of questions and types of strategies used.
• Adjustable difficulty level of the generated question set.
Architecture
22
MCQ Ontology (IMS, 2002; CAA,2010)
23
n m*
*
CAA Centre Resources (2002) http://www.caacentre.ac.uk/resources/ (accessed 12/5/2010)IMS Global Learning Consortium, Inc. IMS Question and Test Interoperability Version 2.1 Public Draft
Generation ProcessInput : Any domain ontologyOutput: MCQ ontology populated with instances of MultipleChoiceQuestion class
24
(any) domain ontology
target (MCQ) ontology
question templatequestion template
Stems, question templates & semantic interpretation
25
Stems using ontology annotations
Stems generated from pre-defined parameterized question templates
Templates based on different levels of Bloom’s (1956) taxonomy of cognitive domains
Strategies for generating distracters
26
Based on ‘text similarity measures’ - the more similar distracters are, the more difficult question becomes (Mitkov et al, 2008)
Mitkov, R. et al, 2006. A computer-aided environment for generating multiple-choice test items. Natural Language Engineering, Vol. 12, No. 2, pp. 177-194.
Which of the following definitions describes the concept Quantitative value? a) This is a conceptual entity that holds together all aspects of the n-ary relation hasWarrantyPromise. A ________ is an entity representing the duration and scope of services that will be provided to a customer free of charge in case of a defect or malfunction of theProductOrService. b) An instance of this class is an actual _________ for a quantitative property of a product. This instance is usually characterized by a minimalvalue, a maximal value, and a unit of measurement. c) A _____ is a numerical interval that represents the range of a certain quantitative Product Or Service Property in terms of the lower and upper bounds for a particular Product Or Service . It isto be interpreted in combination with the respective unit of measurement . d) An instance of this class is an actual _______for a quantitative property of a product. This instance is usually characterized by a minimal value, a maximal value, and a unit of measurement.Examples: The intervals between 10.0 and 25.4 kilogramms" or "10.2 and 15.5 milimeters".e) A __________ is a predefined value for a product characteristic. Examples: the color "green" or the power cord plug type "US"; the garment sizes "S", "M", "L", and "XL”.
Which of the following definitions describe the concept <A> : (x,y,…)
27
Class A
x
Annotation
Class B
yHigh Similarity
Annotation Annotation
Annotation
✔ ✖
Strategies for distractors: Most similar annotation
Which one of the following examples demonstrates the concept <A>:(a,b,c,...)
28
aa
Class A
✔
instanceinstance
instanceinstance
bb
Class B
✖
instanceinstance
instanceinstance
cc
instanceinstance
instanceinstance
Class C
✖
Which one of the following examples demonstrates the concept Payment method credit card? a) Check in advance b) Direct debit c) Pay Swarm d) JCB e) Google Checkout
Strategies for distractors: Instance of a Parent Class, Instance of a Sibling Class …
Read the paragraph <x> and decide which one of the following concepts generalize the concept defined by <x>: (A,B,C,D,E,F,…)
29
Read the paragraph and decide which one of the following concepts generalize the concept defined by it: "A _____ is a conceptual entity that specifies the additional costs asked for settling the payment after accepting a given Offering using a particular Payment Method . A ____ is characterized by (1) a monetary amount per order specified as a literal value of type float in combination with a Currency , (2) the payment method , and (3) a whether this charge includes local sales taxes , namely VAT”
a) Product or serviceb) Actual product or service instance c) Business functiond) Price specificatione) Location of sales or service provisioning
Strategies for distractors: Sibling concept, Child Concept, Parent’s sibling, Grandparent…
Class B
Class A
x
✔
✖Class
E
Class F ✖
✖
Class C
Class D
✖
✖
Read the paragraph and decide which one of the following concepts it defines:"The ________ represents types of services that will be provided free of chargeby the vendor or manufacturer in the case of a defect (e .g . labor and parts ,just parts ), as part of the warranty included in an Offering . The actualservices may be provided by the Business Entity making the offering , bythe manufacturer of the product , or by a third party . Examples: Parts andLabor , Parts" <d>a) Quantitative valueb) Delivery methodc) Payment methodd) Warranty scopee) Warranty promise
Read the paragraph <x> and decide which one of the following concepts it defines: (A,B,…)
30
Class A
x
Annotation
Class B
Annotation High Similarity
Annotation Annotation
Annotation
✔ ✖
Strategies for distractors: Most similar concept
Which one of the following response pairs relates in the same way as <a> and <b> in the relation <R>? : (c,d),(c,b),(a,d),…
31
Which one of the following response pairs relates in the same way as Sunday and Saturday in the relation has previous?
a)Wednesday and Tuesday b) Friday and Saturdayc) Wednesday and Sundayd) Sunday and Mondaye) Monday and Wednesday
aa
bb
R
cc
dd
R ✔
✖✖
Strategies for distractors: Non-existent relation with the same domain, Non-existent relation with the same co-domain
Analyze the following text and decide which one of the following words is acorrect replacement for the blank space in the text: "This subproperty specifies that the upper and lower limit of the given ________ are identical and have the respective value . It is a shortcut for suchcases where a quantitative property is (at least practically ) a single pointvalue and not an interval ."a) Warranty promiseb) Quantitative value integerc) Quantitative valued) Quantitative value floate) Qualitative value
Analyze the text <x> and decide which one of the following words is a correct replacement for the blank space in <x>: (B,C,…)
32
Class B
annotation
annotation
annotation
Class CClass C
annotation
annotation
annotation
High
Similarity
Class A
annotation
x
annotation
Describes
✔✖
Strategies for distractors: Most Similar Annotation
Implementation: opensemcq: generate test
33
Implementation: opensemcq: view test
34
Implementation: opensemcq: evaluate test
35
BenefitsAdding annotations
– Four more question types and three more strategies Adding questions templates
– Avoiding the NLP complexity problemAdding semantic interpretation
– Questions ordered according to increased level of educational objectives (Bloom, 1956)
– Enables adaptive test creation (Lilley & Barker, 2002)Decoupling semantic interpretation from question generation
– Enables change of the underlying learning theory – Kolb’s experiential learning (Kolb, 1984) as a basis for defining
question templates (Barker, 2008) Useful ‘seed’ for further question enhancements
36
Bloom, B. S., Krathwohl, D. R. (1956). Taxonomy of educational objectives. Handbook 1. Cognitive domain. New York: Addison-Wesley.Lilley, M., & Barker, T. (2002). The development and evaluation of a computer-adaptive testing application for English language. 6th Computer assisted assessment conference, July 2002, Loughborough.Kolb, D. (1984) Experiential Learning. Englewood Cliffs. N.J.: Prentice Hall
Issues and LimitationsLack of annotated (test) ontologies
Lack of questions templates for assessing higher order skills (e.g. synthesis and evaluation)
Generation of distracters based on text similarity bear no necessary semantically structured relation to the subject matter domain
– Should be extended with meta-ontological relations, such as: subclass, superclass, ‘friend’ class, ‘related’ classes
Scope of assessment limited to testing the knowledge described in the ontology and the quality of the ontology.
37
Current WorkInvitation for evaluation to be sent to authors of ontologies used for testing
Further invitations to be sent to University of Hertfordshire academics
Improvements to be implemented based on the feedback from the evaluation.
Want to participate? – Go to www.opensemcq.org – Select a Test to evaluate– Click on Evaluate !
Want to develop a new ontology?– Talk to us!
38
Future WorkIntegrate text similarity with more advanced ontological approaches to generation of distracters
– “The distance between concepts in the hierarchy can be used to generate challenging answers covering closely related false answers (distracters)” (Pahl & Holohan, 2009)
Extend and enrich the template base
Add other ontology components, such as rules, axioms, restrictions, events etc. to the meta ontology used for question generations
– Explore Horridge’s et al. (2009) ideas on ontology justifications, for generating question feedback
Extend the empirical dataset to include some larger ‘real-life’ ontologies
Generalize the MCQ ontology to include other types of objective tests e.g. Multiple Response Questions
Further collaborative work planned with the Semantic Web groups from University of Southampton and Manchester; CAA group group University of Dublin , Agean and CMU, and Ontology engineering group from Stanford …
39