aigle ovacs tao icl2012 v 1 0
DESCRIPTION
aigleTRANSCRIPT
From informal knowledge to High Quality e-Assessment process
The OAT Semantic Approach
15th International Conference on Interactive Collaborative Learning and 41st International Conference on Engineering Pedagogy Villach, Austria 26-28 September 2012
Younes Djaghloul, Muriel Foulonneau, Raynald Jadoul
The vision
11/26/12 OVACS-AIGLE-TAO 2
Origins of the experiment
Originally - The generation of assessment items from semantic models - The generation of ontologies from medical guidelines, courses,… Collaboration - TAO implementation in dentistry - Interest in presenting the research at a conference on medical IT
Opportunity = testing our tools and validating our approach in the educational field
- Interest in innovative approach to learning - Interest in creating variability in item banks - Interest in innovative approach to assessment (e.g. for PISA)
11/26/12 OVACS-AIGLE-TAO 3
Overview on approach
11/26/12 OVACS-AIGLE-TAO 4
Knowledge: Informal models
Knowledge: Formal models
1. Model building: data mining, human methodology
Formal but not validated
2. Model validation • Experts for model validation • OVACS: to assist experts and to hide the complexity of the formalism (OWL, description logic )
Final Ontology
Experts, repositories, social media
3. Question generation • AIGLE tool, Automatic QTI-based questions generation • Semantic similarity techniques
List of assessment questions
Validate questions Experts for question validation
4. Delivery strategy • TAO Delivery Module • TAO QTI viewer
The Final test
The process
11/26/12 OVACS-AIGLE-TAO 5
Capturing information from an expert (in our case, a teacher) à Creation of a domain ontology
Validating the ontology à Evaluation by the experts using OVACS
Generating assessment items à Evaluation of the assessment generation
approach (AIGLE) with a teacher
Delivering the assessment items à Delivery of the assessment test using
the TAO platform
The experiment
Stakes: - Validation of the tools that we will create or reuse
- Research questions
- What makes a domain model an educational domain model? - How to optimize the interaction between the model
and the user (teacher in this case) through - formulation and - number of questions?
11/26/12 OVACS-AIGLE-TAO 6
Why we believe there is a future
Real problems in testing and learning - Item variability and item banking - Learning adaptivity
Real problems in the process from modelling to feedback and data collection - We see, for instance, some partners create forms generated
from UML models but that process is not optimized - There is a potential to apply our approach of ontology
validation to data collection
11/26/12 OVACS-AIGLE-TAO 7
The vision on the development of semantic tools
11/26/12 OVACS-AIGLE-TAO 8
Market Artefact:
Science
Solution Artefact
Design Theory
Problem based
Technological Knowledge
Tools to interact with semantic models
Semantic modelling And user interactions
Adaptivity of services
Quality issue of automated processes
OVACS
Ontology VAlidation for Common uSers How to validate formal knowledge
model by questions
11/26/12 OVACS-AIGLE-TAO 9
OVACS architecture
11/26/12 OVACS-AIGLE-TAO 10
Source Ontology OWL
OVACS Engine (Semantic web technologies)
Generated Question (Web based)
Ontology management
feedback
Evaluated ontology
Validated ontology
Expert feedbacks
• Manage history • Get past questions
OVACS interface
11/26/12 OVACS-AIGLE-TAO 11
Defining questions in 3 sub-models
11/26/12 OVACS-AIGLE-TAO 12
Actions on the ontology
11/26/12 OVACS-AIGLE-TAO 13
Questions and ontologies instantiation recovery realization classification
RDFS Classes
rdfs:Resource Y Y N Y
rdfs:Class Y Y N Y
rdfs:Literal Y Y N Y
rdfs:Datatype Y Y N Y
rdf:XMLLiteral Y Y N Y
rdf:Property Y Y N Y
11/26/12 OVACS-AIGLE-TAO 14
Relating patterns to ���question strategies
Instantiation Recovering Realization Classification
Closed questions (i.e., with a textual
context) Y Y Y Y
Graphic questions N N N N Questions with a pre-determined
choice Y Y Y Y
Questions for which a level must be
indicated N N N N
Open questions Y Y Y Y Ordering N N Y N
Setting relations Y Y Y Y Creating (in this case, Modeling) Y Y Y Y
Forms of questions and ���questioning strategies ���for the x propertyOf y ���pattern (i.e., range and ���domain properties)
Multiple questions to verify inter-related sub-graphs Instantiation : is x an instance of y? Recovering : which are all the instances of y? Realization test: among all the objects that x is a type of, which one is the most specific? Satisfaction test: if we say that x is an instance of y, does it create any inconsistency in the
model? Classification test: which types of objects does x belong to?
Implementation of validation strategies
11/26/12 Cc-eSanté 15
-
E.g., based on ontology patterns
Optimization of the validation strategy
11/26/12 OVACS-AIGLE-TAO 16
Pattern P2: Class – Property – Instance
Pattern P1: Class – Subclass – Instance
Next steps
1. Improve the linguistic capabilities of the tool 2. Implement alternative question sequence optimization strategies 3. Implement a knowledge base on question forms 4. Implement a feedback mechanism 5. Test the caries application with Karolinska 6. Test the SANTEC application
11/26/12 Cc-eSanté 17
AIGLE Assessment Item Generator
in Learning Environment
11/26/12 OVACS-AIGLE-TAO 18
AIGLE – Assessment item generator
q Security issue Adding variability to an item no expected variation of the construct
q Model-based learning Generating items from knowledge represented as a model the construct is modified for each item
11/26/12 OVACS-AIGLE-TAO 19
Stem variables
options
key
Auxiliary information
IMS-QTI item generation process
Generating items from Web data sources
11/26/12 OVACS-AIGLE-TAO 20
Test with DBpedia data Q Question model Ontology What was tested Outcome
Q1
What is the capital of {Azerbaijan} Infobox
The lowest level of semantic quality with the InfoBox dataset
3/30 Q/CA pairs not generated for a country 1 Q/CA generated for a country that does not exist any longer 1 capital as URI rather than a literal 1 capital with a labelling issue
Q2
Which country is represented by this flag?
Infobox FOAF YAGO
Improved set of properties Retrieval of pictures from Wikimedia
6 out of 30 items did not include a flag picture because the flag URI had a 404 error
Q3
Who succeeded to { Charles VII the Victorious } as ruler of France ?
Infobox Dbpedia ontology YAGO
The use of the Dbpedia ontology the historical domain
Inconsistent labels duplicates due to multiple labels assigned to each king 1 incorrect statement
Q4
What is the capital of { Argentina }? With feedback
Infobox YAGO
Feedback on items Retrieval of resources through data linkage
Full Flickr dataset unavailable at the time of the query
Q5
Which category does { Asthma } belong to?
SKOS Dublin Core MESH Infobox
Use of SKOS terminology the medical domain
SKOS concept not related to a SKOS scheme Usability of the SKOS structure of limited use as a domain model for education
11/26/12 OVACS-AIGLE-TAO 21
From Q1, 53,33% were directly usable neither a defective prompt nor a defective correct answer nor a defective distractor .
Benchmark from unstructured content: between 3.5% and 21%. Issues
• Ontology issue • Labels • Inaccurate statements • Data linkage (resolvable URIs) • Missing inferences
Measuring data quality
11/26/12 OVACS-AIGLE-TAO 22
Chance that an item will have a defective distractor =
User testing with countries and their capital
11/26/12 OVACS-AIGLE-TAO 23
Calculating the semantic similarity between distractors and a correct answer
Ulan Bator Libreville Manila
11/26/12 OVACS-AIGLE-TAO 24
Gabon – Libreville
Maputo Port Louis Libreville
No SemSim With SemSim
Adapted 3 semantic similarity strategies ��� to large scale semantic graphs
Results of user test Clear decrease of performance in the population
when using SemSim
11/26/12 OVACS-AIGLE-TAO 25
Common vs. uncommon knowledge: annotating the semantic model for educational purpose
Item Construct Estimate T1 Estimate T2
IC1 0.072 -0.288
IC2 -0.298 0.337
IC3 1.961 0.560
IC4 1.755 0.186
IC5 0.084 0.305
IC6 -0.270 -0.349
IC7 -0.737 -1.076
IC8 -1.517 0.751
IC9 -0.284 0.077
IC10 -0.767 -0.502
11/26/12 OVACS-AIGLE-TAO 26
Item difficulty estimate according to IRT
But role of ���the item layer?
Predicting item difficulty? Representation of the semantic distance of options in the items
(item dispersity)
11/26/12 OVACS-AIGLE-TAO 27
Next steps: 1. Refining the semantic similarity metrics 2. Assessing different forms of questions 3. Building an item bank with generated questions 4. Test the value of the questions (with Karolinska)
11/26/12 OVACS-AIGLE-TAO 28
TAO Testing Assisté par Ordinateur
(Computer-Aided Testing)
11/26/12 OVACS-AIGLE-TAO 29
TAO – assessment and feedback loop TAO platform is based on semantic web paradigm, i.e. it manages
question items decorated with any needed ad-hoc properties
The TAO platform delivers questionnaires that can also be featured with any desiderated extra semantic properties
The TAO collects all answers and behaviors of the test-takers
à If extra properties like the “provenance” (i.e. the source model built with OVACS and used by AIGLE) are attached to the question items or to the questionnaire, these properties are stored in tests results
à Analysis of the tests results will be used in a feedback loop for a validation process impacting the AIGLE & OVACS phases.
11/26/12 OVACS-AIGLE-TAO 30
OVACS AIGLE TAO
Initial test with Karolinska
11/26/12 OVACS-AIGLE-TAO 31
Original hypothesis
The creation of the domain ontology can use semi-automatic strategies, or third party encoders, or a collaborative work: can we ask an expert to validate the assertions in the ontology?
- What is lost in the expert’s speech when creating the ontology? - Does the expert understand automatically generated questions? - Does the expert flag the errors?
11/26/12 OVACS-AIGLE-TAO 32
Creating the ontology
An ontology of the caries - A one hour interview where the teacher
explained - the caries, their description, their causes, - how to handle them, - how to prevent them, - how to set a diagnostic
- Definition of a list of concepts / keywords - Creation of classes, instances, and properties - Creation of the OWL ontology
11/26/12 OVACS-AIGLE-TAO 33
Test set up Labels on stand alone Selected a subset of the ontology to keep the test short:
à instanceOf (13 items) and à subClassOf (11 items)
Only Boolean questions (yes/no) + “I do not know” option
24 questions 2 intentional mistakes:
on the content (causes of caries) and spelling (emanel instead of enamel)
Objectives: - check if the teacher would find the validation mechanism usable - verify whether errors would be detected and corrected
Video recording of the teacher
11/26/12 OVACS-AIGLE-TAO 34
Test conclusions
Confusion between the role of domain expert validating knowledge
and the role of teacher who prepares questions for students
According the comments of our expert: “Difficulty level of the generated questions is generally low” “But with very different variations in the difficulty level”
The OVACS validation questionnaire led to: 6 removals (2 subClassOf, 4 instanceOf) 16 acceptances (9 for subclassOf, 7 for instanceOf) 2 answers “I do not know” for subclassOf à meant not relevant
11/26/12 OVACS-AIGLE-TAO 35
OVACS-AIGLE-TAO