wp5: validation anne de roeck diane evans the open university, uk
TRANSCRIPT
![Page 1: WP5: Validation Anne De Roeck Diane Evans The Open University, UK](https://reader031.vdocuments.mx/reader031/viewer/2022032206/56649ed35503460f94be30e2/html5/thumbnails/1.jpg)
WP5: Validation
Anne De RoeckDiane Evans
The Open University, UK
![Page 2: WP5: Validation Anne De Roeck Diane Evans The Open University, UK](https://reader031.vdocuments.mx/reader031/viewer/2022032206/56649ed35503460f94be30e2/html5/thumbnails/2.jpg)
WP5
• Who?• Objectives• Challenges & Risks• Scenario Based Evaluation• Why now?• Progress• Plan
![Page 3: WP5: Validation Anne De Roeck Diane Evans The Open University, UK](https://reader031.vdocuments.mx/reader031/viewer/2022032206/56649ed35503460f94be30e2/html5/thumbnails/3.jpg)
Who?
• All Partners• Open University (WP5 Lead)
– Distance: 200,000 students – 90% on-line– Research and Teaching
• OU Centre for Research in Computing– CRC = Computing + KMi + IET UserLab– NLP: Anne De Roeck; Donia Scott, …– E-learning: Josie Taylor; Patrick McAndrew;
Diane Evans, …
![Page 4: WP5: Validation Anne De Roeck Diane Evans The Open University, UK](https://reader031.vdocuments.mx/reader031/viewer/2022032206/56649ed35503460f94be30e2/html5/thumbnails/4.jpg)
WP5 Objectives
• Aim:– to validate if eLearning systems (LMS) are
improved by adding new functionalities based on language technology tools and ontology-based (multilingual) retrieval.
• Objectives – to assess the extent to which the integration of
these new functionalities affects the effectiveness of the eLearning process.
![Page 5: WP5: Validation Anne De Roeck Diane Evans The Open University, UK](https://reader031.vdocuments.mx/reader031/viewer/2022032206/56649ed35503460f94be30e2/html5/thumbnails/5.jpg)
Challenges
• Focus:– Quantitative vs Qualitative Evaluation– Metrics vs User Experience– Functionalities vs Software Development
![Page 6: WP5: Validation Anne De Roeck Diane Evans The Open University, UK](https://reader031.vdocuments.mx/reader031/viewer/2022032206/56649ed35503460f94be30e2/html5/thumbnails/6.jpg)
Challenges
• Focus:– Quantitative vs Qualitative Evaluation– Metrics vs User Experience– Functionalities vs Software Development
![Page 7: WP5: Validation Anne De Roeck Diane Evans The Open University, UK](https://reader031.vdocuments.mx/reader031/viewer/2022032206/56649ed35503460f94be30e2/html5/thumbnails/7.jpg)
Challenges
• Focus:– Quantitative vs Qualitative Evaluation– Metrics vs User Experience– Functionalities vs Software Development
• Challenge 1: Methodology:– Scenario based validation– Functionality & user context centred– User questionnaires & interviews
![Page 8: WP5: Validation Anne De Roeck Diane Evans The Open University, UK](https://reader031.vdocuments.mx/reader031/viewer/2022032206/56649ed35503460f94be30e2/html5/thumbnails/8.jpg)
Challenges
• Focus:– Language Technology Tools– Ontology retrieval– Improve effectiveness!
• Challenge 2: Search & Retrieval Scenarios– Embed functionality (teachers & learners?)– Establish base-line– Refine scenarios
![Page 9: WP5: Validation Anne De Roeck Diane Evans The Open University, UK](https://reader031.vdocuments.mx/reader031/viewer/2022032206/56649ed35503460f94be30e2/html5/thumbnails/9.jpg)
Challenges
• Focus:– Multi-lingual experience
• Challenge 3: Multilingual Scenarios– Multiple communities– Multi-lingual scenario– Chart the territory!
![Page 10: WP5: Validation Anne De Roeck Diane Evans The Open University, UK](https://reader031.vdocuments.mx/reader031/viewer/2022032206/56649ed35503460f94be30e2/html5/thumbnails/10.jpg)
Risks
• Content!• Assessment??• Performance of Components?• Interface vs Functionality• Cross-cultural issues
![Page 11: WP5: Validation Anne De Roeck Diane Evans The Open University, UK](https://reader031.vdocuments.mx/reader031/viewer/2022032206/56649ed35503460f94be30e2/html5/thumbnails/11.jpg)
Risks
• Content!• Assessment?• Performance of Components?• Interface vs Functionality• Cross-cultural issues
• Technical deliverable vs user requirements
![Page 12: WP5: Validation Anne De Roeck Diane Evans The Open University, UK](https://reader031.vdocuments.mx/reader031/viewer/2022032206/56649ed35503460f94be30e2/html5/thumbnails/12.jpg)
Methodology
• User Scenarios– ‘a story focused on a user or group of users,
which provides information on the nature of the users, the goals they wish to achieve and the context in which the activities will take place’.
![Page 13: WP5: Validation Anne De Roeck Diane Evans The Open University, UK](https://reader031.vdocuments.mx/reader031/viewer/2022032206/56649ed35503460f94be30e2/html5/thumbnails/13.jpg)
Methodology
• User Scenarios– Step beyond Software Development Use Cases– Qualitative evaluation of functionalities …– …in context of user roles– Accessible to all stakeholders – Developed from high level descriptions into
scenario instantiations during implementation– Used in learning applications (eg mobile
learning) & Software Development
![Page 14: WP5: Validation Anne De Roeck Diane Evans The Open University, UK](https://reader031.vdocuments.mx/reader031/viewer/2022032206/56649ed35503460f94be30e2/html5/thumbnails/14.jpg)
Why now?
• Functionalities & System development• LT4eL = New technology• Co-evolve system & validation
– Process of scenario instantiation:– Explore/refine ideas on possible usage– Clarify user requirements– Establish common ground between pedagogists,
developers, NLPers, users– Concurrent with design & development
![Page 15: WP5: Validation Anne De Roeck Diane Evans The Open University, UK](https://reader031.vdocuments.mx/reader031/viewer/2022032206/56649ed35503460f94be30e2/html5/thumbnails/15.jpg)
Progress
• Aim:– Prepare & support user scenario validation– Clarify objectives– Mitigate risk
• Background literature search and survey: – Keyword extraction & validation methods– Ontologies and deployment in search– Scenario-based validation
![Page 16: WP5: Validation Anne De Roeck Diane Evans The Open University, UK](https://reader031.vdocuments.mx/reader031/viewer/2022032206/56649ed35503460f94be30e2/html5/thumbnails/16.jpg)
Progress
• Contribution WP1:– Identifying and supplying English Content within
the domain of e-learning in educational establishments.
• Contribution WP2/3:– running early version of keyword extractor with
English content and feeding back to WP2 leader. – liasing with Malta on English content– identifying qualitative validation tests for Keyword
Extractor application.
![Page 17: WP5: Validation Anne De Roeck Diane Evans The Open University, UK](https://reader031.vdocuments.mx/reader031/viewer/2022032206/56649ed35503460f94be30e2/html5/thumbnails/17.jpg)
Progress
• Start WP5:– Monitor content (ongoing: high risk!)– Draft validation plan for ILIAS + new
functionality– Delivered high level user scenario (starting
point of instantiation)– Preliminary Questionnaire (establish qualitative
base-line)
![Page 18: WP5: Validation Anne De Roeck Diane Evans The Open University, UK](https://reader031.vdocuments.mx/reader031/viewer/2022032206/56649ed35503460f94be30e2/html5/thumbnails/18.jpg)
Progress/Plan
• Questionnaire:– slide presentation supporting 'focus group' – questionnaire (in Word format) covering search
motivation and behaviours – 81 individual student responses from 6
partners
![Page 19: WP5: Validation Anne De Roeck Diane Evans The Open University, UK](https://reader031.vdocuments.mx/reader031/viewer/2022032206/56649ed35503460f94be30e2/html5/thumbnails/19.jpg)
Progress/Plan
• Questionnaire Findings on:– Reasons to search in non Native language– Difficulties encountered when searching– Selecting search criteria – Handling search results– Other issues
![Page 20: WP5: Validation Anne De Roeck Diane Evans The Open University, UK](https://reader031.vdocuments.mx/reader031/viewer/2022032206/56649ed35503460f94be30e2/html5/thumbnails/20.jpg)
Plan
• Process Questionnaire• Challenge2
– Refine search & retrieval scenarios (Workshop – March)
– Integration (WP4)
• Challenge 3– Multi-lingual dimension to functionalities– Uncharted territory !
![Page 21: WP5: Validation Anne De Roeck Diane Evans The Open University, UK](https://reader031.vdocuments.mx/reader031/viewer/2022032206/56649ed35503460f94be30e2/html5/thumbnails/21.jpg)
Plan
0. Preliminary Questionnaire (M12) - Partners
1. Pilot Validation (M16-18) – OU2. Main Validation (M19-23) – Partners (ILIAS Community part)3. Final Validation (M26-28)
• Reports M24, M30