relevance ranking of learning objects
DESCRIPTION
Presentation at EC-TEL 2007 (Crete). The need for relevance rank is presented and basic metrics are presented and evaluated.TRANSCRIPT
Relevance Ranking Metrics for Learning Objects
Xavier Ochoa, ESPOL, Ecuador
Erik Duval, KULeuven, Belgium
Agenda
• Why Relevance Ranking?
• What is Relevance?
• Relevance Ranking Metrics
• Do they work?
• What is next?
Economy of Abundance
Economy of Abundance
Put your LMS here!
Why Relevance Ranking?
Abundance
=
Difficult to find most relevant
Why Relevance Ranking?
Abundance
=
Difficult to find most relevant
Solution:Do it the Google way
Why Relevance Ranking?
Abundance
=
Difficult to find most relevant
Relevance RankingMEANINGFUL & SCALABLE
How we do it now?
• Manual rating:–Meaningful but not Scalable
• Text based Ranking Algorithms:–Scalable but not Meaningful
What is Relevance?
Relevance Ranking Metrics
Relevance Ranking
Relevance Ranking
MetricB
MetricA
MetricC
Context
Relevance Ranking Metrics
Relevance Ranking
Relevance Ranking
MetricB
MetricA
MetricC
Context
Meaningful
Relevance Ranking Metrics
Relevance Ranking
Relevance Ranking
MetricB
MetricA
MetricC
Context
Scalable
Topi
cal R
elev
ance
Personal Relevance
Situational Relevance
Do they work?
• Exploratory Study– 10 users (8 Prof. and 2 R.A.)– MIT OCW learning objects (34,640 LO)– Search and Select Objects to create 10
lessons (10 different topics on CS)– Web application was used
Ranking the Rankings
• Baseline Rank: TF-IDF similarity algorithm
• Re-rank according to Basic Metrics
• All rankings compared with manual relevance ranking
• Kendall Tau was used to measure similaritiy between lists
Results
Basic Topical Relevance MetricTrees
Basic Personal Relevance Metric
Net.
Basic Situational Relevance Metric
XML
Linear Combination
Linear Combination
Results
• Even basic metrics improve the ranking
• Linear Combination should be learnt
• Limited (and Synthetic) Study!
What is next?
• Implementation in ARIADNE NEXT
• Capture user behavior and learn from it for 3 months
• Natural (Real) evaluation of the metrics
• Develop BETTER metrics
Conclusions
• Meaningful and Scalable metrics are possible
• Cheap to implement • Provide significant improvement• While not optimal, they could be
use as a base-line for further research
Thank You! Dank U! Gracias!• Questions, Comments, Critics…
are all welcome!!
Xavier [email protected]
http://www.cti.espol.edu.ec/xavier
Erik [email protected] http://www.cs.kuleuven.ac.be/~erikd