ec-tel 2016: which algorithms suit which learning environments?
TRANSCRIPT
W I S S E N T E C H N I K L E I D E N S C H A F T
http://kti.tugraz.at
Which Algorithms Suit Which LearningEnvironments? A Comparative Study ofRecommender Systems in TELS. Kopeinik, D. Kowald, E. Lex,Graz University of Technology, AustriaKnowledge Technologies Institute, Cognitive Science SectionOctober 24, 2016
2 Outline
A study comparing a variety of recommendation strategieson 6 empirical TEL datasetsConsidering 2 application casesFindings:
The performance of algorithms strongly depends on thecharacteristics of the datasetsThe number of users per resource is a crucial factorA hybrid combination of a cognitive-inspired and apopularity based approach works best for tagrecommendations
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
3
Introduction
Recommender Systems (RS)
... are software components that suggest items of interest or ofrelevance to a user’s needs [Kon, 2004, Ricci et al., 2011].
Recommendations are related to decision making processes:
Ease information overloadSales assistance
Popular examples: Amazon.com, YouTube, Netflix, Tripadvisor,Last.fm, ...
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
4
Introduction
RS in Technology Enhanced Learning
. . . are adaptational tasks to fit the learner’sneeds [Hamalainen and Vinni, 2010].
Typical recommendation services include:
Peer recommendationsActivity recommendationsLearning resource recommendationsTag recommendations
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
5
Introduction
Motivation
So far there are no generally suggested or commonlyapplied recommender system in TEL [Khribi et al., 2015]Learning data is sparse, especially in informal learningenvironments [Manouselis et al., 2011]Available data varies greatly, but available implicit usagedata typically includes learner ids, information aboutlearning resources, timestamps [Verbert et al., 2012]
Research Question 1
How accurate do state-of-the-art resource recommendation algorithms,using only implicit usage data, perform on different TEL datasets?
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
6
Introduction
Motivation
Lack of learning object meta-dataShifting the task to the crowd [Bateman et al., 2007]
Tagging is a mechanism to collectively annotatelearning objects [Xu et al., 2006]fosters reflection and deep learning[Kuhn et al., 2012]needs to be done regularly and thoroughly
Research Question 2
Which computationally inexpensive state-of-the-art tag recommendationalgorithm performs best on TEL datasets?
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
7
Evaluation
Study Setup
Evaluation of two recommender application cases
a) Recommendation of learning resourcesb) Recommendation of tags
For each dataset
1. Sort user activities in chronological order(timestamp)
2. Split data into training and test set
Application Training Set Test SetResources 80% 20%Tags n-1 1
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
8
Evaluation
Algorithms
Well-established, computationally inexpensive tag andresource recommendation strategies
Most Popular (MP) [Jaschke et al., 2007]. . . counts frequency of occurrence
Collaborative Filtering (CF) [Schafer et al., 2007]. . . calculates neighbourhood of users or items
Content-based Filtering (CB) [Basilico and Hofmann, 2004]. . . calculates similarity of user profiles and itemcontent
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
9
Evaluation
Algorithms
Approaches that have been suggested in the context of TEL
Usage Context-based Similarity (UCbSim)[Niemann and Wolpers, 2013]. . . calculates item similarities based onco-occurrences in user sessions
Base Level Learning Equation (BLLAC) [Kowald et al., 2015]. . . mimics human semantic memory retrieval as afunction of recency and frequency of tag use
Sustain [Seitlinger et al., 2015]. . . simulates category learning as a dynamicclustering approach
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
10
Evaluation
Metrics [Marinho et al., 2012, Sakai, 2007]
Recall (R)The proportion of correctly recommended items toall items relevant to the user.
Precision (P)The proportion of correctly recommended items toall recommended items.
F-measure (F)The harmonic mean of R and P.
Discounted Cumulative Gain (nDCG)A ranking quality metric that calculates usefulnessscores of items based on relevance and position.
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
11
Evaluation
Datasets
Six datasets from different application domains:
BibSonomy, CiteULike (Social Bookmarking)KDD15 (MOOCs)Mace, TravelWell (Open Social Learning)Aposdle (Workplace Learning)
|P| |U| |R| |T | |Tp| |ATr | |ATpr | |ARu| |AUr | SPt SPtp
BibSonomy 82539 2437 28000 30889 0 4.1 0 33.8 3 0 100CiteULike 105333 7182 42320 46060 0 3.5 0 14.7 2.5 0 100KDD15 262330 15236 5315 0 3160 0 1.8 17.2 49.4 100 1.1TravelWell 2572 97 1890 4156 153 3.5 1.7 26.5 1.4 3.2 28.7MACE 23017 627 12360 15249 0 2.4 0 36.7 1.9 31.2 100Aposdle 449 6 430 0 98 0 1.1 74.8 1 100 0
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
11
Evaluation
Datasets
Six datasets from different application domains:
BibSonomy, CiteULike (Social Bookmarking)KDD15 (MOOCs)Mace, TravelWell (Open Social Learning)Aposdle (Workplace Learning)
|P| |U| |R| |T | |Tp| |ATr | |ATpr | |ARu| |AUr | SPt SPtp
BibSonomy 82539 2437 28000 30889 0 4.1 0 33.8 3 0 100CiteULike 105333 7182 42320 46060 0 3.5 0 14.7 2.5 0 100KDD15 262330 15236 5315 0 3160 0 1.8 17.2 49.4 100 1.1TravelWell 2572 97 1890 4156 153 3.5 1.7 26.5 1.4 3.2 28.7MACE 23017 627 12360 15249 0 2.4 0 36.7 1.9 31.2 100Aposdle 449 6 430 0 98 0 1.1 74.8 1 100 0
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
12
Discussion
Results: Resource Recommender
Dataset Metric MP CFR CBT CFU UCbSim SUSTAIN SUSTAIN+CFU
BibSonomyP@5 .0154 .0336 .0197 .0410 .0336 .0336 .0467F@5 .0099 .0383 .0238 .0426 .0367 .0363 .0496nDCG@5 .0088 .0416 .0270 .0440 .0371 .0392 .0541
CiteULikeP@5 .0048 .0592 .0353 .0412 .0558 .0503 .0553F@5 .0050 .0694 .0404 .0477 .0627 .0597 .0650nDCG@5 .0048 .0792 .0427 .0511 .0686 .0704 .0717
KDD15P@5 .0018 .2488 .1409 .2355 .2570 .2436 .2377F@5 .0029 .3074 .1612 .3050 .3314 .3025 .3059nDCG@5 .0053 .3897 .1927 .3618 .3529 .3227 .3608
TravelWellP@5 .0127 .0212 .0382 .0425 .0297 .0382 .0382F@5 .0056 .0232 .0240 .0414 .0365 .0427 .0204nDCG@5 .0072 .0220 .0275 .0305 .0491 .0446 .0220
MACEP@5 .0167 .0079 .0023 .0251 .0213 .0065 .0190F@5 .0201 .0079 .0019 .0266 .0177 .0076 .0205nDCG@5 .0248 .0082 .0014 .0264 .0165 .0079 .0215
AposdleP@5 .0 .0 .0 .0333 .0 .0 .0F@5 .0 .0 .0 .0049 .0 .0 .0nDCG@5 .0 .0 .0 .0042 .0 .0 .0
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
12
Discussion
Results: Resource Recommender
Dataset Metric MP CFR CBT CFU UCbSim SUSTAIN SUSTAIN+CFU
BibSonomyP@5 .0154 .0336 .0197 .0410 .0336 .0336 .0467F@5 .0099 .0383 .0238 .0426 .0367 .0363 .0496nDCG@5 .0088 .0416 .0270 .0440 .0371 .0392 .0541
CiteULikeP@5 .0048 .0592 .0353 .0412 .0558 .0503 .0553F@5 .0050 .0694 .0404 .0477 .0627 .0597 .0650nDCG@5 .0048 .0792 .0427 .0511 .0686 .0704 .0717
KDD15P@5 .0018 .2488 .1409 .2355 .2570 .2436 .2377F@5 .0029 .3074 .1612 .3050 .3314 .3025 .3059nDCG@5 .0053 .3897 .1927 .3618 .3529 .3227 .3608
TravelWellP@5 .0127 .0212 .0382 .0425 .0297 .0382 .0382F@5 .0056 .0232 .0240 .0414 .0365 .0427 .0204nDCG@5 .0072 .0220 .0275 .0305 .0491 .0446 .0220
MACEP@5 .0167 .0079 .0023 .0251 .0213 .0065 .0190F@5 .0201 .0079 .0019 .0266 .0177 .0076 .0205nDCG@5 .0248 .0082 .0014 .0264 .0165 .0079 .0215
AposdleP@5 .0 .0 .0 .0333 .0 .0 .0F@5 .0 .0 .0 .0049 .0 .0 .0nDCG@5 .0 .0 .0 .0042 .0 .0 .0
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
12
Discussion
Results: Resource Recommender
Dataset Metric MP CFR CBT CFU UCbSim SUSTAIN SUSTAIN+CFU
BibSonomyP@5 .0154 .0336 .0197 .0410 .0336 .0336 .0467F@5 .0099 .0383 .0238 .0426 .0367 .0363 .0496nDCG@5 .0088 .0416 .0270 .0440 .0371 .0392 .0541
CiteULikeP@5 .0048 .0592 .0353 .0412 .0558 .0503 .0553F@5 .0050 .0694 .0404 .0477 .0627 .0597 .0650nDCG@5 .0048 .0792 .0427 .0511 .0686 .0704 .0717
KDD15P@5 .0018 .2488 .1409 .2355 .2570 .2436 .2377F@5 .0029 .3074 .1612 .3050 .3314 .3025 .3059nDCG@5 .0053 .3897 .1927 .3618 .3529 .3227 .3608
TravelWellP@5 .0127 .0212 .0382 .0425 .0297 .0382 .0382F@5 .0056 .0232 .0240 .0414 .0365 .0427 .0204nDCG@5 .0072 .0220 .0275 .0305 .0491 .0446 .0220
MACEP@5 .0167 .0079 .0023 .0251 .0213 .0065 .0190F@5 .0201 .0079 .0019 .0266 .0177 .0076 .0205nDCG@5 .0248 .0082 .0014 .0264 .0165 .0079 .0215
AposdleP@5 .0 .0 .0 .0333 .0 .0 .0F@5 .0 .0 .0 .0049 .0 .0 .0nDCG@5 .0 .0 .0 .0042 .0 .0 .0
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
12
Discussion
Results: Resource Recommender
Dataset Metric MP CFR CBT CFU UCbSim SUSTAIN SUSTAIN+CFU
BibSonomyP@5 .0154 .0336 .0197 .0410 .0336 .0336 .0467F@5 .0099 .0383 .0238 .0426 .0367 .0363 .0496nDCG@5 .0088 .0416 .0270 .0440 .0371 .0392 .0541
CiteULikeP@5 .0048 .0592 .0353 .0412 .0558 .0503 .0553F@5 .0050 .0694 .0404 .0477 .0627 .0597 .0650nDCG@5 .0048 .0792 .0427 .0511 .0686 .0704 .0717
KDD15P@5 .0018 .2488 .1409 .2355 .2570 .2436 .2377F@5 .0029 .3074 .1612 .3050 .3314 .3025 .3059nDCG@5 .0053 .3897 .1927 .3618 .3529 .3227 .3608
TravelWellP@5 .0127 .0212 .0382 .0425 .0297 .0382 .0382F@5 .0056 .0232 .0240 .0414 .0365 .0427 .0204nDCG@5 .0072 .0220 .0275 .0305 .0491 .0446 .0220
MACEP@5 .0167 .0079 .0023 .0251 .0213 .0065 .0190F@5 .0201 .0079 .0019 .0266 .0177 .0076 .0205nDCG@5 .0248 .0082 .0014 .0264 .0165 .0079 .0215
AposdleP@5 .0 .0 .0 .0333 .0 .0 .0F@5 .0 .0 .0 .0049 .0 .0 .0nDCG@5 .0 .0 .0 .0042 .0 .0 .0
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
13
Discussion
Datasets
|ATr | |ATpr | |ARu| |AUr |BibSonomy 4.1 0 33.8 3CiteULike 3.5 0 14.7 2.5KDD15 0 1.8 17.2 49.4TravelWell 3.5 1.7 26.5 1.4MACE 2.4 0 36.7 1.9Aposdle 0 1.1 74.8 1
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
14
Discussion
Results: Resource Recommender
Dataset Metric MP CFR CBT CFU UCbSim SUSTAIN SUSTAIN+CFU
BibSonomyP@5 .0154 .0336 .0197 .0410 .0336 .0336 .0467F@5 .0099 .0383 .0238 .0426 .0367 .0363 .0496nDCG@5 .0088 .0416 .0270 .0440 .0371 .0392 .0541
CiteULikeP@5 .0048 .0592 .0353 .0412 .0558 .0503 .0553F@5 .0050 .0694 .0404 .0477 .0627 .0597 .0650nDCG@5 .0048 .0792 .0427 .0511 .0686 .0704 .0717
KDD15P@5 .0018 .2488 .1409 .2355 .2570 .2436 .2377F@5 .0029 .3074 .1612 .3050 .3314 .3025 .3059nDCG@5 .0053 .3897 .1927 .3618 .3529 .3227 .3608
TravelWellP@5 .0127 .0212 .0382 .0425 .0297 .0382 .0382F@5 .0056 .0232 .0240 .0414 .0365 .0427 .0204nDCG@5 .0072 .0220 .0275 .0305 .0491 .0446 .0220
MACEP@5 .0167 .0079 .0023 .0251 .0213 .0065 .0190F@5 .0201 .0079 .0019 .0266 .0177 .0076 .0205nDCG@5 .0248 .0082 .0014 .0264 .0165 .0079 .0215
AposdleP@5 .0 .0 .0 .0333 .0 .0 .0F@5 .0 .0 .0 .0049 .0 .0 .0nDCG@5 .0 .0 .0 .0042 .0 .0 .0
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
15
Discussion
Findings
1. The performance of most algorithms (F@5) stronglycorrelates (.958) with the average number of users perresource
2. Good performance values can only be reached for theMOOCs dataset
3. Algorithms based on implicit usage data don’t satisfy therequirements of small-scale environments like Aposdle
4. The performance of algorithms strongly depends on thecharacteristics of the datasets
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
16
Discussion
Results: Tag Recommender
Dataset Metric MPU MPR MPU,R CFU BLLAC BLLAC+MPR
BibSonomyP@5 .1991 .0572 .2221 .2066 .2207 .2359F@5 .2535 .0688 .2814 .2606 .2795 .2987nDCG@5 .3449 .0841 .3741 .3492 .3851 .4022
CiteULikeP@5 .1687 .0323 .1829 .1698 .1897 .2003F@5 .2310 .0427 .2497 .2315 .2597 .2738nDCG@5 .3414 .0600 .3632 .3457 .4016 .4140
TravelWellP@5 .1000 .0366 .1333 .0800 .1300 .1400F@5 .1376 .0484 .1724 .1096 .1708 .1872nDCG@5 .2110 .0717 .2253 .1622 .2525 .2615
MACEP@5 .0576 .0173 .0618 .0631 .0812 .0812F@5 .0799 .0259 .0869 .0893 .1114 .1138nDCG@5 .1146 .0463 .1296 .1502 .1670 .1734
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
16
Discussion
Results: Tag Recommender
Dataset Metric MPU MPR MPU,R CFU BLLAC BLLAC+MPR
BibSonomyP@5 .1991 .0572 .2221 .2066 .2207 .2359F@5 .2535 .0688 .2814 .2606 .2795 .2987nDCG@5 .3449 .0841 .3741 .3492 .3851 .4022
CiteULikeP@5 .1687 .0323 .1829 .1698 .1897 .2003F@5 .2310 .0427 .2497 .2315 .2597 .2738nDCG@5 .3414 .0600 .3632 .3457 .4016 .4140
TravelWellP@5 .1000 .0366 .1333 .0800 .1300 .1400F@5 .1376 .0484 .1724 .1096 .1708 .1872nDCG@5 .2110 .0717 .2253 .1622 .2525 .2615
MACEP@5 .0576 .0173 .0618 .0631 .0812 .0812F@5 .0799 .0259 .0869 .0893 .1114 .1138nDCG@5 .1146 .0463 .1296 .1502 .1670 .1734
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
17
Discussion
Conclusion
Learning Resource Recommendation:
A dense user resource matrix is crucialThe performance of most algorithms (F@5) strongly correlates (.958) with the averagenumber of users per resource
For small-scale learning environments, a thoroughdescription of user and learning resources is necessaryAlgorithms based on implicit usage data don’t satisfy the requirements of small-scaleenvironments like Aposdle
MOOCs are not representative for other, typically sparseTEL environmentsGood performance values can only be reached for the MOOCs dataset
Tag Recommendation:BLLAC+ MPR clearly outperforms the remaining algorithmsMPU,R, an alternative for runtime-sensitive environments
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
18
Discussion
Acknowledgement
Special thanks are dedicated to Katja Niemann who providedus with the datasets MACE and TravelWell. For the KDD15data, we would like to gratefully acknowledge the organizers ofKDD Cup 2015 as well as XuetangX for making the datasetsavailable. This work is funded by the Know-Center and theEU-IP Learning Layers (Grant Agreement: 318209). TheKnow-Center is funded within the Austrian COMET Programunder the auspices of the Austrian Ministry of Transport,Innovation and Technology, the Austrian Ministry of Economicsand Labor and by the State of Styria.
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
19
Discussion
References I
[Kon, 2004] (2004).
Introduction to recommender systems: Algorithms and evaluation.
ACM Trans. Inf. Syst., 22(1):1–4.
[Basilico and Hofmann, 2004] Basilico, J. and Hofmann, T. (2004).
Unifying collaborative and content-based filtering.
In Proc. of ICML’04, page 9. ACM.
[Bateman et al., 2007] Bateman, S., Brooks, C., Mccalla, G., and Brusilovsky, P. (2007).
Applying collaborative tagging to e-learning.
In Proceedings of the 16th international world wide web conference (WWW2007).
[Hamalainen and Vinni, 2010] Hamalainen, W. and Vinni, M. (2010).
Classifiers for educational data mining.
Handbook of Educational Data Mining, Chapman & Hall/CRC Data Mining and Knowledge Discovery Series, pages 57–71.
[Jaschke et al., 2007] Jaschke, R., Marinho, L., Hotho, A., Schmidt-Thieme, L., and Stumme, G. (2007).
Tag recommendations in folksonomies.
In Proc. of PKDD’07, pages 506–514. Springer.
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
20
Discussion
References II
[Khribi et al., 2015] Khribi, M. K., Jemni, M., and Nasraoui, O. (2015).
Recommendation systems for personalized technology-enhanced learning.
In Ubiquitous Learning Environments and Technologies, pages 159–180. Springer.
[Kowald et al., 2015] Kowald, D., Kopeinik, S., Seitlinger, P., Ley, T., Albert, D., and Trattner, C. (2015).
Refining frequency-based tag reuse predictions by means of time and semantic context.
In Mining, Modeling, and Recommending’Things’ in Social Media, pages 55–74. Springer.
[Kuhn et al., 2012] Kuhn, A., McNally, B., Schmoll, S., Cahill, C., Lo, W.-T., Quintana, C., and Delen, I. (2012).
How students find, evaluate and utilize peer-collected annotated multimedia data in science inquiry with zydeco.
In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 3061–3070. ACM.
[Manouselis et al., 2011] Manouselis, N., Drachsler, H., Vuorikari, R., Hummel, H., and Koper, R. (2011).
Recommender systems in technology enhanced learning.
In Recommender systems handbook, pages 387–415. Springer.
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
21
Discussion
References III
[Marinho et al., 2012] Marinho, L. B., Hotho, A., Jaschke, R., Nanopoulos, A., Rendle, S., Schmidt-Thieme, L., Stumme, G.,and Symeonidis, P. (2012).
Recommender systems for social tagging systems.
Springer Science & Business Media.
[Niemann and Wolpers, 2013] Niemann, K. and Wolpers, M. (2013).
Usage context-boosted filtering for recommender systems in tel.
In Scaling up Learning for Sustained Impact, pages 246–259. Springer.
[Ricci et al., 2011] Ricci, F., Rokach, L., and Shapira, B. (2011).
Introduction to recommender systems handbook.
Springer.
[Sakai, 2007] Sakai, T. (2007).
On the reliability of information retrieval metrics based on graded relevance.
Information processing & management, 43(2):531–548.
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016
22
Discussion
References IV
[Schafer et al., 2007] Schafer, J. B., Frankowski, D., Herlocker, J., and Sen, S. (2007).
Collaborative filtering recommender systems.
In The adaptive web, pages 291–324. Springer.
[Seitlinger et al., 2015] Seitlinger, P., Kowald, D., Kopeinik, S., Hasani-Mavriqi, I., Ley, T., and Lex, E. (2015).
Attention please! a hybrid resource recommender mimicking attention-interpretation dynamics.
arXiv preprint arXiv:1501.07716.
[Verbert et al., 2012] Verbert, K., Manouselis, N., Drachsler, H., and Duval, E. (2012).
Dataset-driven research to support learning and knowledge analytics.
Educational Technology & Society, 15(3):133–148.
[Xu et al., 2006] Xu, Z., Fu, Y., Mao, J., and Su, D. (2006).
Towards the semantic web: Collaborative tag suggestions.
In Collaborative web tagging workshop at WWW2006, Edinburgh, Scotland.
S. Kopeinik, D. Kowald, E. Lex, KTI-CSSOctober 24, 2016