recsys 07 doctoral consortium presentation
DESCRIPTION
My PhD research topics and experimental set-up.TRANSCRIPT
![Page 1: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/1.jpg)
Can Social Information Retrieval Enhance the Discovery and Reuse
of Learning Resources?
Riina VuorikariKatholieke Universiteit Leuven, Department of Computer Science
European Schoolnet, Belgium
Doctoral Consortium, RecSys 2007
![Page 2: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/2.jpg)
Outline of the presentation
● Context of the dissertation work
● Main research questions
● Experimental design
● First evaluations so far:– Multi-lingual use of tags– Levels of user engagement
![Page 3: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/3.jpg)
Context of the dissertation work
![Page 4: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/4.jpg)
Context
● European education, especially that of K-12 education, is inherently multilingual and multicultural.
● European teachers have access to multiple repositories of digital learning resources by – Educational Authorities, – publishers, – other teachers,..
![Page 5: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/5.jpg)
EUN partners..image
![Page 6: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/6.jpg)
Context
● Resources – In many different languages– For different national and regional curriculum– Contain metadata (e.g.title, keywords, language)– Of varying quality
● Repositories have formed federations to make resources available– Federated search based on metadata– Harvesting of metada
![Page 7: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/7.jpg)
Challenge for users
● End-users (e.g. teachers) have difficulties to discover and find resources from educational repositories– Metadata does not always match search terms
● Locating content across linguistic and national borders within Europe has proven hard – Despite the use of a multilingual Thesaurus and
controlled vocabularies
![Page 8: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/8.jpg)
Challenges for repositories
● Users become more demanding and expect services that are seen elsewhere (own collections, pedagogical hints, ..)
● European Schoolnet leading projects that build services on top of federation of European repositories– Social bookmarking tool– Tags– My networks
![Page 9: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/9.jpg)
My Main Question
Can Social Information Retrieval Enhance
the Discovery and Reuse
of Learning Resources?
![Page 10: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/10.jpg)
Social Information Retrieval (SIR)
● Refers to a family of techniques that assist users in obtaining information to meet their information needs by harnessing the knowledge or experience of other users.
● Examples of SIR techniques include: – sharing of queries, – collaborative filtering, – social network analysis, – social bookmarking, – subjective relevance judgements such as
tags, annotations, ratings and evaluations, etc.
![Page 11: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/11.jpg)
What is SIR for education?
● Is education as a field of implementation that different from other fields (e.g. music, movies)?
● What are the domain specific requirements, where does the data come from and what are its semantics?
● What are objects of recommendation?
● SIR TEL http://ariadne.cs.kuleuven.be/sirtel/
● My audience are teachers. Metaphor: it's like recommending for DJs?
![Page 12: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/12.jpg)
Context of this dissertation
To empower the social and contextual aspects of teachers' work
Digitalcontent
Education
Digitallibraries
Social Information Retrieval (SIR) methods
Information seeking theories
![Page 13: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/13.jpg)
Main research questions
![Page 14: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/14.jpg)
Main research questions 1
Teachers, tagging, languages:
● How do teachers tag and use social bookmarking in a multi-lingual environment?
● Are those bookmarks and tags useful for discovery of resources?
● How about tags in multiple languages?
![Page 15: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/15.jpg)
Main research questions 2
SIR aspect:
● Can bookmarks and tags be used to connect like-minded teachers cross country and linguistic borders?
● ...and thus used for social information retrieval?
● What are the levels of user engagement with the system?
![Page 16: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/16.jpg)
Main research questions 3
Information Seeking aspect:
● What are the main information seeking tasks that teachers have?
● What are the main SIR retrieval methods that they use for them?
● Can we match a task to a SIR method?
![Page 17: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/17.jpg)
Experimental design
![Page 18: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/18.jpg)
Data source 1
● Calibrate project (http://calibrate.eun.org), now to end of 2007
● K-12 digital learning resources
● Personal collections and tags (not shared)
● 78 pilot schools in Hungary, Austria, Estonia, Czech Republic, Lithuania and Poland
![Page 19: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/19.jpg)
![Page 20: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/20.jpg)
Implementation area and data source 2
● MELT project (http://info.melt-project.eu), from now to March 2009
● K-12 digital learning resources from a federation of about 10 repositories
● Implementation of a social bookmarking tool, annotations and my networks
● About 70 teachers from Austria, Belgium, Finland and Hungary
![Page 21: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/21.jpg)
![Page 22: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/22.jpg)
Data gathering
● Diverse data collection methods to allow triangulation of collected data.
– log files from the portals to see the grand lines, patterns, etc
– complimented by some questionnaires to understand groups or communities
– possible interviews, thinking alouds, observation, etc. on some few users to understand individual behaviour.
![Page 23: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/23.jpg)
Experimental Design
IndependentCondition
Social Condition
● Salganik, M., Dodds, P., & Watts, D. Experimental Study of Inequality and Unpredictability in an Artificial Cultural Market. Science, 311(5762), (2006), 854-856.
![Page 24: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/24.jpg)
Experimental Design
SocialInformationRetrieval
Ranking ofresources
Social navigation based onbookmarks, tags, annotationsand my networks
IndependentCondition
Social Condition
Tag input No tags shown when tagging
Tags shownwithin usersspoken language
Tags shownin alllanguages
![Page 25: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/25.jpg)
Some early analysis
![Page 26: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/26.jpg)
![Page 27: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/27.jpg)
![Page 28: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/28.jpg)
Analysis of User Behavior on Multi-lingual Tagging of Learning Objects
● January 24 to April 21 2007
● 77 teachers /173 total participating
● 459 bookmarks
● 417 multilingual tags
● 320 different learning resources
![Page 29: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/29.jpg)
Cross-border and language use
Tag 1 fi
Tag 2en
Tag 4 fr
LO 1in Fi
Tag 1 fi
Tag 3de
Tag 5 fr
LO 2in Fr
frde
fi fi
fi
![Page 30: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/30.jpg)
Tag 4 fr
Language should not divide..
Tag 1 fi
Tag 2en
Tag 5 fr
Tag 1 fi
Tag 2de
fr defi fi
fi
LO 1in Fi
LO 1in Fi
LO 2in Fr
LO 2in Fr
LO 2in Fr
![Page 31: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/31.jpg)
..but bring like-minded people together
Tag 1 fi
Tag 2en
Tag 1 fi
fr
de
fi fi
fi
Tag 2de
Tag 5 fr
LO 1in Fi
LO 2in Fr
![Page 32: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/32.jpg)
Visualisation tool for cross-country use of bookmarks
● Prototype tool to visualise – Bookmarks (title, classification keyword, country)– Tags (language)– Users (name, country, language)–
● Wanna play around with it?
http://www.cs.kuleuven.ac.be/~hmdb/infovis/calibrate/calibrate.html
![Page 33: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/33.jpg)
![Page 34: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/34.jpg)
Distribution of bookmarks
● Average: 6 bookmarks● Wide distribution:
– 10% “Super users” more than 20
– 15% 20-6 bookmarks– 45% 6-2 bookmarks
– About 30% only experimented (1)
![Page 35: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/35.jpg)
Language analysis
● Out of 417 tags many were with multiple terms, when separated we found 585 terms
● 1/3 in Hungarian
● 26% in English, even though none of the users were native English speakers
● 1/3 in German and Polish
![Page 36: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/36.jpg)
Language analysis
● The language was right in about 70% of cases (from the interface), and found out that...
● ...users tag in many different languages:
– at the same time (e.g. Baum, arbre, tree)
– at different times (once in Pl, other times in En)
– use the interface in different languages (seems like not only to test)
![Page 37: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/37.jpg)
Btw, what do others do?
● del.icio.us, Yahoo.fr, MyWeb.Yahoo.uk, blogmarks.net, MisterWong.de...
● Two different ways to deal with multiple languages can be observed;
– ones taken care of by users (i.e. crowd-sourcing”)
– others where the system supports multiple languages to certain extent
![Page 38: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/38.jpg)
Does the language matter?
● Need for better ways to identify the language– Give rules (if the user first preferred languages is.., then..)– Automate the recognition of languages– Out-source it to users
![Page 39: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/39.jpg)
Semantic analysis
● Factual tags 63%(Golder: item topics, kinds of item, category refinements)
● Subjective tags 29%( Golder: item qualities)
● Personal tags 3% (Golder: item ownership, self-reference, tasks organisation)
● 5% other
● Sen et al. (2006).
![Page 40: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/40.jpg)
Why tag categories?
● In Sen et al. (2006) it was found that tags of different categories can be useful for different tasks
● In our case it is too early to say anything, but ...we'll have an eye on it!
![Page 41: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/41.jpg)
“Travel well” tags
● About 13% of tags contain a general term, a name, place
● e.g. EU, Euroopa, Europa, europe, geograafia, Pythagoras, etc.
![Page 42: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/42.jpg)
What's the point of travel well tags?
● If those tags need no translation or language filtering to be understood, and ..
● ..if they can be identified
● We can be sure to show at least some tags to users – whose language preferences we don't know, and – in which language there are no tags or keywords
available.
![Page 43: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/43.jpg)
Do users find tags useful?
![Page 44: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/44.jpg)
Usefulness of tags..
● Overall, the thesaurus terms performed better than the tags,
● However, it can be argued that tags, after all being produced with no outlay, showed an overall encouraging and potential gain in overall usefulness!
![Page 45: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/45.jpg)
So what is needed?
● HIDE ALL BUT THE RIGHT STUFF!
● In the tagging interface (guided tagging)– Show tags in all languages?– Show only travel well tags?– Show only tags in users' preferred languages
● While viewing the tags– In a tag cloud– For social navigation (resource-user-tag) – Q: does the system translate tags or only when a
user-given translation exist?
![Page 46: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/46.jpg)
Future studies
● Similar language and semantic analysis are planned for a more thorough data in 2008
● Moreover, our goals are to find out:– How do users use the tags (e.g. language and
tag convergence) ?– How are tags and the relation resource-tag-user
used for discovery? – Identify teachers information seeking tasks and a
best fit for a retrieval system.
![Page 47: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/47.jpg)
User engagement
● Inspired by Yahoo!'s START– rating shows the first level of engagement;– then tags it;– user views a page; – forwards it to friends, – and finally writing a review
● How can this be used for recommending purpose?
![Page 48: RecSys 07 Doctoral Consortium Presentation](https://reader034.vdocuments.mx/reader034/viewer/2022051412/54985613b479595b4d8b542c/html5/thumbnails/48.jpg)
User engagement
● In our case these look very different:– views the page – views metadata– bookmarks and tags– rates– actual use?