dangur, hammel, rafaeli, march 2006 1 qsia: online knowledge items in service of learning...

DanGur, Hammel, Rafaeli, March 2006

1

QSIA: Online Knowledge QSIA: Online Knowledge ItemsItems

In Service of Learning In Service of Learning CommunitiesCommunities

Sheizaf RafaeliThe Center for the Study of the Information

Society, University of Haifa

Ilan HammelThe Department of Pathology in Sackler Faculty of Medicine, Tel Aviv University

Yuval Dan-GurThe Center for the Study of the Information

Society, University of Haifa


2

RoadmapRoadmap

• “Recommender Systems” – what are they good for?

• “RS for Knowledge itemsKnowledge items” – what is special here?

• “QSIA” – what is it? Where was it implemented? What does it contain?


3

Roadmap-Roadmap-continuedcontinued

• The Research Project– Research Questions– Research Method– Variables and Analysis– Limitations

• Summary– Findings– Implications

• Q&A


4

Recommender SystemsRecommender Systems

• Recommender Systems (a.k.a reputation or collaboration systems) recommend, in a personalized manner, relevant items to users from large number of alternatives.

• Working examples: web resources, movies, books and ski resorts


5

Are RS Oracles? Are RS Oracles? • They are mostly social -

track what others rank as high/low.

• Also personalized: “know” something about what I like/dislike.

• It “assumes” – that those who liked what I liked, should act as my future recommending group.

• It computes – correlations (to identify my recommenders).


6

““War – What is it good for?”War – What is it good for?”(or: What is special in RS for (or: What is special in RS for Knowledge Knowledge

ItemsItems?)?) • High Risk items - is sharing a

grade of a movie equivalent to sharing a question (potentially in an exam)?

• Ownership of knowledge: – The writer controls who shares.– Student and teachers load items.

• Communities – Preferences of recommenders: (for examples) students>teachers? teachers>teachers?


7

What is special in RS for What is special in RS for Knowledge Knowledge ItemsItems??

(Continued) (Continued)

• Validity - Will grades of an open WEB RS correlate positively with “standard” exams?

• Academic “taste”: – Will different institutions share?– Will different classes share?


8

QSIA - 1QSIA - 1 • QSIA is a collaborative system for

collection, management, sharing and assignment of knowledge items for learning.

• The system supports creation and editing of knowledge items and conducting online educational tasks and includes a recommendation module that assists the students and teachers in filtering relevant information.

• URL - http://www.qsia.orgQSIA was developed in the Center for the Study of the Information Society with the support of the Caesarea Edmond Benjamin de Rothschild Foundation Institute (CRI) for Interdisciplinary Application of Computer Science at the University of Haifa.

http://www.qsia.org/


9

QSIA-2QSIA-2 • QSIA is unique:

– to the best of our knowledge this is one of the first recommender systems that enables user's involvement in the determining the set of the 'neighbors group' for an automated collaborative filtering recommendation.

– QSIA is one of the few systems that enable

immediate usage of the "liked" recommended items in the same system as the next step that follows suggestion of recommendations, and

– QSIA applies recommendation technology to a

novel domain – knowledge items for distance learning and online tests - that are not "natural" for recommender systems


10

QSIA QSIA continuationcontinuation

• QSIA was implemented in numerous courses and several academic institutions. Of concern here is a course in General and Systematic Pathology in the Faculty of Medicine, Tel-Aviv University; Israel.

• QSIA's database and logs comprised of approximately 31,000 records of items-seeking, 3,000 users (mostly students), 10,000 items (mainly medical pathology), and 3,000 rankings by 300 users and knowledge items from 30 domains.

• Only 895 recommendations sought by 108 users were relevant for our study of recommendations. The rest of the data were related to self-browsing.


11

The Research – Main The Research – Main HypothesisHypothesis

• We introduced the term "friends group" to

describe a sub-group of the neighbors group that is not solely rank-dependent.

• The 'friends group' is unique because of the user's involvement in its formation and the user's ability to choose the characteristics of its members.

• The latter aspect is in accordance with the "Social Comparison Theory" and the derived behavioral studies suggesting that 'neighbors' (like-minded group) are relevant for 'low-risk' domains whereas 'friends' (similar on personal characteristics) are more relevant for 'high-risk' domains.


12

The Research QuestionsThe Research Questions

• Our first research question was concerned

with users' preferences concerning control over the recommendation process as opposed to acceptance of recommendations from a "computerized oracle".

• The second research question examined whether the attitude of the recommendation seeker obeys social rules, specifically, the "Social Comparison Theory".

• The last research question was concerned with the characteristics of the members of the 'friends group' that are chosen by the user.


13

The Research MethodThe Research Method

• A two-year long field study, using

QSIA.

• We developed a five-stage conceptual model of users' interaction with the recommendation module of QSIA that describes the processes of 'neighbors' and 'friends' recommendations seeking that the system supports:


14

The Research MethodThe Research Method

User needs to choose(for example: item for bundle)

Recommendations seekingSelf Browsing

(Without recommendation)

'Friends' 'Neighbors'

User chooses characteristics

No user action

Recommendation list is generated

Recommendation list is generated

Acceptance of part of the recommendations

Acceptance of part of the recommendations


15

Variables & AnalysisVariables & Analysis • Our main dependent variables were the "source

of recommendation (SoR)" (namely either 'friends group' or 'neighbors group'), the ratios of accepted and rejected items in each recommendations seeking instance, and the users' choices of the friends' characteristics (group, grade level, and role).

• The independent variables were the iteration number of the recommendations seeking and the SoR.


16

Variables & AnalysisVariables & Analysis

• We analyzed alternative hypotheses concerning the choice of the role of members of the 'friends group' (users will choose teachers' recommendations because of their authority and knowledge expertise) and the choice of grade level (the "reference group" will be comprised of students with higher grades).

• The main statistical methods and tests we used were the Wilcoxon signed-rank test, logistic regression, the GEE models for correlated binary data in logistic regression, and the Runs tests.


17

LimitationsLimitations • Our research has many limitations apart Our research has many limitations apart

from the known drawbacks of any field from the known drawbacks of any field study: the most important one is that we did study: the most important one is that we did not find a relevant comparable field study not find a relevant comparable field study with which to triangulate our results. with which to triangulate our results. Because of its uniqueness, we detailed the Because of its uniqueness, we detailed the weaknesses and limitations that we did weaknesses and limitations that we did recognize in the recognize in the research methodresearch method (the main (the main one being that we did not inquire about one being that we did not inquire about users' motivations for their behavior), theusers' motivations for their behavior), the research toolresearch tool, QSIA (which is hard to , QSIA (which is hard to compare to other recommender systems and compare to other recommender systems and allows processes that result in missing allows processes that result in missing data), and the collected data), and the collected datadata (of which only (of which only minor part was relevant to our study, and its minor part was relevant to our study, and its sparseness can cause other limitations).sparseness can cause other limitations).


18

FINDINGS - 1FINDINGS - 1

• H1: The results of the GEE longitudinal H1: The results of the GEE longitudinal models suggested that users acquire a models suggested that users acquire a tendency to seek recommendations from tendency to seek recommendations from 'friends groups' and the probability 'friends groups' and the probability increases as more recommendations are increases as more recommendations are sought by users.sought by users.


19


• H2: We noted a significant positive difference H2: We noted a significant positive difference in the acceptance level of recommendations in the acceptance level of recommendations by users when they asked for 'friends groups' by users when they asked for 'friends groups' recommendations. In addition, the same recommendations. In addition, the same items were more accepted when offered to items were more accepted when offered to the user by the 'friends group' than when the user by the 'friends group' than when offered by the 'neighbors group'. The offered by the 'neighbors group'. The difference in acceptance was higher for items difference in acceptance was higher for items that were recommended frequently.that were recommended frequently.


20


• H3: The choice of one's own group was the H3: The choice of one's own group was the most important characteristic for users to most important characteristic for users to assign to the advising group members. We assign to the advising group members. We also noted that the majority of users sought also noted that the majority of users sought recommendations from teachers rather than recommendations from teachers rather than from students. About half the time users from students. About half the time users chose participants with higher grades than chose participants with higher grades than their own to populate the advising group and their own to populate the advising group and about half the time users chose participants about half the time users chose participants with similar grades to their own.with similar grades to their own.


21

IMPLICATIONS IMPLICATIONS • The main novel finding is the relationship between The main novel finding is the relationship between

the perceived quality of the recommendation the perceived quality of the recommendation (measured in terms of "usage actions"), and users’ (measured in terms of "usage actions"), and users’ involvement in the formation of the advising group. involvement in the formation of the advising group. We included literature review from a variety of We included literature review from a variety of domains to detail how our findings fit with previous domains to detail how our findings fit with previous research. We point out many studies and papers research. We point out many studies and papers that can be linked to our findings, mainly studies on that can be linked to our findings, mainly studies on accepting advice from an automated machine, HCI, accepting advice from an automated machine, HCI, transparency of systems, applying social rules and transparency of systems, applying social rules and expectations to computers, and the nature of human expectations to computers, and the nature of human taste. taste.

• The findings may be of interest for further The findings may be of interest for further interdisciplinary research on collaborative filtering, interdisciplinary research on collaborative filtering, bridging the gap between "computerized oracles" bridging the gap between "computerized oracles" and social behavior, relating computerized and social behavior, relating computerized collaboration and social theories, economical collaboration and social theories, economical implications of higher acceptance level of implications of higher acceptance level of recommendations, and a motivation to conduct recommendations, and a motivation to conduct additional field studies, especially within the 'high-additional field studies, especially within the 'high-risk' items domain.risk' items domain.


22

Questions?Questions?Comments?Comments?

Observations?Observations?Collaborations?Collaborations?


23

ENDEND


24

BEYOND THE END…..BEYOND THE END…..


25

PAPERS ON QSIA PAPERS ON QSIA 1.1. Rafaeli, S., Dan-Gur, Y. & Barak, M. (2005). Finding friends among Rafaeli, S., Dan-Gur, Y. & Barak, M. (2005). Finding friends among

recommenders: Social and "Black-Box" recommender systems", recommenders: Social and "Black-Box" recommender systems", International Journal of Distance Education Technologies (IJDET), International Journal of Distance Education Technologies (IJDET), Special Issue on Knowledge Management Technologies for E-Special Issue on Knowledge Management Technologies for E-learning: Exploiting Knowledge Flows and Knowledge Networks learning: Exploiting Knowledge Flows and Knowledge Networks for Learning, 3(2), 30-47.for Learning, 3(2), 30-47.

2.2. Barak, M. & Rafaeli, S. (2004). Online question-posing and peer-Barak, M. & Rafaeli, S. (2004). Online question-posing and peer-assessment as means for web-based knowledge sharing in assessment as means for web-based knowledge sharing in learning, International Journal of Human-Computer Studies, 61(1), learning, International Journal of Human-Computer Studies, 61(1), 84-103. 84-103.

3.3. Rafaeli, S., Barak, M., Dan-Gur, Y. and Toch, E. (2004). QSIA - A Rafaeli, S., Barak, M., Dan-Gur, Y. and Toch, E. (2004). QSIA - A web-based environment for learning, assessing and knowledge web-based environment for learning, assessing and knowledge sharing in communities, 43(3), 273-289. sharing in communities, 43(3), 273-289.

4.4. Rafaeli, S., Barak, M. Dan-Gur, Y. & Toch E. (2003). Knowledge Rafaeli, S., Barak, M. Dan-Gur, Y. & Toch E. (2003). Knowledge sharing and online assessment, E-Society Proceedings of the 2003 sharing and online assessment, E-Society Proceedings of the 2003 IADIS conference IADIS e-Society 2003, pp. 257-266.IADIS conference IADIS e-Society 2003, pp. 257-266.

5.5. Rafaeli, S., Dan-Gur, Y., Noy, A., Raban, D., Ravid, G. (2002). Rafaeli, S., Dan-Gur, Y., Noy, A., Raban, D., Ravid, G. (2002). Simulations in Internet Research:Value and Sharing of Simulations in Internet Research:Value and Sharing of Information, Social Facilitation, Friends and Neighbors. A panel in Information, Social Facilitation, Friends and Neighbors. A panel in AoIR 3.0, Maastricht, Netherlands.(unpublished).AoIR 3.0, Maastricht, Netherlands.(unpublished).

6.6. Rafaeli, S., Dan-Gur, Y. (2002). Advising Groups in Recommender Rafaeli, S., Dan-Gur, Y. (2002). Advising Groups in Recommender Systems, Proceedings of the Doctoral Consortium in the 6th Systems, Proceedings of the Doctoral Consortium in the 6th Pacific Asia Conference on Information Systems (PACIS 2002), Pacific Asia Conference on Information Systems (PACIS 2002), Tokyo, Japan. Tokyo, Japan.


26

Hidden SlidesHidden Slides


27


28


29


30

dangur, hammel, rafaeli, march 2006 1 qsia: online knowledge items in service of learning...

Documents

online knowledge items

assignment of knowledge

relevant items

knowledge itemsrs

ownership of knowledge

university of haifa

liked recommended items

high risk items