Download - Am Introduction to Recommendation Systems
Am Introduction to Recommendation Systems
Hasan Davulcu
CIPS, ASU
Recommendation Systems
U: C X S RC: profile: age, gender, incomeS: title, director, actor, year, genre
Recommendations:
Recommender Systems
Content Based
Limitations Too Similar ! New user problem
User - Collaborative Methods U(C,S) is estimated
based on utilities U(Cj,S) by those users Cj who are “similar” to user C.
Limitations New Item Problem Sparsity
Amazon’s Item-to-Item
Comparing Human Recommenders to Online
Systems
Rashmi Sinha & Kirsten Swearingen
SIMS, UC Berkeley
Recommender Systems are technological proxy for a social process
Which one should I read?
Recommendations from friends
Recommendations from Online
Systems
I know what you’ll read next summer (Amazon, Barnes&Noble)
what movies you should watch… (Reel, RatingZone, Amazon)
what music you should listen to… (CDNow, Mubu, Gigabeat)
what websites you should visit (Alexa)
what jokes you will like (Jester)
& who you should date (Yenta)
Method Philosophy: Testing & Analysis as part of the Iterative Design Process
Design
Evaluate
Analyze
Slide adapted from James Landay
Use both quantitative & qualitative methods
Generate Design Recommendations
Taking a closer look at the Recommendation Process
Input User incurs cost in using system:
Time, Effort, Privacy Issues
Receives Recommendation
Cost in reviewing recommendations
Benefit only if recommended item appeals
Judges if he/she will sample recommendation
Amazon’s Recommendation Process
Input: One artist/author name
Output: List of Recommendations Explore / Refine Recommendations
Search usingRecommendations
Book Recommendation Site: Sleeper
Input: Ratings of 10 books for all users Use of continuous Rating Bar
(System designed by Ken Goldberg)
Output: List of items with brief information about each item
Degree of confidence in prediction
Sleeper: Output
What convinces a user to sample the recommendation
Judging recommendations: What is a good recommendation from the
user’s perspective?
Trust in a Recommender System: What factors lead to trust in a system?
System Transparency: Do users need to know why an item was
recommended?
Study of RS has focused mostly on Collaborative Filtering Algorithms
Social Recommendations
Collaborative Filtering Algorithms
Output (Recommendations)
Input from user
Beyond “Algorithms Only” : An HCI Perspective on Recommender Systems
Comparing the Social Recommendation Process to Online Recommender Systems
Understanding the factors that go into an effective recommendation (by studying users interaction with 6 online RS)
The Human vs. Recommenders Death Match
Book Systems
Amazon Books
Sleeper
Rating Zone
Movie Systems
Amazon Movies
Reel
Movie Critic
Method
For each of 3 online systems: Registered at site Rated items Reviewed and evaluated recommendation set Completed questionnaire
Also reviewed and evaluated sets of recommendations from 3 friends each
19 participants, age:18 to 34 years
Results
Defining Types of Recommendations
Good Recs. (Precision)•% items user felt interested in
Useful Recs.•Subset of Good Recs.•User felt interested in and had not read / viewed yet
USEFULNot yet read/viewed
GOOD: User likes
0
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Amazon(15)
Sleeper(10)
RatingZone (8)
Friends(9)
Amazon(15)
Reel(5-10)
MovieCritic (20)
Friends(9)
Books Movies
% Good Recommendations
% Useful Recommendations
Comparing Human Recommenders to RS: “Good” and “Useful” Recommendations
RS AverageAve. Std. Error (x) No. of Recommendations
However users like online RS
This result was supported by post test interviews.
Do you prefer recommendations from friends or online systems?
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
Books Movies
Num
ber
pref
erri
ng s
yste
m
System
Friends
Why systems over friends? “Suggested a number of things I hadn’t heard of,
interesting matches.”
“It was like going to Cody’s—looking at that table up front for new and interesting books.”
“Systems can pull from a large database—no one person knows about all the movies I might like.”
Items users had “Heard of” before
Friends recommended mostly “old” previously experienced items
% Heard of
0
10
20
30
40
50
60
70
Amazon Sleeper RatingZone Friends Amazon MovieCritic Reel Friends
MoviesBooks
What systems did users prefer?
Sleeper and Amazon books average highest ratings Split opinions on Reel, MovieCritic
Would you use system again?
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
Amazon Sleeper RatingZone Amazon Reel MovieCritic
Av
era
ge
Rat
ing
No
Yes
MoviesBooks
Why did some systems…
Provide useful recommendations but leave users unsatisfied? RatingZone, MovieCritic & Reel
Possible Reasons
Previously Enjoyed Items are important: We term these Trust-Generating Items
Adequate Item Description & Ease of Use are important Missing from List: Time to Receive Recommendations &
No. of Items to Rate not important!
What predicts overall usefulness of a System?
0
0.1
0.2
0.3
0.4
0.5
0.6
Good Rec. Useful Rec. TrustGenerating
Rec.
AdequateItem
Description
Ease ofUse
Co
rre
lati
on
All correlations are significant at .05
USEFULNot yet read/viewed
TRUST-GENERATINGPreviously read/viewed
A Question of Trust…
GOOD: User likes Post Test Interviews showed that users “trust” systems if they have already sampled some recommendations
•Positive Experiences lead to “trust
•Negative Experiences with Recommended Items lead to mistrust of system
Difference between Amazon and Sleeper highlights the fact that there are different kinds of good Recommender Systems
A Question of Trust …Number of Trust Generating Items
.00
1.00
2.00
3.00
4.00
5.00
6.00
Amazon Sleeper RatingZone Amazon Reel MovieCritic
Num
ber
of It
ems
Books Movies
Adequate Item Description: The RatingZone Story
0 % of Version 1 and 60% of Version 2 users found item description adequate
% Useful For Both Versions of RatingZOne
0
5
10
15
20
25
30
35
40
45
Version 1: Without Description Version 2: With Description
% U
sefu
l R
ecs
.
An adequate item description, and links to other sources about item was a crucial factor in users being convinced by a recommendation.
System Transparency Why was this item recommended?
Do users understand why an item was recommended
Users mentioned this factor in post test interviews
Effect of System Transparency on Recommendation
0
10
20
30
40
50
60
System Reasoning wasTransparent
System Reasoning NotTransparent
% G
oo
d R
eco
mm
en
da
tio
ns
Discussion & Design Recommendations
Design Recommendations: Justification
Justify your Recommendations
Adequate Item Information: Providing enough detail about item for user to make choice
System Transparency: Generate (at least some) recommendations which are clearly linked to the rated items
Explanation: Provide an Explanation, why the item was recommended.
Community Ratings: Provide link to ratings / reviews by other users. If possible, present numerical summary of ratings.
Design Recommendations:Accuracy vs. Less Input
Don’t sacrifice accuracy for the sake of generating quick recommendations. Users don’t mind rating more items to receive quality recommendations.
A possible way to achieve this: have multilevel recommendations. Users can initially use the system by providing one rating, and are offered subsequent opportunities to refine recommendation
One needs a happy medium between too little input (leading to low accuracy) and too much input (leading to user impatience)
Design Recommendations: New Unexpected Items
Users like Rec. Systems as they provide information about new, unexpected items.
List of recommended items should include new items which the user might not find out in any other way.
List could also include some unexpected items (e.g., from other topics / genres) which the user might not have thought of themselves.
Design Recommendations: New Unexpected Items
Users (especially first time users) need to develop trust in the system.
Trust in system is enhanced by the presence of items that the user has already enjoyed.
Generating some very popular (which have probably been experienced previously) in the initial recommendation set might be one way to achieve this.
Design Recommendations: Trust Generating Items
Systems need to provide a mix of different kinds of items to cater to different users:
Trust Generating Items: A few very popular ones, which the system has high confidence in
Unexpected Items: Some unexpected items, whose purpose is to allow users to broaden horizons.
Transparent Items: At least some items for which the user can see the clear link between the items he /she rated and the recommendation.
New Items: Some items which are new.
Design Recommendations: Mix of Items
Question: Should these be presented as a sorted list / unsorted list/ different categories of recommendations?
Allow users to provide ratings on a continuous scale. One of the reasons users liked Sleeper was
because it allowed them to rate on a continuous scale. Users did not like binary scales.
Design Recommendations: Continuous Scales for Input
Limitations of Study Simulated first-time visit, did not allow system to
learn user preferences over time
Source of recommendations known to subjects—might have biased towards friends
Fairly homogenous group of subjects, no novice users
Future Plans: Second Generation Music Recommender Systems
•Have evolved beyond previous systems
•Use a variety of sophisticated algorithms to map users preferences over music domain
•Require a lot more input from the user
•Users can sample recommendations during the study!
MusicBudha (Mubu.com): Exploring Genres
Mubu.com: Exploring Jazz Styles
Mubu.com: Rating Samples
Mubu.com: Recommendations as Audio Samples
Compare systems, friends and experts Anonymize the source of recommendation
The Turing Test for Music Recommender Systems
Study Design
Goal: Compare recommendations by online RS, Experts (who have same information as RS) & Friends
Music Experts
Friends
Online RS
So far we have heard what this study tells us about Recommender Systems
But what (if anything) does it have to say about human nature?
In conclusion…
Recommender Systems tantalize us with the idea that we are not as unique and unpredictable as we think we are.
Study results show that Recommender Systems do not know us better than our friends!
But …
11 out of 19 users preferred Recommender Systems over Friends Recommendations!
Ultimately, we all want to be tables in a database!
Email: [email protected]
Web address: http://sims.berkeley.edu/~sinha