seesaw personalized web search jaime teevan, mit with susan t. dumais and eric horvitz, msr
Post on 19-Dec-2015
218 views
TRANSCRIPT
Query expansion
Personalization Algorithms
Standard IR
Document
Query
User
Server
Client
v. Result re-ranking
Result Re-Ranking
Ensures privacyGood evaluation frameworkCan look at rich user profileLook at light weight user models
Collected on server side Sent as query expansion
Seesaw Search Engine
query
dog 1cat 10india 2mit 4search 93amherst 12vegas 1
dog cat monkey banana
food
baby infant
child boy girl
forest hiking
walking gorp
baby infant
child boy girl
csail mit artificial research
robotweb
search retrieval ir
hunt
Seesaw Search Engine
query
dog 1cat 10india 2mit 4search 93amherst 12vegas 1
1.6 0.26.0
0.2 2.7
1.3
Search results page
web search
retrieval ir hunt
1.3
Calculating a Document’s Score
Based on standard tf.idf
(ri+0.5)(N-ni-R+ri+0.5)
(ni-ri+0.5)(R-ri+0.5)wi = log
1.30.1 0.5
0.05 0.35 0.3
User as relevance feedback Stuff I’ve Seen index More is better
Finding the Score Efficiently
Corpus representation (N, ni) Web statistics Result set
Document representation Download document Use result set snippet
Efficiency hacks generally OK!
Evaluating Personalized Search
15 evaluatorsEvaluate 50 results for a query
Highly relevant Relevant Irrelevant
Measure algorithm quality
DCG(i) = { Gain(i),DCG(i–1) + Gain(i)/log(i),
if i = 1otherwise
Evaluating Personalized Search
Query selection Chose from 10 pre-selected queries Previously issued query
cancerMicrosofttraffic…
bison friseRed Soxairlines…
Las VegasriceMcDonalds…
Pre-selected
53 pre-selected (2-9/query)
Total: 137
JoeMary
Seesaw Improves Text Retrieval
0
0.1
0.2
0.3
0.4
0.5
0.6
Rand RF SS Web Combo
DC
G
RandomRelevance
FeedbackSeesaw
Further Exploration
Explore larger parameter spaceLearn parameters
Based on individual Based on query Based on results
Give user control?
Making Seesaw Practical
Learn most about personalization by deploying a system
Best algorithm reasonably efficientMerging server and client
Query expansion Get more relevant results in the set to be re-ranked
Design snippets for personalization
User Interface Issues
Make personalization transparentGive user control over personalization
Slider between Web and personalized results Allows for background computation
Creates problem with re-finding Results change as user model changes Thesis research – Re:Search Engine
Thank [email protected]
Study of Personal Relevancy
15 participants Microsoft employees Managers, support staff, programmers, …
Evaluate 50 results for a query Highly relevant Relevant Irrelevant
~10 queries per person
Study of Personal Relevancy
Query selection Chose from 10 pre-selected queries Previously issued query
cancerMicrosofttraffic…
bison friseRed Soxairlines…
Las VegasriceMcDonalds…
Pre-selected
53 pre-selected (2-9/query)
Total: 137
JoeMary
Relevant Results Have Low Rank
1 5 9 13 17 21 25 29 33 37 41 45 49
Rank
Highly Relevant
Relevant
Irrelevant
1 5 9 13 17 21 25 29 33 37 41 45 49
Rank
Relevant Results Have Low Rank
Highly Relevant
Relevant
Irrelevant
Rater 1 Rater 2
Same Results Rated Differently
Average inter-rater reliability: 56%Different from previous research
Belkin: 94% IRR in TREC Eastman: 85% IRR on the Web
Asked for personal relevance judgmentsSome queries more correlated than others
Same Query, Different Intent
Different meanings “Information about the astronomical/astrological
sign of cancer” “information about cancer treatments”
Different intents “is there any new tests for cancer?” “information about cancer treatments”
Same Intent, Different Evaluation
Query: Microsoft “information about microsoft, the company” “Things related to the Microsoft corporation” “Information on Microsoft Corp”
31/50 rated as not irrelevant Only 6/31 do more than one agree All three agree only for www.microsoft.com Inter-rater reliability: 56%
Much Room for Improvement
Group ranking Best improves on
Web by 38% More people
Less improvement
1.2
1.25
1.3
1.35
1.4
1 2 3 4 5 6
Number of People
DC
G
Group
Much Room for Improvement
Group ranking Best improves on
Web by 38% More people
Less improvement
Personal ranking Best improves on
Web by 55% Remains constant
1.2
1.25
1.3
1.35
1.4
1 2 3 4 5 6
Number of People
DC
G
Personalized Group
N
ni
(ri+0.5)(N-ni-R+ri+0.5)
(ni-ri+0.5)(R-ri+0.5)
ri
R
wi = log
Score = Σ tfi * wi
BM25 with Relevance Feedback
(ri+0.5)(N-ni-R+ri+0.5)
(ni- ri+0.5)(R-ri+0.5)
User Model as Relevance Feedback
N
ni
R
ri
Score = Σ tfi * wi
(ri+0.5)(N’-ni’-R+ri+0.5)
(ni’- ri+0.5)(R-ri+0.5)wi = log
N’ = N+R
ni’ = ni+ri
User Model as Relevance Feedback
N
ni
R
ri
World
UserWorld related to query
User related to query
R
Nni
ri
Query Focused Matching
Score = Σ tfi * wi
User Model as Relevance Feedback
N
ni
R
ri
World
UserWeb related to query
User related to query
R
N ri
Query Focused Matching
ni
World Focused Matching
Score = Σ tfi * wi
Parameters
Matching
User representation
World representation
Query expansion
Query focused
World focused
Parameters
Matching
User representation
World representation
Query expansion
Query focused
World focused
User Representation
Stuff I’ve Seen (SIS) index MSR research project [Dumais, et al.] Index of everything a user’s seen
Recently indexed documentsWeb documents in SIS indexQuery historyNone
Parameters
Matching
User representation
World representation
Query expansion
Query focused
World focused All SISRecent SISWeb SISQuery historyNone
Parameters
Matching
User representation
World representation
Query expansion
Query Focused
World Focused All SISRecent SISWeb SISQuery HistoryNone
World Representation
Document Representation Full text Title and snippet
Corpus Representation Web Result set – title and snippet Result set – full text
Parameters
Matching
User representation
World representation
Query expansion
Query focused
World focused All SISRecent SISWeb SISQuery historyNone
Full textTitle and snippet
WebResult set – full textResult set – title and snippet
Parameters
Matching
User representation
World representation
Query expansion
Query focused
World focused All SISRecent SISWeb SISQuery historyNone
Full textTitle and snippet
WebResult set – full textResult set – title and snippet
Query Expansion
All words in document
Query focused
The American Cancer Society is dedicated to eliminating cancer as a major health problem by preventing cancer, saving lives, and diminishing suffering through ...
The American Cancer Society is dedicated to eliminating cancer as a major health problem by preventing cancer, saving lives, and diminishing suffering through ...
Parameters
Matching
User representation
World representation
Query expansion
Query focused
World focused All SISRecent SISWeb SISQuery historyNone
Full textTitle and snippet
WebResult set – full textResult set – title and snippet
All words
Query focused
Parameters
Matching
User representation
World representation
Query expansion
Query focused
World focused All SISRecent SISWeb SISQuery historyNone
Full textTitle and snippet
WebResult set – full textResult set – title and snippet
All words
Query focused
Best Parameter Settings
Matching
User representation
World representation
Query expansion
Query focused
World focused All SISRecent SISWeb SISQuery historyNone
Full textTitle and snippet
WebResult set – full textResult set – title and snippet
All words
Query focused
All SISRecent SISWeb SIS
All SISRecent SISWeb SISQuery historyNone
Full text
All words
Query focused
World focused
Result set – title and snippet
Web
Query focused
All SIS
Title and snippet
Result set – title and snippet
Query focused
Seesaw Improves Retrieval
0
0.1
0.2
0.3
0.4
0.5
0.6
None Rand RF SS Web Combo
DC
G
No user model
RandomRelevance
FeedbackSeesaw
Summary
0
0.2
0.4
0.6
0.8
1
None SS Web Group ?
Rich user model important for search personalization
Seesaw improves text based retrievalNeed other featuresto improve WebLots of room for improvement
future
Personalizing Web Search
MotivationAlgorithmsResultsFuture Work
Further exploration Making Seesaw practical User interface issues
Further Exploration
Explore larger parameter spaceLearn parameters
Based on individual Based on query Based on results
Give user control?
Making Seesaw Practical
Learn most about personalization by deploying a system
Best algorithm reasonably efficientMerging server and client
Query expansion Get more relevant results in the set to be re-ranked
Design snippets for personalization
User Interface Issues
Make personalization transparentGive user control over personalization
Slider between Web and personalized results Allows for background computation
Creates problem with re-finding Results change as user model changes Thesis research – Re:Search Engine
Search Engines are for the Masses
Best common ranking
DCG(i) = { Sort results by number marked highly relevant,
then by relevant
Measure distance with Kendall-TauWeb ranking more similar to common
Individual’s ranking distance: 0.469 Common ranking distance: 0.445
Gain(i), if i = 1DCG(i–1) + Gain(i)/log(i), otherwise