accurately interpreting clickthrough data as implicit feedback joachims, granka, pan, hembrooke, gay...
TRANSCRIPT
![Page 1: Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05](https://reader036.vdocuments.mx/reader036/viewer/2022070306/5517372c550346f5558b5fda/html5/thumbnails/1.jpg)
Accurately Interpreting Clickthrough Data as Implicit Feedback
Joachims, Granka, Pan, Hembrooke, Gay
Paper Presentation: Vinay Goel
10/27/05
![Page 2: Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05](https://reader036.vdocuments.mx/reader036/viewer/2022070306/5517372c550346f5558b5fda/html5/thumbnails/2.jpg)
Introduction
Adapt a retrieval system to users and or collections
Manual adaptation - time consuming or even impractical
Explore and evaluate implicit feedbackUse clickthrough data in WWW search
![Page 3: Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05](https://reader036.vdocuments.mx/reader036/viewer/2022070306/5517372c550346f5558b5fda/html5/thumbnails/3.jpg)
User Study
Record and evaluate user actionsProvide insight into the decision processRecord users’ eye movements : Eye
tracking
![Page 4: Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05](https://reader036.vdocuments.mx/reader036/viewer/2022070306/5517372c550346f5558b5fda/html5/thumbnails/4.jpg)
Questions used
![Page 5: Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05](https://reader036.vdocuments.mx/reader036/viewer/2022070306/5517372c550346f5558b5fda/html5/thumbnails/5.jpg)
Two Phases of the study
Phase I 34 participants Start search with Google query, search for answers
Phase II Investigate how users react to manipulations of search
results Same instructions as phase I Each subject assigned to one of three experimental
conditions Normal, Swapped, Reversed
![Page 6: Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05](https://reader036.vdocuments.mx/reader036/viewer/2022070306/5517372c550346f5558b5fda/html5/thumbnails/6.jpg)
Explicit Relevance Judgments
Collected explicit relevance judgments for all queries and results pages
Inter-judge agreements
![Page 7: Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05](https://reader036.vdocuments.mx/reader036/viewer/2022070306/5517372c550346f5558b5fda/html5/thumbnails/7.jpg)
Analysis of user behavior
Which links do users view and click?
Do users scan links from top to bottom?
Which links do users evaluate before clicking?
![Page 8: Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05](https://reader036.vdocuments.mx/reader036/viewer/2022070306/5517372c550346f5558b5fda/html5/thumbnails/8.jpg)
Which links do users view and click?
Almost equal frequency of 1st and 2nd link, but more clicks on 1st link
Once the user has started scrolling, rank appears to become less of an influence
![Page 9: Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05](https://reader036.vdocuments.mx/reader036/viewer/2022070306/5517372c550346f5558b5fda/html5/thumbnails/9.jpg)
Do users scan links from top to bottom?
Big gap before viewing 3rd ranked abstract Users scan viewable results thoroughly before
scrolling
![Page 10: Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05](https://reader036.vdocuments.mx/reader036/viewer/2022070306/5517372c550346f5558b5fda/html5/thumbnails/10.jpg)
Which links do users evaluate before clicking?
Abstracts closer above the clicked link are more likely to be viewed
Abstract right below a link is viewed roughly 50% of the time
![Page 11: Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05](https://reader036.vdocuments.mx/reader036/viewer/2022070306/5517372c550346f5558b5fda/html5/thumbnails/11.jpg)
Analysis of Implicit Feedback
Does relevance influence user decisions?
Are clicks absolute relevance judgments?
![Page 12: Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05](https://reader036.vdocuments.mx/reader036/viewer/2022070306/5517372c550346f5558b5fda/html5/thumbnails/12.jpg)
Does relevance influence user decisions? Yes Use the “reversed” condition
Controllably decreases the quality of the retrieval function and relevance of highly ranked abstracts
Users react in two ways View lower ranked links more frequently, scan
significantly more abstracts Subjects are much less likely to click on the first link,
more likely to click on a lower ranked link
![Page 13: Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05](https://reader036.vdocuments.mx/reader036/viewer/2022070306/5517372c550346f5558b5fda/html5/thumbnails/13.jpg)
Clicks = absolute relevance judgments?Interpretation is problematicTrust Bias
Abstract ranked first receives more clicks than the second
First link is more relevant (not influenced by order of presentation) or
Users prefer the first link due to some level of trust in the search engine (influenced by order of presentation)
![Page 14: Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05](https://reader036.vdocuments.mx/reader036/viewer/2022070306/5517372c550346f5558b5fda/html5/thumbnails/14.jpg)
Trust Bias
Hypothesis that users are not influenced by presentation order can be rejected
Users have substantial trust in search engine’s ability to estimate relevance
![Page 15: Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05](https://reader036.vdocuments.mx/reader036/viewer/2022070306/5517372c550346f5558b5fda/html5/thumbnails/15.jpg)
Quality Bias
Quality of the ranking influences the user’s clicking behavior If relevance of retrieved results decreases,
users click on abstracts that are on average less relevant
Confirmed by the “reversed” condition
![Page 16: Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05](https://reader036.vdocuments.mx/reader036/viewer/2022070306/5517372c550346f5558b5fda/html5/thumbnails/16.jpg)
Are clicks relative relevance judgments?An accurate interpretation of clicks needs
to take into consideration User’s trust into quality of search engine Quality of retrieval function itself
Difficult to measure explicitlyInterpret clicks as pairwise preference
statements
![Page 17: Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05](https://reader036.vdocuments.mx/reader036/viewer/2022070306/5517372c550346f5558b5fda/html5/thumbnails/17.jpg)
Strategy 1
Takes trust and quality bias into consideration Substantially and significantly better than
random Close in accuracy to inter judge agreement
![Page 18: Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05](https://reader036.vdocuments.mx/reader036/viewer/2022070306/5517372c550346f5558b5fda/html5/thumbnails/18.jpg)
Strategy 2
Slightly more accurate than Strategy 1 Not a significant difference in Phase II
![Page 19: Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05](https://reader036.vdocuments.mx/reader036/viewer/2022070306/5517372c550346f5558b5fda/html5/thumbnails/19.jpg)
Strategy 3
Accuracy worse than Strategy 1 Ranking quality has an effect on the accuracy
![Page 20: Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05](https://reader036.vdocuments.mx/reader036/viewer/2022070306/5517372c550346f5558b5fda/html5/thumbnails/20.jpg)
Strategy 4
No significant differences compared to Strategy 1
![Page 21: Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05](https://reader036.vdocuments.mx/reader036/viewer/2022070306/5517372c550346f5558b5fda/html5/thumbnails/21.jpg)
Strategy 5
Highly accurate in the “normal” condition Misleading
Aligned preferences probably less valuable for learning Better results even if user behaves randomly
Less accurate than Strategy 1 in the “reversed” condition
![Page 22: Accurately Interpreting Clickthrough Data as Implicit Feedback Joachims, Granka, Pan, Hembrooke, Gay Paper Presentation: Vinay Goel 10/27/05](https://reader036.vdocuments.mx/reader036/viewer/2022070306/5517372c550346f5558b5fda/html5/thumbnails/22.jpg)
Conclusion
Users’ clicking decisions influenced by search bias and quality bias
Strategies for generating relative relevance feedback signals
Implicit relevance signals are less consistent with explicit judgments than the explicit judgments among each other
Encouraging results