Download - Sigir 2014-mobile-eye-tracking-slides
Towards Better Measurement of Attention and Satisfaction
in Mobile Search
Dmitry Lagun, Chih-Hung Hsieh, Dale Webster, Vidhya Navalpakkam
Thanks!
Vidhya Navalpakkam Dale WebsterChih-Hung Hsieh
2
Mobile is popular!
• 25% of Web page visits come from mobile[Statcounter.com, 2014]
• Mobile browsing grew five fold since 2010 (5%)[Statcounter.com, 2014]
• One in every 5 search queries is issued from a mobile device[RKG Digital Marketing Report, 2013]
3
Our Study
Attention MeasurementSatisfaction with Rich Results
KnowledgeGraphResult
4
Satisfaction with Rich Results on Mobile:Background
• Long history of using clicks for measurement of search satisfaction and result relevance[Joachims et al., SIGIR 2005; Agichtein et al., SIGIR 2006]
• Result relevance and implicit indicators (mouse cursor hover, touch & swipe)[Huang et al., CHI 2011; Lagun et al., SIGIR 2011; Guo et al., SIGIR 2013]
• Rich Answers do not require to click and mouse hoversdo not exist on mobile – What other implicit metrics can we use to infer result
relevance/satisfaction without clicks/hovers?
5
User Study Design
• Two Factor (within Subject)
– Relevance
– Presence
• 20 Search Tasks
• Users were asked to provide explicitsatisfaction score for each task (1-7 scale)
KG RelevantKG Not
Relevant
KG Present 5 Tasks5 Tasks
KG Absent5 Tasks 5 Tasks
6
User Study Details
• Participants– 24 users (diverse background, age, occupation)
• Mobile Eye Tracker Setup
• Calibration Directlyon Phone Screen
7
Can Implicit User Metrics Indicate Answer Relevance?
• Page and Task metrics – Time on SERP– Number of Scrolls– Time on Task
• Gaze Metrics– Time on Rich Result (and %)– Total Time below Rich Result (and %)
• Viewport Metrics– Time on Rich Result (and %)– Total Time below Rich Result (and %)
Kn
ow
ledge G
raph
Resu
lt
8
KG is Not Relevant More Scrolling
10
KG is Relevant Faster Search(answer is found in KG without a click)
11
No Impact on User Satisfaction when KG is Not Relevant!
12
Gaze Metrics vs. KG Relevance
13
Relevant Not Relevant
More Time Below the KG Result
%Viewport Time Below vs. KG Relevance
15
0
5
10
15
20
25
30
Not Relevant Relevant
% V
iew
po
rt T
ime
Be
low
KG
More time on results below Not Relevant KG
Satisfaction with Rich Results:Summary
• We can use Page and Viewport metrics to infer KG relevance and satisfaction
• No impact on user satisfaction when Not Relevant KG is shown
• Users view more results below the KG, when it is Not Relevant
16
Our Study
Attention MeasurementSatisfaction with Rich Results
KnowledgeGraphResult
17
Attention Measurement in Search:Background
• Eye Tracking – accurate, but limited in scale[Granka et al., WWW 2004; Buscher et al., SIGIR, CHI 2008-2010]
• Mouse Cursor Tracking – less accurate, but scalable [Huang et al., CHI 2011, 2012; Lagun et al., SIGIR 2011; Guo et al., CHI 2010, WWW 2012; Navalpakkam et al., WWW 2013]
• Viewport Tracking – accurate (???), scalable(on mobile)
18
Viewport Time Calculation: Primer
• Display Time = 10 sec
• ViewportTime(R1) = ?
• Coverage– % of screen area occupied by the result
(e.g. Coverage(KG) > Coverage(R1))
• Exposure– % of result area visible on the screen
(e.g. Exposure (R2) < 1.0)
• ViewportTime(R) = DisplayTime * Coverage(R) * Exposure(R)
19
KG
R1
R2
%Gaze Time
%V
iew
po
rt T
ime
Vie
wp
ort
Tim
e
Gaze Time
Can we use Viewport Time to measure time spent on each result?
Pearson R = 0.57 Pearson R = 0.69
one search result
Correlation is high can use Viewport Time to accuratelymeasure time spent on individual search result at scale
20
Are attention patterns similar on desktop and mobile?
?
Granka et al., WWW 2004
?21
Viewing Time vs. Result Position
Granka et al., WWW 2004
On desktop:
Why?
22
Short Scroll Effect
23
Short Scroll Effect
24
Short Scroll Effect
25
Do users have position preferencewhen reading on a mobile phone?
26
Conclusions
• Viewport and Page metrics can be used to measure Rich Answer Relevance and Satisfaction
• Viewport time provides accurate (R=0.69)estimate on time spent on search result
• Users prefer to position content on top half of the phone’s screen
27
Results Summary
Attention MeasurementSatisfaction with Rich Results
28
%Gaze Time
%V
iew
po
rt T
ime
Viewport ≈ Gaze(on mobile)
Pearson R = 0.69
Top half of the screen receives more Attention
“Short-Scroll” effect
Granka et al., WWW 2004
DesktopMobile
Relevant Not Relevant
More results are viewed if Answer is Not
Relevant
No Impact on User Satisfaction when KG is Not
Relevant!