Towards Better Measurement of Attention and Satisfaction

in Mobile Search

Dmitry Lagun, Chih-Hung Hsieh, Dale Webster, Vidhya Navalpakkam

Thanks!

Vidhya Navalpakkam Dale WebsterChih-Hung Hsieh

2

Mobile is popular!

• 25% of Web page visits come from mobile[Statcounter.com, 2014]

• Mobile browsing grew five fold since 2010 (5%)[Statcounter.com, 2014]

• One in every 5 search queries is issued from a mobile device[RKG Digital Marketing Report, 2013]

3

Our Study

Attention MeasurementSatisfaction with Rich Results

KnowledgeGraphResult

4

Satisfaction with Rich Results on Mobile:Background

• Long history of using clicks for measurement of search satisfaction and result relevance[Joachims et al., SIGIR 2005; Agichtein et al., SIGIR 2006]

• Result relevance and implicit indicators (mouse cursor hover, touch & swipe)[Huang et al., CHI 2011; Lagun et al., SIGIR 2011; Guo et al., SIGIR 2013]

• Rich Answers do not require to click and mouse hoversdo not exist on mobile – What other implicit metrics can we use to infer result

relevance/satisfaction without clicks/hovers?

5

User Study Design

• Two Factor (within Subject)

– Relevance

– Presence

• 20 Search Tasks

• Users were asked to provide explicitsatisfaction score for each task (1-7 scale)

KG RelevantKG Not

Relevant

KG Present 5 Tasks5 Tasks

KG Absent5 Tasks 5 Tasks

6

User Study Details

• Participants– 24 users (diverse background, age, occupation)

• Mobile Eye Tracker Setup

• Calibration Directlyon Phone Screen

7

Can Implicit User Metrics Indicate Answer Relevance?

• Page and Task metrics – Time on SERP– Number of Scrolls– Time on Task

• Gaze Metrics– Time on Rich Result (and %)– Total Time below Rich Result (and %)

• Viewport Metrics– Time on Rich Result (and %)– Total Time below Rich Result (and %)

Kn

ow

ledge G

raph

Resu

lt

8

KG is Not Relevant More Scrolling

10

KG is Relevant Faster Search(answer is found in KG without a click)

11

No Impact on User Satisfaction when KG is Not Relevant!

12

Gaze Metrics vs. KG Relevance

13

Relevant Not Relevant

More Time Below the KG Result

%Viewport Time Below vs. KG Relevance

15

0

5

10

15

20

25

30

Not Relevant Relevant

% V

iew

po

rt T

ime

Be

low

KG

More time on results below Not Relevant KG

Satisfaction with Rich Results:Summary

• We can use Page and Viewport metrics to infer KG relevance and satisfaction

• No impact on user satisfaction when Not Relevant KG is shown

• Users view more results below the KG, when it is Not Relevant

16

Our Study

Attention MeasurementSatisfaction with Rich Results

KnowledgeGraphResult

17

Attention Measurement in Search:Background

• Eye Tracking – accurate, but limited in scale[Granka et al., WWW 2004; Buscher et al., SIGIR, CHI 2008-2010]

• Mouse Cursor Tracking – less accurate, but scalable [Huang et al., CHI 2011, 2012; Lagun et al., SIGIR 2011; Guo et al., CHI 2010, WWW 2012; Navalpakkam et al., WWW 2013]

• Viewport Tracking – accurate (???), scalable(on mobile)

18

Viewport Time Calculation: Primer

• Display Time = 10 sec

• ViewportTime(R1) = ?

• Coverage– % of screen area occupied by the result

(e.g. Coverage(KG) > Coverage(R1))

• Exposure– % of result area visible on the screen

(e.g. Exposure (R2) < 1.0)

• ViewportTime(R) = DisplayTime * Coverage(R) * Exposure(R)

19

KG

R1

R2

%Gaze Time

%V

iew

po

rt T

ime

Vie

wp

ort

Tim

e

Gaze Time

Can we use Viewport Time to measure time spent on each result?

Pearson R = 0.57 Pearson R = 0.69

one search result

Correlation is high can use Viewport Time to accuratelymeasure time spent on individual search result at scale

20

Are attention patterns similar on desktop and mobile?

?

Granka et al., WWW 2004

?21

Viewing Time vs. Result Position

Granka et al., WWW 2004

On desktop:

Why?

22

Short Scroll Effect

23

Short Scroll Effect

24

Short Scroll Effect

25

Do users have position preferencewhen reading on a mobile phone?

26

Conclusions

• Viewport and Page metrics can be used to measure Rich Answer Relevance and Satisfaction

• Viewport time provides accurate (R=0.69)estimate on time spent on search result

• Users prefer to position content on top half of the phone’s screen

27

Results Summary

Attention MeasurementSatisfaction with Rich Results

28

%Gaze Time

%V

iew

po

rt T

ime

Viewport ≈ Gaze(on mobile)

Pearson R = 0.69

Top half of the screen receives more Attention

“Short-Scroll” effect

Granka et al., WWW 2004

DesktopMobile

Relevant Not Relevant

More results are viewed if Answer is Not

Relevant

No Impact on User Satisfaction when KG is Not

Relevant!