Attention and selection behavior on "universal search" result pages based on proposed Google commitments of Oct. 21, 2013 Report about an eye tracking pilot study commissioned by ICOMP Initiative for a Competitive Online Marketplace

Introduction 2

Short report 4

Setup of the study 5

Detailed results of the study 14

References 23

Institute of Communication and Media Research

GERMAN SPORTS UNIVERSITY COLOGNE

- November 21st, 2013 -

  2  

1. Introduction

The pilot study as described in the following

illustrates an empirical approach regarding the

distribution of attention and the selection

behavior on Google "Universal Search" result pages.

The ISO Norm line 9241-11states as the three

criteria for usability: effectiveness, efficiency

and satisfaction. One tries to adjust the

interface design to these criteria for example by

user tests, interviews, heuristic evaluation by

experts or modeling the expected user behavior.

In markets with a large variety of offers and little

possibility of differentiation, providers can gain

a decisive competitive advantage by user

oriented interfaces. A precondition of this is that

relevant information can be obtained for entrepreneurial decisions to this regard.

  3  

Commissioned by the Initiative for a

Competitive Online Marketplace ICOMP, the

Institute of Communication and Media

Research (IKM) at the German Sports

University Cologne (DSHS) developed and realized a suitable procedure. In the following

• the main findings are briefly presented

• the methods are discussed

• individual results are briefly explained

A pilot study could be conducted within the time

and budget constraints, which already presents clear results.

  4  

2. Short report

(1) Google placed "Sponsored" Google-own page elements grab a pre-dominant amount of the total visual attention.

(2) The "alternative search sites" even with small logos do not evoke enough visual attention to stimulate the users to click on them.

(3) Map services can not compete against Google because the placed Google Map pane and Google Images thumbnails get visual attention of users more, earlier and longer than all other page elements.

(4) Google Ads presented as lateral skyscraper-shaped link lists are vastly irrelevant for visual attention as well as for mouse clicking behavior.

(5) The visual attention for organic links on the SERPs is negligible compared to those Google elements placed above them, enhanced with pictures. With browser windows not opened wide, organic links may go unseen.

  5  

3. Setup of the study

The study presented here represents a user

oriented study using a special kind of observation.

By interviewing, general usage habits regarding

the users' preferred web browsers, a specific

search engine or a specific online service can

be learned. Data on questionnaires cannot

represent the behavior retrospectively because

this is affected, for example, by memory effects

(Felix, Hink & Minor, 2001; Möller & Schierl,

2012). Therefore an eye tracker was used

additionally. This was a high resolution device

of the company Tobii, based on the cornea

reflex method (Duchowski, 2007). The use of

this elaborate method is necessary in order to

catch spontaneous user behavior that is not

adulterated by willful processes (Byrne et al.,

1999;  Graham & Jefferey, 2011;  Ferreira et al.,

  6  

2011). In respect to methodology Just and

Carpenter (1980) assumed that the users'

visual attention is focused on the object that is

also the object of the cognitive processing (eye-mind hypothesis). It is further assumed that

the time of fixation corresponds to the time of

cognitive processing (immediacy assumption).

35 persons were recruited as test subjects, 20

men and 15 women. The specification was to

measure eye movements on search engine

result pages (SERPs) given as examples in the

October 21, 2013 commitment proposal by

Google, Annex V, of the Case COMP/C-

3/39.740 – Foundem and others, and an

additional SERP depicting Google Maps. For

this, the following three SERPs were used as target pictures.

  7  

Fig.1: Target picture "iPod shopping" (from Annex V, P. 35)

  8  

Fig.2: Target picture "map London" (screenshot from actual Google web search engine)

  9  

Fig.3: Target picture "flight search" (from Annex V, p.39)

In order to have the subjects behave as they

would in normal, everyday Internet use, the

purpose of the test was not revealed to them

and to preserve their attention for the stimuli

  10  

used during the entire measuring cycle, a special test setting was created.

The subjects sat about 2 meters in front of a 46-

inch plasma screen monitor that showed

various images. The eye tracker stood about 70

centimeters (= 2,297 ft) in front of the subjects

and therefore was outside the regular sight field

of the participants. All instructions, stimuli and

questions were only shown on the screen to

avoid any distortions caused by the

experimenter. In order to get the subjects used

to the situation and gain their attention they

were informed that they were participating in a

perception experiment and were presented

several images and questions unrelated to the

SERPs to be examined; They were presented

with a Stroop test of word recognition and

thereafter with a geometrical-optical illusion

image. Then the subjects were asked to look at

an advertisement for eight seconds. In the

advertisement, a spokesperson with a visible

  11  

physical handicap was promoting a fictional

automotive brand. Besides the fictional brand,

we questioned the subjects about any

conspicuous issues concerning the

spokesperson. Then the subjects got written

instructions to click on a link they desired in an

on-line advertisement. Then they were asked

about any sponsors visible on this site. Next the

subjects were shown a picture of a table tennis

athlete in a tuxedo and asked to assign a name

to him from a list of choices. Then the

participants were shown a picture of the same

athlete performing his sport and were asked for

a name choice. After this subjects were asked if

they are iPod users, if they usually buy those

electronic devices in specialised shops,

discount shops, or online shops. After this

subjects were instructed to assume intention to

buy an iPod nano and select one of six search

queries by mouse click. The instruction

continued that after their choice a Google

  12  

search result page will be visible and that

subjects should click on it to proceed (cf. fig. 1,

taken from Annex V, p. 35) buying an iPod

nano. Next instruction was to memorise as

many details of an upcoming iPod ad. The ad

was visible for six seconds. Then subjects had

to answer how many apps were visible on the

display of iPod that was presented in the ad.

After that subjects were asked to use a SERP

to choose a map to find Apple Stores in London

(cf. fig.2, screenshot from actual Google web

search engine). Then they were asked if they

ever booked a flight on the web. The

information was given that web booking is very

usual in the US. The next SERP showed

different flight offers (cf. fig.2, taken from Annex

V, p. 39). Subjects should choose one of the

offerings, again by clicking as usual. After this

subjects were thanked for their participation and dismissed.

  13  

The eye movements and fixations were

recorded at a rate of 120 Hertz during the entire

measuring section. Video and audio of all

subjects was recorded with a webcam in

parallel and integrated with the stimuli

presented, the mouse clicks and the tracking data into an overall view.

For each target picture the following parameters were computed from the raw data:

1) Fixation count

2) Absolute duration of the fixations

3) Time to first fixation.

A fixation occurs when an eye movement rests

for at least 200 milliseconds on an area of 50 pixels.

The results arrived at are described in detail in the following chapter.

  14  

4. Detailed results of the study.

As the first analytical step, the left mouse clicks

of the subjects were marked in the three target

pictures as symbolized ( ) mouse-left-click.

At the same time these data were to be

complemented by the data of the eye

movement. Therefore corresponding heat maps

were generated. In these heat maps, a color

code indicates the diverse intensities of the

visual attention triggered by the SERPs. Analog

to a traffic light, the color red represents the

maximal viewing time and the color green the

minimal duration. In all areas without color

assignment there was no fixation. For optimal

comparability, each heat map is shown on a

single page followed by a table containing the

corresponding data. Results will be discussed below.

  15  

Fig.4: Mouse clicks and total visual attention on target picture "iPod shopping"

  16  

Table 1: Overview of the recorded parameters on target picture "iPod shopping"

Area of

interest (AOI)

Fixation

duration (FD)

Time to

First

Fixation (TFF)

Fixation

Counts (FCʼ)

Mouse

Clicks (MC)

ad on top of SERP

0.54 1.08 192 0

Googles

own

product

search results

1.10 0.27 715 18

“alternative sites”

0.67 3.11 311 1

organic blue links

1.60 0.35 459 13

  17  

ig.5: Mouse clicks and total visual attention on the target picture "map London"

  18  

Table 2: Overview of the recorded parameters on target picture "map London"

AOI FD TFF FC MC

Google map 0.77 0.46 286 15

“Google images” 0.98 0.62 348 12

organic blue links 1.39 2.87 322 6

  19  

Fig.6: Mouse clicks and total visual attention on the target picture "flight search"  

  20  

Table 3: Overview of the recorded parameters on target picture "flight search"

AOI FD TFF FC MC

ad on top of SERP 0.85 1.50 143 2

Googles own product search results

2.03 0.10 556 15

“alternative sites” 0.91 5.54 110 4

organic blue links 1.16 3.35 196 11

Skyscraper-shaped ads 1.66 3.93 174 3

  21  

The SERP for the search term "iPod" reveals

that thumbnail product pictures guide the

spontaneous visual attention of users to the

"Google Shopping Results". For this area the

time to first fixation is least: 0.27 seconds. This

concurs with the amount of mouse clicks: 56%

of the participants clicked into this area with

Google's own product search. The average

amount of visible attention (the total fixation

duration) is 1.1 seconds. The maximum total

duration of fixations on the organic links is 1.6

seconds. 41% clicked one of the organic blue

links, mostly the first (Apple.com). One has to

keep in mind that the organic links were fixated

for the first time only after 3.5 seconds in

average! This is a long period, even for highly

involved users. To adhere to Google's example,

and to save time, the original mockup

screenshot from their commitments paper was

used, which contained Apple.com as first

ranked organic blue link. Further testing on

  22  

SERP examples without the very well known

brand Apple leading the organic blue links,

might show less attention and clicks to this position and even more in the product images.

The related ad at the top of the SERP had no

clicks and the "alternative search sites" were

scanned visually even less and only clicked

once. Both show a very small time in fixation

duration of round about half a second. One may

conclude that these areas are evoking no interest from recipients.

On the target SERP "map London" 46% of the

participants clicked on the Google map

whereas 36% clicked on the area of "Google

Images" which refers to another Google

service. Also the total duration of fixations for

both areas is comparable (0.77 seconds for the

Google map, 0.98 seconds for the "Google

images"). But the Google map gets visual

attention of the user faster (0.49 seconds)

  23  

compared to the "Google Image" (0.62

seconds). Number one organic link

mylondonmap.com is a site using the

syndicated Google Maps API and received two

clicks. The famous, official Transport for

London maps received no click. Further down

the SERP organic links again referring to Google Maps received four clicks together.

The "flight search" SERP in fig. 3 is the

visually most complex stimulus in the study

design. Similar to the SERP "iPod shopping"

only few mouse clicks are on the ad on the top

of the SERP (two clicks = 6%). Most clicks are

on the Google Flight Search "sponsored links"

area (43%), followed by the "organic links" area

(31%). The "alternative search sites" had little

visual attention with correspondingly only four

clicks (11%) and the ads on the right side just

three clicks (9%). Next to the amount of mouse

clicks, the total visual attention (2.03 seconds)

as well as the very small time to first fixation

  24  

(0.10 seconds) indicates that the Google Flight

Search area is the most attractive one of this

SERP. It is remarkable that even the organic

links at the bottom of the Page get more visual

attention (1.16 seconds) and faster (3.35

seconds) than the "alternative search sites"

(duration 0.91 seconds; time to first fixation = 5.54 seconds).

Finally all target pictures suggest that the

implemented "Sponsored (i)" labels do not

reach the beholder’s visual attention. This

finding is supported by the fact that those

labels were not clicked.

  25  

5. References

Byrne, M.D., Anderson, J.R., Douglass, S. and Matessa, M. (1999). Eye tracking the visual search of click-down menus. In Human Factors in Computing Systems: CHIʼ99 Conference Proceedings, ACM Press, New York, pp. 402-409.

Duchowski, A.(2007). Eye Tracking Methodology. Theory and Practice. Springer, London.

Felix, R., Hink, W. and Minor, M. (2001). Hereʼs looking at you kid: Eye-tracking applications in consumer information processing research. In Proceedings of the Annual Meeting of the Society for Marketing Advances, pp. 157-161.

Ferreira, P., Rita, P., Morais, P., Oliveira, J., Gamito, P., Santos, N., Soares, F. and Sottomayor, C. (2011). Grabbing attention while reading website pages: the influence of verbal emotional cues in advertising. Journal of Eyetracking, Visual Cognition and Emotion, 1, 1, pp. 64-68.

Graham, D.J. and Jefferey, R.W. (2011). Location, Location, Location: Eye-Tracking Evidence that Consumers Preferentially View Prominently Positioned Nutrition Information. Journal of the American Diet Association. 111, 11, 1704-1711. doi:

  26  

10.1016/j.jada.2011.08.005 Just, M.A. and Carpenter, P.A. (1980). A

theory of reading: from eye fixation to comprehension. Psychological Review, 87, 4, pp. 329–354.

Möller, C. and Schierl, T. (2012). Attention and selection behavior on "universal search" result pages. Journal of Eyetracking, Visual Cognition and Emotion, 2, 1, pp. 1-10.

  27  

Institute of Communication and Media Research German Sport University Cologne Am Sportpark Müngersdorf 6 50933 Cologne Germany Contact: Dr. Carsten Möller c.moeller@dshs-koeln.de +49(0)221 4982 6243 www.ikm-dshs.de