effects of age and think-aloud protocol on eye-tracking data and usability measures
TRANSCRIPT
Effects of Age and
Think-Aloud Protocol on
Eye-Tracking Data and
Usability Measures
Jennifer Romano Bergstrom @romanocog
Fors Marsh Group @forsmarshgroup
&
Erica Olmsted-Hawala
U.S. Census Bureau
EyeTrackUX | Las Vegas, NV | June 2012
Usability Testing
• Technique in user-centered design to
evaluate a product by testing it with actual
users of the product
• Value is getting direct feedback on how
real users work with the product
How Usability is Measured
• International Organization for
Standardization (ISO) 9242:11
The extent to which a product can be used by
specified users to achieve specified goals with
effectiveness, efficiency and satisfaction in a
specified context of use.
Usability Testing
• Metrics
– Task accuracy
– Task efficiency
– User satisfaction
• Usability professional (test administrator)
identifies problem areas of product based
on user performance & verbalized
feedback
5
Think-Aloud Protocol
• Primary tool used by usability professionals when conducting usability tests
• Test administrator — Listens to what participant says — Watches what participant does — Cannot directly observe what a participant
thinks
6
Concurrent Think Aloud (CTA)
Protocol
• Participant encouraged to “think aloud” while working on task – Running commentary of thought-action
process – Participant articulates as they work on task
Retrospective Think-Aloud (RTA)
Protocol
• Participant completes session in silence
– Watches a video replay of session
• Sometime with eye tracking
– Articulates while watching video replay
Research on CTA and RTA
• Usability studies have compared effects of
CTA and RTA on usability performance
– Accuracy
– Efficiency
– Satisfaction
– Verbal data quality
• Results vary by study
Pros and Cons of Think Aloud CTA
• Metrics might not be accurate
• Emotions
• Users describing the what
• Real sense of users’
experience with site
• Shorter sessions
• Real time responses
RTA
• Metrics “more pure”
• Reflection
• Users describe their own
interpretation or
understanding of why
• Users’ insight into what the
design issues are
• Longer sessions
• Memory is fallible
Cognitive Aging
Aging: The Facts
• The population is increasing.
The Facts
250,000
270,000
290,000
310,000
330,000
350,000
370,000
390,000
2000 2005 2009 2010 2015 2020 2025 2030
Po
pu
lati
on
of
US
in t
ho
usa
nd
s
The Facts
• The population is increasing.
• Older adults are the fastest growing part of
the population.
The Facts
0
20,000
40,000
60,000
80,000
100,000
120,000
2000 2005 2009 2010 2015 2020 2025 2030
Po
pu
lati
on
of
US
in t
ho
usa
nd
s
18 to 24 years
25 to 44 years
45 to 64 years
65 years and over
The Facts
Percent Change in US Population and in Older Adults: 2000-2029.
The Facts
• The population is increasing.
• Older adults are the fastest growing part of
the population.
• More people are using the Internet.
The Facts
0
10
20
30
40
50
60
70
80
90
1997 2000 2001 2003 2007 2009
Pe
rce
nt
of
ho
use
ho
lds
wit
h In
tern
et
The Facts
The Facts
• The population is increasing.
• Older adults are the fastest growing part of
the population.
• More people are using the Internet.
• Cognitive decline is inherent with age.
The Facts
• The population is increasing.
• Older adults are the fastest growing part of
the population.
• More people are using the Internet.
• Cognitive decline is inherent with age.
Cognition and Internet Examples
• Spatial skills
– Where things are located
• Short-term memory
– Remembering the purpose
• Working memory
– Remembering information entered earlier to respond to later questions
• Sustained attention
– Complex Web sites
The Facts
• The population is increasing.
• Older adults are the fastest growing part of
the population.
• More people are using the Internet.
• Cognitive decline is inherent with age.
• Web site designers often do not take
cognitive decline into account when
designing Web sites and surveys.
Cognitive Aging,
Think-Aloud Protocol
and Usability Performance
Challenges of Dual tasks
• Performance for older adults decreases
when they attempt to complete dual tasks
Research Questions
• Does the dual task of thinking aloud while
working on finding information impact
usability performance and eye fixation
patterns?
• What think-aloud protocol (CTA / RTA)
should practitioners use when conducting
usability studies with older adults?
Our Study
• 2 (CTA / RTA) x 3 (Age) between-subjects
study.
• Outcome Variables
– Accuracy
– Efficiency
– Satisfaction
– Eye-tracking data
– Verbal reports
Participants
• 95 participants
• Metro DC area
• Three age groups
– Young adults ages 18-28
– Middle-age adults ages 40-50
– Older adults ages 64-76
• One year of prior experience using Internet • Reimbursed $40.00
Participant Characteristics
Young adults Middle-age adults Older adults
CTA RTA CTA RTA CTA RTA
Gender 3M / 8F 6M / 15F 5M / 7F 9M / 10F 5M / 6F 9M / 12F
Mean Age 22 22 45 46 67 68
Mean Years of
education 15 15 15 14 16 16
Ease of learning
a new Web site* 1.36 1.47 1.83 1.42 2.36 2.38
Ease of
navigating the
Internet*
1.09 1.05 1.33 1.16 1.72 1.57
*Scale: 1 (Not difficult at all) – 5 (Extremely difficult).
Methods • 5 information-gathering tasks
– Typical tasks users come to site to accomplish
– Tasks organized by easy to hard
– Collected accuracy and efficiency measures
– Eye tracking on all tasks
• Satisfaction questionnaire and debriefing at end
• CTA group – CTA on all tasks
• RTA group – RTA on last task only
– Gaze video playback
• Test administrator and participant were in separate rooms
Website Tested
• US Census Bureau’s American FactFinder
– Primary website about population, housing,
and economy of the US
– Legacy version of site
AFF Website: http://factfinder.census.gov
Example Tasks
• First easy task: You want to learn more
about Maryland, and specifically about
how many people live there. How many
people live in Maryland?
• First hard task: You are doing a report on
schooling in the U.S. What percent of the
population in Florida, California and Texas
completed college in 2008?
Results
• Task accuracy
• Task efficiency
• User satisfaction
• Eye Tracking
• Verbal reports
Young and middle age adults
completed more tasks successfully
than older adults
0%
10%
20%
30%
40%
50%
60%
70%
80%
Young adults Middle-age adults Older adults
CTA
RTA
**
*
*p < 0.05; **p < 0.01
Middle-age adults in CTA had higher
accuracy in hard task than in RTA
0%
10%
20%
30%
40%
50%
60%
70%
80%
Middle-age adult--1st hard task
CTA
RTA
*
*p = 0.07
Young adults completed tasks
faster than older adults
0:00
0:28
0:57
1:26
1:55
2:24
2:52
3:21
3:50
4:19
Young adults Middle-age adults Older adults
CTA
RTA
**
**p = 0.05, *p = 0.10
Middle-age adults in RTA completed
hard task faster than in CTA
0:00
1:12
2:24
3:36
4:48
6:00
7:12
8:24
9:36
10:48
Middle age adults--1st hard task
CTA
RTA
*
*p = 0.10
No significant age-related
differences in user satisfaction
0
1
2
3
4
5
6
7
8
9
Young adults Middle-age adults Older adults
CTA
RTA
Speed / Accuracy Trade Off
• Cognitive demands influence the speed
and accuracy with which adults process
information
• Our findings for the middle-age adults are
consistent with earlier cognitive work.
– Higher accuracy for hard task with CTA
– Slower for hard task with CTA
Interaction of Age and Think Aloud
• Older adults’ performance decreases
when they perform dual tasks.
• Our findings for the middle age adults are
consistent with earlier cognitive work.
– Older adults: floor level
– Young adults: ceiling level
Eye Tracking
• Total number of fixations per AOIs
– Task 1
• Total number of fixations on main page
– Task 1
AOIs
Left navigation Population clock
Top banner Top navigation
Whole screen
Left navigation: No difference
between CTA and RTA
0
5
10
15
20
25
CTA RTA
Left navigation: Older adults had
fewer fixations than young adults
for CTA only *
*p < 0.05
0
1
2
3
4
5
6
7
8
9
10
Young adults Middle-age adults Older adults
CTA
RTA
Top banner: More fixations during
CTA
0
1
2
3
4
CTA RTA
CTA
RTA
p = 0.06
Top banner: No significant
differences, but pattern is the same
0
1
2
3
4
5
Young adults Middle-age adults Older adults
CTA
RTA
Top navigation: No difference
between CTA and RTA
0
1
2
CTA RTA
CTA
RTA
Top navigation: Older adults had
fewer fixations than young adults
for CTA only *
*p < 0.05
0
1
2
3
4
Young adults Middle-age adults Older adults
CTA
RTA
Population clock: No difference
between CTA and RTA
0
1
2
3
4
5
6
CTA RTA
CTA
RTA
Population clock: No significant
differences, but CTA pattern is similar
0
1
2
3
4
5
6
7
8
Young adults Middle-age adults Older adults
CTA
RTA
Whole screen: No difference
between CTA and RTA
0
10
20
30
40
50
60
CTA RTA
CTA
RTA
Whole screen: Middle-age adults
had fewer fixations than young
adults for CTA only
*
*p = 0.05
0
10
20
30
40
50
60
70
80
Young adults Middle-age adults Older adults
CTA
RTA
Seconds to complete Task 1: No
significant differences
0
20
40
60
80
100
120
140
160
Young adults Middle-age adults Older adults
CTA
RTA
Accuracy for Task 1: No significant
differences
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Young adults Middle-age adults Older adults
CTA
RTA
Language Analysis
• Present tense and past tense
verbalizations
• Also examined
– Positive emotion words (e.g., good, helpful)
– Words of assent (e.g., absolutely, cool)
– Negative emotion words (e.g., annoy, awful)
– Negations (e.g., aren’t, can’t)
– Affective processing (e.g., caring, insult)
Participants in CTA spoke more present tense;
Participants in RTA spoke more past tense
0%
2%
4%
6%
8%
10%
12%
14%
Past tense Present tense
CTA
RTA
Young adults in CTA spoke 5%
fewer utterances in present tense
0%
2%
4%
6%
8%
10%
12%
14%
16%
Present tense
Young CTA
Middle-age CTA
Older CTA
Young adults in RTA follow similar pattern:
fewer utterances of present tense
0%
2%
4%
6%
8%
10%
12%
Present tense
Young RTA
Middle-age RTA
Older RTA
Words of Insight and Cognition
• Test administrator observations during sessions – CTA more immediate what they were doing
– RTA reflections on actions
• Insight words – Complex
– Meaning
– Prove
• Cognitive processes – Ambiguous
– Hesitate
– Solution
Participants in RTA spoke more
words of insight and cognition
0%
5%
10%
15%
20%
25%
Insight words Cognitive processes
CTA
RTA
Summary of Usability Results I
• Accuracy – Young and middle-age adults
completed more tasks successfully (higher
accuracy) than older adults.
• Efficiency – Young adults completed tasks
faster than older adults.
• Satisfaction – Age did not affect
satisfaction.
Summary of Usability Results II
• Condition effect – Middle-age adults who
completed tasks in silence (RTA)
completed the most difficult task faster and
with higher accuracy than middle-age
adults who thought aloud (CTA) while
completing tasks.
Summary of Eye-Tracking Results
• When age was not a factor, there appeared
to be no differences in fixations between CTA and RTA
• When age was taken into account – Older had fewer fixations than young in CTA only
• Left navigation
• Top navigation
• Similar pattern for other AOIs
– Middle-age had fewer fixations than young in CTA only
• Whole screen
– Not related to accuracy or efficiency
Implications
• Recruit participants of varying age groups for usability
testing; conduct usability tests with older adults.
• Be aware of cognitive demands and effects that thinking
aloud has on aging adults.
• Consider pros and cons when creating a protocol and
selecting a think-aloud method.
– Use CTA for a richer verbal commentary, particularly with respect to
participants’ emotional reaction to the site.
– Use RTA when interested in verbal feedback related to participants’
insight on the issues they had with the application.
– Use RTA when pure eye-tracking data is necessary.
Future Analysis
• Fixation duration
• Sum across all tasks – Easy vs. hard tasks
• Performance in first 10 seconds
• Correlations – Education
– Experience
– Cognitive abilities
• Verbalizations – Phrases
– Number of words, phrases
• Click data
• Identified usability issues
Future Research
• Include wider ranges of age to test the
assertion that the age-related effect is
incremental and occurs slowly through a
lifetime
• Alter RTA protocol
• Replicate with different web sites
• Are there specific gaze patterns by age
and think-aloud protocol?
Thank You!
Jennifer Romano Bergstrom
Fors Marsh Group
571.858.3795
@romanocog
Proceedings paper from STC re verbalizations is available.
Eye-tracking paper is forthcoming.