Design and Usability Center
175 Forest Street, Waltham, MA 02452 Tel: 1-800-523-2354
Beyond the Usability Lab: Exploring Large-Scale User Experience Research
Bill AlbertDirector, Design and Usability Center
Lena DmitrievaSenior Consultant, Design and Usability Center
2
Design and Usability Center
Acknowledgments
• Thanks to Tom Tullis and Donna Tedesco, Fidelity
Investments
• Based on upcoming book: “Beyond the Usability Lab:
Conducting Large-Scale User Experience Studies” (published by Morgan Kauffman/Elsevier in January, 2010)
3
Design and Usability Center
• Основан в 1999 году
• Оказывает консалтинговые услуги для корпоративных клиентов
• Работает с Master’s program - Human Factors in Information Design
• Консультанты
Bill Albert, PhD – Director Bill Gribbons, PhD - Sr. Consultant
Rich Buttiglieri - Usability Consultant Lena Dmitrieva - Usability Consultant
Kris Engdahl – Usability Consultant Chris Hass – Usability Consultant
9 Research Associates
О центре
4
Design and Usability Center
Наши клиенты
5
Design and Usability Center
Наши технические возможности
• Две юзабилити лаборатории
• Прямая трансляция юзабилити сессий
• Eye tracking
• Возможность удаленного тестирования
• Оборудование, предназначенное для
тестирования мобильных и медицинских
устройств
6
Design and Usability Center
Мы предлагаем
Изучение потребительских нужд
• Анализ пользователей - Персоны
• Интернет опросы
• Исследования вне лаборатории
• Фокус-группы
Тестирование, оценки, дизайн
• Количественные и качественные исследования
• Исследования по-русски
• Экспертная оценка интерфейсов
• Юзабилити-тестирование: в лаборатории, удаленное, без участия ведущего
• Информационная архитектура
• Доступность информационно-коммуникационных технологий
User Еxperience Обучение
• Корпоративные треннинги
• Интенсивные треннинги
7
Design and Usability Center
1. Context of UX research
2. What is large-scale UX research?
3. Strengths, limitations and myths
4. Complements with other UX research methods
5. How to conduct a study (planning, design, pilot/launch,
and analysis)
6. Keys to success
Agenda
Design and Usability Center
Context of UX Research
Beyond the Usability Lab:
9
Design and Usability Center
The Goal of UX Research
• Making smart and quick decisions
• Informing design and strategy
• Sparking innovation
• Methods don’t matter
10
Design and Usability Center
I am a big fan of traditional usability
11
Design and Usability Center
Missing Piece in our Toolkit
12
Design and Usability Center
Simplified Toolkit
13
Design and Usability Center
Big Picture
Design and Usability Center
What is Large-Scale
UX Research?
Beyond the Usability Lab:
15
Design and Usability Center
What do you call this?
Unmoderated Usability
Online UsabilitySelf-Guided Usability
Remote UsabilityAutomated Usability
Asynchronous Usability
Large-Scale Usability
16
Design and Usability Center
5 basic ingredients
1. Interactive system (usually a website)
2. A lot of participants (n>50 per segment)
3. An online tool to moderate study and collect data
4. Tasks (not just attitudes/opinions)
5. Capturing data about their experience
17
Design and Usability Center
Welcome Page
18
Design and Usability Center
Screener Questions
19
Design and Usability Center
Task Instruction
20
Design and Usability Center
Dialog
21
Design and Usability Center
Post Task Questions
22
Design and Usability Center
Follow-up Questions
23
Design and Usability Center
Open-ended Questions
24
Design and Usability Center
Types of Studies
• Comprehensive evaluation
• UX benchmark
• Competitive evaluation
• Live site vs. prototype comparison
• Feature/function test
• Discovery
25
Design and Usability Center
Types of Data
• Performance
• Success, time, efficiency
• Self-reported
• Task ratings, SUS, expectations, verbatims, etc
• Clickstream
• Pages, times, clicks, data entry, abandonment, etc
26
Design and Usability Center
Outcomes
• What are the usability issues, and how big?
• Which design is better, and by how much?
• How do different customer segments differ?
• What is the user experience like?
Design and Usability Center
Strengths, Limitations & Myths
Beyond the Usability Lab:
28
Design and Usability Center
Strengths
1. Comparing products
2. Measuring user experience
3. Finding the right participants
4. Focusing on design improvements
5. Insight into the user’s real experience
29
Design and Usability Center
Limitations
1. Not well suited to rapid, iterative design
2. Need a deep understanding of issues
3. Studies that require long sessions
4. Lose control over prototypes
5. Internet access
30
Design and Usability Center
Myths
1. Only test with websites
2. It is very expensive
3. Only gather quantitative data
4. A lot of noise in the data
5. Same as any market research
study
Design and Usability Center
Compliments to Other
UX Methods
Beyond the Usability Lab:
32
Design and Usability Center
• Use lab test to identify issues, iterate, and the run
online study to validate
• After an online study, use lab to do a deep dive on
issues
Traditional Lab Testing
33
Design and Usability Center
• Use focus groups to generate design concepts, then
use online study to validate
Focus Groups
34
Design and Usability Center
• Conduct an expert review to identify the most
significant issues, then test online
Expert Reviews
35
Design and Usability Center
Web Traffic Analysis
• Web traffic analysis can be used to identify areas of
concern, followed by an online test as a deep dive into
the problems
• Web traffic can
also follow an
online test to
validate design
changes
Design and Usability Center
Planning a Large-Scale
UX Study
Beyond the Usability Lab:
37
Design and Usability Center
Setting Goals
• Compare prototype designs
• Benchmark
• Identify issues and prioritize
• Validate new design against
current design
• Compare specific design
features/functions
38
Design and Usability Center
Budget
• Recruiting costs• $2 - $20 per complete (panel)
• No cost if using internal list
• Incentives• $5 - $10 per complete
• Drawing to win
• No incentive
• Technology• Annual license or per study for commercial service
• Discount approach
39
Design and Usability Center
Timelines
ID Task Name Start Finish Duration
Jan 3 2010 Jan 10 2010 Jan 17 2010
4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
1 3d01/06/201001/04/2010Plan Goals and Structure of Study
2 5d01/11/201001/05/2010Build prototypes (2 Wizard Flows, with
slight difference)
3 3d01/11/201001/07/2010Build tasks and questions into existing
online study tool
4 3d01/13/201001/11/2010Pilot and Test tool (2 iterations)
2d01/15/201001/14/2010Launch Study5
2d01/19/201001/18/2010Analysis
7
6
1d01/20/201001/20/2010Present Results
21 22
ID Task Name Start Finish Duration
Mon Jan 4
8 9 10 11 12 1 2 3
1 1h01/04/201001/04/2010Point online study tool to prototypes
and create tasks and questions
2 1h01/04/201001/04/2010Pilot with team members and iterate
3 2h01/04/201001/04/2010Launch study internally
4 2h01/04/201001/04/2010Analyze and report back results
40
Design and Usability Center
Commercial Services
41
Design and Usability Center
Discount Approach
Design and Usability Center
Designing a Study
Beyond the Usability Lab:
43
Design and Usability Center
Introducing the Study
• Purpose
• Sponsor/contact
information
• Time estimate
• Incentive
• Technical requirements
• Legal stuff
• Instructions
44
Design and Usability Center
Screeners
• Screener questions focus study on target users
• Most common screens based on product experience
and demographics
• Screening questions need to be grouped together so
they are hard to guess
• Careful about
fraudulent
participants
45
Design and Usability Center
Starter Questions
• Customer
segmentation
• Product experience
• Expectations
• Avoid sensitive
questions
• Conditional logic
46
Design and Usability Center
Tasks
• Open-ended (discovery)
• “Learn about how carbon offsets work”
• Close-ended (measuring success)
• “What is the ISBN number of the book “Measuring the User
Experience”
• Self-generated (discovery)
• “What would you like learn on this website?...”
• Self-selected
• “Choose 5 tasks that are most important to you”
47
Design and Usability Center
Mental Cheaters
• About 10% of participants go through the study as fast
as possible to get the incentive
• Speed traps
• Consistency checks
48
Design and Usability Center
Post-Task Questions
• Ease of use
• Confidence
• Open-ended
• Proxy measures
49
Design and Usability Center
Post Session Questions
• Overall
assessment
(SUS)
• Demographics
• Open-ended
(focus on
specifics)
Design and Usability Center
Piloting and Launching
Beyond the Usability Lab:
51
Design and Usability Center
Piloting and Launching
Technical checks
Usability checks
Full pilot with data checks
Timing the launch
Singular launch
Phased launches
Monitoring results
Design and Usability Center
Data Analysis
Beyond the Usability Lab:
53
Design and Usability Center
Data Cleanup
Removing participants
Incomplete data
Fraudulent participants
Mental cheaters
Removing or modifying data for individual tasks
Outliers
Contradictory responses
Removing a task for all participants
Modifying task success
54
Design and Usability Center
Data Analysis
• Task performance
• Self-reported
• Clickstream
• Combined metrics
55
Design and Usability Center
Task Performance
• Task success
• Task time
• Efficiency (tasks per minute)
• Learnability (efficiency over time)
Number of Correct Tasks per Minute
(Error bars represent the 90% confidence interval)
1.80
1.85
1.90
1.95
2.00
2.05
2.10
2.15
2.20
Static TOC Expand-Collapse TOC Auto-collapse TOC
56
Design and Usability Center
Self-Reported Data
• Rating Scales
• Likert Scale (disagree – agree)
• Semantic Differential (easy –
difficult)
• Open-ended questions
• Based on conditional logic Before and After Task Ease Ratings
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0
Before Rating
Aft
er
Rati
ng
Harder Easier
Easier
Harder
Harder Easier
Easier
Harder
T1
T2
T3 T4
T5
Fix It FastBig Opportunity
Promote It Don’t Touch It
Task Ease Ratings (1-5, Higher=Easier)
(Error bars represent the 90% confidence interval)
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
Ferret Food Store Location Gift Card Shipping Pine Cat Litter
Site A
Site B
57
Design and Usability Center
Clickstream Analysis
• Percentage of participants who used specific paths
• Intended paths, acceptable paths, unacceptable paths
• Abandonment (rates and pages)
• Total page visits (compared to minimum)
• Lostness metric# Pages Beyond Minimum
0
2
4
6
8
10
12
Task 1 Task 2 Task 3 Task 4
58
Design and Usability Center
Combined Metrics
• Convert each
metric to a
percentage, then
average all
percentages
• Z-score
transformation
• Jeff Sauro’s SUM
method
Participant Condition
Mean Task
Ease Rating
Percent
Correct
Mean Task
Time (sec) Time %
Task
Rating %
Combined
Task %
P382 Static TOC 4.0 100% 20 50% 75% 75%
P383 Static TOC 4.4 100% 24 42% 85% 76%
P384 Auto-collapse TOC 4.4 100% 19 51% 85% 79%
P385 Expand/Collapse TOC 4.0 100% 47 21% 75% 65%
P386 Static TOC 3.8 90% 26 39% 70% 66%
P387 Auto-collapse TOC 4.2 100% 29 35% 80% 72%
P388 Auto-collapse TOC 4.1 90% 33 30% 78% 66%
P389 Expand/Collapse TOC 3.2 100% 19 53% 55% 69%
P390 Static TOC 4.9 100% 23 43% 98% 80%
P391 Expand/Collapse TOC 4.1 80% 50 20% 78% 59%
P392 Static TOC 4.3 90% 22 46% 83% 73%
P393 Static TOC 4.4 100% 17 60% 85% 82%
P394 Auto-collapse TOC 4.0 100% 41 24% 75% 66%
P395 Static TOC 3.2 100% 25 40% 55% 65%
P396 Auto-collapse TOC 4.6 100% 44 23% 90% 71%
P397 Auto-collapse TOC 4.1 89% 60 17% 78% 61%
P398 Auto-collapse TOC 2.3 60% 41 24% 33% 39%
P399 Expand/Collapse TOC 4.3 90% 24 41% 83% 71%
P400 Expand/Collapse TOC 4.8 90% 36 28% 95% 71%
P401 Expand/Collapse TOC 4.0 80% 35 29% 75% 61%
Overall Task % (Accuracy, Speed, Task Ease)
(Error bars represent the 90% confidence interval)
67%
68%
69%
70%
71%
72%
Static TOC Expand/Collapse TOC Auto-collapse TOC
Design and Usability Center
Keys to Success
Beyond the Usability Lab:
60
Design and Usability Center
Keys to Success
1. Choosing the right tool
2. Thinking beyond the web
3. Test early
4. Compare alternatives
5. Consider the entire user experience
6. Use your entire toolkit
7. Explore your data
8. Sell your results
9. Trust your data (within limits)
10. Just dive in
61
Design and Usability Center
Resources
• www.measuringux.com
• www.remoteusability.com
• www.userzoom.com
• www.measuringusability.com
• http://keynote.com/product
s/customer_experience/
Design and Usability Center
175 Forest Street, Waltham, MA 02452 Tel: 1-800-523-2354
Спасибо Thank you
Bill AlbertDirector, Design and Usability Center
Lena DmitrievaSenior Consultant, Design and Usability Center