better ux surveys part 1
DESCRIPTION
The first part of a workshop on user experience surveys. Topics: (1) how to improve the questions in surveys and (2) how to assess UX using a survey. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.TRANSCRIPT
![Page 1: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/1.jpg)
Better UX
surveysHow to ask better questions, andhow to assess UX using surveys
A workshop led by Caroline Jarrett
![Page 2: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/2.jpg)
Introductions (I’m Caroline Jarrett)
• Your name and role
• A random thing about yourself
2
![Page 3: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/3.jpg)
3
Agenda Introductions
What is a survey?
How to ask better questions
Break
How to assess user experience using surveys
Wrap up
![Page 4: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/4.jpg)
Is this a survey or something else?
• Work in pairs
• Review this survey
• Decide whether it is a survey or something else
4
![Page 5: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/5.jpg)
5
Survey = Questionnaire + Process
![Page 6: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/6.jpg)
A basic survey process
6
Goals• Establish
your goals for the survey
Questions you need
answers to
Users• Interview
users about the topics in your survey
Questions users can
answer
Build• Final version
of questions• Build the
questionnaire
Questionnaire
Deploy • Run the
survey from approach to follow-up
Data
Analyse• Extract
useful ideas• Share with
others
InsightSome questions
“Let’s do a survey”
![Page 7: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/7.jpg)
• Final version of questions
• Build the questionnaire
Today we’re starting withquestions users can answer
Goals Users Build Deploy Analyse
7
• Establish your goals for the survey
• Interview users about the topics in your survey
• Run the survey from approach to follow-up
• Extract useful ideas
• Share with others
Questions you need
answers to
Questions users can
answerQuestionnaire Data Insight
![Page 8: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/8.jpg)
• Final version of questions
• Build the questionnaire
Then we’re going on to goals,specifically for satisfaction surveys
Goals Users Build Deploy Analyse
8
• Establish your goals for the survey
• Interview users about the topics in your survey
• Run the survey from approach to follow-up
• Extract useful ideas
• Share with others
Questions you need
answers to
Questions users can
answerQuestionnaire Data Insight
![Page 9: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/9.jpg)
We won’t have time for much on these topics
Goals Users Build Deploy Analyse
9
• Establish your goals for the survey
• Interview users about the topics in your survey
• Run the survey from approach to follow-up
• Extract useful ideas
• Share with others
Questions you need
answers to
Questions users can
answerQuestionnaire Data Insight
• Final version of questions
• Build the questionnaire
![Page 10: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/10.jpg)
Forms and questionnaires differ in how the answers are used
Form:
answers used individually
Questionnaire:
answers aggregated
10
![Page 11: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/11.jpg)
11
Better questions
Survey =
Questionnaire
+ Process
![Page 12: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/12.jpg)
There are four steps to answer a question
Adapted from Tourangeau, R., Rips, L. J. and Rasinski, K. A. (2000)“The psychology of survey response”
Read and understand the
question
Find an answer
Judge whether the answer fits
Place the answer
![Page 13: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/13.jpg)
13
ReadFindJudgePlace
![Page 14: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/14.jpg)
Review these two questions from a survey
• Think about the four steps of answering a question:– Read and understand the question– Find the answer– Judge whether the answer fits– Place the answer
• Any problems with either question?
• If so, which step(s) are problematic?
14
![Page 15: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/15.jpg)
15
Any ideas?
![Page 16: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/16.jpg)
16
The approximate curve of forgetting
Long ago
Hig
h
Recent
Low
Time since event
Qualityof data
Major life event
Occasional, salientUnremarkable,repetitive
![Page 17: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/17.jpg)
17
This survey asks some ‘how often’ questions
• Where do these questions fit on the ‘curve of forgetting’?– Major life event– Occasional, salient– Unremarkable, repetitive
• Do we think the answers are useful?
![Page 18: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/18.jpg)
18
“Place the answer” is mostly about using the right widget to collect the answer
Use ForRadio buttons A single known answerCheck boxes Multiple known answersText boxes Unknown answers
Allen Miller, S. J. and Jarrett, C. (2001) “Should I use a drop-down?”http://www.formsthatwork.com/files/Articles/dropdown.pdf
“knownx answers”
you’ve interviewed users until you’ve
found all their answers
![Page 19: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/19.jpg)
19
Do these questions have the right widgets?
• Review the type of widgets used for these questions
• Are they the right ones?
• How confident are we that the answer options reflect
the answers that users are likely to have?
![Page 20: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/20.jpg)
Grids are often full of problems at all four steps
20
![Page 21: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/21.jpg)
21
Grids are a major cause of survey drop-out
35%
20%
20%
15%
5%5%
Total incompletes across the 'main' section of the ques-tionnaire
(after the introduction stage)
Subject MatterMedia DownloadsSurvey LengthLarge GridsOpen QuestionsOther
Source: Database of 3 million+ web surveys conducted by Lightspeed Research/Kantar
Quoted in Coombe, R., Jarrett, C. and Johnson, A. (2010) “Usability testing of market research surveys” ESRA Lausanne
![Page 22: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/22.jpg)
22
But it’s the topic that matters most
35%
20%
20%
15%
5%5%
Total incompletes across the 'main' section of the ques-tionnaire
(after the introduction stage)
Subject MatterMedia DownloadsSurvey LengthLarge GridsOpen QuestionsOther
Source: Database of 3 million+ web surveys conducted by Lightspeed Research/Kantar
Quoted in Coombe, R., Jarrett, C. and Johnson, A. (2010) “Usability testing of market research surveys” ESRA Lausanne
![Page 23: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/23.jpg)
Response relies on effort, reward, and trust
23
Trust
Perceivedeffort
Perceivedreward
Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method”
![Page 24: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/24.jpg)
An interesting subject helps in all the areas
24
Shared interestsinspire trust
Interestingtopicstakeless effort
Interesting subject =
intrinsicallyrewarding
![Page 25: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/25.jpg)
Compare survey invitations
• Which survey invitation is the best? Why?
• (Optional harder exercise: write a better invitation)
25
![Page 26: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/26.jpg)
26
Your answers to this survey are important for our work
But what’s in it for me? And I’m really ready for a coffee.
![Page 27: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/27.jpg)
27
Agenda Introductions
What is a survey?
How to ask better questions
Break
How to assess user experience using surveys
Wrap up
![Page 28: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/28.jpg)
Survey = questionnaire + process
28© Caroline Jarrett and Effortmark Ltd
![Page 29: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/29.jpg)
We’ve got a lot of different goals to consider
What the organisation
wants to achieve
What the user wants to do
Our aims in doing a survey
29
![Page 30: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/30.jpg)
• Final version of questions
• Build the questionnaire
The survey process goals are about what we want to achieve with the survey
Goals Users Build Deploy Analyse
30
• Establish your goals for the survey
• Interview users about the topics in your survey
• Run the survey from approach to follow-up
• Extract useful ideas
• Share with others
Questions you need
answers to
Questions users can
answerQuestionnaire Data Insight
![Page 31: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/31.jpg)
Usability is about people doing things easily
The extent to which a product
can be used by specified users
to achieve specified goals with
effectiveness, efficiency and satisfaction
in a specified context of use
(ISO 9241:11 1998)
31
This assumes that we agree on the goals
![Page 32: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/32.jpg)
ISO 9241-210 definition of user experience
2.15 user experience
person's perceptions and responses resulting from the use
and/or anticipated use of a product, system or serviceNOTE 1 User experience includes all the users' emotions, beliefs, preferences, perceptions, physical
and psychological responses, behaviours and accomplishments that occur before, during and after
use.
NOTE 2 User experience is a consequence of brand image, presentation, functionality, system
performance, interactive behaviour and assistive capabilities of the interactive system, the user's
internal and physical state resulting from prior experiences, attitudes, skills and personality, and the
context of use.
NOTE 3 Usability, when interpreted from the perspective of the users' personal goals, can include the
kind of perceptual and emotional aspects typically associated with user experience. Usability criteria
can be used to assess aspects of user experience.
32
![Page 33: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/33.jpg)
Or putting it more simply:user experience is the satisfaction bit of usability
33
The extent to which a product
can be used by specified users
to achieve specified goals with
effectiveness, efficiency and satisfaction
in a specified context of use
(ISO 9241:11 1998)
As the user, I get to choose the goals
![Page 34: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/34.jpg)
34
Satisfaction is a complex matter
Compared experience to what? Resulting thoughts
(nothing) Indifference
Expectations Better / worse / different
Needs Met / not met / mixture
Excellence (the ideal product) Good / poor quality (or ‘good enough’)
Fairness Treated equitably / inequitably
Events that might have been Vindication / regret
Adapted from Oliver, R. L. (1996) “Satisfaction: A Behavioral Perspective on the Consumer”
![Page 35: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/35.jpg)
Example: bronze medal winners tend to be happier than silver medal winners
35
Nathan Twaddle, Olympic Bronze Medal Winner in Beijing
Photo credit: peter.cipollone, Flickr
Matsumoto D, & Willingham B (2006). The thrill of victory and the agony of defeat: spontaneous expressions of medal winners of the 2004 Athens Olympic Games.
![Page 36: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/36.jpg)
Memorable experiences are also complex
• Think about the experience of
helping to run a stand at an exhibition– What would you expect to happen?– What would you need to happen?– What would the ideal experience be?– How would you expect to be treated
compared to other people at the event?– If you didn’t do this, what else might have happened?
36
![Page 37: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/37.jpg)
Memorable experiences are also complex
• We have some candidate areas to explore
• Let’s compare with some actual questions
37
![Page 38: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/38.jpg)
The challenge of UX and surveys: which bit to measure?
The extent to which a product
can be used by specified users
to achieve specified goals with
effectiveness, efficiency and satisfaction
in a specified context of use
(ISO 9241:11 1998)
???
![Page 39: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/39.jpg)
39
Find out about users’ goals
Tip
![Page 40: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/40.jpg)
40
Ask about recent vivid
experienceTip
Image credit: Fraser Smith
![Page 41: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/41.jpg)
41
Interview first
Tip
![Page 42: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/42.jpg)
Survey = Questionnaire
+ Process
Remember my definition?
![Page 43: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/43.jpg)
43
Bonus
TipSurvey = Questionnaire
+ Process
Successful
That involveslots of testing
![Page 45: Better UX Surveys part 1](https://reader036.vdocuments.mx/reader036/viewer/2022062511/54c65cdf4a795934598b45c3/html5/thumbnails/45.jpg)
More resources on http://www.slideshare.net/cjforms
45