10 tips for a better survey at stc2011
DESCRIPTION
Ten tips for surveys: on questions, process, and testing your survey. Books mentioned are listed here: http://rosenfeldmedia.com/uxzeitgeist/lists/cjforms/10-tips-for-a-better-survey-stc2011 This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.TRANSCRIPT
![Page 1: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/1.jpg)
10 tips for a better survey
Caroline Jarrett
FORMS
SURVEYS
STC Summit 2011
![Page 3: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/3.jpg)
3
Survey = Questionnaire + Process
Successful
That involveslots of testing
![Page 4: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/4.jpg)
Outline Tips about questions
Tips about process
Tips about testing
4
![Page 5: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/5.jpg)
5
120 characters
48 characters
![Page 6: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/6.jpg)
6
![Page 7: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/7.jpg)
Tech writing challenge: rewrite in 140 characters
• XXX is conducting research to better understand and
meet the needs of the users of their site. Would you like
to participate?
• Responses are anonymous, view our privacy policy
7
![Page 8: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/8.jpg)
Tip 1 One introduction is
enough
(and think 140 characters)
8
![Page 9: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/9.jpg)
Try answering this question, then tell me how you worked out your answer
9
![Page 10: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/10.jpg)
10
An example of scale effects:How many hours a day do you study?
Low scale• ½ hour or less
• From ½ to 1 hour
• From 1 to 1 ½ hours
• From 2 to 2 ½ hours
• More than 2 ½ hours
High scale
• 2 ½ hours or less
• From 2 ½ to 3 hours
• From 3 ½ to 4 hours
• From 4 to 4 ½ hours
• More than 4 ½ hours
Schuman, H. and Presser, S. (1996) “Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context”
70%
30%29%
71%
![Page 11: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/11.jpg)
11
An example of scale effects:How many hours a day do you study?
Low scale• Very lazy
• A bit lazy
• Middling
• A bit diligent
• Very diligent
High scale
• Very lazy
• A bit lazy
• Middling
• A bit diligent
• Very diligent
Schuman, H. and Presser, S. (1996) “Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context”
70%
30%29%
71%
![Page 12: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/12.jpg)
Tip 2 Do not ask questions
about unremarkable
repetitive behaviours
12
![Page 13: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/13.jpg)
What’s missing from this question?
13
![Page 14: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/14.jpg)
Tip 3 Always include ‘other’ in
category questions– Don’t know– Prefer not to answer– All of the above
14
![Page 15: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/15.jpg)
Watch the order of these two questions
15
![Page 16: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/16.jpg)
16
![Page 17: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/17.jpg)
Tip 4 Think of your questions
as a conversation with
your respondent
17
Jarrett, C. and Gaffney, G (2008) “Forms that work: Designing web forms for usability”
Janice (Ginny) Redish (2007) “Letting go of the words:Writing web content that works”
![Page 18: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/18.jpg)
18
![Page 19: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/19.jpg)
Outline Tips about questions
Tips about process
Tips about testing
19
![Page 20: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/20.jpg)
Response relies on effort, reward, and trust
20
Trust
Perceivedeffort
Perceivedreward
Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method”
![Page 21: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/21.jpg)
How are response, effort and trust working here?
21
![Page 22: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/22.jpg)
In mail surveys, a token incentive works better than no incentive or a prize draw
22
![Page 23: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/23.jpg)
Tip 5 Try to find an incentive
that gives an immediate
reward
23
![Page 24: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/24.jpg)
24
![Page 25: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/25.jpg)
25
![Page 26: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/26.jpg)
26
![Page 27: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/27.jpg)
27
![Page 28: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/28.jpg)
Tip 6 One good incentive:
allowing the respondents
to express their views
28
![Page 29: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/29.jpg)
Write down your answers to this questionnaire
1. How many user surveys have you run?
NONE 1 to 5 6 to 10 more than 10
2. What is your top tip for a better user survey, based on
experience of writing or answering?
__________________________________
__________________________________
29
![Page 30: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/30.jpg)
Now try it as an interview.In pairs: use this is a script for an interview
1. How many user surveys have you run?
NONE 1 to 5 6 to 10 more than 10
2. What is your top tip for a better user survey, based on
experience of writing or answering?
__________________________________
__________________________________
30
![Page 31: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/31.jpg)
Now tell each other a story about your experience relevant to these questions
1. How many user surveys have you run?
NONE 1 to 5 6 to 10 more than 10
2. What is your top tip for a better user survey, based on
experience of writing or answering?
__________________________________
__________________________________
31
![Page 32: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/32.jpg)
Now let’s share data and stories
• What numbers have we collected?
• What open answers?
• What stories?
32
![Page 33: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/33.jpg)
Tip 7 Interview some users
to find out what
they want to tell you
33
![Page 34: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/34.jpg)
Did you answer the same questionthat I asked?
1. How many user surveys have you run?
34
![Page 35: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/35.jpg)
Does your household have access to running water?
35Picture credits: James Mooney, Caroline Jarrett
![Page 36: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/36.jpg)
“Shared reference”: both sides interpret in the same way
36
Service area?
![Page 37: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/37.jpg)
Tip 8 Ask users for stories to
explore vocabulary and
concerns
37Quesenbery, W and Brooks, K (2010) “Storytelling in User Experience Design”
![Page 38: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/38.jpg)
38
© Caroline Jarrett and Effortmark LtdIf you use this cartoon in another context, please credit Caroline Jarrett and Effortmark Ltd.
![Page 39: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/39.jpg)
What recommendation will you make based on your results?
• What decision will you make?
• What happens if …– The users all agree with your gut feel?– The users all disagree with your gut feel?– The users don’t care either way?
39
![Page 40: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/40.jpg)
Tip 9 Try writing your
presentation first
(before the questions)
40Natalie Webb, conversation at UPA2010
![Page 41: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/41.jpg)
Outline Tips about questions
Tips about process
Tips about testing
41
![Page 42: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/42.jpg)
42
“If you don’t have time to do the pilot,you don’t have time to do the study”
• Oppenheim, A. P. (1992) “Questionnaire Design,
Interviewing and Attitude Measurement “– And in each subsequent edition
• Dillman, D.A. (2000) “Internet, Mail and Mixed Mode
Surveys: The Tailored Design Method”– And in each subsequent edition
• Gillham, B (2000) “Developing a Questionnaire”– And in the second edition
• Jarrett, C (2010) “I wish I’d run a pilot”
![Page 43: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/43.jpg)
Tip 10 Run a pilot
43
![Page 44: 10 tips for a better survey at STC2011](https://reader033.vdocuments.mx/reader033/viewer/2022051819/54c65d024a7959e9438b4631/html5/thumbnails/44.jpg)
Caroline Jarrett
Twitter: @cjforms
Slideshare: cjforms
I’m a consultant, hire me:Consultancy: www.effortmark.co.uk
Training: www.usabilitythatworks.com
Free stuff:Forms advice: www.formsthatwork.com
Editing: www.editingthatworks.com
Columns: www.uxmatters.com
“Good Questions”
www.usabilitynews.com
“Caroline’s Corner”
44