surveys
DESCRIPTION
TRANSCRIPT
Surveys
Christopher S. LaRocheMIT, Usability Consultant
Northeastern University, Senior [email protected]
Copyright © 2012 - 2014 - Christopher S. LaRoche & Katherine Wahl
Survey Overview
• Surveys are an efficient way to gather a high volume of information and specific data from your users
• Surveys are multi-faceted. You can create simple surveys and get massive amounts of data; but to get statistically significant information you often have to parse your demographics and question wording (Goodman, p. 327)
• Surveys can range from simple to incredibly complex
2
Survey Overview
• Most questions are closed (multiple choice) vs. open-ended (open text box)
• Try to strip out subjectivity and bias. (Goodman, p. 331-343)
3
Online tools
• Online tools make surveys much easier and efficient
• One of the most popular online survey services: www.surveymonkey.com
• Not only do online survey sites allow you to conduct the survey online, they provide tools to help you analyze the data
4
Types of Surveys
• There are several different kinds of surveys:– Profile, when you want to get a snapshot of the composition of your current user population
– Satisfaction, when you want to find out what what works well and what doesn’t
– Value, when you want to know what people find important (Goodman, p. 328)
5
When a survey is a good choice
• A survey is a good choice when you want to get answers to specific questions by reaching a large group of users
• It’s not a good idea to use a survey when you have broad questions about your users or want to understand comprehensive workflow
• A survey is a good method when you want quantitative [numbers] data, not qualitative [text] data
6
Disadvantages of surveys
• Quality of survey data can be uneven, as users will sometimes skip questions or respond quickly with out fully considering questions
• Questions MUST be well-written and free of bias
• Analysis can be time consuming and complex
• Users don’t always report truthfully in surveys
7
Getting started
• Determine the goals for your survey. What do you want to learn?
• Make sure your stakeholders know what they will and will not learn from the survey
• Clearly define the profiles of the survey participants and how you will reach them
8
Sample size
• How many participants should you have for your survey? This is one of the hardest questions to answer and, as with many user research related questions, the answer is ‘it depends’
• You want to make sure you are covering the audiences for your project and that they are all represented in your sample group. This means your sample size will vary
• If you don’t get enough responses, you need to report that even though it may invalidate the survey
9
10
Closed questions• Closed questions are those where users pick the response from a list
• You want users to pick a specific answer or answers from a list of possibilities. You need to make sure the list of possible answers is complete and that there is no overlap among the answers so users will not get confused. (Goodman, p. 333)
• Always include an “other” option in your list and include a text box for participants to write in what that option is. That way, the list is really complete!
11
Open-ended questions
• Open-ended questions are those where participants write their responses as free form text
• These yield more complete and complex data but they are difficult and time consuming to analyze
• Participants may be likely to skip them if they are in a hurry
• Some of the data may be difficult to interpret if the answer uses unfamiliar terminology
12
Open-ended questions
• Include at least one open-ended question at the end of the survey, even if it’s “Do you have any thoughts, comments, or suggestions?” This gives participants a nice ‘release valve’ to give any feedback they wanted to share but didn’t see covered in the survey
Writing questions
• Keep questions positive (avoid negative statements). Negative questions are more difficult to understand (Goodman, 336)
• Keep each question one question – do not have multiple questions in one. This may involve breaking one question into multiple questions to get at what you want to know
13
Writing questions
• Be consistent in how questions are formatted
• Use Likert scales of 1 – 5 or 1 – 7. Make sure the scale is consistent throughout the survey
• Include an opt-out option so that participants can indicate that the question does not apply to them
• Also limit required questions to not distract users who might prefer to skip questions
14
Writing Questions
• Revise, revise, and edit, edit questions over and over to get any potential leading or subjective wording removed
• This usually takes many versions of questions to actually come up with a final version. Your own bias and subjectivity is often quite subtle, especially to you!
• This is the most difficult part of the process (Goodman, p. 331-341)
15
Constructing the survey
• Once you have your survey questions, it’s time to think about how they will flow in a survey
• If you are asking for demographic information in the survey, include it at the end (the only exception to this rule is if you are using it to determine other questions)
16
Survey logic
• Survey logic can help you guide participants through the survey and answer sets of questions based on answers to previous questions
• You can use ‘skip logic’ to have participants skip over sections of the survey that are not relevant to them
17
Survey logic
• If you find yourself writing questions like “If you answered ‘no’ to the previous question, skip to question 6,” using logic is a good idea!
• Most online survey tools include logic as part of their paid packages, not the free ones
• Test your survey logic thoroughly to make sure it works the way you intended. Having faulty logic can ruin your survey data!
18
Organizing the survey
• Make sure your questions flow logically. Ask the more general questions up front and the more specific, detailed questions in the middle. The final questions should be more general and summary in nature
• Make sure your survey is not too long
• A general rule is to limit the survey to 20 questions
19
Testing the survey
• Once your survey is constructed, test it! Have a few people try the survey to make sure the instructions are clear and the logic works
• If you are sending an invitation to participants via email, make sure the link to the survey is correct and working
20
Launching the survey
• If you are sending your survey via email, think about when you are going to send it out. Sending a survey on Thursday afternoon means it may be buried under a long list of other emails. Having a survey go out first thing on Monday morning means it will be at the top of the potential participant’s inbox
21
Closing the survey
• You want to leave the survey open long enough to give participants enough time to complete it. Usually five business days to a full week is sufficient
• If you find that your response rate is low, you may want to send out a reminder to participants who haven’t completed the survey
22
Incentives
• Sometimes, offering potential participants a chance to win a prize after completing the survey will help boost response rates. You could also provide a small incentive to all participants
• Usually participants opt in to the incentive by providing an email address after completing the survey
• Be sure to complete the drawing and send out the incentive as soon as the survey closes
23
Analysis
• You want to analyze the data as soon as you can. It’s much easier to analyze while the data is fresh
• Some online survey tools also provide components that help you count and compare the data to see trends
• Be careful with the conclusions you draw! Using numbers as evidence does not mean a rigorous process produced them (Goodman, p. 359)
24
Analysis
• Look at the raw data first. You want to scan the data to look for obvious trends or errors
• Consider missing data (from questions skipped or ‘no answer’ being selected). It can be as important as completed data
• Look for trends and consistent themes
25
Reporting
• Draft an executive summary. If your audience only reads this portion of the report, you want to make sure this includes the most important points
• In the more detailed body of the report, include your goals, methods, and the participant profile(s)
• Summarizing the data you’ve collected in tables and graphs will help your readers quickly scan the information
26
Examples of Poor Survey Questions• What is wrong with this question?
27
• Question is leading (‘the right technological depth’)
• If this question is used/kept, should be a Likert scale (1-5 or 1-7 choices)
• Wording is poor in the last option (“Ok” and “Bad/No Comments”)
Examples of Poor Survey Questions
• What is wrong with this question?
28
• Question is leading (‘that take away American jobs’) and for this type of (Likert) scale, questions should be written in a manner such as: ‘American should buy imported automobiles’ or ‘Buying imported automobiles takes away American jobs’
• Follows a sort of Likert scale (1-5 choices), but should be strongly agree, agree, neutral, disagree, strongly disagree
Examples of Poor Survey Questions
• What is wrong with this question?
29
• Question is fine, but the way they used the various mechanism to choose the option is convoluted. You should have one drop-down list box asking the sex of your child (Male or Female) and another drop-down list box to choose their age (0-15). That would be only two drop-down list boxes and be much easier to read and use much less space
BibliographyBaty, Steve. (2009). User Research for Personas and Other Audience Models. UX Matters, User Dialogs, April 2009. http://www.uxmatters.com
Courage, Catherine & Kathy Baxter. (2005). Understanding Your Users: A Practical Guide to User Requirements Methods, Tools, and Techniques. Amsterdam, Netherlands: Morgan Kaufmann/Elsevier.
Goodman, Elizabeth, Mike Kuniavsky, and Andrea Moed. (2012). Observing the User Experience: A Practitioner’s Guide to User Research - Second Edition. San Francisco, California: Elsevier/Morgan Kaufmann.
Morville, Peter & Louis Rosenfeld. (2006). Information Architecture for the World Wide Web – Third Edition. Sebastopol, California: O’Reilly Media, Inc.
Nielsen, Jakob. (1993) Usability Engineering. San Francisco, California: Morgan Kaufmann.
Nielsen, Jakob. (1993) Usability Engineering. San Francisco, California: Morgan Kaufmann.
31
Bibliography
Pearrow, Mark. (2007). Web Usability Handbook, Second Edition. Boston, Massachusetts: Charles River Media.
Schumacher, Robert. Editor (2009). Handbook of Global User Research. Amsterdam, Netherlands: Elsevier/Morgan Kaufmann.
Wilson, Chauncey. (2009). User Experience – Re-mastered. Amsterdam: Netherlands: Elsevier/Morgan Kaufmann.
Wilson, Chauncey. (2007). “Designing useful and usable Questionnaires: You Can’t Just ‘throw a Questionnaire together.’” Interactions May/June 2007: pp. 48-49, 63.
32