surveying macrh 16 - uc berkeley school of...
TRANSCRIPT
Surveys and Usability
I214 March 11, 2010
What Info Can Be Collected via Surveys?Facts
Characteristics of respondentsDemographics, experience, employment...
Self-reported behaviorThis instanceGenerally/usuallyPastAnticipated (who will you vote for?)
Opinions and attitudes:Preferences, opinions, satisfaction, concerns, perceptionsTheir expectations of their future behavior
KnowledgeWhat do they they know about x, y, z
Some sources of error
Sample, respondents
Question choice
Question wording
Question order
Method of administrationSurveyor, if administered person to person
Inferences from the data
Users’ interests in influencing results“vote and view the results”
Response Rates
% of sample who actually participatelow rates may indicate bias in responses
Whom did you miss? Why?Who chose to cooperate? Why?
How much is enough? For statistically valid sample
Babbie: 50% is adequate; 70% is very good
Web surveys tend to be 3%, 5%....
Increasing response rates
Good designShortEasy to answer immediatelyNOT an extra step Harder to say ‘no’ to a personExplanation/request
Explain purpose of studyDon’t underestimate altruism
Why you need themIncentives
Reporting back to respondents as a way of getting responseMoney; entry in a sweepstakes
Follow up on non-respondents (if you can)
Ways of Administering Surveys (roughly in order of expected response rate)
In person
Phone
Paper, in person
Email (usually with a link)
Web
Mail.
Active vs passiveActive: solicit respondents
Send out email (with questions, or link), letters, phoneUse sampling frame to develop a sample, I.e. list
Ideal:Keep track of who respondsFollow up on non-respondents if possible
Compare respondents/non-respondents looking for biasesPassive – “poll”
Popup box: “would you take a few minutes to help us…”Link on a site
Passive: problems may include
Response rate probably unmeasurableMay be difficult to compare respondents to population as a wholeLikely to be biased (systematic error)
Frequent users probably over-representedBusy people probably under-representedDisgruntled and/or happy users probably over-represented
What to look for in web survey software/site
# of questions you can ask
# of respondents
How long the survey will be available
Variety of question designs/formats, or at least the designs you need
Kind of data analysis they support
Reports they produce for you
Whether you can download data into Excel etc
How long the data will be available to you
Most have free/lowcost versions and premium
Web survey problems
Loss of context – what exactly are you asking about, what are they responding to?
Are you reaching them at the appropriate point in their interaction with site etc?
Incomplete responses Multiple submissions
NOW (PBS) using cookies to prevent repeated votingResponse rate problems
Low rateHard to calculate
Is your sampling unit the user or the interaction?
Questionnaire construction
ContentGoals of study: What do you need to know?
What can respondents tell you?
Conceptualization
Operationalization – e.g., how exactly do you define “household with access to internet”?
Question design
Question ordering
Short, short, short!
Respondent characteristics include:Demographics:
age, sex, race…IncomeFamily/living circumstances
Living conditionsWhereUrban/suburban/ruralHouse, apartment…etc etc.
Experience, expertiseOccupation
Know how you’ll use the data. If you don’t need to know, don’t ask
Respondent characteristics
Pew Burning Man Survey
What is your gender? M, F
What is your current gender?
Use standard categories OR Match the user’s wording
Pew Burning Man
What is your race? White
Black or African-American
Asian or Pacific Islander
Mixed race
Native American/American Indian
Other (SPECIFY)
Don't know/Refused
Do you consider yourself to be a person of color?
Do you consider yourself to be white?
Income: PewLast year, that is in 2009, what was your total family
income from all sources, before taxes. 1 Less than $10,0002 $10,000 to under $20,0003 $20,000 to under $30,0004 $30,000 to under $40,0005 $40,000 to under $50,0006 $50,000 to under $75,0007 $75,000 to under $100,0008 $100,000 or more9 Don't know/Refused
Respondent characteristics: specific
Internet experience 2
Behavior, Activity
In general
Specific to this topic
Specific to this product
BehaviorTasks (e.g., what did you do today?)
Site usage, activityFrequency; common functions – hard to answer accurately
Self-reports vs observations
Time:This event
Today
The last time you...
The last week
The last month
Generally...
Pew: Activity (open-ended)
We'd be interested in hearing about the most amazing thing that has ever happened while you were online.
It could be a story about how something on the Web changed your life, something interesting you discovered on the Web, or something really important or even really amusing that happened during an exchange of email.
Opinions, Evaluation
Opinions, preferences, concerns about this product
About the site: Content, organization, architecture, interface
Ease of use
Perceived needs
Preferences
ConcernsE.g., security
Success, satisfactionSubdivided by part of site, task, purpose…
Other requirements
Suggestions
PEW: Opinions1. Some people say they feel overloaded with information these days, considering all the TV news shows, magazines, newspapers, and computer information services. Others say they like having so much information to choose from.
How about you... do you feel overloaded, or do you like having so much information available?
Feel overloaded
Like having so much information available
2. Overall, do you think that computers and technology give...
people MORE control over their lives, LESS control over their lives, or don't you think it makes any difference?
More
Less
No difference
Summary Evaluation – and asking for follow-up
Writing Questions: Guidelines
Writing questions involves:Deciding on topics to be addressed
Ordering the topics
Writing a specific SET of questions
Deciding on the type of question, the format
Writing the ANSWERSExhaustive list
Unambiguous
Mutually exclusive (unless you ask “check all that apply”)
Picky (but important) decisions
Only one answer per question?Force them to answer to be able to continue? (NO!)Allow neutral answer? (YES)Include “other” – open-ended (USUALLY)Do you need:
“don’t know”“not applicable”“no experience” or something like that
Remember!
People skimPeople answer quickly
Types of questions
http://www.surveymonkey.com/
Bad survey questionsBarreling:
Bad: Is this site clear and useful?Good: Do you find the information on this site clearly presented? Is it useful to you?
Loose Bundling vs. Anchoring:Questions are too general and not grounded in specific behaviors.
Bad: How often do you use the internet?Good: For the following internet uses, please indicate how often…LIST: email, online shopping, online banking, reading blogs…etc etc
AssumptionsBad: How has blogging changed your life?Good: Do you blog? If yes, how well does this statement describe you? “Blogging has changed my life.”
Writing questions – some guidelines
Unambiguous questions
Caution re “what would you do”
Not too many items for ranking
Not too many ratings (with neutral!)
No more than 7; preferably 5 or fewer
Writing: Parallel questions
Is easy to use?
Provides information at the right level?
White space: attractive, easier to read
Design: attractive; white space
Avoid convoluted instructions, questions, answers
People SKIM
How interesting would each of the following displays be to you?
* Historic displays with old-fashioned and famous police artifacts from the beginning of America, through the Wild West, to the mobster era and more recent times
* Modern displays where visitors go behind the badge and experience what it's like to walk in an officers shoes today, featuring the latest crime-fighting equipment and tactics
(Another ex) Avoid convoluted instructions, questions, answers
(Another ex) Avoid convoluted instructions, questions, answers
Questions should be broadly applicable (or use a skip pattern)
Recognition rather than recall
P
Please evaluate the topics covered in this workshop:
Topic 1: ____________How useful was this topic?How would you rate the presenter?What did we NOT cover that you were hoping to learn about?
Topic 2: _______________
More guidelines
Edit! Use good English syntax.
Focused questions people can answer briefly.
Reporting Survey Data
Some principlesReader must know how many total respondents are represented, andmust be able to calculate from data provided any numbers they may be interested in
Reader must know which responses/respondents are included and which not
Reader needs exact wording, answer choices offered, since these affect precise answers
Don’t know/refused: with large #s, there are always some. Are they included in %ages?
Any table or graphic should still make sense separated from the report.
Readers don’t always read text; some just look at the “pictures”Tables and graphics get copied and re-used
Questionnaire with the data filled in
% % % %
Pew – networked workers
Pew – networked workers
% % % %% %
Types of Variables and Measures of Central Tendency
Type Examples Measures of Central Tendency
Nominal gender, race, college major
mode
Ordinal preferences, satisfaction, opinion; age or income in intervals
mode, median
Interval (arbitrary zero point )
Year (calendar date)temperature Celsius or Fahrenheit
mode, median, or mean
Ratio age, income, time mode, median, arithmetic mean
Ratio or Nominal?
Ratio or Nominal?
1 2 3 4
Pew – Nominal variable
Burning Man - nominal
Ordinal Variable
Ratio
Tables
Questionnaire with the data filled in
Tables
Pew – cross tabs
Independent variableDependent variable
Table Summarizing Key Findings
Respondent characteristics
Pew Typology Quiz Burning Man Survey
What is your gender? M, F
What is your age? Fill in
Where do you live? Drop down
What is your educational background?
drop down – from currently in middle school to grad degree
Burning Man: Race