surveying macrh 16 - uc berkeley school of...

Surveys and Usability

I214 March 11, 2010

What Info Can Be Collected via Surveys?Facts

Characteristics of respondentsDemographics, experience, employment...

Self-reported behaviorThis instanceGenerally/usuallyPastAnticipated (who will you vote for?)

Opinions and attitudes:Preferences, opinions, satisfaction, concerns, perceptionsTheir expectations of their future behavior

KnowledgeWhat do they they know about x, y, z

Some sources of error

Sample, respondents

Question choice

Question wording

Question order

Method of administrationSurveyor, if administered person to person

Inferences from the data

Users’ interests in influencing results“vote and view the results”

Response Rates

% of sample who actually participatelow rates may indicate bias in responses

Whom did you miss? Why?Who chose to cooperate? Why?

How much is enough? For statistically valid sample

Babbie: 50% is adequate; 70% is very good

Web surveys tend to be 3%, 5%....

Increasing response rates

Good designShortEasy to answer immediatelyNOT an extra step Harder to say ‘no’ to a personExplanation/request

Explain purpose of studyDon’t underestimate altruism

Why you need themIncentives

Reporting back to respondents as a way of getting responseMoney; entry in a sweepstakes

Follow up on non-respondents (if you can)

Ways of Administering Surveys (roughly in order of expected response rate)

In person

Phone

Paper, in person

Email (usually with a link)

Web

Mail.

Active vs passiveActive: solicit respondents

Send out email (with questions, or link), letters, phoneUse sampling frame to develop a sample, I.e. list

Ideal:Keep track of who respondsFollow up on non-respondents if possible

Compare respondents/non-respondents looking for biasesPassive – “poll”

Popup box: “would you take a few minutes to help us…”Link on a site

Passive: problems may include

Response rate probably unmeasurableMay be difficult to compare respondents to population as a wholeLikely to be biased (systematic error)

Frequent users probably over-representedBusy people probably under-representedDisgruntled and/or happy users probably over-represented

What to look for in web survey software/site

# of questions you can ask

# of respondents

How long the survey will be available

Variety of question designs/formats, or at least the designs you need

Kind of data analysis they support

Reports they produce for you

Whether you can download data into Excel etc

How long the data will be available to you

Most have free/lowcost versions and premium

Web survey problems

Loss of context – what exactly are you asking about, what are they responding to?

Are you reaching them at the appropriate point in their interaction with site etc?

Incomplete responses Multiple submissions

NOW (PBS) using cookies to prevent repeated votingResponse rate problems

Low rateHard to calculate

Is your sampling unit the user or the interaction?

Questionnaire construction

ContentGoals of study: What do you need to know?

What can respondents tell you?

Conceptualization

Operationalization – e.g., how exactly do you define “household with access to internet”?

Question design

Question ordering

Short, short, short!

Respondent characteristics include:Demographics:

age, sex, race…IncomeFamily/living circumstances

Living conditionsWhereUrban/suburban/ruralHouse, apartment…etc etc.

Experience, expertiseOccupation

Know how you’ll use the data. If you don’t need to know, don’t ask

Respondent characteristics

Pew Burning Man Survey

What is your gender? M, F

What is your current gender?

Use standard categories OR Match the user’s wording

Pew Burning Man

What is your race? White

Black or African-American

Asian or Pacific Islander

Mixed race

Native American/American Indian

Other (SPECIFY)

Don't know/Refused

Do you consider yourself to be a person of color?

Do you consider yourself to be white?

Income: PewLast year, that is in 2009, what was your total family

income from all sources, before taxes. 1 Less than $10,0002 $10,000 to under $20,0003 $20,000 to under $30,0004 $30,000 to under $40,0005 $40,000 to under $50,0006 $50,000 to under $75,0007 $75,000 to under $100,0008 $100,000 or more9 Don't know/Refused

Respondent characteristics: specific

Internet experience 2

Behavior, Activity

In general

Specific to this topic

Specific to this product

BehaviorTasks (e.g., what did you do today?)

Site usage, activityFrequency; common functions – hard to answer accurately

Self-reports vs observations

Time:This event

Today

The last time you...

The last week

The last month

Generally...

Pew: Activity (open-ended)

We'd be interested in hearing about the most amazing thing that has ever happened while you were online.

It could be a story about how something on the Web changed your life, something interesting you discovered on the Web, or something really important or even really amusing that happened during an exchange of email.

Opinions, Evaluation

Opinions, preferences, concerns about this product

About the site: Content, organization, architecture, interface

Ease of use

Perceived needs

Preferences

ConcernsE.g., security

Success, satisfactionSubdivided by part of site, task, purpose…

Other requirements

Suggestions

PEW: Opinions1. Some people say they feel overloaded with information these days, considering all the TV news shows, magazines, newspapers, and computer information services. Others say they like having so much information to choose from.

How about you... do you feel overloaded, or do you like having so much information available?

Feel overloaded

Like having so much information available

2. Overall, do you think that computers and technology give...

people MORE control over their lives, LESS control over their lives, or don't you think it makes any difference?

More

Less

No difference

Summary Evaluation – and asking for follow-up

Writing Questions: Guidelines

Writing questions involves:Deciding on topics to be addressed

Ordering the topics

Writing a specific SET of questions

Deciding on the type of question, the format

Writing the ANSWERSExhaustive list

Unambiguous

Mutually exclusive (unless you ask “check all that apply”)

Picky (but important) decisions

Only one answer per question?Force them to answer to be able to continue? (NO!)Allow neutral answer? (YES)Include “other” – open-ended (USUALLY)Do you need:

“don’t know”“not applicable”“no experience” or something like that

Remember!

People skimPeople answer quickly

Types of questions

http://www.surveymonkey.com/

Bad survey questionsBarreling:

Bad: Is this site clear and useful?Good: Do you find the information on this site clearly presented? Is it useful to you?

Loose Bundling vs. Anchoring:Questions are too general and not grounded in specific behaviors.

Bad: How often do you use the internet?Good: For the following internet uses, please indicate how often…LIST: email, online shopping, online banking, reading blogs…etc etc

AssumptionsBad: How has blogging changed your life?Good: Do you blog? If yes, how well does this statement describe you? “Blogging has changed my life.”

Writing questions – some guidelines

Unambiguous questions

Caution re “what would you do”

Not too many items for ranking

Not too many ratings (with neutral!)

No more than 7; preferably 5 or fewer

Writing: Parallel questions

Is easy to use?

Provides information at the right level?

White space: attractive, easier to read

Design: attractive; white space

Avoid convoluted instructions, questions, answers

People SKIM

How interesting would each of the following displays be to you?

* Historic displays with old-fashioned and famous police artifacts from the beginning of America, through the Wild West, to the mobster era and more recent times

* Modern displays where visitors go behind the badge and experience what it's like to walk in an officers shoes today, featuring the latest crime-fighting equipment and tactics

(Another ex) Avoid convoluted instructions, questions, answers

(Another ex) Avoid convoluted instructions, questions, answers

Questions should be broadly applicable (or use a skip pattern)

Recognition rather than recall

P

Please evaluate the topics covered in this workshop:

Topic 1: ____________How useful was this topic?How would you rate the presenter?What did we NOT cover that you were hoping to learn about?

Topic 2: _______________

More guidelines

Edit! Use good English syntax.

Focused questions people can answer briefly.

Reporting Survey Data

Some principlesReader must know how many total respondents are represented, andmust be able to calculate from data provided any numbers they may be interested in

Reader must know which responses/respondents are included and which not

Reader needs exact wording, answer choices offered, since these affect precise answers

Don’t know/refused: with large #s, there are always some. Are they included in %ages?

Any table or graphic should still make sense separated from the report.

Readers don’t always read text; some just look at the “pictures”Tables and graphics get copied and re-used

Questionnaire with the data filled in

% % % %

Pew – networked workers

Pew – networked workers

% % % %% %

Types of Variables and Measures of Central Tendency

Type Examples Measures of Central Tendency

Nominal gender, race, college major

mode

Ordinal preferences, satisfaction, opinion; age or income in intervals

mode, median

Interval (arbitrary zero point )

Year (calendar date)temperature Celsius or Fahrenheit

mode, median, or mean

Ratio age, income, time mode, median, arithmetic mean

Ratio or Nominal?

Ratio or Nominal?

1 2 3 4

Pew – Nominal variable

Burning Man - nominal

Ordinal Variable

Ratio

Tables

Questionnaire with the data filled in

Tables

Pew – cross tabs

Independent variableDependent variable

Table Summarizing Key Findings

Respondent characteristics

Pew Typology Quiz Burning Man Survey

What is your gender? M, F

What is your age? Fill in

Where do you live? Drop down

What is your educational background?

drop down – from currently in middle school to grad degree

Burning Man: Race