introduction to questionnaire design

2013-2014 Workshop in Methods

November 8, 2013

Introduction to Questionnaire Design

Ashley Bowers, Stacey Giroux, and Lilian Yahng

Why Questionnaire Design Matters

• Do you think the sports media treat African American athletes differently than white athletes?

– YES– NO

60% of respondents answer “YES, DIFFERENTLY”

• Do the sports media treat African American athletes and white athletes the same or differently?

– SAME– DIFFERENTLY

40% of respondents answer “DIFFERENTLY”

Center for Survey Research November 8, 2013

Cognitive Aspects of Survey Methodology

Question-Answering Process:

• Comprehension• Retrieval• Judgment• Reporting:

• Formatting into response options

• Editing answer

Tourangeau (2000)

November 8, 201Center for Survey Research

• Captures both the ideal and nonideal case – e.g., misunderstanding, satisficing

• Respondents can go wrong at any of these stages

• Not necessarily an orderly progression; several can be happening simultaneously

• Survey as artificial construct • Survey within a social context

• (social exchange theory)• (cooperative/conversational principle) • (social desirability)

Example: Do you think the sports media treat African American athletes differently than white athletes? (Yes/No)

TOP TEN

1. Conceptual Variability

2. Problematic Wording

3. Vague Quantifiers

4. Response Options

5. Order Effects

6. Recall/Retrieval Difficulty

7. Estimation Difficulty

8. Attitudinal Questions

9. Sensitive Topics

10.Formatting


#1 - Conceptual Variability• Words have many meanings:

• variability across respondents• divergence from researcher’s definition

• Suessbrick, Schober & Conrad (2000) administered CPS Tobacco Use Supplement followed by post-test about interpretation:

Most frequent interpretation held by only 53.8%!

23%

23%

54%

Only cigarettes youfinished

Cigarettes you finishedor partly smoked

Even just one puff


Do you think children suffer any ill effects from watching programmes with violence in them, other than ordinary Westerns?

Belson (1981) determined that respondents interpreted children, ill effects and violence in numerous ways

• e.g. “children”: < 8 years, < 19 - 20 years• children as students• only 8% interpreted question as intended


The “same” question, with some definitions:

Do you think that children suffer any ill effects from watching TV with violence in them, other than ordinary Westerns? By children I mean people under 14, by ill effects I mean increased aggression at school or at home, increased nightmares, inability to concentrate on routine chores, and so on. By violence I mean graphic depictions of individuals inflicting physical injuries on themselves or others, depictions of individuals wantonly damaging property or possessions, abusive behaviors or language to others, and so on.

Improved?

Additional words can clarify intended meaning but this may lead to unwieldy questions, as above.

• Tradeoff between clarity and complexity • Possible confusion when everyday terms are defined (Gricean

implicatures)November 8, 2013Center for Survey Research

#2 - Problematic Wording

AVOID:• Double-barreling • Negatives• Complex syntax• Hidden assumptions• Leading questions

Aim for a ~6th grade reading level.


Are you satisfied with our prices and customer service? Do you favor or oppose not allowing drivers to use cell phones while driving? Given the world situation, the government protects too many documents by classifying them as SECRET or TOP SECRET. (Agree/Disagree)

How many minutes does it usually take you to commute to work? In the past month, have you crossed the street from one side to another in order to avoid going near someone you thought was frightening?

Now that you've seen how you can save time, would you buy our product?

#3 - Vague Quantifiers

• Non-numerical terms for quantity have different numerical interpretationsBelson (1981) found “few” (in over the last few years) meant:

• “no more than two years” (7/59 respondents) • “seven or more years” (19/59)• “ten or more” (11/59)

• Particularly problematic in response optionsHow often do you feel really excited? Very often, pretty often, not

too often or never?If R says more than never, Schaeffer & Bradburn asked for

numberFor educated and younger Rs, “pretty often” and “very often”

were associated with larger numbers


#4 - Response Options

Don’t Forget to Check: Balanced scale Number of response options: 5 to 7?

(Krosnick et al.) Label all scale points Mutually exclusive and exhaustive Midpoint (e.g., Neither/Neutral) option? “Other (Please Specify)” option? “Don’t Know” option? “Not Applicable” option?


Why Response Options MatterResponse Alternatives and Television-Watching (Schwartz & Bienias, 1990)

#5 - Order Effects

• Ordering of Questions• Part/whole effects (marital satisfaction/general satisfaction)

– ask general question first• Related content – Fewer people say taxes are too high

when first asked several questions about whether government spending for various programs should be increased

• Ordering of Response Options• Recency effect: tendency to endorse last option in list

• most likely when interviewer reads to respondent

• Primacy effect: tendency to endorse first option in list• most likely when respondent reads to self (self-administered)

or predictable scale usedNovember 8, 2013Center for Survey Research

#6 - Recall/Retrieval Difficulty

April 19, 2023Center for Survey Research

When did you purchase your car radio?

Tourangeau (2000)

• Recall tasks can involve a complicated process for respondents (cognitive burden) • Telescoping• Situating an event between landmarks

• More recent, more likely to recall• Greater impact/salience,

more likely to recall


• Is the respondent in a position to know the answer? (e.g., parent’s income on college student survey)

#7 - Estimation Difficulty• In the past twelve months, since [DATE], how many different times have you been admitted to a

hospital as a patient overnight or longer? • In the past year, how many times have you seen or talked with a medical doctor or a physician’s

assistant about your health? • During the past 4 weeks, how much have you been bothered by emotional problems (such as

feeling anxious, depressed, or irritable)?

• How many hours of television did you watch yesterday?• How many hours of television did you watch last month?• In a typical week, how many hours of television do you watch?

• We all, from time to time, compare our lives with those of other people, such as co-workers, family members, friends, neighbors, people you went to school with, and so on. We would now like to ask you some questions about how you see yourself in relation to others.

• Counting, general impressions, and hypotheticals can be difficult for respondents

• Reference period/point?• Ask in a series of questions?• Try a longer question (using familiar words, examples)? November 8, 2013Center for Survey Research

#8 - Attitudinal Questions• Do you think marijuana should be made legal, or not?• On the average, [racial minority] have worse jobs, income, and housing than white

people. Do you think these difference are mainly due to discrimination?• How satisfied are you with how your life has turned out so far? Please use a scale of

1 to 10, where 1 is extremely dissatisfied and 10 is extremely satisfied.

• Factual vs. attitudinal questions• Measuring subject states of respondents

• “No right or wrong answers”• May not be fixed• No mental “card catalog” of topics• Capturing weak attitudes and nonattitudes

– “…or have you not thought much about this?”– Midpoint, No Opinion, Don’t Know options


#9 - Sensitive TopicsThe Casual Approach:

Do you happen to have murdered your wife?

The Numbered Card Approach:

Would you please read off the number on this card that corresponds to what became of your wife? (Hand card to respondent.)

(1) Natural death (2) I killed her (3) Other (What?)

The Everybody Does It Approach:

As you know, many people have been

killing their wives these days. Do you

happen to have killed yours?

The Other People Approach:

Do you know any people who have

murdered their wives? How about yourself?Bradburn (2004) November 8, 2013Center for Survey Research

Do you happen to jog or not? (Gallup)

In talking to people about elections, we often find that a lot of people were not able to vote because they weren’t registered, they were sick, or they just didn’t have time. How about you—did you vote in the elections this November? (ANES)

Some people say that most people can be trusted. Others say you can’t be too careful in your dealings with people. How do you feel about it? (GSS)

• Sensitive answers• A short list of topics shown to be sensitive:

• Pap smears, mammograms, disease prevention, library card ownership, voting, reading, exercise, income, charitable giving, drinking, gambling, sexual activity, illegal activity.

• Note: List possibly susceptible to culture, time, gender, etc.

• Demographics: early or later in questionnaire?• Income: try broad ranges, if initially refused.


#10 - Formatting


KEEP IT SIMPLE

Visual Cues and Heuristics

For web surveys:

1. Middle means typical

2. Left and top mean first

3. Near means related

4. Up means good

5. Like means close


Special Issue: Topics on Survey Measurement and Public OpinionPOQ (2013) 77 : S1

The Good, the Bad, and the Ugly 20

• Do you have access to high speed internet, either in your home, at work, or somewhere you can easily get to like your public library or a nearby relative?

• Do you agree or disagree that teens should not be fined for not obeying the local outdoor smoking ordinance?

• What is the status of this house? (Indonesian Family Life Survey)– Self-owned– Occupying– Occupying illegally without rent– Occupying illegally with rent– Rented/contracted

• Do you agree or disagree with the following statement: the federal government has gotten totally out of control and threatens our basic liberties unless we clear house and commit to drastic change. (FOX News Poll)Center for Survey Research November 8, 2013

Workshop

• Scenario: The Happiness Task Force has hired your team to develop a survey to gauge respondents’ happiness.

• Rules: Draft five survey questions, and you must include at least:• One demographic question• One question that uses a Likert scale (e.g.,

1=strongly disagree …. 5=strongly agree)• One additional closed-ended question• One open-ended question



Focus group

Cog

nitiv

e ex

pert

rev

iew C

ognitive interview

Field pretest

Focus Group


http://lovestats.wordpress.com/dman/

Focus Group• Guided discussion among 6-10 led by a

moderator • How do people think about this topic?• Have we missed something? • Language people use• Technology: Online, teleconference, two-

way

• Study protocol development


Sample Focus Group Protocol


• What is your job title and what kind of work do you do?• Main activities?• Change much from day to day?

• When you think about job skills, what kinds of things come to mind?• Skills shared with others in this

group?• Skills required?

Cognitive Expert Review

• Experienced survey researcher/methodologist or cognitive psychologist• Not substantive expert

• Written review with follow-up discussion• Measurement problems• Solutions (e.g., rewording, reordering)• Testing options


CSR: Might need to be a little more specific – an alternative from the Current Population Survey is: Are you employed by government, by a private company, a nonprofit organization, or are you self-employed or working in the family business?


CSR: This will be a tough question and may be subject to inaccuracy. Would recommend asking if they have provided help in a specific area (e.g., help with food) – and then, how many times have they provided help in that specific area and to how many different people.



CSR: Do you need both Q5 and Q6? I am not sure how much additional information Q6 gains you for additional respondent burden.

CSR: Will respondent know the level of political influence of other people?

CSR: Might be good to be consistent across the items – you use two years here, then five years, then last year – may confuse respondents.

CSR: Large number of scale points. Would be good to ensure that respondents are able to make distinctions between such a large number of points. I would expect that there would be concentration of responses at a limited number of points. Might also consider giving the respondent a showcard with the response scale as this might make it easier for them to provide a response that uses more of the scale.

Cognitive Interview

• One-on-one interview with trained interviewer(s)/researcher(s)

• Protocol: survey questions + item-by-item probes• Respondent “thinks aloud” as he/she

answers each question• Directed probes after questions

• Protocol just a guide


“Thinking Aloud”

• Windows example before interview• Early probing: “Tell me more about that”,

“Tell me what you are thinking”• Concurrent• May be difficult

Common Directed Probes (from Willis 2005, p. 48)

Cognitive Probe Example

Comprehension/ interpretation probe

What does the term “outpatient” mean to you?

Paraphrasing Can you repeat the question I just asked in your own words?

Confidence judgment How sure are you that your health insurance covers drug and alcohol treatment?

Recall probe How do you remember that you went to the doctor five times in the past 12 months?

Specific probe Why do you think that cancer is the most serious health problem?

General probes How did you arrive at that answer?I noticed that you hesitated. Tell me what you were thinking.Was that easy or hard to answer?

Examples – AVS


• Revised: Scientists should modify human genes to cure serious diseases.

• Original: How good does it make you feel when you see the American flag flying?

EXTREMELY GOOD

VERY GOOD

NOT VERY GOOD

NOT AT ALL

Finding: R feels neutral

Revised: Changed scale and included neutral option• Original: If I have worked hard, I deserve to have time for

fun and pleasure.

Finding: Pleasure not viewed positively

Revised: If I work hard, I deserve time to enjoy life.


Examples – AVS

Pilot/Field Pretest/Pretest

• Small scale version of main study (n=20-30+)

• Mirror procedures of main study• Use debriefing questions

• Identify common problems from debriefing forms (observe/talk with interviewers)

• Examine your survey data


Debriefing Questions• Were there any words or phrases in any of the questions

that you felt were unclear?• Please list any questions that you felt were not easy to

read or required you to read them more than once.• Were there any questions that you felt you did not have

enough knowledge to answer?• Were there any questions that you felt were too personal

or were uncomfortable answering?• One of our major challenges in conducting this study will

be obtaining participation from busy executives like yourself. Please provide any thoughts you might have about what we could do to encourage participation.


Debriefing Questions• Did you have technical difficulties with any of the following

tasks while completing this survey? (Select all that apply.)

Seeing the text of a question Seeing all of the responses for a question Entering an answer to a question Going back to a previous question Changing an answer to a question Exiting the survey Finding out how to contact us Figuring out how far along you were in the survey I didn’t have any technical difficulties with the survey


• Original: How important to you are greenways and trails?• Finding: Rs do not understand greenway.• Revised: Add a definition.

• Original: How satisfied or dissatisfied are you with greenways and trails?• Finding: New residents unable to answer.• Revised: Must live in the city for 6+ months.

• Original: What is your date of birth?• Finding: Rs uncomfortable with question.• Revised: Use age range.

• Original: How would you rate: Timely arrival of products on back order?• Finding: Never had back ordered product.• Revised: Asked screener question. Added reference

period.



Blank

Reliability and Validity (from Singleton and Straits 2005, pp. 90-105)

• Reliability• Internal consistency• Test-retest reliability

• Construct validity• Intercorrelations• Discriminant validity• Convergent validity• Known groups


ReferencesQuestionnaire design: Bradburn, N. et al. (2004). Asking Questions.

Converse, J., and Presser, S. (1986). Survey Questions: Handcrafting the Standardized Questionnaire.

Fowler, F.J. (1995). Improving Survey Questions.

Tourangeau, R. et al. (2000) . Psychology of Survey Response.

Focus groups: Krueger, R., and Casey, M. (2000). Focus Groups.

Cognitive interviewing: Willis, G. (2005). Cognitive Interviewing: A Tool for Improving Questionnaire Design.

Reliability/validity: Singleton, R.A., Jr., and Straits, B.C. (2005). Approaches to Social Research.

Data collection:

Dillman, D et al. (2009). Internet, Mail and Mixed Mode Surveys.

Couper, M. (2008). Designing Effective Web Surveys.

Oishi, S.M. (2003). How to Conduct In-Person Interviews for Surveys. November 8, 2013Center for Survey Research

Thank You!

For more information:

Ashley Bowers Stacey Giroux Lilian [email protected] [email protected] [email protected]

Center for Survey Researchhttp://csr.indiana.edu


Cognitive Testing Plan• Recruiting and scheduling subjects

• How? Convenience sample, usually recruit through personal networks, Craigslist, social media, flyers, newspaper ads, community groups

• What types? Representative of the population you are studying, make sure some respondents will go down all paths in questionnaire

• Incentive usually given (unless volunteers) - $25-$50• Generally, 5-25+ subjects per round (often only time

and $$ for 1 round but 2-3 is desirable)• Train multiple interviewers

• Involve in questionnaire development• Learn background on technique and probing


Cognitive Testing Plan (cont’d)• Mode of data collection? In-person may work

better but can do by telephone• Where to conduct? Quiet lab or non-lab

setting• Record interview and/or take notes for each

interview• How long of an interview? 15 minutes – 2 hours.

1 hour recommended (may be able to get through a 20 minute questionnaire)

• Hold debriefing and may write a report


introduction to questionnaire design

Documents