february 11, 2015 survey design and administration catherine r. messina phd
TRANSCRIPT
February 11, 2015
Survey Design and Administration
Catherine R. Messina PhD
“The scientist is not a person who gives the right answers, he's one who asks the right questions.” ― Claude Lévi-Strauss (1908-2009), French anthropologist
No matter how good your study design, your sampling strategy, your response/participation rate, etc …………
The quality of your measurement instruments will directly impact your ability to draw meaningful conclusions about relationships studied
What is a survey or questionnaire?
…….. a systematic method for collecting information from a sample of individuals for the purpose of quantitatively describing the attributes of the larger population of which the sample are members.
Groves, Fowler, Couper, Lepkowski, Singer and Tourangeau, 2004.
Benefits of surveysCan reach a large number of people relatively easilyProvide quantifiable answersRelatively easy to analyse
Drawbacks of surveysProvides only limited insight into problem
Information obtained is limited by questions asked Always possible that wrong questions are asked
Varying response – respondent may misunderstand or misinterpret questionsNeed to get it right first time
Can’t chase after missing data – or go back and try again
Making inferences from survey data
Respondent answers question(s)
You infer….
Characteristic(s) (attribute(s)) of Respondent
Characteristic(s) (attribute(s)) of Sample
You infer….
Characteristic(s) (attribute(s)) of Population
Measurement is a systematic and repeatable process that quantifies or classifies events or characteristics
Measurement tools (i.e., surveys/questionnaires) need to yield measurements that are reproducible (reliable) and accurate (valid)!
Reliability- the property of reproducability does the measure consistently yield a similar
finding? Validity- does the measure really assess (i.e.
measure) the concept/object that it is intended to? How accurate is the measure?
Making inferences from survey data
Measurement is represented by – 3 related concepts: observed value (aka the “measured value”) true value and measurement error
Observed value = true value + measurement error
Well designed measurement tools (i.e., surveys) minimize measurement error so that the observed value is very close to the true value
Making inferences from survey data
Creating a survey or questionnaire First, determine clearly what it is you want to
measure in order to answer your research question(s) or hypotheses Be very clear about what you want to know
Decisions about relevant variables are based on the research question (or hypothesis) Research question / hypotheses should be
stated in measureable terms
Creating a survey or questionnaire List all relevant variables!
Dependent variables Independent variables
Covariates - A secondary variable (and of interest to the researcher) that is observed and which can affect the relationship between the IVs and the DV E.g. age can effect the relationship between
CRC-screening reminders and the likelihood of having a colonoscopy
Identifying Existing Measures
Next, determine if questions that you want to include have already been developed (and tested) by other researchers
Review the literature!!!!… you do not have to reinvent the wheel!
Saves development time and can facilitate comparisons with other studies
Identifying Existing Measures
Use standardized measures as available Review the literature for standardized measures Look for validity and reliability information,
appropriate to your sample Usually not a good idea to “adapt” or change – but if
you do, describe this is your Methods section and note this as a potential limitation in Discussion section
Be cautious if using only a subset of questions from an existing survey as this may change the meaning of the summary scores. Check whether a shorter version of the instrument exists that has also been validated
Re: primary research articles – if questions not included look for on-line supplemental info
What if there are no existing measures on my topic, and I need to develop my own survey or questionnaire?
Developing Surveys – General Approach Determine the mode of survey administration:
self-administered or interviewer adminsitered Each has its own advantages and
disadvantages …… and will determine your survey format
Types of surveys
Self-administered•Hand out•Mailed•Email / web-based
Interviewer administered•Face to face•telephone
Self-administered surveys Advantages:
Relatively inexpensive and easy to administer Preserves confidentiality (can be anonymous) Can be completed at respondent's convenience No influence by interviewer
Disadvantages: Low response rate – email surveys sift to bottem of
inbox very quickly Questions can be misunderstood No control by interviewer
Interviewer-administered surveys Advantages:
Facilitates participation by people with low literacy Interviewer can clarify ambiguity
Disadvantages: May introduce interviewer bias Needs more resources – more expensive Short surveys - especially on telephone Difficult for sensitive issues
Developing Surveys – General Approach Create a pool of new questions or revise existing
questions – write more questions than will be included in the final draft
Put questions in sequence and format survey draft This includes all instructions for filling out survey
and possible skip patterns! Include appropriate white space to get a good
estimate of the length
Developing Surveys – General Approach Decided on operational definitions
How the researcher chooses to measure a particular variable E.g.: age – can be operationally defined as”
DOBYearsAge group
Use measure that offers most flexibility
Developing Surveys
If possible, ask experts to review for content Content validity: do the items in a questionnaire
adequately represent the universe of items relating to a specific construct? How accurately does a measurement instrument tap into the various aspects of a specific construct?
Developing Surveys Get feedback (e.g., advisor; peers) on your 1st draft Face validity: is the measurement logical on the
“face of it”? The degree to which the purpose of a measurement instrument is obvious to those using it or responding to it, i.e., “looks like” or appears to respondents as a measure of depression. Advantage of good face validity: respondent really
understands the point of the questions and interprets them accurately
Disadvantage of good face validity: respondent really understands the point of the questions and “tailors” their responses to “fake good” or “fake bad”
Developing Surveys Get feedback (e.g., advisor; peers) on your 1st draft –
con’t Which questions measure IV(s)? Which questions measure DV(s)? Which questions measure covariate(s)? Are questions clear and “answerable”? Are any questions potentially offensive? Does format make sense? Is it logical? Length of survey?
REVISE!! Get more feedback (e.g., advisor; peers) on your
revised draft
Developing Surveys Pilot-test on a small sample of respondents who are
similar to your study sample Are questions clear and “answerable”? Are any questions potentially offensive? Questions should be
culturally sensistive!! Does format make sense? Is it logical? Length of survey and how long to complete? (consider
participant burden) Cognitive interviewing and “think aloud”
Ask a few people in pilot sample to read the questions out loud to you and then explain to you, what each question means to him/her and how he/she thought about their answer
Test-drive your coding system – prelim analyses REVISE!
Developing Questions If possible, pilot-test the near final draft of the survey
with new pilot sample
Pilot-test all procedures and logistics for administering survey Self or interviewer administered? Who will administer? Who is responsible for collecting completed
surveys? Where will these be stored before pick-up?
Developing Surveys – Specifics
Your goal is to write survey questions that •Respondents will interpret the same way (that you interpret them) •Respondents are able to respond accurately •Respondents are willing to answer
Only include questions you plan to use Too many questions (especially if they seem
unrelated to the purpose of your survey) can irritate respondents (an IRB issue too)
Avoid lengthy or complex questions – keep it simple Write at a low reading level – understand the
literacy level of your respondents Avoid jargon and acronyms
Use words relevant to respondents - terms and concepts should be familiar to respondents and easy for them to understand
Writing good survey questions
Instructions need to be placed exactly where needed - not at the beginning of the entire survey.
Use italics or bolding or underline to emphasize instructions and directions
Writing good survey questions
Use standard demographic questions Facilitates comparing your sample to others in the
literature Examples from national health surveys Look at the way these are asked in similar studies Only ask for necessary demo information
Race / ethnicity – comprised of two questions (NIH format):1. Are you Hispanic or Latino? (yes vs. no)2. What is your race? (allow for multiple answers)
Writing good survey questions
Avoid “and” “or” (double-barreled) questions Trying to respond to a question measuring two or
more ideas is confusing to respondent and their response is difficult for the researcher to interpret E.g., Do you think that increasing physical activity and
(or) losing weight is beneficial to you child’s health? E.g., Do you think that eye contact and (or) using words
you understand improve communication between you and your child’s doctor
E.g., Were you satisfied with the timeliness and quality of the health care service your child received?
Writing good survey questions
Avoid leading questions
Do you think that the food in the hospital made your child sick?
Do you agree that the hospital staff were overworked?
Writing good survey questions
Avoid bias by using both positive and negative sides in the question stem
“To what extent do you agree….”
vs.
“To what extent do you agree or disagree with this statement…. “
Writing good survey questions
Avoid vague quantifiers
How often did you use aspirin for pain relief in the last month?
Never
Rarely
Occasionally
Regularly
Vs……..
Writing good survey questions
How often did you use aspirin for pain relief in the last month?
Not at all About once in the last month Two or three times in the last month About once a week More than once a week
Writing good survey questions
Ask questions that people are willing to answer! Do not be too personal or risky unless absolutely
needed Be culturally sensitive
People are more likely to answer questions about behaviors and health compared to income or education Using income categories rather than asking for
exact $ amounts is a better strategy
Writing good survey questions
Ordering questions Two key considerations:
Questions should be ordered such that the response to one question does not influence the response to the question that follows it
Questions should be ordered such that respondents are more likely to complete the survey Fact-based questions before opinion based questions Start with non-threatening or least sensitive questions Most important questions should be placed first Vary question format to reduce likelihood that
respondents will “auto-pilot” response sets
Writing good survey questions
Good questions follow comfortably from the previous question. Requires good writing skills Smooth transitions between questions or groups
of questions Group similar questions together
Avoid non-response (i.e., missing data) by including response options for “Don’t Know” and “Refused” or “Prefer not to answer”
Writing good survey questions
Recall of past events – people have limited recall of past events (this will vary with type of event)
Improve recall by providing reasonable prompts Past year Past 6 months Past 3 months Past 2 weeks
Use addition prompts, if needed: “What season was it when your child was last
vaccinated?” “Was it near a holiday or someone’s birthday?”
Writing good survey questions
FORMAT OF QUESTIONS
Two main question formats Closed format forced choice
Yes Always No Sometimes Don’t know Never
Open format free text
Please describe your child’s most distressing symptom? ________________________________________________________________________________________________
Close-Ended Response Formats
the Likert-type response format “Indicate the strength of your agreement for each of the following statements, on a scale from 1 to 5, where a response of 1 indicates that you strongly disagree, a response of 5 indicates that you strongly agree, and 3 indicates that you are uncertain.” Odd number of categories is preferred – captures
respondents who are indifferent or undecided Reduces non-response No more than 5 response options
Close-Ended Response Formats Semantic Differential Scale - used to assess the
meaning of a variable to the respondent uses a pair of discrete descriptor words, which name
opposing positions, as anchors for the response scale Everything in between represents the continuum of
choices between the 2 anchors
Very satisfied __I__I__I__I__I__I__I__I__ Very dissatisfied
Very confident __I__I__I__I__I__I__I__I__ Not at all confident
Strongly agree __I__I__I__I__I__I__I__I__ Strongly disagree
NOTE: odd number of spaces
Advantages: Simple and quick Easy to code, record, analyse Easy to compare Easy to report results
Disadvantages: Restricted number of possible answers Loss of information
Close-Ended Response Formats
Open – ended response questions Advantages
Rich and detailed response – respondents are not constrained by researchers choices
Unexpected responses possible Good for exploring knowledge and attitudes
Disadvantages Cognitively demanding for respondent and
researcher (coding is challenging and time intensive) Interviewer bias Difficult to analyse Difficult to compare groups
Asking questions about: Knowledge Attitudes and beliefs Behavior
Developing Surveys
Measuring Knowledge True/False and multiple choice “fact” questions Open-ended response formats
E.g., list all of the problems related to X that come to mind
Close-ended response formats (response options are provided by the survey) Quicker, less burden compared to open-ended Less detail but easier to code and analyze
If you plan to intervene on knowledge, include questions at pre and post-test that measure knowledge specifically addressed by intervention
Measuring Attitudes and Beliefs
Close-ended response formats are very common when measuring attitudes / beliefs
Attitude questions can be stand-alone items but … Responses often combined (i.e., summed) as a
“Scale” which means that many items are used to assess a single attitude or belief
often using the same response format… e.g. strongly agree …… strongly disagree
Methods to measure behavior Self report: “I did not smoke any cigarettes today.”
By phone, mail, computer, interview Observation - Report of respondent’s behavior by
others (family member, health professional) “I saw him/her smoke cigarettes today.”
Survey design can go a long way in reducing non-response!
Consider using surveys that you think are attractive as a template Professional looking format - good use of “white space” Space response scales widely enough so that it is easy to circle or
check the correct answer without the mark accidentally including the answer above or below. Open-ended questions: the space for the response should be big enough to
allow respondents with large handwriting to write comfortably in the space. Closed-ended questions: line up answers vertically and precede them with
boxes or brackets to check, or by numbers to circle, rather than open blanks. Use larger font size (e.g., 14) and high contrast (black on white).
Proof read!! No typos! Get help if this is not your thing.
Always conclude your survey with THANK YOU!