Download - Designing survey instruments
COLLECTING DATA ON A SAMPLE OF RESPONDENTS
Designing survey instruments
Remember:
The sample survey is a research technique used to estimate population parameters
A number of people are asked a series of questions from a standardized instrument concerning a variety of topics, and the answers are analyzed Interviews v. questionnaires
Options for administration
Personal interviewsPhone interviewsMail questionnairesComputer-mediated questionnaires
Survey design is strongly influenced by administration option chosen
The importance of quality instruments
A well-designed instrument is crucial to the success of any sample survey or census
With few exceptions, poorly written instruments represent a much greater threat to the validity of your conclusions than do sampling problems or other common threats to survey research validity
How do we go about developing the instrument?
The most important thing is knowing what we want to know What are the most significant questions
we have? What information is crucial to answering
these questions? Guided by literature review
What information can best be obtained in other ways?
How do we go about developing the instrument?
Prioritize the importance of the informationCrucial v. important v. useful but not
necessary Getting really good information on a limited set of constructs may be more valuable than a lot of scattered facts and incomplete ideas on many constructs
Don’t go fishing!
Too often researchers include many more questions than necessary Too many ‘interesting but unnecessary’
questions Tradeoff between additional questions and
completion rateTry to avoid a phone survey that lasts
more than 15 minutes Less for questionnaires
Typical instrument format
Greeting and introductionEarly general questions
Develop rapport Get the respondent thinking about the
topics of study Generate ‘top-of-mind’ data
Early to mid surveyFocus on topics of interest
Use closed-ended but still somewhat general questions
Move from category to brand questions, medium to station to program questions, etc.
Late survey questions More open-ended questions Questions dependent upon deeper review
of content Respondent has been thinking about
topics for a while Edgiest questions
Greatest level of rapport necessary
Final questionsMost embarrassing/difficult
questionsDemographics (often lose
respondents when you ask about income, etc.)
Closing Thank your respondents for their time Reassure them that their answers were
helpful and useful regardless of the quality of responses/level of knowledge of the respondent
Any necessary debriefing, especially if you were relatively guarded at the beginning or engaged in some sort of misleading or deceptive instructions, etc.
Impact of data collection method
Construct the instrument according to the method of data collection Self-administered questionnaires require
more instructions on the instrument itself than do interviews
When the instrument is administered by an interviewer, written instructions to the interviewer (usually placed in the margin) can be useful in improving response and/or validity of the information gathered
Skip Patterns
When you want to avoid asking inappropriate questions, you first ask a ‘screening question’ to identify appropriate respondents Those for whom the question is
inappropriate are not asked the question and ‘skip’ forward to the next appropriate question on the instrument
Example
If you want to ask questions about the kinds of beer someone drinks, you would first identify respondents who do not drink beer and have them skip to the next question appropriate to them
Example:1. Do you ever drink beer?
Yes (continue) No (skip to Q. 3)
2. Which brands do you drink? ______________
3. Do you ever drink coffee?
Writing the questions
Levels of measurement The higher the level of measurement, the greater the
ability to use powerful statistics to analyze your dataScale typesResponse options
Yes/no Agree/disagree
(Strongly, somewhat) Numeric responses v. categories
Question formats
Try to develop repetitive response options Increases speed Makes respondent feel more comfortable Reduces errors Boxes, check-offs, white space
Question wording
The wording of questions on any research instrument is crucial. There is usually no more important decision than how a question is asked. Results from surveys have been shown to
be strongly influenced by even subtle differences in question wording.
Instrument format
Matters of crowding, use of white space, inclusion of instructions, typeface, and so on are important.
Test the instrument
Pre-test the instrument Have trained interviewers try out your draft on
each other, Determine whether questions are misleading,
ambiguous, biased Determine how long the instrument takes to
completeUse this information honestly when recruiting
respondents Ask how the respondent interprets the question Allow for open-ended responses where closed-
ended questions have been written
Developing the instrument
Revise and reorganize the instrument based on the feedback from the testing
Try it out on respondents from the population being studied
Make final adjustments Always keep track of the changes you made
Developing the instrument
Obtain IRB approval (if this is an academic study)
Enter the field and gather data Monitor the success of the instrument and make
changes if absolutely necessary Changes must be run by the IRB
A number of concerns influence the structure of the instrument:
First and foremost the essential information needed to answer the research questions/hypotheses
Data collection method Interview, self-administered questionnaire, online v.
paper, etc. Amount and type of instruction, skip patterns,
incentivesThe normal progression of the interview/
questionnaire Developing rapport/trust
Dynamics of respondent memory Question ordering
A number of concerns influence the structure of the instrument:
Respondent facility with language and/or advanced concepts Kids Recent immigrants
Sensitivity of question topics Forms and methods of providing confidentiality
Efficiency in asking questions