Introduction to survey question testing

Debbie Collins

• Survey terminology

• Important ideas and assumptions

• Why we might want to pretest survey questions

• Overview of testing options

Content

Terminology

44

“A survey is a systematic method for gathering information from (a sample of) entities for the purpose of constructing quantitative descriptors (statistics) of the attributes of the larger population of which the entities are members.”

Groves et al (2004) Survey Methodology. Wiley Interscience.

What is a questionnaire?

Means of collecting information

Means by which researchers can quantify, e.g. behaviours and opinions

A sequence of questions

Script for a conversation between the researcher & respondent

The script is standardised

Question stem Overall, how satisfied are you with your life

nowadays?

Answer scale 1 10

Not at all Completely

happy happy

Multi –item scales/ indexes/ latent variables Used to represent complex constructs that

can’t be summarised in a single Q, e.g.

satisfaction, attitudes, health, personality

Filtering/ routing/ skip patterns Instructions that determine who gets asked

which questions

Instructions Provide additional information e.g. to only

select one answer

Terminology

Important concepts and ideas

• The process by

which observations

are collected and

recorded to become

data

• Two important

considerations:

– Level

– Accuracy

Measurement

Common measurement levels in social surveysMeasurement level Example question

Nominal What is your sex?

1 Male

2 Female

Ordinal How easy or difficult would you find explaining the following terms to a

friend…‘Interest rates’?

1 Very easy

2 Fairly easy

3 Fairly difficult

4 Very difficult

Interval How many hours per week does (child) spend being looked after by (your

childcare provider) during school term time?

Record hours

Measurement accuracy -Reliability and validity high

Measurement accuracy –Reliable but not valid

Measurement accuracy –valid but not reliable

Standardisation

“The goal of standardization is that each respondent be exposed to

the same question experience, and that the recording of answers

be the same, too, so that any differences in the answers can be

correctly interpreted as reflecting differences between respondents

rather than differences in the process that produced the answer.”

Fowler F. J. & Mangione T.W. 1990. Standardized Survey Interviewing: Minimizing

Interviewer-Related Error. ASRMS Vol: 18. Sage.

Why pretest survey questions and questionnaires?

Components of a ‘good’ survey questionnaire

– Valid

– Reliable

– Sensitive

– Unbiased

– Complete

Components of measurement error

Questions misunderstood

that cannot be answered (accurately)

that respondents will not answer

Interviewers do not read questions as worded

probe directively

record answers inaccurately

R’s memory

Survey communication process

Answer

Question

wording

Interview setting

Interviewer

R’s judgement

Response

options

Wider

context

Why do we need to test survey questions or instruments?

• To assess reliability, validity, bias, sensitivity

• To ensure that respondents:

– have the information being sought

– understand questions, concepts or tasks in a consistent way

– feel able and willing to answer

• To ensure questionnaire is user-friendly

Pretesting & evaluation methods

Questionnaire evaluation methods

Quantitative

• Behaviour coding

• Analytical methods e.g. Item Response Theory, Latent Class Analysis

• Split ballot experiments

• Field piloting

Qualitative

• Focus groups

• Cognitive Interviewing

• Interviewer debriefing

• Respondent debriefing

• Usability testing

Desk based review

• Questionnaire Appraisal System

• Expert review

Mixed methods

• Eye tracking with cognitive interviewing

• Field piloting with respondent debriefing/web probing

• Spilt ballot experiment with cognitive interviewing

Traditional field piloting

Usually testing whole survey process

Questionnaire usually examined in terms of:

length

flow

respondent interest

ease of administration

response

May detect overt problems but not reasons for them

Some problems may go undetected…

Choosing pretesting method(s)

Aims of pretest/evaluation

Resources available (e.g. time, money, your skills)

Type(s) of evidence you need

Further resources

Presser, S., Rothgeb, J.M, Couper, M.P, Lessler, J.T., Martin, E., Martin, J., Singer, E. Eds (2004) Methods for Testing and Evaluating Survey Questionnaires. Hoboken, NJ: Willey.

Madans, J., Miller, K., Maitland, A., Willis, G. Eds. (2011) Question Evaluation Methods: Contributing to the Science of Data Quality. Hoboken, NJ: Willey

Fowler, F.J. & Mangione, T.W. (1990) Standardized Survey Interviewing: Minimising Interviewer-related Error. Newbury Park, Sage.

Fowler, F.J. (1995) Improving Survey Questions: Design and Evaluation. Thousand Oaks, CA: Sage.

Oksensburg, L., Cannell, C., Kalton, G. (1991) New strategies for testing survey questions. Journal of Official Statistics, 7, 349-365.