Survey Response Rates: Trends and Standards

Karen Donelan, ScDSenior Scientist in Health Policy

Massachusetts General Hospital/Harvard Medical School

AcademyHealth - June 27th, 2006

Overview

What is a response rate? Why do we care about response rates? Trends Does nonresponse = biased response? How should rates be calculated and

reported? How can we improve?

What is a Response Rate?

The number of complete interviews with reporting units divided by the number of eligible reporting units in the sample

Other Rates Cooperation rates - The proportion of all cases

interviewed of all eligible units ever contacted.

Refusal rates - The proportion of all cases in which a sampled unit or the respondent refuses to be interviewed, or breaks-off an interview, of all potentially eligible cases.

Contact rates - The proportion of all cases in which some responsible sampled unit member was reached.

Why Do We Care About Response Rates?

Sources of Error in Surveys Coverage, Measurement, Nonresponse, Processing,

Sampling Practical implications

Generalizability Publishability (often cutoff or major test of quality) Credibility Fundability

What Do We Know About Response Rate Trends?

http://www.census.gov/prod/2000pubs/tp63.pdf

Current Population Survey (CPS)

Behavioral Risk Factors Survey (BRFSS) : Telephone

Survey

Source: Groves et al. 2004

U Michigan: Survey of Consumers

Source: Groves et al. 2004

Why Does Nonresponse Happen?

Who is in the sample? (demographics, SES, lifestyle) # Contacts Schedule of contacts Mode of contact Respondent selection Respondent cooperation (refusal) Incentives ($, benefit) Respondent burden (time, boredom, frustration) Salience Respondent ability to respond to questions Sponsorship Privacy concerns

How Should Rates Be Calculated and Reported? Disclosure

Units drawn and attempted Addresses mailed Phone numbers called Households approached Patients invited

Sample Disposition: what happened? Eligible and ineligible respondents Bad contact information How many attempts, at what intervals # Never reached, # actually contacted # completed, refused, never got a

decision

www.aapor.org

Does Nonresponse = Biased Response? Low response rates may not be problematic Representativeness doesn’t necessarily increase with

response rate Sample representativeness is a function of the

difference between respondents and nonrespondents on the statistic(s) of interest

High response rates can can yield an unrepresentative sample (high nonresponse bias)

Low response rates can still yield a representative sample

Keeter et al. (2000), Curtin, Presser and Singer (2000), Merkle and Edelman (2002)

Measuring Nonresponse Bias

With response rates falling, understanding the impact on study findings is essential

Assess quantitatively if possible, but at least consider qualitatively

Methods for assessing nonresponse bias: Response rate comparisons across groups Follow-up interviews with non-respondents Comparing early vs. late respondents Comparisons to similar estimates from other

sources Post survey adjustment for nonresponse

Ways Not to Approach Nonresponse

Ignore it Obscure it Omit it Fail to measure/comment on

potential associated bias Give up on it

Improving Response Rates

Multiple contacts Multiple modes of contact Interviewer training Advance Notification/Endorsement Incentives Reduce respondent burden Increase relevance to respondent Time and money to get quality

Reporting Response Rates

Know and apply standards (AAPOR, CASRO)

Engage and manage with vendors Structure sample disposition Response and nonresponse analysis Publish enough information to allow others

to do the calculations Peer review: build awareness of standards