How Much is Too Much?How adding information to a question may narrow, instead of expand, understanding

Carol Cosenza

Center for Survey Research

University of Massachusetts Boston

Presented at: QUEST 2007

Ottawa

How We Try to Help R’s Understand

• Give definitions

• Add common examples

• Add less common examples

Definition• The next question is about strenuous tasks done around your home.

By strenuous, we mean any chores or projects that made you feel very tired by the time you finished them. In the past 30 days, on how many days did you do strenuous tasks in or around your home?

• A pneumonia shot, also called the pneumoccoccal vaccine, is usually given only once or twice in a person’s lifetime and is different from the flu shot. Have you ever had a pneumonia shot?

How We Try to Help R’s Understand

• Give definitions

• Add common examples

• Add less common examples

(Common) Examples

• The next question is about vegetables you have eaten, including corn, green beans, spinach, salad vegetables, or any others.

• The next question is about vegetables you have eaten, including vegetables eaten by themselves, mixed in with other dishes, salad vegetables, or any others.

How We Try to Help R’s Understand

• Give definitions

• Add common examples

• Add less common examples

(Less Common) Examples

A health provider is anyone you would see for health care, including eye doctors, mental health workers, and physical therapists.

The Experiments

Wave 1

• Split Ballot design (Spring 2005)• 454 respondents from an RDD sample• 39.6% male• 36.3% with high school education or less• 51.0% over age 50 (mean=51.3 years)

Wave 2

• Split Ballot design (Spring & Fall 2006)• 425 respondents from an RDD sample• 37.9% male• 29.2% with high school education or less• 60.0% over age 50 (mean=53.7 years)

Skin Growth

VERSION 1:

In the last 12 months, did you see or talk to a doctor about a tumor, cyst or growth of the skin?

VERSION 2:

In the last 12 months, did you see or talk to a doctor about any kind of growth on your skin?

Skin GrowthVERSION 1: In the last 12 months, did you see or talk to a doctor about a tumor, cyst or growth of the skin?

VERSION 2: In the last 12 months, did you see or talk to a doctor about any kind of growth on your skin?

Version 1(tumor, cyst, or growth)

Version 2(any growth)

YES – had growth 8.7% (n=13) 18.2% (n=27)

p=.015

Problems Breathing

VERSION 1:

During the past 12 months, did you see or talk with a doctor about any problems with coughing or breathing including asthma, pneumonia, or chronic lung disease?

VERSION 2:

During the past 12 months, did you see or talk with a doctor about any problems with coughing or breathing?

Problems Breathing

Version 1(with examples)

Version 2(no examples)

YES – had problems 16.7% (n=35) 19.9% (n=43)

VERSION 1: During the past 12 months, did you see or talk with a doctor about any problems with coughing or breathing, including asthma, pneumonia, or chronic lung disease? VERSION 2: During the past 12 months, did you see or talk with a doctor about any problems with coughing or breathing?

p = n. s.

Arthritis & Joint PainDuring the past 30 days, have you had symptoms of pain, aching, or stiffness in or around a joint?

Did your joint symptoms first begin more than 3 months ago?

Have you ever seen a doctor or other health professional for these joint symptoms?

Have you ever been told by a doctor or other health professional that you have some form of arthritis, rheumatoid arthritis, gout, lupus, or fibromyalgia?

Arthritis & Joint Pain1. During the past 30 days, have you

had symptoms of pain, aching, or stiffness in or around a joint?

Did your joint symptoms first begin more than 3 months ago?/Have you ever seen a doctor or other health professional for these joint symptoms?

2. Have you ever been told by a doctor or other health professional that you have some form of arthritis, (rheumatoid arthritis, gout, lupus, or fibromyalgia)?

1. Have you ever been told by a doctor or other health professional that you have some form of arthritis, (rheumatoid arthritis, gout, lupus, or fibromyalgia)?

2. During the past 30 days, have you had symptoms of pain, aching, or stiffness in or around a joint?

Did your joint symptoms first begin more than 3 months ago?/Have you ever seen a doctor or other health professional for these joint symptoms?

Number of times talked to a doctor

VERSION 1:

In the past 12 months, how many times have you seen or talked on the telephone with a {DOCTOR} about your {health}?

VERSION 2:

In the past 12 months, how many times have you seen or talked with a {DOCTOR} about your {health}?

Number of times talked to a doctorVERSION 1: In the past 12 months, how many times have you seen or talked on the telephone with a {DOCTOR} about your {health}?

VERSION 2: In the past 12 months, how many times have you seen or talked with a {DOCTOR} about your {health}?

n Range Mean

Version 1 (phone) 350 0 - 60 times 3.34 (SD=6.31)

Version 2 (no phone) 341 0 – 50 times 3.72 (SD=4.93)

Number of times talked to a doctor

When number of times talking to doctor is top-coded at 12:

p=.008

Mean Standard Error of Mean

Version 1 (phone) 2.71 (SD=3.10) .166

Version 2 (no phone) 3.36 (SD=3.27) .177

VERSION 1: In the past 12 months, how many times have you seen or talked on the telephone with a {DOCTOR} about your {health}?

VERSION 2: In the past 12 months, how many times have you seen or talked with a {DOCTOR} about your {health}?

Number of times talked to a doctor

• The original intent of adding “telephone” was the fear that Rs who

only talked on the phone but not say “yes” to “seen or talked with”

• This test shows that there is a difference – in the opposite way.

• Respondents seem to be distracted by “telephone” in Version 1

• This result was validated by cognitive testing.

VERSION 1: In the past 12 months, how many times have you seen or talked on the telephone with a {DOCTOR} about your {health}?

VERSION 2: In the past 12 months, how many times have you seen or talked with a {DOCTOR} about your {health}?

Conclusions• Hard to know whether the extra “stuff” we add to questions

helps broaden a respondent’s thinking or narrows it.

• There is some evidence that adding more “serious” health conditions as examples in a question may make it harder for respondents to say yes

• Split Ballot experiments help us figure this out.

• Cognitive testing, behavior coding and other evaluative techniques can provide us with additional clues.

Thank you