Survey Design

So Many Surveys

Political ‘push polls’– Trying to influence, not collect data

Marketing polls– Some are just advertisements

News Polls

How to write a good survey

Think in terms of ‘questionnaire’, not questions– How are questions related to each other?

Have a clear topic– A research question– Know what concepts you are measuring

Think in terms of hypotheses, IVs and DVs – Every question should have a clear reason for

inclusion

The Instrument – the whole package

Make it look/sound appealing– Clear directions that make it sound

interesting/important– shorter is always better (i.e. 15 minutes)

Consistency is good– Use similar response formats for questions

Introductions and Instructions

What is the study about? Informed and Voluntary Consent

– Institutional Review Boards Contact information for researcher Let them know they can have the results

Example of InstructionsThis survey is part of a research project about your parish directed by Dr. Matthew Loveland of Le Moyne College. Completing the survey indicates your voluntary, informed consent to participate in the project. Your responses are anonymous and confidential. If you are under 18, please do not complete the survey. Please circle the number to indicate your answer.

1. How often do you attend mass at this parish?

1. Daily 2. Weekly

3. Few times a month 4. Few times a year

5. I’m a visitor today 6. I don’t know

Thanks for your participation. More information about the research project can be found at: www.lemoyne.edu/parishproject. With questions, call Dr. Loveland at 315 445 5462, or by Email at lovelam@lemoyne.edu

Question Order

Early questions make identities salient– Example: Political ID and Political Issues

So, Put space between the questions Early questions should not be threatening

– How often do you watch pornography?– Have you ever been convicted of a crime?

Early questions should not be boring– How old are you?

Of course, this all depends– Different survey settings call for different strategies– It’s a judgment call

Bad question order example (and probing)

1. Are you of the Catholic faith? 1. Yes 2. No (end) 3. Not sure (Do Not Read) (end) 2. Aside from weddings and funerals, how often do you attend Mass? (Do not Read Responses, Use Categories as Probes if Necessary) 1. Never (skip to question 9) 2. once a year or less

3. only on religious holidays 4. several times a year 5. once a month 6. 2 to 3 times a month 7. Every week

8. More than once a week 9. Not sure

Do previous questions influence answers?

The following set of items asks about your attitudes about church leadership. For each of these statements, please tell me whether you strongly agree, somewhat agree, somewhat disagree, strongly disagree, or have no opinion. [Read No Opinion/ Don’t Know] 27. Pope Benedict is doing a good job leading the Church 1. Strongly agree 2. Somewhat agree 3. Somewhat disagree 4. Strongly disagree 5. No Opinion/DK 28. The American bishops are doing a good job leading the U.S. Church. 1. Strongly agree 2. Somewhat agree 3. Somewhat disagree 4. Strongly disagree 5. No Opinion/DK 29. [Skip if q24=1]The pastor of the parish I attend is doing a good job leading our congregation. 1. Strongly agree 2. Somewhat agree 3. Somewhat disagree 4. Strongly disagree 5. No Opinion/DK 30. I think it would be a good idea if parishes were to choose their own priest from among available ordained priests. 1. Strongly agree 2. Somewhat agree 3. Somewhat disagree 4. Strongly disagree 5. No Opinion/DK

Social Desirability

Sensitive Topics– sex, drugs, crime, embarrassment, church

attendance, etc…. People will sometimes tell you who they wish

they were, not who they really are Advice: write questions so that undesirable

answers are made to sound ok.

Social Desirability Example

1. How often do you attend mass at this parish?

1 Daily2 Weekly3 Few times a month4 Few times a year5 I’m a visitor today6 I don’t know

Open and Closed Ended

Closed ended: give responses Open ended: let respondent give comments If closed:

– Mutually exclusive and exhaustive Allow don’t know / not sure ?

If open:– Will have to code after data collection– Might effect length of interview

Could be quicker or slower

Open vs. Closed Example

• How many people do you consider close friends? ________

• How many people do you consider close friends?

1. 02. 1 to 53. 6 to 104. More than 10

Avoid Double Barreled Questions

A single question that really asks two separate things.– Watch out for this if you use the word “and”– Try not to preface any question with something

people might disagree with Do you think the United States government

should raise taxes and spend more money on education?

Double Barreled

The government should guarantee jobs to insure a basic standard of living 1. Strongly Agree 2. Agree 3. Neutral 4. Disagree 5. Strongly Disagree

Skip Patterns

Contingency Questions– Answer to a question sends you down a different

path If yes, go to question number X

Use it to shorten survey for respondent Too many can get complicated

– Use with caution on a self administered survey

Skip Pattern Example

These questions are about your previous place of worship. If you do not have a previous place of

worship, you may skip to question # 14.

11. How long were you involved at your previous place of worship?

1. Less than 5 years 2. 5 to 10 years 3. 11 to 20 years 4. 21 to 40 years 5. More than 40 years 6. No prior place

Negative or Biased Terms

Negative terms may be confusing The death penalty should be illegal

– SDA DA N A SA– What does it mean if you disagree?

Avoid Biased Terms– Biased questions may encourage a certain

answer– Depends on connotation of words

“Forbid” versus “Not Allow”

Do you think the U.S. should forbid public speeches against democracy?

Do you think the U.S. should allow public speeches against democracy?

“Americans are much more willing to not allow speeches than they are to forbid them.”

Some general tips

If asking about the past, use a time period people can remember– “In the last week…”

Only ask about personal experiences– Don’t ask about other people’s impressions

Beware of hidden contingencies– In the past month, have you crossed the street to

avoid someone you found frightening?

Response Rate

% of those sampled who participated Larger RR usually means more

representative data General Guidelines, no clear rule

– Shoot for 70%– Happy with 50%– Don’t be surprised with 30% or lower

Methods of Administration – Mail Survey

Self Administered– Relies on Participant Enthusiasm

Low Response Rates Appearance is Most important in this case Give an Incentive? Slow

Send a prior letter, maybe call Send the survey Send a reminder letter to those who haven’t replied Contact again

Methods of Administration –Phone Survey

Random Digit Dial (RDD)– Very Fast– Very Expensive

Participation gets worse every year Cooperation Rates

– 20% to 30%, or lower– Statistically, in 30% range is not that different from

in 60% range as long as sample is representative

Methods of Administration-In person interview

Best response rate & Most Expensive– Contact ahead of time to arrange meeting– Sit with respondent while collecting data

Well trained interviewers– Dress, Race, Gender, etc.– Threatening Situations

Mixed Methods– Computer Assisted– Self administered portions of an in person interview

Methods of Administration-Web Survey

Still a new method, very few clear guidelines Visually, I’d say follow same guidelines as a

a mail survey How do you get your sample?

– Do you trust online identities? Can be very affordable

– Survey Monkey

General comments on survey error

Sample surveys are way to generate estimates

There are multiple sources of error– Sampling error– Low response rates– Poorly written questions