1

Survey ResearchUAPP702: Research Design for Urban & Public PolicyClass Notes

Based on Earl Babbie, The Practice of Social Research

Danilo YanichCenter for Community Research & ServiceSchool of Public Policy & AdministrationUniversity of Delaware

2

Topics appropriate for survey research

Used for exploratory, descriptive & explanatory purposes

Probably best method to collect original data for describing a population too large to observe directly

3

Guidelines for Asking Questions Choose appropriate question form: two options

Open-ended--R is asked to provide own answer to question

Closed-ended--R is asked to select an answer from among a list provided by researcher

Closed-ended questions require the categories of answers to be exhaustive and mutually exclusive

4

Guidelines for Asking Questions, p.2

Make items clear

Babbie uses examples of "proposed peace plan"...Which peace plan?

And, employment "last week"...Monday through Friday or Sunday through Saturday as the Census Bureau defines week?

5

Guidelines for Asking Questions, p.3

Avoid double-barreled questions

Example: "Did you walk to work or carry your lunch?" …..Yes or No

Regardless of the answer you get from the respondent, you don’t know what information you have.

6

Double-barreled to the max Question on 1986 Harris poll re: American

public opinion about Libya

If Libya now increases terrorist acts against the U.S. and we keep inflicting more damage on Libya, then inevitably it will all end in the U.S. going to war and finally invading that country which would be wrong.

Respondents were given opportunity to answer “Agree”, “Disagree” or “Not sure”

7

Double-barreled to the max, p.2

Notice the elements contained in the complex statement

Will Libya increase its terrorist acts against the U.S.? Will the U.S. inflict more damage on Libya? Will the U.S. inevitably or otherwise go to war with Libya? Would the U.S. invade Libya? Would that be right or wrong?

These elements offer the possibility of numerous points of view—far more than the three alternatives offered to the R’s

8

Guidelines for Asking Questions, p.4

Respondents must be competent to answer Babbie uses example of asking students to indicate the percentage of fees to be

used for a long list of activities, about which the respondents have little knowledge

Respondents must be willing to answer There may be a risk to the respondent in answering the question

Reluctance may be due to nature of the information sought, but good science and human subjects requirements dictate caution.

Example: asking a staff member of a non-profit agency her/his opinion about the leadership qualities of the administrator

Using a technique that guarantees anonymity or confidentiality greatly increases R’s willingness to answer questions

9

Guidelines for Asking Questions, p.5 Questions should be relevant to the Respondent

Babbie uses example of fictitious person, Tom Sakumoto

9% of R's said they were familiar with him. Shows R’s tendency to be helpful to researcher

Short items are best

Avoid negative items Possibility of misinterpretation is great

Babbie uses example of asking R to say if they agree/disagree with statement that "The U.S. should not recognize Cuba".

Often, R’s will “read over” the word “NOT”

10

Guidelines for Asking Questions, p.6

Avoid biased items and terms

Questions should not encourage a particular response or discourage another.

Be wary of "social desirability" of answers

Attitudes re: government spending for programs in the General Social Survey, 1989

More support Less support

Assistance to the poor Welfare reform

Halting rising crime rate Law enforcement

Dealing with drug addiction Drug rehabilitation

Solving problems of big cities Assistance to big cities

Improving conditions of blacks Assistance to blacks

Protecting social security Social security

Surveys “gone wild” National Park Survey

11

NRA Survey 1

NRA Survey 2

NRA Survey 3

12

Questionnaire Construction General questionnaire format should be

uncluttered

Squeezed-together questionnaires are disastrous

Formats for respondents should be clean and clear

13

Questionnaire Construction, p.2

Contingency question

A question that only applies to the R’s who answered a previous question in a particular “qualifying” manner

Make sure that the directions for answering the subsequent questions are clear for the respondent

14

Contingency question, example

Have you ever been abducted by aliens? [ ] Yes [ ] No (go to ques. # 4)

If yes: Did they let you steer the ship?[ ] Yes[ ] No

If yes: How fast did you go?[ ] Warp speed[ ] Weenie speed

15

Questionnaire Construction, p.3

Matrix questions

Advantages Uses space efficiently, R's will probably complete it faster

Format may increase comparability of responses to different questions

Disadvantages May encourage researcher to use matrix format when

another form would be better

Can foster a response-set among R's in which they may develop a pattern

16

Matrix question, example Contact with criminal justice system

Have you ever…Yes No Ref/DK

Been a defendant in a criminal case 1 2 7

Been a witness in a criminal case 1 2 7

Reported a crime to the police 1 2 7

Been the victim of a crime 1 2 7

17

Matrix question, example 2

Using the A, B, C, D, F grading system that we learned in school, please rate the importance of the following services in this hotel.

Hotel service A B C D F Ref/DK

Dining services 5 4 3 2 1 7

Laundry service 5 4 3 2 1 7

Room service 5 4 3 2 1 7

18

Questionnaire Construction, p.4

Ordering questions in a questionnaire

The appearance of one question can affect answers given to later ones

As remedy, do not make the questions random, that will look chaotic to the R and confuse the issue

Safest solution is sensitivity to the problem

19

Questionnaire Construction, p.5

Questionnaire Instructions

Must be clear, concise and COMPLETE

Pre-testing the questionnaire

ALWAYS PRE-TEST!!!!

20

Comparing different survey methodsChoosing among mail questionnaire, personal interview and telephone survey

Factors influencing coverage and information gathered

Mailed questionnaire

Personal interview

Telephone survey

Lowest relative cost 1 3 2

Highest % return 3 1 2

Highest accuracy of information 2 1 3

Largest sample coverage 3 1 3

Completeness, including sensitive material 3 1 2

Overall reliability & validity 2 1 3

Time required to gather information 3 2 1

Ease of gathering information 1 3 2

Total number of rankings—1,2,3 2,2,4 5,1,2 1,5,1

Note: 1=most favorable ranking, 2=intermediate ranking, 3=least favorable rankingSource: Delbert Miller. Handbook of Social Research Design and Measurement, 5th Edition,1991, p. 168

21

Strengths of survey research Useful in describing large populations

Are flexible in that you can ask many questions regarding your topic

Standardized questionnaires have strength regarding measurement generally

Survey researcher must ask the same question of all subjects and…

Impute the same intent to all respondents giving a particular response

22

Weaknesses of survey research Standardized questionnaire items often

represent the least common denominator in assessing people’s attitudes, etc.

Does not deal well with the context of social life

Can be inflexible…cannot change the survey instrument if field conditions change

23

Weaknesses of survey research, p.2

Subject to artificiality

A person giving a conservative answer to a questionnaire does not necessarily mean the person is conservative

Artificiality has two aspects

Topic of study may not be amenable to measurement through a questionnaire

Studying the topic may affect it…

Asking someone whether they think the governor should be impeached when R may have given no thought to it until asked for the opinion

24

Ch 14: Quantifying data , Coding System to translate the responses to questions to

numeric expressions that the computer can process

Two basic approaches to coding process

Begin with developed coding scheme Babbie uses occupation as example...but must use scheme

that is appropriate for research question.

Generate codes from data that you collected.

25

Using a developed coding scheme, example

Occupation

The question to the R would be: What is your occupation?

The R’s response is then placed in one of several categories that you have already identified

26

Using a developed coding scheme, example, p.2

What is your occupation?

Let’s say that the answer was “nurse”

Scheme 1: by type Professional, managerial, clerical, semi-skilled, etc.

In this scheme, the R’s occupation would be placed in the “professional” category

Scheme 2: by sector of the economy Manufacturing, health, education, commerce, etc.

In this scheme, the R’s occupation would be placed in the “health” category

27

Using a developed coding scheme, example, p.3

Note: record the response of the R verbatim

Place the response into one of your pre-determined categories during the data manipulation phase

Use “recode” facility in SPSS

Remember, you can always “aggregate” data

Cannot “disaggregate” data

28

Using a developed coding scheme, example, p.4

What is your occupation?

Verbatim responses

R1: “nurse” R2: “sell shoes” R3: “build cars” R4: “manager at Wendy’s” R5: “physician” R6: “computer programmer” R7: “retired” R8: “soldier”

Type Sector

professional health

sales commercial

semi-skilled labor commercial

managerial commercial

professional health

professional information tech

retired retired

military military

29

Generating codes from data collected, example

Student responses to “biggest problem facing college today”

Responses can be coded as “academic” or “non-academic”Academic Non-academic

Tuition is too high x

Not enough parking spaces x

Faculty don’t know what they’re doing x

Advisors are never available x

Not enough classes offered x

Cockroaches in the dorms x

Too many requirements x

Cafeteria food is infected x

Books cost too much x

Not enough financial aid x

30

Codebook construction End product of the coding process is the conversion of data

items into numerical codes

These codes represent attributes composing variables which....

Are assigned locations within a data file

Location means the specific column of the data file where, for example, the responses for “gender” would occur

A codebook is a document that describes the locations and lists the assignment of codes to the attributes composing those variables

It is the fundamental document of the research process

31

Codebook serves two functions

Primary guide for the coding process

Guide to locating variables in the data file during analysis

32

Example of Coding InstructionsPublic Attitudes: Crime, Drugs & Public ServicesEnterprise Community/Wilmington/Statewide Survey

Variable Name (column location) Value label

1. ID# (1-3) Continuous (001 through n)

2. Area (4)1= Westside, 2= West Center, 3= Delaware Avenue4= Southwest, 5= Eastside, 6= Northeast7= Northwest, 8= New Castle, 9= Kent, 10= Sussex

3. Defendant in criminal case (5) 1=Yes, 2=No, 7=Ref/DK, 8=NA

4. Witness in a criminal case (6) 1=Yes, 2=No, 7=Ref/DK, 8=NA

5. Report crime to police (7) 1=Yes, 2=No, 7=Ref/DK, 8=NA

6. Victim of crime (8) 1=Yes, 2=No, 7=Ref/DK, 8=NA

33

Data Cleaning...a fundamental activity The process of detecting and correcting coding errors

Two types

Possible-code cleaning--for any given variable there are only a specified set of codes possible

Example, gender--computer program would "beep" when an erroneous code is entered and refuse the code

Family Court experience with gender codes

Contingency Cleaning--process of checking only those cases that should have data on a particular variable do in fact have such data