introduction to questionnaire design

Training Course on Basic Statistics for Research

August 24-28, 2009

STATISTICAL RESEARCH AND TRAINING CENTERJ and S Building, 104 Kalayaan Avenue, Diliman, Quezon City

Introduction toQuestionnaire Design

2

Statistical Research and Training Center Training Course on Basic Statistics for ResearchAugust 24 - 28, 2009

Questionnaire

refers to any list of questions, which can be self-administered by the respondent or read to the respondent by an interviewer

3


Process of Constructing the Questionnaire

1. Decide on the type of questionnaire that will be constructed.

2. Write down the questions.

3. Combine all questions and decide on which order to present them.

4. Write the introductory statement or cover letter.

5. Write the instructions to interviewers or respondents.

6. Design the Form.

7. If the study is a nationwide survey, translate the questionnaire to other major languages.

8. Pretest the questionnaire.

4


Strategies in Writing the Questions

Use the research objectives as the basis for your questions.

You may borrow questions that were already asked in past studies.

Anticipate the possible problem that you might encounter in extracting information from the respondents.

Consider the resources available. Consider the medium used in acquiring the responses Consider the level of measurement required by the

analysis that you have in mind.

5


Types of Questionnaire

Structured wording and order of questions are uniform for all respondents

Unstructured wording and order of questions can vary for different subjects; usually used for qualitative studies like FGDs, case studies, etc.

6


Types of Questions in Survey Research

Open-Ended Questions

Closed-Ended Questions

7


Open – ended :response categories are not specified; the respondents are free to answer as they please

Example:We would like to get your opinion regarding the effectiveness of the project in your barangay.

a. What do you think are the strengths of the project? Why did you say so?

b. How do you think can the project still be improved to meet its objective?

Types of Questions

8


Closed - ended respondent selects one or more of the specific categories provided by the researcher or the question is asked as an open question and the interviewer allocates the answer to the appropriate code category.

Example:

Do you practice family planning?

__ Yes __ No

If YES, what methods did you use?

__ Condom __Pills

__ Ligation __IUD

__ Injectables __Natural

__ Others, specify ____________________

Types of Questions

9


Advantages of Closed-Ended Questions

The answers are standard, and can be compared from person to person.

The answers are much easier to code and analyze, and often can be coded directly from the questionnaire.

A respondent who is unsure about the meaning of the question can often tell from the answer categories what is expected.

Irrelevant responses are avoided.

10


It is very easy for a respondent who does not know the answer or has no opinion to try to guess the appropriate answer or even to answer randomly.

The respondent may feel frustrated because the appropriate category for his/her answer either is not provided at all or is not provided in sufficient detail .

Variations in answers among the different respondents may be eliminated artificially by forced-choice responses.

Disadvantages of Closed-Ended Questions

11


They can be used when all of the possible answer categories are not known, or when the investigator wishes to see what the respondent views as appropriate answer categories.

They allow the respondent to answer adequately, in all the detail he/she likes, and to qualify and clarify his/her answer.

They can be used when there are too many potential answer categories to list on the questionnaire.

They allow more opportunity for creativity and self-expression by the respondent.

Advantages of Open-Ended Questions

12


They may lead to collection of worthless and irrelevant information.

Coding is often very difficult and subjective, leading to low inter-coder reliability.

Open-ended questions require superior writing skills, better ability to express one’s feelings verbally, and generally a higher educational level than do closed-ended questions.

Open-ended questions require much more of the respondent’s time and effort, and may engender a high refusal rate.

Disadvantages of Open-Ended Questions

13


Examples of Pitfalls in Question Construction

Incorrect ordering of questionsQ7. How would you rate the scent of lotion X?Q8. What do you think of lotion X?

Improvement: ________________________________________ ________________________________________

Double-barreled questions

“Does your department have a special recruitment policy for ethnic minorities and women?”

Improvement: ____________________________________ _____________________________________

14



Words with double meaning

“Does your household participate in the activities of the barangay?”

The National Statistics Office defines the household as an aggregate of persons generally but not necessarily bound by ties of kinship, who live together under the same roof and eat together or share common the household food.

15



Difficult words or technical terms

“What proportion of your evening viewing time do you spend watching news programs?”

Improvement: ___________________________________________ ___________________________________________

16


Presuming questions

“Do you think the Clean Air Act is being implemented properly by local authorities?”

Improvement: _____________________________________ ____________________________________


17


Sensitive or Threatening Questions

“Do you masturbate? Yes _____ No _____ If yes, how often? Once a week, once a month, everyday?”

Improvement: ___________________________________________ ________________________________________


18


Unrealistic questions

“What brand of perfume do you think you will be using three years from now?”

Improvement: ___________________________________________

__________________________________________


19


Incomplete/non-exhaustive listing

“Did you learn about the brand from TV, radio, newspaper, or friends?”

Improvement: ____________________________________________

____________________________________________


20


Biased Wording

“Do you think that decent, low-cost funerals are sensible?

Improvement: ___________________________________________ _________________________________________


21


Leading questions

“The majority of physicians in the Philippines feel that smoking is harmful; do you agree?”

Improvement:__________________________________ ______________________________________


22



Double Negatives especially in agree/disagree items

“Please tell me whether you agree or disagree with the following statement about teachers in public schools: Teachers should not be required to supervise students in the halls, the lunchroom and school parking lots.”

Improvement: ____________________________________________

____________________________________________

23


Dangling Alternatives

“Would you say that it is very often, frequently, seldom, or never that you and your boss disagree about advertising campaigns?”

Improvement: _______________________________________________

_______________________________________


24



Insufficiently Specific or Unidimensional

“Are you satisfied with your canteen?”

Improvement: ___________________________________________

__________________________________________

25


Hypothetical Questions

“If you are the president of the Philippines, would you push through with the Charter Change?”

Improvement: _______________________________________________

_______________________________________


26


Vague questions

“How often do you wash your hair?”

Very often

Often Not too often Never

Imrovement: ______________________________________ __________________________________________


27


Example to Avoid Unrealistic Questions

Getting information on number of hours spent shopping

(i) Did you go shopping in the past week? (IF YES) How many hours did you spend shopping in the past week?

(ii) On a typical week, how many hours did you spend shopping? Or, on the average, how many hours do you spend shopping in a week?

(iii) Do you go shopping? (IF YES) On your last shopping trip, how many hours did you spend shopping?

28


Use of Simple Words

WORD SIMPLER WORDacquaint inform, tell

assist helpconsider thinkinitiate begin, startmajor important, chief, main,

principalmaterialize come about, happen, occurpurchase buysufficient enoughterminate end

29


Exercise:

Evaluate the following statements:1. According to the Chief Justice of the Supreme Court, the court

should be free of bribery and corruption.2. Do you think that the Congress, as an institution, is definitely free

from external controls, influence or pressure?3. Does the government agency have criminal and civil cases?4. What is the nature of the complaint against the Chief of the

Division?Have you ever filed an administrative complaint against a Division Chief?

5. Do you accept money for a favorable decision? If yes, how often?

6. Approximately how many administrative cases did you have in your agency from 2005 to 2008?

30


Filter and Contingency Questions

A contingency question is a question whose relevance to the respondent is determined by the response to an earlier filter question.

There are some questions that are not relevant to all of the respondents in the study, particular if the population under study is very heterogeneous.

Instead on using separate questionnaires for the different subgroups, only one questionnaire containing

filter and contingency questions is used.

31


Possible Format of Filter and Contingency Questions

Instructions telling which to answer next in the event of the category being chosen are placed beside each response category.

In the past two months, did you have any illness that kept you in bed, indoors, or away from your usual activities?

Yes …………(ASK A)……………1

No…………..(SKIP TO Q.25)……2

Or

Yes…………………….1 ASK A

No ……………………. 2 SKIP to Q.25

32


Possible Format of Filter and Contingency Questions

Used boxed contingency questions with no arrows leading to them.

Have you ever had a miscarriage?

( ) Yes

( ) No

If YES: How old were you when you had your first miscarriage? ____________

33


Response Category Format

Response categories should be exhaustive. Response categories should be mutually exclusive. Response categories should be made easy to data collectors,

respondents, or coders to select appropriate options. Response categories should include the “Other” option with

sufficient space for writing answers. Response categories should anticipate analytical needs and

enable the collection of data that are suitable for those analyses.

It should be decided whether single or multiple answers are to be allowed.

34


Response Category Format

Number of Categories There is no optimal number of categories. 5 to 6 categories are commonly used

Odd or Even Number of Categories If odd, usually the middle category is the neutral position. If even, there will be no neutral category. Thus, the

respondent is forced to take a position.

35


Some Types of Category Formats

Verbal rating scale This requires a respondent to indicate their position by

selecting among verbally identified categories. Examples:

Strongly agree, Agree, Neither Agree nor Disagree, Disagree, Strongly Disagree

Very important, Important, Somewhat Important, Not at all Important

Excellent, Good, All right, Poor, Bad

36



Graphic Rating Scale This requires a respondent to indicate his position on a

continuum which ranges from one extreme of the attitude in question to the other extreme.

Since the respondent is not using a predetermined set of response categories, the researcher cannot determine how a particular respondent has subdivided the continuum.

Example: “To what extent do you feel employees should have a voice in determining performance evaluation content?”

Employees should make Performance evaluation matters

the decisions about 0 1 2 3 4 5 6 7 8 9 10 should be left entirely

performance to the Administration

evaluation matters

37



Rank-Order Scale This involves having the respondent rank various objects with respect to

the attitude in question. This is actually a forced scale. It is possible that respondent actually dislikes all of the objects they were asked

to rank according to a favorable characteristic. This attitude is not captured by the scale.

Example: The following are some of the problems faced by the local government. Please rank them in terms of importance, from 1 (most important) to 5 (least important).

_____ Pollution

_____ Garbage

_____ Traffic

_____ Drug Addiction

_____ Crime

38



Checklist Question provides more than two alternatives from which the respondent can

choose as many responses that apply to them. Example: What kind of ball sports do you like to play? basketball volleyball football tennis baseball water polo

others, pls. specify _____________

39


The order of questions should be in logical sequence so that the trend of thought of the respondent is not broken.

The first part of the questionnaire are on Identification Information consisting of the geographic identification, interviewer’s record and the information about the sample.

Sensitive questions are placed towards the end of the interview schedule or questionnaire.

Question Order

40


Approaches to Question Sequence

Funnel Approach Uses a general-to-specific question flow.

Work Approach Requires that difficult to answer questions are placed deep

inside the questionnaire.

Sections Approach Arranges the questionnaire into sections or topic category

divisions that are dictated by the objectives of the study

41


Approaches to Question Sequence

Quintamensional Design This 5-question approach was developed by George Gallup to

cover the most essential features of an opinion:

Open-ended question on awareness/knowledge of the issue

Open-ended question on the attitudes toward the issue

Closed-ended question on attitude towards specific issues

Open-ended question on reason behind attitude

Closed-ended question on the intensity of attitudes

42


Questionnaire Lay-out & Design

Flow of questions should be respondent-friendly .

Vary questions by length and type.

Place scale items according to response required.

Separate reliability-check question pairs.

43


Physical Design of a Questionnaire

(1) The appearance of the form should be simple and well spaced;

(2) The size and type of font used should be readable;

(3) The font size and type used for questions and directions should be different.

44


Basic Contents of the Cover Letter/Introduction

Identification of the persons or organization conducting the study

Explanation of the importance of the study and reason/s why it is being conducted

Reason why the respondent should be the one to answer the questionnaire

Assurance of no right or wrong answers Guarantee the confidentiality of answers

45


Example of an Introduction

Good morning/afternoon/evening. I am Josie Almeda from the UP School of Statistics Research Foundation. This study seeks to know how Filipinos feel about various issues and problems affecting their lives. This information is valuable both for planning and for scientific research. Your household is very important to this survey because it represents hundreds of others which are not in our sample. Everything you tell us will be strictly confidential. Your name will be in no way connected to the findings of this important study.

46


The Pretest

Objective: To determine if there are errors in the questionnaires or areas that need improvement such as ordering of questions, wording, and the natural flow from one section to another.

Respondents: No rules on who the respondents should be in the pretest and on the size of the sample.

47


Types of Pretest

Participating Pretest Respondents are told that this is a practice run, and are asked

to explain their reactions and answers.

This allows for a very detailed probe on certain questions such as:What did the whole question mean to you? What did the (term) make you think of?

Undeclared Pretest The respondent is not told that this is a questionnaire under

construction, and the interviewer plays it straight.

48


Evaluating the Pretest

Before the pretest, tell interviewers to write marginal comments on the questionnaire.

Conduct an oral debriefing after the pretest so that the interviewers can report on their experience.

Some of the issues that can be discussed are:Were there questions that made the respondent uncomfortableWhich questions have to be repeatedDid the respondent misinterpret any questionWhich questions were the most difficult or awkward to readDid any of the sections bore the respondentWere there questions that respondent found stupidWere there any section in which the respondent would have liked the opportunity to say more

49


Evaluating the Pretest

Review answers to open-ended questions

Tabulate the number of no responses, unable to answer, and don’t knows for each question.

Investigate questions that yield too many such answers.

Watch out for response sets.

Look at the variability of responses.

50


WORKSHOP

Write a short questionnaire to measure employees’ job satisfaction on their current position. Incorporate topics on the employees’ preferences. Put questions regarding the personal characteristics of employees like age, sex, department, position, and others. Maximum of 15 questions only. Keep in mind the guidelines on questionnaire construction.

Include a question that allows for multiple response.

51


Data Coding

Coding is a process of grouping the responses to a question into categories and assigning numbers, characters, and/or other symbols, called codes, to these categories.

52


REASONS FOR CODING

To simplify the recording of information.

To preserve the confidentiality of the responses either for personal, industrial, or security reasons.

To make it easier to enter the data into the computer, minimize storage space, and speed up processing.

To facilitate the management of files (e.g., sorting, merging, transforming, listing, etc.) using the coded values of one or more key variables.

To perform data analysis using computer software like MS Excel, STATISTICA, MICROSTAT, etc., which require numeric inputs.

53


Basic Principles in Coding

Consistent codes for identical answers should be used throughout the instrument.

Consistent conventions should be used to record missing data, and when appropriate, to indicate the reasons data are missing.

Assign codes that minimize transformations during data analysis

Select standard or commonly used classification whenever possible

54


TYPES OF CODES

Listing codes

Bracket codes

Scale codes

Series codes

System codes

55


LISTING CODES

Listing codes simply correspond to a list of choices or possible responses.

Successive integers, beginning with 0 or 1, are assigned to the items in the list of possible responses.

56


Example:

For questions which allow only one answer, each possible answer can be assigned a code from 1 to n.

Example : Religion

[1] Catholic [3] Iglesia ni Kristo[5] Others, specify

[2] Protestant [4] Fundamentalist

57


BRACKET CODES

Bracket codes represents ranges or class intervals.

These are frequently used for numerical responses like income and age in which the respondent may not be able to provide an exact answer.

58


Example:

Q: What is your monthly income ?(1) less than P 10000.00(2) P 10000.00 to less than P 30000(3) P 30000.00 to less than P 50000(4) P 50000 or more

Provide the codes of the following answers:Ans 1: "My income is P 5000."Ans 2: "My income is between P 15000 and P 35000.”Ans 3. "My income is not more than P 50000".Ans 4. "My income is P 49999.75".

59


SCALE CODES

Scale codes represent grouping of responses which vary in degree or intensity along a continuum.

We represent each segment by a number which serves as the code for any response falling within it.

We often use these for coding responses to attitudinal questions.

60


Example:

1. Two-point scale

Q: "How do you rate the overall performance of this administration?"

1 2

__________________

Low High

Ans: "Inspite of the economic crisis, the performance of

this administration is above my expectation."

Code: ______

61


Example:

2. Three-point scale

(a) This scale has a middle or neutral position

Q: "Do you get along well with your supervisor?"

1 2 3__________________________________

YES YES IN SOME WAYS, NO

NO TO OTHERS

Ans: "Sometimes."

Code: _______

62


Example:

3. Four-point scale(a) This scale has no middle or neutral position.

Q: "How sure are you about the outcome of the election in your province?"1 2 3 4

__________________________________Certain Fairly Fairly Uncertain

certain uncertain

Ans: "I have no idea."Code: _______

63


Example:

4. Five-point scale(a) We use a five-point intensity to code responses with

degree of feeling.Q: "How satisfied are you with your present job?" 1 2 3 4 5

_____________________________________________Very Satisfied Neutral Dissatisfied VerySatisfied Dissatisfied

Ans: "Neither satisfied or dissatisfied."Code: ______

64


Series Codes

Multi-digit codes that are divided into two sets – the first set of leading digits represents a major classification while the second set of trailing digits represents a specific category within the major classification.

This coding scheme can be extended to more than 2 levels.

65


Example of Series Codes

Why do you think that now is a good time to buy a car?

Response:

PRICE and CREDIT Prices are lower now compared to before 11 Uncertain about price in the future – may go up 12 Credit is easy to get 13 Attractive car loan terms 14

66


Continuation of Example of Series Codes

FEATURE of CAR Quality is good/better 21 New models have better features 22 Small cars are available 23

GOOD ECONOMIC PERFORMANCE OF COUNTRY People can afford to buy 31 High employment rate 32

67


SYSTEM CODES

System codes are used primarily in computer data processing to identify and/or classify respondents, households, products, accounts and other data items.

They are usually combinations of numbers and letters (called alphanumeric).

By using the digit 0 through 9 and the 26 letters of the alphabet, a total of 36 characters, 1,679,616 unique four-character codes can be generated.

68


Example:

The Philippine Standard Occupation Classification (PSOC) uses group classification codes which consist of three levels: major, minor, and unit groups.

The major groups are the highest level, of occupation aggregate and represent broad fields of work.

Each major group has a one-digit code.

Minor groups represent the second level of occupation aggregate and each minor group has a two-digit code.

Finally, unit groups are the third level of occupational groupings with each unit group assigned a three digit code.

69


Example:

List of Philippine Standard Classifications Philippine Standard Geographic Classification

Philippine Standard Industrial Classification

Philippine Standard Commodity Classification

Philippine Standard Occupational Classification

70


Example:

Major Group 0/1: Professional, Technical, and Related

Workers

0-8 Mathematicians, Statisticians, System Analysts and

Related Workers

0-81 Mathematicians and actuaries

0-82 Statisticians

0-83 Systems analyst

0-84 Statistical and Mathematical technicians

71


CODING OF NO RESPONSE

Types of No Response:

A. No Opinion

B. Don't Know

C. Refuse to answer

D. Not applicable

E. Missing (in general)

72


NO RESPONSE

A. NO OPINION NO OPINION should be coded NO or given a code

number directly following the other categories.

Example:

Did you find Mr.______adequate in helping you solve your problems?

________Yes ________No

Answer: No Comment.

We could give Yes a code (1); No a code (0) and a No Opinion/No Comment a code (2).

73


NO RESPONSE

B. DON'T KNOW

occurs when we ask Knowledge questions

assign a 'DON'T KNOW' response a code number

directly following the other categories.

74


NO RESPONSE

Example: How much did you and your household spend on

transportation this past month ?_____(1) less than P200_____(2) P200 to less than P500_____(3) P500 to less than P1000_____(4) P1000 to less than P2000_____(5) more than P2000_____(6) don't know

75


NO RESPONSE

C. REFUSES TO ANSWER We sometimes, unfortunately, encounter not-so

receptive respondents who do not like to readily volunteer their responses.

At times it may be due to an offensive question or altogether sensitive topics.

In cases like these, we assign it a code '9', or '99', or '9999' following the number of columns allotted for the variable, i.e., if no other category has been previously assigned to this.

76


NO RESPONSE

Example:How much was your family income this past month ?

Answer: refuses to answer

Code: _______

77


NO RESPONSE

D.NOT APPLICABLE Some responses to our questions may be dependent

upon answers to previous questions or to some attributes of our respondents. So that if the attribute is not present, the question becomes irrelevant.

In cases like these, we code it as NA; or as '9',or '99' or even '99999', again depending on the number of columns allotted to our variable.

78


NO RESPONSE

Example :Do you own a farm ? _____Yes _____NoWhat is the land area of your farm ? _____(hectares)

Code: ________

79


NO RESPONSE

E.MISSING

refers to responses which were not recorded for various reasons, which could not be verified. Hence, we assign to it a special "missing" value symbols.

Following are some characters which are recognized automatically as missing values by three major statistical software packages:

BMDP: blankSAS : blankSPSS : blank

80


Coding Multiple Response

Multiple Dichotomy Method For each possible response, create a dichotomous

variable with a value of 1 to indicate that the response was given and a value of 0 if not.

81


Example of Multiple Dichotomy Method:

Q: What type of TV shows do you watch ? (Please check your answers )[ ] action[ ] drama[ ] game[ ] sport

[ ] none[ ] Pls. specify __________________

A: "I watch action and game shows.”

Code: _____________

82


Example Of Multiple Dichotomy Method

Field Name CodeAction 1 watched 0 did not watchDrama 1 watched

0 did not watchGame 1 watched

0 did not watchSport 1 watched

0 did not watchNone 2 does not watch

83


Some Rules for Designing Code Frames

Create a sufficient range of codes and variables so that the coder need not force data into categories.

Include an “Others” category if you are not assured of the completeness of codes.

Monitor the frequency of the “Others” category. A large frequency indicates a failure in the coding scheme.

Group together related categories of information and use series codes.

84


Quality Control

Training of Encoders Encoders should be trained and supervised

Procedures to be used in coding and entering data should be reviewed in sequence and in detail.

Encoders should be provided with codebooks

85


Quality Control

Range Checking This checks the variable values against predefined

maximum and minimum bounds.

This is often done to catch spurious values or keyboard entry errors.

Cleaning rules can be formulated as logical expressions making various assertions about the data.

86


Quality Control

Consistency Checks Using Contingency Tables The pairs of variables whose answers should be

consistent with each other must be cross-tabulated. The cells representing inconsistent responses must be

empty. After cleaning, the contingency tables are constructed

once more to check that all answers are consistent with each other.

87


Quality Control

Reentering the data file A very expensive method but this is recommended if a

close-to-zero encoding error is desired.

Some software include a “verify” mode in which the data being entered the second time are automatically compared with those entered the first time.

88


WORKSHOP

Construct a code frame for the questionnaire constructed in the last workshop. For each part of the questionnaire, provide the variable name, variable label and code of response.

89


Example

Respondent Identification Variables

Variable Name

Variable Label Code – Response

ID ID Number Encode as is

Brgy Barangay1 – Ibayo2 – Labac

Type Type of Barangay1 – Urban2 – Rural

Resp Age Classification1 – Adult2 – Youth

Intervr Name of Interviewer Encode as is

90


Example

Demographics. Personal Information

Variable Name

Variable LabelCode – Response

A1 A01Name Name

A2 A02SexSex 1 – Lalaki

2 – Babae

A3 A03AgeAge Input number in years

A4 A04CS

Civil Status1 – Single2 – Married3 – Separated4 – Widowed5 – Others

introduction to questionnaire design

Documents