A-team Spring Session #2Questionnaire Design

February 28, 2007

Questionnaire or Survey?

• Questionnaire is an actual instrument: I.e. web questionnaire (Perseus)

• Survey is actually a verb/method: “to study or examine comprehensively”

• Your questionnaire is actually “surveying”

• Often used interchangeably

Rea & Parker (1997)

Yields Different Types of Data:

• Descriptive– Socioeconomic parameters to better understand the

larger population represented by the sample. (e.g., income, age, college/school, major, class standing)

• Behavioral– Patterns of use, recreation, entertainment, personal

behavior. (e.g., # UGA bus rides per day/week)

• Preferential– Opinions & preferences about socio-political issues.

(e.g., opinion about new parking plan)

Rea & Parker (1997); Schuh & Upcraft (2001)

Types of Responses/Measurement Scales

• Nominal Scales – Used to categorize objects; “name” them– Object is in a category or it is not– No order implied along any dimension– Response sets that are nominal:

• Yes/No (dichotomous)• Can be “Choose one,” “Choose all that apply,”

“Choose one and specify,” or “Choose all and specify” from listing of characteristics

Schuh & Upcraft (2001)

Examples of Nominal Scales

• Immediately after aerobic exercising I generally

feel: ___Exhausted ___Invigorated ___Thirsty

___Sweaty ___Overheated ___Nauseated

[Note: make sure (+) and (-) options offered in listing]

• Indicate your sex: ___Male ___Female

• Have you ever resided in Brumby Hall?

___Yes ___No

Schuh & Upcraft (2001)

Measurement Scales (Cont’d)

• Ordinal Scales (a.k.a. rank, order, rank-order)

– Used to rank objects according to amount of characteristic the object possesses

– Order reflects varying amounts or levels– Rank reflects range from high to low amounts– Ranking has no absolute zero– Intervals from one rank to next not the same– Likert scales are ordinal but sometimes treated as

interval scales (judgment call)

Schuh & Upcraft (2001)

Examples of Ordinal Scales

• Order of finish in a horse race

• Rank in class (e.g., achievement)

• Highest degree earned

• Order of preference

• A higher number indicates a higher rank, e.g., “more” of characteristic possessed

• Watch for (reverse) coding

Schuh & Upcraft (2001)

Examples of Ordinal Scales

• Rank-order your on-campus living preferences for the next academic year, with 1 = first priority, 2 = second priority, and so on:

___ Single Room ___One-bedroom Apartment ___ Double Room ___ Multiple-bedroom Apartment ___ Suite (2 double rooms separated by a bathroom)

• Rank-order your reasons for attending this workshop, with 1=most influential reason, 2=second greatest influence, and so on:

___ Surveys are my life ___My boss sent me ___ To get the handouts ___ My thirst for knowledge ___ To earn CEUs ___ To get the free gift

Schuh & Upcraft (2001)

Measurement Scales (Cont’d)

• Interval Scales– A ranking/rating using interval score values– The difference between intervals is equal– The difference between 1 & 2 is the same as the

difference between 4 & 5– Still a focus on the amount of a characteristic an

object possesses– Likert-type (pronounced Lick-ürt) scales often treated

like interval scales (although considered ordinal):• 5=strongly agree, 4=agree, 3=no opinion, 2=disagree,

1=strongly disagree

Schuh & Upcraft (2001)

Examples of Interval Scales

• Likert Scale Example:Parking on campus should be free.__Strongly Agree __Agree __Neither Agree nor

Disagree __Disagree __Strongly Disagree

• Non-Likert Scale Example– When driving a UGA van, the safest following

distance under ideal conditions (in seconds) is__1.5 __3 __4 __8 __10 __25

Schuh & Upcraft (2001)

Examples of Interval Scales

• Non-Likert Scale Example (cont’d): Please rate your satisfaction with the following

student activities on a scale of 1 to 5, with 1=very dissatisfied, 2=dissatisfied, 3=neither dissatisfied nor satisfied, 4=satisfied and 5=very satisfied. [could add not applicable or did not attend option].

__Welcome Week __Dawgs After Dark __Movie-O-Rama __Concert on the Quad __ Halloween “I Vant to Drink Your BlooooooDrive”

Rea & Parker (1997); Schuh & Upcraft (2001)

Measurement Scales (Cont’d)

• Ratio Scales– Empirically meaningful zero/absolute zero; true

absence of characteristic; (e.g., height, weight; more common in physical/biological sciences)

– Have all characteristics of nominal, ordinal, and interval scales

– Can be converted to ordinal scales– Can be converted to categories– Education examples: income, age, # years of

education, # meetings with academic advisor

Schuh & Upcraft (2001)

Examples of Ratio Scales

• Indicate the number of times you accessed the University Health Center in the last 30 days: ___

• Indicate your age: ___

DeVellis (1991, p. 68-70))

Likert Scales

• Present question/item stem in a declarative sentence (one statement under consideration).

• Response options represent varying degrees of agreement or endorsement of one statement.

• Response options should be worded to represent approximately equal intervals; use equal # positive and negative possibilities.

• The question stem doesn’t have to span the range of the construct (as in Thurstone or Guttman); response options infer levels of phenomena.

DeVellis (1991, pp. 68-70)

Likert Scales

• Often 5, 7, or 9 response-options sets • A 6 response-options set is also common

– Strongly disagree– Moderately disagree– Mildly disagree– Mildly agree– Moderately agree– Strongly agree

DeVellis (1991, pp. 68-70)

Likert Scales

• Midpoint often used but optional• What does midpoint wording imply?

– Neither agree nor disagree: Apathy?– Agree and disagree equally: Strong paradox?

• Common midpoint wording– Neither agree nor disagree– Agree and disagree equally– Neutral

DeVellis (1991, pp. 68-70)

Likert Scales

• Most used in surveys of opinions, beliefs, attitudes

• Useful if statements are fairly strong (but not extremely)

• Everyone can agree, have no opinion, or have little opinion about a mild statement

• Write clear statements that reflect true differences of opinion

DeVellis (1991, p. 70)

Likert Scale Examples

• Exercise is an essential component of a healthy life-style.1=Strongly Disagree, 2=Moderately Disagree, 3= Mildly

Disagree, 4=Mildly Agree, 5=Moderately Agree, and 6=Strongly Agree

• Combating drug abuse should be a top national priority.1=Completely True, 2=Mostly True, 3=Equally True and

Untrue, 4=Mostly Untrue, and 5=Completely Untrue

DeVellis (1991, p. 70-71)

Semantic Differential Scales

• Response options consist of one but usually several adjective pairs

• One adjective is negative, the other positive; each serves as the (-) or (+) end of a continuum that characterizes the stimulus

DeVellis (1991, pp. 70-71)

Semantic Differential Scales

• Individual lines/points are placed between the two extremes (adjectives)

• 7 or 9 lines/points are common• Respondents check/select lines or points closest

to the adjectives if they hold extreme views• Respondents check/select lines or points toward

the middle of the continuum if they hold more moderate views

DeVellis (1991, pp. 70-71)

Semantic Differential Examples

• Automobile Salesmen• Honest __ __ __ __ __ __ __ Dishonest • Quiet __ __ __ __ __ __ __ Noisy• Friendly __ __ __ __ __ __ __ Not Friendly• Fair __ __ __ __ __ __ __ Unfair• Trustworthy __ __ __ __ __ __ __ Untrustworthy

Length of a Survey

• Sufficient to capture needed data• Short enough to hold participants’ attention• Type of survey affects length• Types of questions affect length• Quantitative/Qualitative/Mixed approach affects

length• Participant characteristics affect length

Measurement Scales: More Tips

• Avoid providing categories/options that overlap; difficult or impossible to analyze

• Frequently happens with age, income, class hours, years of service, hours worked, etc.

• Example:– Select the category that best describes your

annual, gross income: __$0-$10,000 __$10,000 – $30,000 __$30,000 - $60,000

DeVellis (1991); Schuh & Upcraft (2001)

Measurement Scales (Cont’d)

• Be thoughtful with Use of “other” or “does not apply” or “not applicable” in listing of characteristics/options

– Positive: Obtain option you may not have considered– Positive: Prevents forced responses – Negative: Can give response already listed or

spurious data– Potential Negative: Adds to analysis time

AnalysisType of Data Measure of

Central Tendency

Measure of Variability

Examine Responses of 1 Group to 1

Question

Interval

Ratio

Scale

Mean or Median(if skewed)

Variance or

Standard

Deviation

Frequency Distribution Bar or Line Graphs

Ordinal

Rank

Rank-Order

Median Frequency Distribution

Bar Graph

Line Graph

Freq. Dist.

Bar or Line Graphs

Nominal

Categorical

Dichotomous

Mode Frequency Distribution

Bar Graph

Freq. Dist.

Bar Graphs

Schuh & Upcraft (2001); Upcraft & Schuh (1996)

Analysis (Cont’d)Type of Data Compare

responses of 2 Grps to 1 Q

Compare responses of 1 Grp to 2 Qs

Degree of Relshp Bet. Response to

2 Qs

Degree of Relshp. Among responses to 3

or > Qs

Interval

Ratio

Scale

Paired Frq.

Distribution or Graphs

Scattergram Pearson’s

Product Moment Correlation

Multiple Corr. Partial Correl.

Ordinal

Rank

Rank-Order

Paired Frq.

Distribution

or Graphs

Cross-tab

Table

Spearman’s Rank-Order Correlation Coeff.

Kendall’s Partial Rank Correl.

Nominal

Categorical

Dichotomous

Paired Frq.

Distribution or Graphs

Cross-Tab

Table

Cramer’s Index of Contingency

References• DeVellis, R. F. (1991). Scale development: Theory and applications. Newbury Park,

CA: Sage.

• Miller, T. K., (1999). CAS: The book of professional standards for higher education. Washington, DC: Council for the Advancement of Standards in Higher Education.

• Payne, D. (1992). Measuring and evaluating educational outcomes. New York: Macmillan.

• Rea, L. M. & Parker, R. A. (1997) Designing and conducting survey research (2nd Ed.). San Francisco: Jossey-Bass.

• Schuh, J. H. & Upcraft, M. L. (2001). Assessment practice in student affairs: An applications manual. San Francisco: Jossey-Bass.

• Upcraft, M. L. & Schuh, J. H. (1996). Assessment in student affairs: A guide for practitioners. San Francisco: Jossey-Bass.