to survey or not to survey? that’s a really good question!

STUDENT AFFAIRS RESEARCH AND ASSESSMENT

TO SURVEY OR NOT TO SURVEY? THAT’S A REALLY GOOD QUESTION!It is tempting, if the only tool you

have is a hammer, to treat everything as if it were a nail.

~ Abraham Maslow, 1966

SARA Assessment Brown BagFebruary 10, 2011


Why is assessment important?Helps determine if you are meeting

your educational objectivesHelps ensure that you have the

resources you needHelps prioritize effortsCan contribute to our understanding

of student learning and development


Key steps - PDCA Identify outcomes Identify appropriate measuresChoose an appropriate assessment methodChoose and appropriate research designCollect the dataAnalyze the dataDisseminate the findingsTake actionWash, Rinse, Repeat!


Presentation outlineWhy do you ask? Review types of outcomes and

measuresBenefits and drawbacks to

surveysOther types of assessmentsIt’s your turn - practice makes

perfect!


Why not use a survey?Ease of web-based surveys has

lead to their proliferationResult is that surveys are the “go-

to” research tool, but…◦Not every question is a nail◦Survey fatigue is a significant threat◦Multiple types of evidence (assessment

triangulation) build a stronger case


Starting questions1. What type of outcome do you want to

measure? ◦ Cognitive (knowledge)◦ Affective (attitudes)

2. What type of data do you want to collect? ◦ Psychological (personal traits)◦ Behavioral (observable activities)

3. Time frame?◦ Short-term◦ Long-term


Taxonomy of Student Outcomes (Astin, 1993)

TYPE OF DATA

TYPE OF OUTCOMECognitive Affective

Psychological Subject-matter knowledgeAcademic abilityCritical thinking abilityBasic learning skillsSpecial aptitudesAcademic achievement

ValuesInterestsSelf-conceptAttitudesBeliefsSatisfaction with college

Behavioral Degree attainmentVocational achievementAwards or special recognition

LeadershipCitizenshipInterpersonal relationsHobbies and avocations


Time: Examples of Short- and Long-term Outcomes (Astin, 1993)

TYPE OF OUTCOM

ETYPE OF

DATASHORT-TERM

(DURING COLLEGE)LONG-TERM

(AFTER COLLEGE)

Cognitive BehavioralCompletion of AlcoholEdu (vs. noncompletion)

Exhibits responsible drinking behaviors

Cognitive Psychological MCAT score

Score on medical licensing exam

Affective Behavioral Participation in student government

Involvement in local or national politics

Affective Psychological

Satisfaction with college Job satisfaction


Types of measures

Direct measuresIndirect measures

Norm-referencedCriterion-referencedSelf-referenced

Although direct measures are typically preferred, practically speaking, your overall assessment plan should contain a mix of these


Your question should guide your choice of assessment tool, and not the other way around!


Surveys

“Surveys represent one of the most common types of quantitative, social science research. In survey research, the researcher selects a sample of respondents from a population and administers a standardized questionnaire to them.” http://writing.colostate.edu/guides/research/survey/index.cfm

http://writing.colostate.edu/guides/research/survey/index.cfm


Surveys can be a good tool if you are interested in:PerceptionsBeliefsMotivationsFuture plansPast behaviorPrivate behavior


What about student learning? Research indicates that aggregate self-reports of

learning can provide a reasonable estimation of actual learning. *

**Self-reported data is most valid when:◦ the information is known to the respondents,◦ the questions are unambiguous and refer to recent activities,◦ the respondents take the questions seriously, and ◦ responding has no adverse consequences nor does it

encourage socially desirable, rather than truthful, answers.

* (Anaya, 1999; Kuh, Kinzie, Schuh, Whitt, & Associates, 2005; Laing, Swayer, & Noble, 1989; Pace, 1985; Pike, 1995). **(Kuh et al., 2005 & Pike, 1995)


Validity and reliabilitySurveys tend to be weak on validity and strong on reliability. The artificiality of the survey format puts a strain on validity. Since people's real feelings are hard to grasp in terms of such dichotomies as "agree/disagree," "support/oppose," "like/dislike," etc., these are only approximate indicators of what we have in mind when we create the questions. Reliability, on the other hand, is a clearer matter. Survey research presents all subjects with a standardized stimulus, and so goes a long way toward eliminating unreliability in the researcher's observations. Careful wording, format, content, etc. can reduce significantly the subject's own unreliability.http://writing.colostate.edu/guides/research/survey/index.cfm



RepresentativenessIf your respondents are not

representative of your population, then your results may be misleading◦Example: You want to survey all

undergraduate students about their attitudes towards a student honor code. Your friend is in charge of the FYS program and offers to have your survey passed out in all first-year seminars. You’re excited to get this direct push for your survey, but how might this effect your results?


Strengths of surveysRelatively inexpensiveCan reach large numbers of peopleLarge numbers allow for multivariate

analysesCan ask many questions relatively quicklyStandardized instruments (like NSSE)

allow for comparisons between groupsCan be confidential or anonymousCan have high reliability


Strengths continuedCan provide

student/alumni/employers perspective of the institution/program.

Can make respondents feel that their opinions matter.

Ease of response can provide information from hard to reach individuals

Results easily understood


Weaknesses of surveysQuestions have to be general enough to

apply to all or most respondentsInflexible - Forced-response choice may

not allow respondents to express their true opinions

Require good response rates to achieve representative results

May be hard for respondents to recall information or answer truthfully

Can seldom deal with “context”


Weaknesses continuedValidity can be questionable – results

tend to be highly dependent on wording of items, salience of survey, and organization of the instrument

Socially desirable responsesIndirect evidence which may have less

legitimacy with stakeholdersBetter for measuring and comparing the

responses of groups rather than individuals


Finally, it might not be a survey if….The questions you want to ask

don’t have a limited number of known, well-defined possible answers

You want to be able to ask about relationships rather than inferring them

Your population of interest differs in culture or language from the majority


Some other types of assessments Standardized exams Test of abilities or knowledge Simulation or performance appraisals Interviews and focus groups External examiners Archival records and transcript analysis Portfolios Behavior observations Student self-evaluations Reflective writing Minute papers/muddiest point


Final thoughts on choosing an assessment toolGo back to your assessment

question(s)◦What do you want to know?◦What are the resource limitations?

(e.g., time, money, staff)◦One shot or longitudinal?◦Experimental design?◦What type of analysis is appropriate?


NOW IT’S YOUR TURN


Think about a program you want to assess…What do you want to know? What type of measure?How many students are involved? What type of evidence do you

already have (if any)? What type of evidence is most

effective with your intended audience?


Sample assessment questionsDo participants in an alternative spring

break (n=20) develop an increased awareness of social injustice and subsequently a greater commitment to working for social justice?

Do students (n=5,000) who go through an alcohol intervention drink less as a result?

Do participants in the PRCC’s Learning Circle (n=15) exhibit improved ability to engage in positive cross-racial interaction with other participants?


Resources Astin, A. W. (1993). Assessment for excellence: The philosophy

and practice of assessment and evaluation in higher education. American Council on Education Series on Higher Education. Phoenix, AZ: Oryx Press.

Improving Educational Programming online training module: https://www.sa.psu.edu/workshops/edprogram/index.htm

Pope, R. L., Reynolds, A. L., Mueller, J. A., & Cheatham, H.E. (2004). Multicultural competence in student affairs. San Francisco: Jossey-Bass.

SARA Assessment Resources website: http://studentaffairs.psu.edu/assessment/resources.shtml

Writing Guide: Survey Research, Colorado State University: http://writing.colostate.edu/guides/research/survey/index.cfm

Yin, A. C., & Volkwein, J. F. (2009). Assessing General Education Outcomes. In J. F. Volkwein (ed.), Assessing Student Outcomes: Why, who, what, how? New Directions for Institutional Research Assessment Supplement (pp. 79-100). San Francisco: Jossey-Bass.

https://www.sa.psu.edu/workshops/edprogram/index.htm

http://studentaffairs.psu.edu/assessment/resources.shtml



QUESTIONS?

to survey or not to survey? that’s a really good question!

Documents