s tudent a ffairs r esearch and a ssessment the role of assessment in student affairs practice csa...

STUDENT AFFAIRS RESEARCH AND ASSESSMENT

THE ROLE OF ASSESSMENT IN STUDENT AFFAIRS PRACTICE

CSA 502: Organization and Administration in Student AffairsFebruary 10, 2011

Building a culture of evidence:


Presentation outlineWhat is assessment & why do we

do it?

The role of SARA and assessment offices in general

Assessment planning & the assessment cycle

It’s your turn - practice makes perfect!


Learning objectives: At the end of this session, you should be able to:

Identify why assessment is important in all aspects of student affairs.

Describe the general role of an assessment office.

Describe the assessment cycle.

Perform the basic steps to create an assessment plan.


What is assessment?It is “any effort to gather, analyze,

and interpret evidence which describes…effectiveness (Upcraft & Schuh, 996)

Upcraft & Schuh (2002) argue that research informs theory while assessment informs practice.

But what does that really mean in practical terms??


Why is assessment important?Helps determine if you are meeting

your educational objectives

Helps ensure that you have the resources you need

Helps prioritize efforts

Can contribute to our understanding of student learning and development


Student Affairs Research & Assessment (SARA)Mission

Student Affairs Research and Assessment provides leadership and service to the Division of Student Affairs in the areas of assessment, learning outcomes, educational programming, and strategic planning. SARA collects and disseminates data about Penn State students, their experiences and learning, and their perceptions of the campus environment. SARA enhances students’ educational experiences through the design, implementation, and evaluation of programs that support the University’s Cocurricular Learning Outcomes.

http://www.sa.psu.edu/hr/pdf/intorgchart.pdf

http://www.sa.psu.edu/hr/pdf/intorgchart.pdf


What does an assessment office do?Conducts assessments, duh!

Provides training and professional development to others

Environmental scanning

Consulting

Brings together evidence

Contributes to strategic planning and accreditation activities


Assessment planning

There is a great deal of pressure to DO assessment, but where do you start?


First, review your goals

Clues can be found in your:

Mission, vision, and objectives

Strategic plan

Stated learning outcomes

Where do you want to

be?


It is really helpful to get out of your silo

You can only learn so much from documents.

Institutional knowledge:

◦Who?

◦What?

◦How?


Next, develop a plan.


Start with a model

1. Janusian Duality

For internal improvement

For External accountability

II. Five Questions

1. Are you meeting your goals?

2. Are you improving?

3. Do you meet prof. standards?

4. How do you compare?

5. Are you efforts cost-effective?

III. Select Methods & Measures

For Each Level:-Institution-Unit-Individual students

For Academic Effectiveness:

Student learning,

Research, and Scholarship

For Administrative Effectiveness:

Strategic Planning &

Resource Mgmt.

IV. Collect & Analyze

EvaluationEvidence

V. Communicate

Results

Take Academic &

Administrative Action

Improve & Strengthen Programs

The Volkwein Model (2009)


The assessment cycle

Identify outcomes

Gather evidence

Interpret evidence

Refine efforts


Key steps - PDCA Identify outcomes

Identify appropriate measures

Choose appropriate assessment methods

Choose an appropriate research design

Collect the data

Analyze & interpret the data

Disseminate the findings

Take action

Wash, Rinse, Repeat!


Think long-termChances are you cannot assess

all of your outcomes every semester or even every year.

Be realistic about your resources and develop a comprehensive, multi-year plan.

Don’t forget to incorporate all steps of the cycle into your plan.


A good plan:Collects multiple types of evidence

(triangulation),

Takes into consideration your population(s) of interest,

Takes into consideration your audience,

Has a realistic timeline,

Identifies its products,

Incorporates dissemination & action, and

Is always subject to revision!


A matrix can be a helpful to organize your strategy and look for holes in your plan

Outcome Survey Reflective writing

Expert evaluatio

n

Knowledge /skills

test

Critical thinking FA2011

Leadership SP2012 FA2013

Communication FA2013 SP2014

Civic engagement SP2014

Healthy lifestyle FA2011 SP2012

Social justice perspective SP2015 FA2014


Once you have your multi-year plan, you can bite it off in chunks.

Identify a subset of outcomes

Identify appropriate measures

Choose an appropriate assessment method

Choose an appropriate research design

Collect the data

Analyze & interpret the data

Disseminate the findings

Take action

Wash, Rinse, Repeat!


Your turn…

Identify outcomes

Gather evidence

Interpret evidence

Refine efforts


Example from the PRCC:By the end of the World Cultural

workshop students will be able to describe four unique traditions, values and beliefs of other races and cultures.

By the end of the social justice retreat, students will demonstrate the ability to incorporate social justice techniques.


AudienceBy the end of the World Cultural

workshop students will be able to describe four unique traditions, values and beliefs to other races and cultures.



BehaviorBy the end of the World Cultural

workshop students will be able to describe four unique traditions, values and beliefs of other races and cultures.



ConditionBy the end of the World Cultural

workshop students will be able to describe four unique traditions, values and beliefs to other races and cultures.



Next…

Identify outcomes

Gather evidence

Interpret evidence

Refine efforts


Determine appropriate measuresAttitudes?

Experiences?

Knowledge?

Abilities?

Retention?

Graduation?

Employment?


Some types of assessments Standardized exams

Test of abilities or knowledge

Simulation or performance appraisals

Interviews and focus groups

External examiners

Surveys

Archival records and transcript analysis

Portfolios

Behavior observations

Student self-evaluations

Reflective writing

Minute papers/muddiest point


Tests

◦Can be used to measure a specific skill or knowledge base

◦“Objective”

◦Feasible?

Surveys

◦Attitudes

◦Experiences

◦Self-reported abilities & knowledge

◦Can be confidential/anonymous

◦Can reach large numbers of people


Interviews

◦Rich & detailed information

◦Body language can be helpful

◦Respondents may feel inhibited

◦Repeated interviews can build trust/comfort

◦Time consuming

Focus groups

◦Many of the same benefits as interviews, but not as in-depth

◦Participants “feed off of” each other

◦Can reach more people than interviews


Back to the PRCC: By the end of the World Cultural Workshop students will be able to describe four unique traditions, values and beliefs of other races and cultures.

Measure: Knowledge

Some possible assessment methods:

◦ Knowledge test (direct evidence)

◦ Survey self assessment of knowledge (indirect)

◦ Portfolio

◦ Reflective writing

Research design could be ex-post facto, pre-test/post-test, etc.


Next…

Identify outcomes

Gather evidence

Interpret evidence

Refine efforts


What if:World Cultural Workshop

participants rated their knowledge of other cultures more highly BEFORE the workshop than after (pre-/post-test design)?

In a reflective writing exercise you notice that participants are commonly mismatching the culture/race and its unique characteristics?


Next…

Identify outcomes

Gather evidence

Interpret evidence

Refine efforts


Potential refinements to the World Cultural Workshop based on our assessmentHave a single assessment in

which students focus on what/how much they learned in the workshop, rather than a pre-/post-test comparison.

Cover a smaller number of races/cultures and spend more time on each.

Consider refining objectives


And again...

Identify outcomes

Gather evidence

Interpret evidence

Refine efforts


Questions?If you are driving at the speed of light

and you turn on your headlights, do they do anything?

How your boss is likely to make assessment plans:


Resources & References Astin, A. W. (1993). Assessment for excellence: The

philosophy and practice of assessment and evaluation in higher education. American Council on Education Series on Higher Education. Phoenix, AZ: Oryx Press.

Improving Educational Programming online training module: https://www.sa.psu.edu/workshops/edprogram/index.htm

Pope, R. L., Reynolds, A. L., Mueller, J. A., & Cheatham, H.E. (2004). Multicultural competence in student affairs. San Francisco: Jossey-Bass.

SARA Assessment Resources website: http://studentaffairs.psu.edu/assessment/resources.shtml

Volkwein, J. F. (2009). A model for Assessing Institutional Effectiveness. In J. F. Volkwein (ed.), Assessing Student Outcomes: Why, who, what, how? New Directions for Institutional Research Assessment Supplement (pp. 13-28). San Francisco: Jossey-Bass.

https://www.sa.psu.edu/workshops/edprogram/index.htm

http://studentaffairs.psu.edu/assessment/resources.shtml

s tudent a ffairs r esearch and a ssessment the role of assessment in student affairs practice csa...

Documents

division of student

student affairs practicecsa

aspects of student affairs

role of assessment

assessment important

assessment cycleits

assessment offices

areas of assessment