Travis Olson 2013 Summer NUFP Intern Social Justice and Leadership Programs Residence Life, University Housing University of Illinois at Urbana-Champaign

7/26/2013

MOVING FORWARD WITH A

COMPREHENSIVE

ASSESSMENT PLAN FOR

RESIDENTIAL LIFE

PARAPROFESSIONAL STAFF Understanding How Our Students Learn about Social Justice Topics

Currently there is no assessment plan in place for the EOL 199 course for Resident Life paraprofessionals. This report is designed for practitioners interested in developing and implementing an assessment plan. It contains an introduction to assessment best practices, a review of related assessment initiatives at select peer institutions, and a list of suggested next steps toward establishing an assessment plan. Any questions or comments can be directed to Travis Olson at travis.h.olson@gmail.com.

7/26/2013

1

Moving Forward with a Comprehensive Assessment Plan for Residential Life Paraprofessional Staff Understanding How Our Students Learn about Social Justice Topics

INTRODUCTION Ten years ago the Multicultural Advocates (MAs) joined Resident Assistants (RA’s) as paraprofessional staff members in our residence halls. Since then they have become a hallmark of the University of Illinois at Urbana-Champaign’s Residence Life programs. The full-time staff in University Housing knows that the MA’s role as programmers and peer counselors in every residence hall on campus has thoroughly improved the climate for both their RA coworkers and residents. Unfortunately, there currently is little existing data to empirically prove this fact.

There is an increasing demand from outside the academe and from prospective students that colleges and universities begin to prove student learning (Selingo, 2013). As this pressure mounts, we as student affairs professionals with our academic affairs peers will have to develop new ways to demonstrate and communicate that paying tuition to our institution (and rent to University Housing) will give students a unique advantage worthy of their investment. As a nationally competitive Research I institution, the ability to speak intelligibly about how all Illinois student affairs units are meeting student learning outcomes will increasingly become part of our daily work.

One of the easiest ways to begin preparing for this shift in professional culture will be to begin collecting and analyzing existing data about programs

“… more colleges are, for the first time, making serious attempts to answer the question ‘What am I buying with this degree?’”

(J. Selingo, 2013, p. 140)

7/26/2013

2

we know to be effective from our own daily observations, such as the MA and RA training. Part of the requirements for being a paraprofessional staff member is participation in a ten-week academic course that is facilitated by University Housing professional staff. Along with training paraprofessional staff and providing professional development opportunities to our full-time staff, the completed assignments from this course (EOL 199) is a promising source of data that can be used to begin assessing how well students are retaining social justice concepts and if they are successfully integrating them into their work and personal lives.

The EOL 199 course syllabus outlines four major objectives to be completed by the time the class is finished, including an exploration of:

! Our own cultural identities. ! The similarities and differences in our social and cultural

experiences. ! The institutional, historical and cultural factors that influence

individual and group experiences in the context of structural and social inequality.

! The roles and factors in working with and across differences in a community setting to promote Social Inclusion.

In addition to these objectives, the course syllabus lists six student learning outcomes (SLOs):

! Exploration of self and personal reflection of self in relation to others.

! Knowledge of the role of power, privilege and oppression on the histories and experiences of self and others.

! Understanding the interrelationship of social identity and community development.

! Understanding communication as a tool for learning. ! Enhanced skills for the development of social inclusion. ! Practice in planning initiatives to promote social inclusion,

community building, and assessment of community needs. From these objectives and SLOs an assessment plan can be developed that uses student work as its raw data.

7/26/2013

3

ASSESSMENT BEST PRACTICES All assessment plans should have three basic parts: identifying goals, determining how to collect information, and using data to implement action (Walvood, 2010). As a course that is designed to teach cultural competency, any assessment of EOL 199 would also be subject to additional best practices that measure the learning process in addition to just outcomes.

Identifying Goals

The type of assessment, its duration, and how results are communicated will all be determined by what goals are identified by Residence Life staff and who the audience of the information being generated is determined to be (Walvood, 2010). The existing literature on establishing assessment programs suggests that all plans are tied not only to SLOs that are being measured directly, but also to the central mission of the unit carrying out the assessment plan in order to demonstrate congruency at every level of the organization. Our University Housing stake holders will have to ask, “What is most important for us to know in order to advance the Residence Life mission?” in order to limit the scope of the investigation to a manageable size.

Additionally, there is an increasing trend to measure how competency in students is being developed, as opposed to simply measuring content memorization (Dunn, McCarthy, Baker & Halonen, 2011). Understanding how students are not only remembering, but also deploying knowledge of

social identities, power, and privilege is especially important for a program designed to address the challenges of lived systemic oppression.

One way to ensure that competency goals are being met is to examine the cycles of learning that MAs and RAs are undergoing while participating in EOL 199.Learning about social justice is a complicated

Effective this fall Iowan state law will “require universities overseen by the Board of Regents … to formally monitor student learning at the course level and prepare annual reports on those courses for the Board of Regents.” (C. Flaherty, 2013) Inside Higher Ed

7/26/2013

4

process that involves cycles of defensively rejecting new information that is threatening and eventually integrating that very same information into new world views (Bell & Griffin, 2007; Mahooney & Schamber, 2004). Asking the student staff to reflect and report on these experiences is one way we can measure the effectiveness of the course.

Determining How to Collect Data

One of the biggest hurdles to collecting data is deciding where to begin and a common mistake in assessment planning is to over complicate the process to the point where any progress is frozen by logistical considerations (Walvoord, 2010). To avoid this we should start modestly, collecting existing data, with plenty of room for growth into new types of data collection down the line (Love & Estanek, 2001). Currently, the students enrolled in EOL 199 are required to complete three assignments, an introductory reflection paper, poster presentation and a community development plan. Each of these assignments could be used for assessment purposes without significantly changing the course structure. All that is needed is a common rubric that measures how students are demonstrating competency across different sections of the course and over the course of future semesters. Over time, course assignments and new instructional elements can be added to better capture how students develop over their time in class and after the sessions

have ended.

After an assessment plan has been established and put into place it will need to be sustained through ongoing support from not only administrative and staff stakeholders, but also student participants. The quality and quantity of data that is collected will suffer if students see assessment efforts as burdensome and of no personal benefit.

“Systems cannot be separated from their environment, and all systems are recognized as being open, that is, the boundaries of the system are not barriers and are in effect arbitrary, even if everyone agrees upon where the boundaries are.” (p. 103, Love & Estanek, 2004)

“Assessment plans should provide explicit and public statements and goals about institutional values and expectations for student learning and administrative practice.” (p. 108, Love & Estanek, 2004)

7/26/2013

5

Assessment can serve as another opportunity to develop discipline-specific vocabulary and skills when facilitators are explicit about what is competencies are being targeted in a course and measured through assessment (Dunn, McCarthy, Baker & Halonen, 2011). Using assessment tools that encourage reflection and make data available to participants will allow paraprofessional staff opportunities to demonstrate their own professional development and add value to the EOL 199 course.

Using Data to Implement Action

Assessment should be conceived of as an ongoing and cyclical process, not a linear operation focused solely on a bottom line (Love & Estanek, 2001). Therefore, the final step in an assessment plan consists of using data analysis to adjust the program and reinvest in further assessment. In order to ensure that this stage can be completed successfully, however, key players who have the ability to make these changes and can provide perspective on some crucial questions must be included in the process from an early stage.

Some of these questions that must be discussed with senior staff before assessment is carried out include:

! What are the non-negotiable goals of the EOL 199 course? ! For whom is this assessment important? ! Whose responsibility will it be to carry out this assessment after it is

developed? ! What are the political consequences of this assessment?

(Love & Estanek, 2011)

Some of the individuals who should be consulted on these topics in Residence Life may include: Herb Jones, Patricia Anton, and Nathan Sanden.

After an assessment plan has been developed with the support of these individuals and we have collected some data, facilitators and curriculum designers can begin answering questions related to the effectiveness of the RA/MA course, including:

! Does the curriculum reflect the status of the field? ! Is the course sequenced appropriately? ! Do students know why the course is designed the way it is?

(Dunn, McCarthy, Baker & Halonen, 2011)

7/26/2013

6

ASSESSMENT PRACTICES AT PEER INSTITUTIONS Many of our peer institutions have been engaging in assessment for RA training for years, including the two schools below who have graciously shared their data with us.

The University of Michigan – Ann Arbor, MI

The University of Michigan’s “Res. Staff Class” is one of the models that UI turned to when developing EOL 199. The Michigan professional residence life staff has been collecting assessment data in the form of student evaluations since 1998. Some of the statements that they have students respond to through a five point Likert-scale include:

! “I became more aware of multiple perspectives on issues of diversity.”

! “I learned to think critically about difficult issues of diversity.” ! “The facilitators were sensitive to multicultural issues in the

classroom.” ! “The facilitators promoted meaningful discussions of issues of

diversity.”

Section Summary

Identifying Goals

! Maintain congruency across the program through connecting assessment to the mission statement.

! Measure participant competency, not content memorization.

Determining How to Collect Data

! Start now modestly and then grow into new types of data collection.

! Take advantage of assessment as a developmental and learning opportunity that benefits participants.

Using Data to Implement Action

! Involve key stakeholders early on to ensure that data can be used to make the most impactful changes possible.

7/26/2013

7

Demonstration of the trends created by Michigan’s assessment data over the past15 years.

Even though the data is incomplete, residence life staff at Michigan are able to demonstrate how student staff opinions on their training have changed over time and how satisfaction with the teaching of diversity topics correlates to other areas of training.

The Ohio State University – Columbus, OH

Currently The Ohio State University’s residence life staff goes through a cultural competency training program that is organized and sponsored by the campus’ Multicultural Center. The program is called “Open Doors” and is broken into three modules that cover “Identifying and Interrupting Bias”, “Sharing Identities/Empathetic Listening”, and “Working with Persons in Distress.” More information on the program’s goals and design can be found at www.opendoors.osu.edu.

Bowen Marshall (personal communication, July 18, 2013), a PhD candidate at OSU who is heavily involved with the planning and maintenance of the Open Doors program, shared that currently the program only does a post-test analysis. Practitioners at OSU have found that before the program participants are not familiar enough with the language necessary to conduct

7/26/2013

8

pre-test analyses. OSU is looking to expand their assessment efforts of Open Doors by beginning to collect qualitative, as well as quantitative data.

Samples of OSU’s post-test assessment data.

In his correspondence, Mr. Marshall noted:

Even after experiencing such a powerful training as Open Doors, more than a quarter to virtually thirty percent of the participants were still unsure that they would report bias incidences for themselves or others reflects a continuing need for Open Doors and other training/programs on this campus for every student, faculty, and staff member (personal communication, July 18, 2013).

This is an important fact to consider for EOL 199. Since we are currently not collecting assessment data, we have no way to validate such statements about Residence Life’s trainings.

7/26/2013

9

MOVING FORWARD The following four suggestions have been developed from the literature on assessment best practices and conversations with assessment professionals both here at Illinois and other peer institutions. They are organized from least time and effort intensive to most.

Create Universal Rubrics

In order to begin collecting comparable data from existing EOL 199 requirements, the course facilitators must be using the same set of rubrics in every section of the course. In order to allow for assessment of the course, these rubrics should explicitly state how students effectively demonstrate competencies that are tied to the course SLOs.

Here is an example using EOL 199’s first SLO and the first writing assignment:

SLO RELATED

COMPETENCY

STUDENT

SCORE

SLO 1: EXPLORATION OF

SELF AND

PERSONAL

REFLECTION OF

SELF IN RELATION

TO OTHERS.

COMPETENCY: CRITICAL

ANALYSIS

GRADE OF 4.0 OR A: DEPTH OF ANALYSIS IS

SIGNIFICANT

OFFERING BOTH AREAS

OF STRENGTH &

LIMITATION …

MEASURE OF SLO ACHIEVEMENT

MEASURE OF SLO ACHIEVEMENT

7/26/2013

10

Examples of rubrics can be found throughout Walvood (2010) and in select chapters of Dunn, McCarthy, Baker & Halonen (2011, p. 76 & 97).

Deploy Consistent Assessment-Minded Language

In order to effectively communicate what is being learned in EOL 199 it is important to use specific and meaningful language. In reviewing the course’s syllabus, Special Assistant to the Dean of Students for Assessment Dr. Belinda De La Rosa found that the current SLOs use vague language, making assessment hard to carry out (B. De La Rosa, personal communication, July 3, 2013). Specifically, she suggested that “understanding” and “skills” can be read as vague in certain contexts and be clarified. Dr. De La Rosa suggested that we turn to Bloom’s Taxonomy of Learning Outcomes to rewrite the EOL 199 SLOs in a way that is universally intelligible to education professionals.

A Summary of Bloom’s Taxonomy of Learning Outcomes

(ranked in order of cognitive complexity)

Adapted from (Iowa State University Center for Excellence in Learning and Teaching, 2012) and (UNC Charlotte Center for Teaching and Learning, 2013).

Outcome

Description Associated Verbs

Remember Retrieve relevant information

from long-term memory. List, Recognize, Recall,

Identify

Comprehension Construct meaning from instructional messages.

Summarize, Classify, Clarify, Predict

Apply Carry out or use a procedure in

a given situation. Respond, Provide, Carry

Out, Use

Analyze

Break material into constituent parts and determine how parts relate to one another and to an

overall structure or purpose.

Select, Differentiate, Integrate, Deconstruct

Evaluate Make judgments based on

criteria and standards. Check, Determine,

Judge, Reflect

Create Put elements together to form a coherent whole; reorganize into

a new pattern or structure.

Generate, Assemble, Design, Create

7/26/2013

11

With Bloom’s language the SLOs could be changed in the following ways:

! SLO 1: Students will comprehend their own identities and how those identities affect their relationships with others.

! SLO 2: Students will analyze the roles power, privilege and oppression have played historically in relations between different groups and how those forces still affect their experience of self and others today.

! SLO 3: Students will comprehend the interrelationship of social identity and community development.

! SLO 4: Students will apply communication tools in order to learn about social justice and identity.

! SLO 5: (We need to clarify what we mean by “skills”: interpersonal, intrapersonal, programming, counseling, communication, etc.)

! SLO 6: Using the competencies built during this course, students will create an initiative that promotes social inclusion, community building, and assessment of community needs.

These slight adjustments to the SLOs would allow both more precise assessment and future benchmarking against other programs.

Tie Assessment to Professional Development

As stated earlier, it is in the best interest of both the students in the EOL 199 course and the facilitators to design an assessment regimen that is beneficial to all parties. Integrating assessment into professional development for para-professional staff can be as easy as including assessment data in one-on-one meetings or encouraging students to collect personal reflections for professional development portfolios.

“An ePortfolio may function as a venue for collecting and sharing academic work with faculty members, a tool for inviting collaboration and feedback, a professional portfolio to share with prospective employers, or a private log of academic progress.” (Loyola University Chicago Center for Experiential Learning, 2013)

7/26/2013

12

Creating a more standardized procedure that both collects assessment data for practitioners and organizes and displays evidence of professional development for student staff is possible, however. Many universities are doing this through electronic portfolio or “e-portfolio” programs. Depending on the software used, students could potentially use e-portfolios to store reflections, graded rubrics, and self-assessments and then staff could tap into these files for assessment purposes. Additionally, after the completion of the course, student staff is able to create materials to market themselves to potential employees.

The University of Michigan even includes building a “M-portfolio” as a part of the RA staff class, while other universities such as Loyola University Chicago are exploring how to deploy e-portfolios to measure the success of student affairs programs. More information on how these universities are deploying e-portfolio software can be found at the following sites:

University of Michigan “M-Portfolio” Program

http://www.mportfolio.umich.edu

Loyola University Chicago

http://www.luc.edu/eportfolio

E-portfolio software can be purchased and used with existing campus-wide software such as Blackboard, but free services are also available online. Free providers could be used to pilot an e-portfolio program:

Pathbrite

http://www.pathbrite.com

CONCLUSION

Developing a new assessment plan for EOL 199 will take some effort, but there are many resources available at both the University of Illinois and peer institutions to do so effectively. Following the best practices outlined in this document will allow practitioners to begin collecting data this semester, develop long-term plans to collect more robust data, and eventually use assessment practices to reinvest in the course and paraprofessional development.

7/26/2013

13

WORKS CITED Bell, L. A. & Griffin, P. (2007). “Designing Social Justice Education Courses.” In Adams, M., Bell, L.A. & Griffin, P. (Eds.), Teaching for Diversity and Social Justice (67-87). New York: Routledge.

Dunn, D., McCarthy, M., Baker, S. & Halonen, J. (2011). “The New Architecture of Learning Design: Focusing on Student Learning Outcomes.” In Using Quality Benchmarks for Assessing and Developing Undergraduate Programs (71-91). San Francisco, CA: Jossey-Bass.

Dunn, D., McCarthy, M., Baker, S. & Halonen, J. (2011). Evaluating Curricula. In Using Quality Benchmarks for Assessing and Developing Undergraduate Programs (92-119). San Francisco, CA: Jossey-Bass.

Flaherty, C. (July 19, 2013). “Assessment: It’s the Law.” Inside Higher Ed. Retrieved from www.insidehighered.com/news/2013/07/19/iowa-state-legislators-mandate-course-level-continuous-improvement-reporting-mixed

Iowa State University Center for Excellence in Learning and Teaching (2012). A Model of Learning Objectives. Retrieved from www.celt.iastate.edu/pdfs- docs/teaching/RevisedBloomsHandout.pdf.

Love, P. & Estanek, S. (2004). Developing an Assessment Mindset. In Rethinking Student Affairs Practice (83-117). San Francisco, CA: Jossey-Bass.

Loyola University Chicago Center for Experiential Learning. (2013). About ePortfolios @ LUC. Retrieved from www.luc.edu/experiential/eportfolio/about/

Mahooney, S. & Schamber, J. (2004). Exploring the Application of a Developmental Model of Intercultural Sensitivity to a General Education Curriculum on Diversity. The Journal of General Education, 53(3/4), 311-334.

Selingo, J. (2013). College (Un)Bound: The Future of Higher Education and What it Means for Students. Boston, MA: New Harvest.

UNC Charlotte Center for Teaching & Learning (2013). Writing Objectives Using Bloom’s Taxonomy. Retrieved from teaching.uncc.edu/articles-books/best-practice-articles/goals-objectives/writing-objectives-using-blooms-taxonomy.

Walvood, B. (2010). Assessment Clear and Simple: A Practical Guide for Departments, Institutions, and General Education. San Francisco, CA: Jossey-Bass.