Leveraging Partnerships to Enhance Assessment

Practices

Beth M. Lingren Clark, Ph.D.

Director: Orientation & First-Year Programs

University of Minnesota-Twin Cities

Outcomes

• Evolution into assessment framework• Developing assessment plans • Involving key campus partners/stakeholders• Examples of using the data to increase the

effectiveness of measures, themes and impact institutional projects

Old practices in assessment• Orientation & First-Year Programs (OFYP) discovered

assessment methods only measured the level of satisfaction with our program/specific components.

• OFYP asked questions unveiled that participants were satisfied to very satisfied.

• Satisfaction not outcome orientated or as helpful when trying to enhance or improve your program.

• OFYP aligned orientation program questionnaires. We worked with the Office of Measurement Services to help us identify common questions across all questionnaires.

Old practices in assessment

• Shared data with presenters to enhance content

• Shared with collegiate representatives to enhance college components

• Influenced orientation program content and communications

WITH THIS DATA…

Needed more information to identify content, make enhancements/improvements, inform partners to all populations served

TAKING A BROADER APPROACH

Think about what motivates a study(topic) and what shapes a study (question or problem).

“Research is a structured process of inquiry that draws upon existing knowledge and theory to advance general understanding or impact practices in the field at large.”

-Henscheid, J.M.& Keup, J.R. (2011, p.ix)

Research: Topics/Questions

• How does what we implement impact retention?

• Is parent involvement important?• Why do students leave?• How are we impacting personal and

academic success?• What key factors predict transfer student

success?

Process: Topics/Questions

• Who completes online Orientation?• Who attends on campus Orientation?• Who lives in Living Learning Communities• Who has completed the Strengths

assessment?• What attended Transfer Welcome Day?• Which International students are coming?

Content: Topics/Questions

• What are the gaps between expectations and reality of experiences?

• How can we reduce anxieties/concerns?• How can we increase parent involvement?• How do we create a sense of belonging?

EVOLUTION TO ASSESSMENT FRAMEWORK

Where we started…

Purpose of Committee: (Continued) – Identify existing instruments to collaborate data collection– Discuss implementation of the program evaluation– Discuss attendance tracking at events

Representatives from the following areas: Provost’s office, Orientation & First-Year Programs, Office of Measurement Services, Institutional Research, College of Liberal Arts, Student Activities, Center for Teaching and Learning, Department of Educational Policy & Administration, Career Center, Student Leadership & Engagement

Welcome Week Committee Structure

The process• Followed cohorts from old program – New Student Weekend (2001 and

2002 cohorts)

• Welcome Week planning occurred simultaneously with the development of the First Year Assessment Plan (baseline cohort 2007)

• Did not want to implement WW in isolation• Support the efficacy of WW

• Created a mapping processing and aligned with what mattered at the institution

• Identified what else existed system-wide to support first year content and themes

Assessment Cycle • Identify learning objectives/goals for program• Map learning objectives/program goals to Student Learning

and Development Outcomes• Conduct gap analysis• Change/enhance content based upon gap analysis and

program survey results from previous year (Common questions)

• Implement program introducing objectives and outcomes to population

• Evaluate program • Analyze and share results• Repeat

How Do We Create a Common Language of Assessment?

• Goal: Initiating undergraduates into Transformative Education– Student learning & development outcomes

• Two primary transformations…– Transforming the campus into the curriculum– Transforming the undergraduate experience

• Assessing first-year programs– Have goals we care about– Know what you know– Map to goals

Understanding the assessment framework

How Do We Create a Common Language of Assessment?

• Mapping process is a conversation– Meant to be iterative; refined over time– Each step informs the next

• What we needed to begin mapping– Schedule and description of

goals/sessions/events– List of student learning and

development outcomes– Mapping spreadsheet (essentially a

matrix of sessions/outcomes)

Mapping Outcomes

How Do We Create a Common Language of Assessment?

• Was done on a large scale (for all of Welcome Week’s content)• Mapping also done on a smaller scale

– College Day - College of Liberal Arts

Mapping Outcomes

The Power of the Common Question

• We wanted to:– Create a common language beyond

orientation programming– Create an intentional plan for content and

evaluative measures– Implement transformational practices on

campus– “We wanted to speak a common language”

The Power of the Common Question (continued)

• We wanted to:– Create a profile of people completing our

program questionnaires– Create longitudinal data from one year to

the next as well as within the cohort/year– Ask more specific questions, AND ask

participants to elaborate on items where they may not have been satisfied

– Understand the utility of common mapping

DEVELOPMENT OF A FORMAL FIRST-YEAR ASSESSMENT PLAN

First-Year Assessment Plan

Intent of FYA Plan• The intent of this process provides data on

student behavior related to:• Matriculation process• Student participation• Student satisfaction• Student success both personally and academically• Longitudinal data utilized• Measure the impact on student retention and other

programs

• The plan helps to inform institution on the things we were learning about from our students and their experiences

Involving Campus Partners

• Office of Measurement Services• Undergraduate Education• Collegiate Units• Academic Support Resources• Admissions, Housing, Athletics, Honors,

International • Office of Institutional Research• Strengths• Office for Student Affairs

EXAMPLES OF USING THE DATA

Matriculation

• Tracking…– Online Orientation completion– Orientation attendance– Document check for international– Parent attendance– Welcome Week/Transfer Welcome Day

attendance– Strengths completion

ParticipationClass of 2016 5514 (10th day enrolled)

Commuter 754 13.7%

Residential 4760 86.3%

MN 3493 63.3%

Out of State 1714 31.1%

International 285 5.2%

UHP 572 10.4%

Athletes 139 18.4%

SOC 1077 19.5%

Band 74 1.3%

First-generation 1382 25.7%

NHS Attended OR 5572 103.8% 5501 (10th Day Enrolled)

NHS Parents Attended OR 4791

June OR 3626 67.5%

July OR 1276 23.8%

Attended OR 8/22 129 2.34% (3 non 10th day, 53 international)

Attended OR 8/23 189 3.43% (4 non 10th day, 76 international)

Attended OR 8/27 307 5.57% (139 international)

0 Attended Parent OR 2267 41.1%

1 Attended Parent OR 1810 32.8%

2 Attended Parent OR 1370 24.8%

3 Attended Parent OR 66 1.2%

4 Attended Parent OR 1 0.1%

Survey Data• Pre-Orientation Survey

– Expectations/Welcome messages– Orientation content (student and parent)

• Common Questions– Cross college– Theme/Program content

Survey Data• Program Evaluations

– Improvement– Outcomes based

• Check-In Surveys– Reality of expectations– Additional support and resources– Newsletter content

Institutional Data• National Survey on Student Engagement

– Comparing freshmen and transfer student experiences

• Student Experience in a Research University– Welcome Week and Sense of Belonging– Transfer Research Grant results

Institutional Data• Leaver’s Analysis

– Longitudinal data/Collegiate specific– Importance of Parent/Family Engagement– @ Home in MN

• Student Retention• Transfer Grant

– Influence overall experience and engagement

Additional Themes…

• Interactions with faculty are an important part of the college experience.

• The majority of the students spend between 11-20 hours attending class per week.

• On average, students spend up to 10 hours per week on social networking (facebook).

• Transfer students have been successful adjusting to the academic demands and developing effective study skills but have had more of a difficult time getting to know their peers and their faculty

Additional Themes…• Student’s top major anxiety/concerns:

– academic coursework– making friends – financial concerns– deciding on a major – developing skills related to note-taking/studying – time management – stress management– transfer of credits**

Additional Themes…

• The University can feel like a big place so it is important for students to create community.– Students can do this in their residence hall– Through the Commuter Connection program– In class– At work– Joining a student organization. – Especially important for students who are not from the

Twin Cities area.

Additional Themes…• New students leave the U for various reasons:

– Financial concerns– Academic– Medical– Family– Other personal issues

Additional Themes…

• At the six to eight week mark, students are:– seeking more contact with advisers– want help with study skills– more opportunities to socialize**– information on campus libraries**– more interactions with faculty**

Sharing the Data

– College/unit level data is reported to committees and administrators

– Mapping Process revisited to ensure meeting goals and desired programmatic outcomes and University Student Development and Learning Outcomes

Sharing the Data

– Program evaluation data will be distributed to content creators and future planning committees

– Data presented to the students (e.g., Advice from the Class of 2016/Transfer?)

– Annual reports disseminated to leadership and decision-makers within Undergraduate Education

Where Do We Go From Here?

• Utilize the Data– Use the findings for continuous improvement efforts

• Did we do what we set out to do?• What can we do better?• What was successful and should be continued?• How do we alter content to address new issues?

– Compare findings year to year• Are students attitudes/perceptions changing?• Are there new transformational issues?

– Longitudinal Analysis• Has Welcome Week impacted the student experience?• Are there correlations between student attendance, satisfaction

retention, graduation rates, etc.? (Utilize orientation database)

Inspiring Our Work

• Influence program content, delivery, and messaging

• Integrate data results directly into what we share with students and families

• Share with colleges and units to influence the way we do our work

• Strengthens partnerships• Provide evidence to work with units to make

strategic changes

“Educators who work to improve the experience of college students, especially students facing transitional periods, are typically driven by passion for the work. At their best, these educators combine this passion with a solid understanding of how students experience college, what helps or hinders the students’ progress, and what outcomes result from their experience.

-Henscheid, J.M.& Keup, J.R. (2011, p.17)

Resourceshttp://www.ofyp.umn.edu/en/welcome-week.html

http://www.ofyp.umn.edu/en/survey-results.html

http://www.ofyp.umn.edu/ofypmedia/pdfs/highered/reports/2011_OFYPAnnual.pdf

http://www.oir.umn.edu/

http://www.oir.umn.edu/surveys/seru

Reference

Henscheid, J.M. & Keup, J.R. (2011). Crafting and conducting research on student transitions. Columbia, SC: University of South Carolina, National Resource Center for the First-Year

Experience and Students in Transition.

“When done well, quantitative research has the potential to enhance knowledge and inform programs, policies and pedagogies that educators use to serve students and contribute to their success”

(Henscheid & Keup, 2011, p.84)

Questions/Comments

Beth Lingren Clark, Ph.D.Director: Orientation & First-Year Programs

lingr004@umn.edu