Got Benchmarking Data? Now what?

NIRSA Pre-conference

April 1, 2009

Overview

• Introductions: Take a note of colleges/universities represented that might be most like your campus.

• General overview of benchmarking/intended outcomes• Definition of and types of benchmarking • Considerations and challenges of benchmarking• NIRSA/StudentVoice Benchmarking Study • Making sense of benchmarking data• Assessment reporting • Panel of campus recreation professionals• Discussion/questions throughout!

Intended Outcomes

• Participants will leave with an increased understanding of the value of benchmarking

• Participants will get hands-on experience viewing and understanding benchmarking data

• Participants will learn how to utilize benchmarking data for public relations, marketing, and importantly, they will leave with ideas on how to report data in meaningful ways to their various audiences

• Armed with new confidence around understanding and interpreting benchmarking data, participants will leave with an increased love of assessment and data!

Session Warning Labels• Three hours is a long time… especially if you are talking assessment!

(We have one planned break around 9:45, but take breaks as needed.)

• Warning: High doses of assessment are likely.• Some of the information is not going to make sense until your

college/university actually participates in benchmarking. • Getting the most out of the group exercise (of interpreting

benchmarking data) is going to be based on what emotions ‘data’ and assessment invoke in you!

• Set some of your own goals/outcomes for the session: Do you want to show others on your campus how to use the data? Do you want to learn best practices from other campuses? Do you want to be able to go back to your campus and produce a report? What else?

• It’s ok if you don’t get ‘it’ all! StudentVoice benchmarking webinars are available each May and December for free. Or I can do a webinar for you and others on your campus specific to your data.

Benchmarking

• Comparing performance and other measures across organizations/units.

• Benchmarking is done primarily to benefit from wisdom and success of others (best practices) so that you don’t have to “reinvent the wheel.”

• Allows institutions or departments to document what they are doing well (in comparison to national or peer data), as well as document opportunities for improvement.

Uses of Benchmarking

• Improvement; learning of best practices• Short and long term planning• Forecasting and predicting trends• Generating new ideas• Provide data for analyzing and determining affordability

and cost effectiveness • Exposes areas that are in need of follow-up/more in-depth

assessments

Types of Benchmarking

Types (typical in higher education)

1) Internal: departmental comparisons

2) External: comparisons to peer institutions

• Operational benchmarking (comparing staffing, facilities, programs, budgets) – may occur both internally and externally

• Student data benchmarking (comparing student responses) – may occur both internally and externally

Benchmarking Considerations

• Common questions asked across institutions • Timing of the data collection • Sampling methods • Selecting comparable institutions for comparisons • Many benchmarking opportunities have very strict

guidelines for participating campuses (rightly so – to ensure that the data is valid and reliable)

History of the NIRSA/StudentVoice Benchmarking Project

• Inspired by assessment efforts of Juliette Moore, Director of Recreational Sports at the University of Arizona and NIRSA Past President, the Campus Recreation Student Outcomes Benchmarking Project provides a benchmarking mechanism to capture ways in which campus recreation enhances the lives of students.

• Online assessment to random sample of students (sometimes faculty, staff, other stakeholders).

• Over 85 colleges and universities have participated in the last five years.

• Data from over 70,000+ students, faculty, staff, and community members.

Campus Use of the Data

• Data on the impact of campus recreation programs and services on recruitment and retention

• Document student learning outcomes • Justify new facilities, expansion, new programs• Change and enhance existing programs and services• Determine support for fee increases• Gauge interest in new programs and services• Implications for staff training and professional development • Spotlight areas of achievement and areas for improvement• Guide future/follow-up assessments

Benchmarking Topics

• Importance of recreation on recruitment and retention• Usage of facilities, services, programs, and offerings• Barriers to participation • Satisfaction• Outcomes• Needs assessment • Marketing/Promotions• Feasibility questions (if appropriate)• Demographics

Are we really measuring outcomes?

• Program Outcomes examine what a program or process is to do, achieve or accomplish for its own improvement; Generally needs/satisfaction driven.

• Learning Outcomes examine cognitive skills that students develop through department interactions; Measurable, transferable skill development; what students are expected to demonstrate in terms of knowledge, skills, and attitudes upon completion of a program, course, interaction, or activity.

Direct versus Indirect Assessment Methods

• Direct Methods - Any process employed to gather data which requires subjects to display their knowledge, behavior, or thought processes.

• Indirect Methods - Any process employed to gather data which asks subjects to reflect upon their knowledge, behaviors, or thought processes.

Process• Sign-up for the NIRSA/StudentVoice benchmarking project.

• Customize instrument as appropriate – keeping in mind that any changes to the instrument may impact the ability to benchmark.

• Obtain institutional approval/IRB – since your data will become part of an aggregate dataset.

• Launch the project; All data is collected online preferably with a random sample of students.

• Institution-specific data is available in real-time for immediate use.

• Benchmarking data and reports are available once all campuses are done collecting data.

• No college or university names are revealed alongside any college or university data.

Viewing Benchmarking Data

Data Reporting Tools

Benchmarking Wizard

Making sense of multiple layers of data

• Institutional data including filtered views to understand sub-populations or respondents with certain characteristics

• Longitudinal data• National results• Peer comparison data (who do you want to be able to

compare with?)• Filtered views by sub-populations across national and

peer comparison data

What’s in a StudentVoice benchmarking report?

• Frequency tables

• For scaled items:

• Means and standard deviations

• Whether the difference in means is statistically significant (green = ‘your’ mean is statistically higher at the p<.05; red = ‘your’ mean is statistically lower). NOTE: Statistical significance occurs with large sample sizes and the significance indicators need to be considered with this in mind. Advanced statistical analysis should be conducted in a stats package such as SAS or SPSS.

• Total N (number of responses), if this number is not ‘private’

• Top 2 and Bottom 2 percentages (Top 2 = Strongly agree plus somewhat agree)

• Ranking of ‘your’ mean score

• No basis to judge / Not applicable are excluded from all mean calculations. All benchmarking questions utilize Likert scales of 1-5, 1-4, or 1-3.

Stats 101

What basic stats knowledge do you need (or at least need to be aware of) to interpret a benchmarking report?

• Sample size and response rates

(http://www.touchpoll.com/calculator.htm)

• Generalizability

• Descriptive stats: Basic frequency tables

• Likert scales versus categorical data

• Means and standard deviations

• T-tests: Do two mean scores differ?

• Statistical significance

More advanced uses of assessment data:

• Factor analysis

• Correlation studies

• Regression

Small Group Work

• Find a work group with similar types of campuses or similar types of recreation facilities and programs

• Maximum of four-five people per group will be best• Knowing that each small group might approach the

exercise differently, if you are attending with another colleague from the same institution, it might make sense to split up and then share experiences afterwards.

Group Exercise Scenario

Let’s look at page 1 of the handout labeled Benchmarking Data Questions 2 and 3 together in a large group to make sure all is clear about how the data is presented and what is in the report.

Let’s also look at page 1 of the handout labeled Benchmarking Data Question 4 together.

Reporting out your findings

• Create manageable sections/chunks of data.• Assign/delegate to staff or teams and come up with an

appropriate strategy to make the data analysis feasible and manageable.

• Explore possibility of hiring a graduate assistant to assist with data analysis or find other campus partners.

Assessment Reporting

1) Determine the ‘client,’ purpose, and audiences for the report

2) Determine what data and information are available or must be gathered

3) Select the type of report and report format

4) Decide how to depict data and information

5) Produce report

6) Dissemination

Findings vs. Results vs. Implications

• Findings: Data represented as value-free facts. What is the data?

• Results: Interpretations of the findings. What do the findings mean?

• Implications: Uses of the results. Who should care about these results and what should they do with the results?

Knowing your Audience

• What do my audiences need to know about this subject?• What do my audiences want to know about this subject?• What do I want them to know about this subject?• What decisions will or might my audience make based on

this report?• What other audiences might my primary audiences send

this report to, even if I haven’t intended for them to receive the material.

• What other audiences might be interested in the same subject?

(Bers & Seybert, 1999)

Report Components1) Meaningful title2) Executive summary3) Table of contents4) Introduction and purpose5) Methodology6) Findings7) Summary, conclusion, implications, and

recommendations8) References9) Glossary10) Appendices

Graphical Representations of Data

Graphs are depictions of numerical relationships to demonstrate the following:

1) Size (e.g., largest to smallest)

2) How things change over time

3) What is typical or exceptional

4) How one thing predicts or is related to another

Avoid chart clutter/chart junk….

Huh?

Working - Studying & GradesTransfers - Fall 2006

0%

10%

20%

30%

40%

50%

60%

70%

80%

0 1-5 6-10 11-15 16-20 21-25 26-30 31-3536-40 >40

Work Hour Category

% S

tud

yin

g 1

0+

Ho

urs

per

Week

2.00

2.20

2.40

2.60

2.80

3.00

3.20

3.40

1st

Sem

este

r G

PA

StudyingMore than10 HoursPer Week

AverageFirstSemesterGPA

Presenting Clear Visual Representations of Data

Panelists

• Amy Davenport: University of Oklahoma

• Mary Chappell: University of Kansas

• Tina Clawson: Oregon State University

Opportunities to share best practices after today:

https://www.studentvoice.com/app/views/community/

Online Resource Center

Additional Resources

Bender, B. E., & Schuh, J. H. (2002). Using benchmarking to inform practice in higher education. New Directions for Higher Education no. 118. San Francisco: Jossey-Bass

Taylor, B. E., & Massey, W. F. (1996). Strategic indicators for higher education. Princeton, NJ: Peterson’s.

Upcraft & Schuh, (1996). Assessment in Student Affairs. San Francisco: Jossey-Bass.

Contact Information

Kim VanDerLinden

www.studentvoice.com

Email: kvanderlinden@studentvoice.com

716.652.9400 and then press 1