assessment institute dr. don levy – director, siena college research institute october 28, 2013...
TRANSCRIPT
Assessment Institute
Dr. Don Levy – Director, Siena College Research Institute
October 28, 2013Indianapolis, IN
• Fulfilling Higher Education’s Mission for Community Engagement
• NASCE Methodology
• NASCE Findings
• Utility of the NASCE as Institutional Assessment
• Institutional Case Study
Agenda
NASCE as Institutional AssessmentNASCE is an Institutional Assessment – uses individual-level data to assess
institutional community engagement
Use NASCE Data…As backbone of Strategic Plan
To assess campus culture of service and facilitate community engagement discussions with faculty and administrators
To establish realistic and structured community engagement goals
To implement real change at the systemic level and move all students along the continuum of service
As bi-annual assessment tool to continuously evaluate institutional progress in community engagement
Assess type, frequency, depth, motivations for, and obstacles to service on the part of students
Assess effects of institutional structures and programs on student community service
Generates individual level data that allows institutions to draw conclusions about student subsets
Comparative analysis with national dataset of more than 27,000 students and counting
NASCE as Assessment Tool
NASCE
• Assessment of institutional expression of service – the Capacity Contribution
• By measuring the behavior of students across 9 service dimensions
• In order to– Have a valid and
reliable measure– Publicize the
findings across campus
– Plan necessary implementations
– Re-measure
Belief that community engagement is essential to undergraduate education
Lack of substantive data on community impact in higher ed.
Disconnect between institutional mission and civic action
Authors saw need for community engagement measurement
NASCE uses student-reported service for quantitative analysis
NASCE Background
The university enriches the worldthrough scholarship, artistic expression,community engagement, and entrepreneurship.
Through academic programs, research, artistic expression, public service and community-based learning, the university serves as an educational, cultural and economic leader for the region.
As a leading metropolitan university, we achieve our mission by creating a learning environment where curricula are connected to societal issues through civic engagement.
We inspire our students to be citizens of character who demonstrate professional and civic leadership. We contribute to the vitality and well-being of the communities we serve.
We offer a unique blend of traditional classroom knowledge and innovative experiential learning that provides students with a solid foundation for a lifetime of learning, service, and leadership while valuing social responsibility and diversity.
Mission Statements
We view the three traditional pillars of the public higher education mission—research, education, and service—as interdependent endeavors that continually enrich and inform each other. Groundbreaking research, transformative educational experiences, and deeply engaged service to all communities…
George KuhColleges that over-perform on engagement live their mission
Mission and Engagement
Defining Community Engagement
Community Engagement
The extent to which college and university students serve the unmet human needs of people in their communities (i.e. Housing, Health Issues, Hunger, etc.)
Includes community service and engagement activities that require the time and effort of the participant, with the ultimate goal of bettering the community
Individual Efforts, Work or Internship, Clubs or Organizations, and Classes
Survey template customized to
school
All students sent four email invitations with link to survey
Students individually take
survey online
NASCE Methodology
Male32%
Fe-male68%
GenderClass Year
I would describe myself as:
Freshman24%
Junior24%
Senior29%
Class Year
Sophomore23%
Caucasian76%
Black or African-Amer-
ican8%
Asian7%
Hispanic or Latino
5%
Native Amer-ican<1%
Multi-racial4%
Other 2%
I would describe myself as:
81%
12%
5%
Which of the following best de-scribes your high school expe-
rience?I attended a public school for my entire time in high school
I attended a private school for my entire time in high school
I attended both public and private schools during my high school
Current Sample
Total n for study= 27,038
Average Age: 23 yearsAverage GPA: 3.26
12% non-traditional students
• Through Spring 2013: 46 different schools across 14 states
• Ranging in size from 800-23,000• 6 re-assessments (52 assessments in
total)• Large and diverse new cohort
• Including:• 15 New York Campus Compact
Schools• 6 SUNY schools • 20 Bonner schools
• 15 public schools• 9 schools with over 10,000 enrolled
students• 13 religiously affiliated schools
Participating Colleges and Universities
1. “DO YOU SERVE?”
Please consider the community service in which you have or are engaged. Remember, by community service we mean the following: any activity, including
internships, work study, or co-ops, in which you participate with the goal of providing, generating and/or sustaining help for individuals and groups who have unmet human
needs in areas like shelter, health, nutrition, education, and opportunity. Do you engage in community service here at XYZ?
NASCE Instrument
Homelessness Hunger Religious
Environmental Health Civic
Youth Elder Economic Justice
Measures student service across 9 areas
NASCE Instrument
NASCE Instrument
For each of the community service activity types that you checked, how often would you say you did that type of service. Would you say….
Once or twice a yearSeveral times during the yearAbout once a monthSeveral times a monthAbout once a weekMore than once a week
2. “HOW OFTEN DO YOU SERVE?”
Community Involvement
ONE SHOT
Volunteer once at a food drive
REGULAR BASIS FOR A PERIOD OF TIME
Mentor Middle School students every week for a
semester
DEEPLY COMMITTED
Volunteer at a Senior Living facility twice a week since
you were 16
NASCE Instrument3. “AT WHAT DEPTH DO YOU SERVE?”
For each of the types of service that you selected, which of the following describes your level of involvement? [CHECK ALL THAT APPLY]
I would participate at an event or short term drive. Usually it was a one-shot type of involvement.
I was involved on a regular basis for a period of time. One example would be a regular commitment to be there once a week for an entire semester, or another would be to participate on a service trip for most of each day for a period of time.
I was deeply involved in a project or cause and dedicated to it. Rather than thinking of my service as a chore or time commitment, I was drawn to serve by the issue or problem and worked towards its resolution.
The NASCE computes the POP Score by assigning values to those students who perform service in each area:
1. Engaged in any of the nine areas. • Yes (1)• No (0)
2. Frequency. • Once or twice a year (1)• Several times a year or once a month (2)• Several times a month (3)• Weekly or more (4)
3. Depth. • An event or drive; one-shot (1)• Regular basis for a period of time or a service trip (2)• Deeply involved and dedicated to a project or cause (3).
Percent of the Possible (POP) Scores
An individual’s responses are multiplied to create area level individual scores ranging from 0-12. These totals are summed across the institution and divided by the maximum score.
The raw score for each area is expressed as:
(Service * Frequency * max(Depth))12
The institutional raw POP Score is computed where n=the number of areas:
(Service * Frequency * max(Depth))n*12
The area level scores are averaged to create the institutional percent of the possible. Both institutional and area scores are then normalized with .33 equaling a POP Score of 100.
Percent of the Possible (POP) Scores
60+Impressive
• Impressive Service: Students serve frequently and maintain substantive connections with service sites beyond individual projects. Your institution is making a difference.
40-60High
• High Level Service: Students are engaged. Small gaps in participation, frequency, or depth hinder peak service. Potential for greater contribution exists.
20-40Moderate
• Moderate Level Service: Students are serving; however, significant gaps in student participation, frequency, or depth present opportunity for improvement. Data and the POP scores identify areas of weakness.
0-20Low
• Low Level Service: Little student involvement. Service takes the form of "one-shot" activities with little commitment. If the school wishes to expand its service contribution, data points the way.
POP Scores
• Each year, John takes part in one Saturday morning environmental cleanup and then months later sees a sign for a food drive and drops by with some cans of soup.
• Areas of Need John Served: Hunger and Environmental
Environmental
Hunger Total
Participation
1 1 ------------
Frequency 1 1 ------------
Depth 1 1 ------------
Overall Contributi
on
1 1 2
Food
Drive
Total
Raw POP .0208
Individual POP
6
Overall POP 6
Frequency=LowDepth=Low
Frequency=LowDepth=Low
Example of POP Score
• Mary is also involved in the same environmental cleanup day and canned food drive as John each year. However, she also has tutored elementary school students every Tuesday afternoon for an entire semester.
• Areas of Need Mary Served: Hunger, Environmental, Youth
Example of POP Score
Environmental
Hunger Youth
Total
Participation
1 1 1 ---------
Frequency 1 1 4 ---------
Depth 1 1 2 ---------
Overall Contributi
on
1 1 8 10
Total
Raw POP .1042
Individual POP 32
Overall POP 32
Food
Drive
Frequency=Low
Depth=Low
Frequency=Low
Depth=Low
Frequency=Low
Depth=Low
• Alex has worked diligently to create a program in which students are linked to elder members of the community. He has also spent his time weekly at a local city mission and cleaning up at a local park.
• Areas of Need Alex Served: Elder, Homelessness, Environmental
Frequency=Impressive
Depth=Impressive
Frequency=Impressive
Depth=High
Frequency=Impressive
Depth=High
Example of POP Score
Elder
Homelessness
Environmental
Total
Participation
1 1 1 ---------
Frequency 4 4 4 ---------
Depth 3 2 2 ---------
Overall Contributi
on
12 8 8 28
Total
Raw POP .2917
Individual POP 88
Overall POP 88
• Each student provides a certain amount of service measured in units.
Each student can provide 12 units of service per service area for a total of 108 units.
Example: College XYZ has 1000 students enrolled and every student participates in a food drive once a year.Individual Student’s Contribution: 1 unit per personInstitutional Contribution: 1000 units The total amount of service this institution could have done
theoretically was 12(9 areas) x 1000 = 108,000 units.
Each surveyed student’s unit of service contributes to the sample total which is generalized to the total student population.
6 units 32 units 88 units
Units of Service
School Comparisons
Variables School A School B School C
Institutional POP 52 28 13Hunger POP 60 26 13
Homelessness POP 41 25 8Religious POP 26 7 8
Environmental POP 42 32 11Civic POP 46 21 15
Health POP 38 35 25Youth POP 151 66 18Elder POP 9 9 5
Participated in Service In College 87% 67% 34%
Drop off From High School 9% 20% 47%
Institutional
Civic
Elder Care
Environmental
Health
Homelessness
Hunger
Religious
Youth
Economic
0 20 40 60 80Low Moderate High Impressive
17
8
22
12
15
16
40
17
16
Student POP Scores Across NASCE Sample
7
n = 27,038 students
52 total assessments between 2009 and Spring 2013
6 re-assessments
0% 5% 10% 15% 20% 25% 30% 35% 40%
POP Scores
Institutional POP Scores Across NASCE Sample
POP Score Range: 3 – 52Institutional Average: 21
0 - 10
11-20
21-30
31-40
More than 40
2%
37%
38%
10%
13%
0.2% 7.5% 14.8%22.2%29.5%36.8%44.1%51.5%58.8%66.1%73.4%80.8%88.1%95.4%0
10
20
30
40
50
60Capacity Contribution: Institutional
Percent of Respondents
PO
P S
co
re
54% of students report doing no service.
46% of students con-tribute to a POP score
of 17.
Capacity Contribution Examples
0.2% 7.2% 14.3%21.3%28.3%35.3%42.4%49.4%56.4%63.4%70.5%77.5%84.5%91.5%98.6%0
10
20
30
40
50
60Capacity Contribution: Youth
Percent of Respondents
POP
Scor
e
27% of stu-dents con-tribute to a
POP score of 44.
10% of students account for 63% of the Youth POP score.
73% of students do no service addressing youth needs.
Capacity Contribution Examples
Low Service
0 20 40 60 80
8
Low Moderate High Im-pressive
XYZ POP Score
Yes23%
No77%
Do you participate in community service here
at XYZ?
A School with a Low POP Score will:• Have a low Percentage of Students performing Service• Have a small percentage of students contributing to a high percentage of the overall POP Score
0.2% 7.1% 14.0%20.9%27.8%34.7%41.6%48.5%55.4%62.3%69.2%76.1%83.0%89.9%96.8%0
5
10
15
20
25
30Capacity Contribution: Institutional
Percent of Respondents
PO
P S
core
77% of students report do-ing no service.
23% of students contribute to a POP score of 7.
23% of students report doing service
10% of Students Account for 84% of the total service score
XYZ POP Score
0 20 40 60 80Low Moderate High Impressive
25
Yes70%
No30%
Do you participate in com-munity service here at XYZ?70% of Stu-dents Report Doing Service
0.4% 9.9% 19.4%28.9%38.4%47.9%57.4%66.9%76.4%85.9%95.4%0
10
20
30
40
50
60
70Capacity Contribution: Institutional
Percent of Respondents
PO
P S
core
30% of students re-port doing no ser-
vice.
70% of students contrib-ute to a POP score of 25.10% of
students account for 35% of the total service score
A School with a Moderate POP Score will:• Have a majority of their students reporting service.• Have a small percent of students contributing to less than half of the overall POP Score.
Service in Need AreasModerate Service
High Service
XYZ POP Score
0 20 40 60 80Low Moderate High Impressive
52
Yes87%
No13%
Do you participate in com-munity service here at XYZ?
0.4% 7.9% 15.4%22.9%30.4%37.9%45.5%53.0%60.5%68.0%75.5%83.0%90.5%98.0%0
10
20
30
40
50
60
70Capacity Contribution: Institutional
Percent of Respondents
PO
P S
core
13% of students report doing no
service.
87% of students contribute to a
POP score of 52.
87% of students report doing service.
10% of students account for 28% of the total service score.
A School with a high POP Score will:• Have the vast majority of their students reporting service.• Have a small percent of students contributing to a relatively small percent of the overall POP score.
0% 25% 50% 75% 100%0
5
10
15
20
25
30
35
40
45
50
Capacity Contribution: Institutional
Percent of Respondents
Pop
Scor
e
100% of students contribute to a POP score of 50.
0% of students report doing no service.
Ideal Capacity Contribution
Across the entire sample, 46% of college students participate in community engagement and service, a drop of 40 percentage points from high school.
SERVICECollegeHigh School
Drop in Service after High School; “The Melt”The percentage of students performing service and the institutional POP scores often decrease drastically from high school to college
Potential factors: Less structure and organization Decreased promotion of activities Fewer requirements Campus culture Other activities, including studying and social events, take up more time
The total Institutional POP drops 16 points from high school to college
43% 3%
43% 11%
Yes: High School No: High School N
o: C
olle
ge
Yes
: Col
lege
46 percent of Students Report Doing Service in
College
From High School to CollegeFour Types of Student Transitions
Volunteer
Yes/No and Yes/Yes: What’s the difference?
Continuing volunteers are more likely to be involved in clubs in college (90%) compared to high school only volunteers (62%)
Continuing volunteers are significantly more satisfied with their personal level of involvement in college than are high school only volunteers (77% vs. 45%)
Service by Need AreasElder Care and Economic Justice
are neglected need areas.Youth Services is the most
impactful area.Religion and Youth represent
deepest commitments.Environment and Health have
high rates of participation, but low frequency and depth
Percentage of students serving once a week or more:o Religion (48%)o Youth (38%)o Economic (21%)o Elder Care (19%)
Civic11%
Elder Care5%
Health14%
Hunger10%
Religion10%
Youth26%
Econ5%
Total Service by Needs Area
Environment10%
Homelessness 8%
Institutional Beliefs
Student Opinions (Agree or Strongly Agree):
84%: Overall, I would say that XYZ University promotes community service among the student body.
72%: I think the college or university does an appropriate job of informing students of all the ways they can be engaged in the community.
59%: Overall, I am satisfied with my personal level of involvement in community service here at XYZ University.
Previous bad experience
It makes me uncomfortable
What I would like to do is not available
Too busy with friends
Lack of transportation
Not enough energy
I don't know what is available
I have to work
Too busy with other activities
I have to study
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
63%
48%
44%
42%
32%
28%
18%
13%
84%
74%
Obstacles to Service
I don't want to feel guilty
Because my friends do it
I have been required to
It is important to have on my record
I meet people through participating
I want to change the world
It makes me feel good about myself
It is the right thing to do
I believe I can help people who are in need
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
96%
95%
94%
83%
83%
75%
59%
53%
24%
Motivations for Service
Campus Awareness
Campus News-paper
My RA or RD
Residence life
Organizational Fair
Through a club
In a class
Flyers
Word of mouth
Campus emails
0% 10% 20% 30% 40% 50% 60% 70%
22%
24%
29%
30%
49%
54%
62%
63%
68%
How have you heard about service opportunities available here at XYZ?
71% involved in at least one type of club
Type of Club Percentage Reporting Membership
Individual sports 30%
Academic clubs 28%
Cultural groups 22%
Religious groups 21%
Organized sports 18%
Political or environmental advocacy 13%
Student Government 8%
Community Youth Groups 7%
Student Clubs and Organizations
• 57% of students involved in a club or activity do service in college
• Only 18% of students not involved in any clubs or activities do service in college
“Ideally, institutions would structure the curriculum and other learning opportunities so that one high-impact activity is available to every student every year…In the short term, making high-impact activities more widely experienced should have a demonstrable impact in terms of student persistence and satisfaction as well as desired learning outcomes.”
-- George Kuh High Impact Educational Practices, AAC&U
Service-Learning – High Impact Practice
84% of students agree that one of the obstacles to volunteering more is having to study
Of the students who serve, 48% say they’ve done so “as part of a course.”Students who do service as part of a course:
o 88% are also involved in at least one type of club or activity in collegeo 76% are satisfied with their personal level of involvement (sample-wide = 59%).
Service learning, campus culture of service, and college structures as a solution to:o Having to studyo Too little timeo No transportationo Don’t hear about activities o What I’d like to do is not available
High Performers and the RestPOP Scores:
High Performers (top 5% of Sample) n = 1,437
The Rest of Students who Serve:n = 10,953
Institutional: 107 Institutional: 28
Hunger: 109 Hunger: 23
Environmental: 94 Environmental: 28
Youth: 190 Youth: 74
Civic Awareness: 121 Civic Awareness: 27
Religion: 108 Religion: 25
Elder Care: 61 Elder Care: 11
Homelessness: 94 Homelessness: 17
Health: 120 Health: 38
Economic Justice: 65 Economic Justice: 8
High Impact Community Engagement
Leadership Qualities
Organized or planned a service project that involved other volunteers:
Helped facilitate a group service project as a group leader:
Participated in a service-based trip:
34%
31%
25%
76%
75%
58%
HP’s The Rest
NASCE Reliability
15e. “About how many hours do you spend in a typical 7 day week doing community service or volunteer work?”
First Year Students
Seniors
0 hours 56% 49%
1-5 hours 33% 36%
6-10 hours 6% 8%
11 hours or more
4% 7%
NASCE Data
Freshmen Seniors
Yes 36% 51%
No 64% 49%
Q28. “…On average during your time at XYZ, how many hours a MONTH do you engage in community service?”
Q22. “…Do you engage in community service here at XYZ?”
Freshmen Seniors
More than 0 but less than 10
25% 30%
Between 10 and 20 6% 11%
More than 20 5% 11%
NSSE Data
Institutional
Civic
Elder Care
Environmental
Health
Homelessness
Hunger
Religious
Youth
Economic
0 20 40 60 80 100 120 140 160 180 200
Bonner POP Scores
Low Moderate High Impressive
84
28
60
88
88
34
200
NASCE Validity – An Example Bonner Data
Bonner Sample: n = 198
100% Report doing service in College
97% Reporting Service Prior to College
97% Involved in clubs in College
70% organized/planned a service project that involved other volunteers
70% helped facilitate a group service project as a group leader
60% Participated in service as part of a course
78% Participated in a service-based trip
76
57
76
Using NASCE to Enhance Community Engagement
Developmental Pathway
0
1
23
NASCE Data
Strategic Plan
Move students along “service continuum”
Increase service involvement and POP
NASCE
NASCE
NASCE
Improve Students’ Participation,
Frequency and Depth of Service
Target Areas of Need
LIVE the Mission
Participation
Frequency Depth
Win-Win-Win
2013 Data
HS Participation Rate: 93%College Participation Rate: 64%
POP Scores
Institutional: 26Civic: 28
Elder Care: 7Environmental: 17
Health: 30Homelessness: 26
Hunger: 25Religious: 15
Youth: 73Economic: 13
Institutional Case Study2009 Data
HS Participation Rate: 93%College Participation Rate: 49%
POP Scores
Institutional: 14Civic: 14
Elder Care: 3Environmental: 9
Health: 21Homelessness: 14
Hunger: 10Religious: 9Youth: 32
Economic: -- HighImpressive
Low
Moderate
• Established Central Engagement Office• Developed Large Bonner Program• Institutionalized Service-Learning
– Community Engaged First-Year Seminar– Certificate in Community Development– Interdepartmental Collaborations
• Increased Service Programs/Opportunities• Increased Service Staff/Administrators
Institutional Case Study
The How and Why:
Is it Perfect Assessment? No.
• NASCE overstates service
• POP places relatively large emphasis on frequent, deeply committed service compared to sporadic involvement
• Some institutions administer NASCE and fail to effectively use the data
Evaluating NASCE
But…
• NASCE is largest compiled dataset on community engagement in higher education
• NASCE goes more in-depth than any existing assessment
• NASCE is affordable, customizable, and informs data-driven solutions
Higher Ed’s Decision• Does Community Engagement matter?• Are we here to have a meaningful impact on the communities in which we are
nested?• Are we here to foster responsible, active citizens during and after college?
Data paves the way!• Assessment and evaluation is first necessary step to making informed decisions
about the effectiveness and importance of community engagement initiatives on campus.
• Assessment and evaluation provide the necessary structure to foster dialogue with all relevant parties about community engagement on your campus
• Is it Higher Ed’s duty to play an engaged role in the community?• \
• Do we wish to fulfill our Institutional Missions by creating a more sustained connection between classroom learning and community involvement?
How can we retain volunteerism at the high school level? 54% of students reported not doing serviceo How can we get these students involved?
How can we increase frequency and depth of service?Pinpoint students with deep commitment to service in admissions
Facilitate the transition from high school to college
Create structure and awareness
Enhance academic involvement and service learning
Future Paths and Recommendations
NASCE Final Analysis
• Final Report• Comparative analyses to national sample• Additional data breakdowns of subsets• Data Cross tabulations• Data Frequency Tables• Raw dataset• Phone calls/in-person meetings with NASCE Staff• Continued Support• Assessment Tool to incorporate into strategic plan• Follow-up Assessments to measure progress
www.siena.edu/sri