Moving from Concept to Meaningful Assessment:

Using Logic Models to Maximize Efforts & Clarify

ResultsAshley Finley, Ph.D

Senior Director of Assessment & Research, AAC&U

National Evaluator, Bringing Theory to Practice

Bringing Theory to Practice Well-Being Research Project Webinar

April 29, 2015

BTtoP 2015-2017 Grant Emphases (http://www.bttop.org/sites/default/files/public/bttoprfp20152017.pdf)

O Category I: Well-Being Seminar GrantsO Up to $1,000) O Provide support for campus-wide planning discussions (seminars) that

give focused attention to a particular dimension of the well-being of students, faculty, or other leaders in creating and maintaining an engaged culture for learning. Seminars should deepen participants’ understanding of the institution’s commitment to whole-person development and the learning-related policies and practices, both inside and outside the curriculum, that support such endeavors

O For examples of other campus seminars, see: http://www.bttop.org/grants-funding/category-1-seminar-mini-and-student-grants

O Category II: Well-Being Research Project GrantsO Up to $10,000, plus institutional matching) O Implement and assess a campus-based two-year research project that

gives focused attention to the well-being of students, particularly those traditionally underserved by higher education. Projects should, through the gathering of evidence, provide justification for best practices for deepening and sustaining the institution’s commitment to whole-person development

O http://www.bttop.org/grants-funding/category-2-program-start-or-research-initiative-grants

Requirements for Well-Being Research Project Grants

Well-Being Research Project Grants proposals will be expected to include the following three components and shouldn’t exceed 8 pgs: O 1) Name and contact information of the primary contact

person (PI). Dates and timeline of implementation. Amount of funds requested (budget requirements listed in RFP).

O 2) Proposal Narrative—organized in the following categories: O Purpose of and rationale for the project O Participants by campus area and rationale for their selection O Facilitation process and proposed initial guiding questions that

reflect the context and culture of the institution O Anticipated outcomes and subsequent action steps. A logic

model outline or its equivalent is expected.O How the proposed work will link the research objectives to

advancing the larger campus strategy of attending to well-being as a core purpose

O Evaluation and reporting information (requirements listed in RFP; additional evaluation is always encouraged)

O 3) Budget (guidelines in RFP)

Questions?

Outcomes

AssessmentCurriculum

Co –Curriculum

The problem with program assessment…

Why Logic Models?

Assessment ChallengesO What are the long-term impact goals (the

vision for change) vs. more immediate outcomes?

O How is what we propose to do actually connected to the changes (outcomes) we propose to get?

O What types of evidence are needed to say something meaningful about outcomes?

O How do we think holistically beyond grant resources to other resources on campus that can help facilitate change toward outcomes?

Moving from Goals to Expected Outcomes

Would

LOVE to see

Would

LIKE to see

EXPECT to see

The need for collaboration necessarily increases as you move toward LOVE – ability to affect outcomes requires

greater resources and input

Long-TermIntermediateShort-Term

Expected Changes: short, inter-mediate

Products needed to assess outcomes, “countables”:

Actions needed to produce outputs:

Resources needed to start or keep going: OUTCOME

S(What should

improve as a

direct result of efforts

that contribute to the

long-term vision?)

OUTPUTS(What

counts as good

evidence?)

ACTIVITIES(What will students be asked to do?)

INPUTS(What is

needed for the

process?)

Impact Goals

(What is the hope for the future

for students

, for faculty, for the

institution?)

Long-term vision for change

•General education•Teaching and learning center•Counseling center•Student affairs•Student orientation•Career center•Alumni center

•Group dialogue•Small group problem-solving or applications•Engagement with contemporary issues, real-life questions

•Individual/ group Reflection•Summary of group dialogue•Critical application of well-being to course material•Multimedia

Expected Changes: short, intermediate:

Products needed to assess outcomes:

Actions needed to produce outputs:

Resources needed to start or keep going: OUTCOM

ESOUTPUTS

ACTIVITIES(What will

students be asked to

do?)

INPUTS(What is

needed for the

process?)

•Flourishing •Self-esteem•Resilience•Persistence•Self-authorship•Satisfaction with life•Hope[Resource: University of Pennsylvania Positive Psychology Center]

Impact

•Build culture of well-being at institution •Increase institutional commitment to whole student development•Increase commitment to “flourishing” across campus community

Questions?

Basic Principles of Logic ModelsO Should be developed collaborativelyO Intended to simplify plan for development

and function as communication toolsO Attentive to maturation of projects over

timeO Start with outcomes, not inputsO Assumptions underlie all dimensions

Selected Logic Model Readings & Resources:• Grayson, Thomas. 2012. “Program Evaluation in Higher Education.”

in Handbook on Measurement, Assessment, and Evaluation in Higher Education. Charles Secolsky & Brian Denison (Eds).

• Finley, Ashley. 2012. “Assessment and Evaluative Studies as Change Agents in the Academy.” in Transforming Undergraduate Education. Donald Harward (Ed).

• Kellogg Foundation Logic Model Development Guide: http://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide

A Case Study: Dartmouth Mentoring with Purpose

O Mentoring With Purpose = collaborative project to foster a world view of looking out for others; supporting the recently-launched Dartmouth By-Stander Initiative (against sexual assault) by engaging students as allies; and assessing the effects of cascade mentoring on the campus, community, and participants.

O A coordinated mentoring reflection program will allow students—and faculty and staff members--to work in a "cascade fashion" with people above them, beside them, and below them.

O Through an annual mentoring summit, dialogue groups, classroom discussions, and their own mentoring of others, students--with faculty, staff, community leaders--will learn more about themselves; about those whom they mentor; & about the possibilities & need for social change. They will address significant life questions while mentoring, helping them to flourish and to nurture others in flourishing as well.

O Aim of the program is to involve 200 students in first mentoring summit & 100 students in early discussion groups with faculty & staff. Then expand to include larger numbers of students & mentees and in surrounding communities, with eventual aim of constituting a national civic engagement model.

IMPACT

Outcomes

Activities

Resources

IMPACT

Defining Long-Term Impact, Intermediate, & Short-Term Outcomes:

Mentoring with Purpose Program

Would LOVE to see

Would LIKE to see

EXPECT to see

Who needs to be involved to reach goals of programmatic expansion and national impact?

• assess effects of cascade mentoring on campus, community, & participants

• Expand [program] to include larger #s of students & mentees & in surrounding communities

Engage students as allies, to work in a "cascade fashion" with people above them, beside them, and below them

Collaborative project to foster a world view of looking out for others…[with]…Eventual aim of constituting a national civic engagement model

Impact Goals

OUTPUTSACTIVITIES

INPUTS • Involved

students better understand who they are & who they might become•Dartmouth is known as a “mentoring” college•More connection bet. academic theory & learning outside classroom•Stronger community on campus across faculty, staff & comm. Members• Increase # students committed to helping create a just society

Outcomes

For all participants:•articulate value of mentoring & rel. bet. mentoring theory & practice

Students will:•Dem. under. of CE & social change through mentoring• Increased awareness of potential for personal growth•Dem. know. of mentoring skills•Apply curr. Learning to exp, outside classroom•More committed to create just society

Faculty/Staff/CL•Dem. Increased know. of student lives outside classroom

•Plan overall program w/ stakeholders•Glean learning from guest lecturers•Plan summit•Create recruitment training, implementation, debriefing, assess. plan for dialogue grps•Train presenters & facilitators•Annual mentoring summit•Annual mentoring dialogue grps. •Assessment•Write summary report

•New pres. w/ strong commitment to experiential educ.•Critical # of committed fac., staff & students•Coord. efforts of Tucker Foundation, Council on Service & Eng. & Dean of College•Existing mentoring programs•Relevant academic courses (CBL)•Far-reaching awareness of mindfulness

•Survey data on mentoring programs’ success vs. past success•Pre/post survey on participants’ understanding of personal values & goals•Pre/post survey on ability to connect learning outside of class•# of students, fac., staff, CLs as mentors•# of cross-campus & comm. Partner collaborations•# of younger people/peers mentored•Media coverage of mentoring activities

Institutional• Current assess.• Inst. Research• Student Affairs• Counseling center• Teaching Center

Faculty & Staff• No. of faculty & staff involved• Faculty & staff dev. resources

Student• Program resources• Inputs from student affairs?•Curricular inputs?

Mapping Outcomes Beyond the Student Level

Institutional• Assessment

workshops•

Communication strategies to promote awareness

• Transparency

INPUTS ACTIVITIES OUTPUTS

Faculty & Staff• Faculty & staff

orientation• Faculty & staff

development workshops

Student• Reflection papers• Summary pts from group discussion among mentors • Collaborative work?

Faculty & Staff• # of courses/progs w/ best practices• # of comm. contact hrs./integration with course material• Faculty/staff dev. hrsInstitutional• increase in assign/courses focused on well-being• Posters/banners on well-being• recognition event

Student• connection of well-being & mentoring to coursework•Group work• Activities in co-curr

Student• Increased flourishing• Increased self-esteem•Resilience

Faculty & Staff• Use of pedagogies to support mentoring• Awareness of Well-being• Bldg. Communities of Practice

Institutional• Evidence of connection of well-being & learning & CE• Retention• Campus awareness

OUTCOMES

Questions?

Summary Considerations for Working with Logic Models

O Have you separated long-term impact goals from intermediate and short-term outcomes

O What are the activities of the program? What is happening to effect change in students? What about faculty? At the institutional level?

O What are students doing as a result of their participation that can be counted as evidence? What other sources of evidence will be used to assess outcomes? Ditto at faculty and institutional levels.

O What resources exist that contribute to project goals and outcomes?

Additional Assessment ResourcesO Logic Models: http://

www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide

O Bringing Theory to Practice Toolkit Instrument & Overview document (includes the flourishing scale): http://www.aacu.org/bringing_theory/assessmenttools.cfm#Tlkt

O University of Pennsylvania Positive Psychology Center: http://www.positivepsychology.org/ppquestionnaires.htm

O AAC&U VALUE Rubrics (direct assessment of learning outcomes, including civic engagement & intercultural competence): http://www.aacu.org/value/index.cfm

O Wabash Center for Inquiry in the Liberal Arts: http://www.liberalarts.wabash.edu/study-instruments/

Post-Webinar Questions?

O Questions about grant guidelines or due dates: info@bttop.org

O Questions about logic models or project assessment: Ashley Finley, finley@aacu.org