Mentoring Partnership of Minnesota

Quality in Action

May 4, 2011

Mentor Program Evaluation

22

Webinar Logistics

Asking Questions & Sharing Comments During the Webinar

1. “Raise your hand” & MPM Organizers will unmute you

2. Or, type questions (and comments) in the question/answer section and submit; we will respond directly to you or possibly share your question with all attendees

When unmuted, please monitor your background noise

April Riordan, Director of Training and Community Partnerships

Recordings of past webinars are available on our YouTube Channel.

2

Overview Introduction & Best Practices

Elements of Effective Practice Evaluation Plan & Logic Model

Tips for Selecting Instruments Oregon Mentors Evaluation Toolkit

Important Rules for Administering Instruments

Additional Resources

3

What are Programs Measuring?

4

Mat

h Gra

des

Readin

g Gra

des

Scienc

e Gra

des

Social

Scie

nce

Grade

s

Overa

ll Gra

des

Schola

stic

Appea

l

Schola

stic

Compe

tenc

e

Educa

tiona

l Exp

ecta

tions

Absen

ces

Tardin

ess

Suspe

nsion

Quality

of W

ork

High S

choo

l Gra

duat

ion

0%

10%

20%

30%

40%

50%

60%

70%

80%

Percentage of Programs Measuring Academic Per-formance

What are Programs Measuring?

5

Misc

ondu

ct

Proac

tive

Behav

ior

Hyper

activ

ity

Youth

Sub

stan

ce U

se

Youth

Pre

gnan

cy/P

aren

ting

Youth

Gan

g In

volve

men

t

Positiv

e Rela

tions

hips

with P

aren

ts

0%

10%

20%

30%

40%

50%

60%

70%

80%

Percentage of Programs Measuring Behavior

What are Programs Measuring?

6

Emot

ional

Self-e

ffica

cy

Self C

ompe

tenc

y

Develo

pmen

tal A

sset

s

Emot

ional

Proble

ms

Peer

Relatio

nship

s with

Pee

rs

Mat

ch Q

uality

: Men

tors

help

cop

e with

pro

blem

s

Mat

ch Q

uality

: Unh

appin

ess

with m

ento

r

Mat

ch Q

uality

: Clos

enes

s to

men

tor

Relatio

nship

Qua

lity

Mat

ch Q

uality

: Clos

enes

s to

men

tee

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Percentage of Programs Measuring Psychological and Social Functioning

What are Programs Measuring?

7

Less than 5 5 through 10 10 through 15

15 through 20

20 through 25

Over 25 0%

5%

10%

15%

20%

25%

30%

Total Number of Indicators Measured (Out of 31)

Elements of Effective Practice

Ensure program quality and effectiveness:Develop a plan to measure program process:• Select indicators such as training hours,• meeting frequency and relationship duration; and• Develop a system for collecting and managing

specified data.Develop a plan to measure expected outcomes:• Specify expected outcomes;• Select appropriate instruments to measure outcomes• Select and implement an evaluation design.Create a process to reflect on and disseminate evaluation findings:• Refine the program design and operations based on

the findings; and• Develop and deliver reports to program constituents,

funders and the media

8

Mike GarringerAdvisor

National Mentoring Center Education Northwest

Celeste JanssenProgram Director

Oregon Mentors

Panelists

9

Measuring Effectiveness

How well you run

your program

How well your

volunteers mentor the

youth

Quality of match

relation-ships

Mentee’s attitudes

on certain topics

Youth behavior

(outcomes you wish

to achieve)

10

Tips for Evaluating Mentoring Programs

Let’s look at the mechanisms of mentoring…

11

Missed Opportunities in Mentoring Program Evaluation

1. Service deliveryDid they get the mentoring promised?

2. Participant satisfaction

3. Mentoring relationshipDid they form the bond needed to make any of this work?

4. Intermediary outcomes are OK!

12

What is a Logic Model?

Systematic picture of how you believe your program works

“…helps you visualize and understand how human and financial investments can contribute to achieving your intended program goals.”

13

Sample Vacation Logic Model

14

Sample YouthBuild Mentoring Logic Model

15

Tips for Selecting Good Instruments

• Try to use instruments that have been tested for reliability and validity

• Make sure they fit your logic model, especially in terms of proximal and distal outcomes

• Note the complexity of the tool: – Age range– Reading level– Concepts

• Can you provide a comparison group of some kind?

16

Quantitative Tools

Indicator: a variable with the purpose

to measure change in a phenomena or process

Valid: Does the tool measure

what it claims to measure? Reliable:

Does the tool give consistent results?

Pre and Post Test

17

Example

Pretest Posttest

Pretest Posttest

Live Demo - Oregon Mentors Evaluation Instrument Toolkit

Browse for specific instruments (19 “domains”)

Simple format for each instrument “record” Tips for using Links to other online repositories

18

Toolkit Domains of InterestLive Demo of Toolkit & Instruments

Personal Attitudes and Beliefs:Antisocial attitude scaleDecision making skills scaleGoal Setting Skills ScaleProsocial Involvement, Opportunities and Rewards

Mentoring relationship quality:Match Characteristics Questionnaire (MCQ)Youth Mentoring Survey (YMS)Youth Survey: Measuring the Quality of Mentor-Youth Relationships

19

Comprehensive Surveys:Youth Outcome SurveyYouth Risk Behavior Surveillance System

Connectedness:Hemingway Scale of Adolescent Connectedness

School and Academics:Education Expectations and Aspirations ScaleMotivated Strategies for Learning Questionnaire (MSLQ)

Tips for Administering Instruments

1. Plan for problems

2. Don’t hand-pick which questions to use

3. Don’t reword questions

4. Get parent permission

5. Provide a private space to administer

6. Don’t provide guidance

7. Administer the instruments in a consistent, uniform way

8. Keep the answers anonymous/ sort out any coding before administration

9. Plan for data storage and safety

20

Resources

21

Resources

22

Resources

W.K. Kellogg Foundation Evaluation Handbook

W.K. Kellogg Foundation Logic Model Development Guide

23

Stand by Me: The Risks and Rewards of Mentoring Today's Youth (The Family and Public Policy) by Jean E. Rhodes

Buying Information Reviews Endorsements

Handoutshttp://www.slideshare.net/traininginstitute

1. Using Standardized Instruments2. Comparing Program Outcomes to National

Standards (Appendix B – 36)3. Blank Logic Model Template4. Sample YouthBuild Mentoring Logic Model5. Using Your Logic Model to Focus the Evaluation6. Checklist for Selecting an Evaluator7. Sample Contact & Information Release Form8. Sample Evaluation Press Release9. Evaluating Mentoring Programs – P/PV Brief10. Additional Reading and Resources for Evaluating

Your Mentoring Program11. Glossary of Measurement and Evaluation Terms

24

Other Resources

This presentation & others www.slideshare.net/traininginstitute

MPM Training www.mpmn.org/traininginstitute

Web sites & PDFs www.delicious.com/traininginstitute

25

Next Quality in Action Webinar

June 1 – Eat Well, Stay Active, Have FunA new resource from the Harvard School of Public Health, "Eat Well, Stay Active, Have Fun: A Guide for Mentors," provides mentors with information and guidance on how to make healthy eating and physical activity natural parts of the mentor-mentee relationship. Join this webinar to learn more about the guide and how local mentors are using it.

26

Mentoring Program Evaluation & GoalsMonday, May 16 - 10:00-11:15am Pacific TimeGo to Friends for Youth's Mentoring Institute to

register online for this FREE webinar!