1

HEP Evaluation: Measurement of

Strategy Implementation

Office of Migrant EducationUnited States Department of Education

HEP Technical Assistance Webinar 28 September 2011

2

Purpose of the Webinar

1 12

1

45

7

4

9

6

0

2

4

6

8

10

No.

of G

rant

ees =

40

Stratified GPRA 1 Chart: 2010 APR HEP Data We have variation

in our results.

3

Purpose of the Webinar 2010 Target = MET!

National Target = 69% of students attain a GED 2009-2010 National results = 70% Performance results reflect an increase of 9%. Performance results excluded first year (2009) projects.

Share Practices from Improving or High Performing Projects

IMPROVE HEP results nationwide, so that we may effectively and efficiently assist Migrant students obtain their GED’s and place them on the path toward success.

4

Today’s Objectives Review the meaning of program evaluation. Review the components of program

evaluation. Individual HEP project directors present

effective ways to measure strategy implementation.

Questions and answers for our presenters. Leave our webinar with ideas to improve

measurement of strategy implementation!

5

The Meaning of Program Evaluation

Evaluation means systematically and methodically collecting information about a program in order to improve the program or make decisions about the merit or worth of the program.

6

Components of Program Evaluation

1. To Determine Overall Effectiveness in Meeting Program Goals/Objectives (Are we on track?)

2. To Determine the Level Program Activities Are Being Implemented (Did we really deliver what we agreed to deliver in the application? How do we know?)

3. To Identify Strengths and Weaknesses in Program Implementation and Program Effectiveness (What strategies are working, and which ones are not?)

4. To Provide Recommendations for Change in Program, in Order to Improve Results (What can we do differently to change the results?)

7

Components of Program Evaluation

To Determine Overall Effectiveness in Meeting Program Goals/Objectives

Are program activities effective, leading to the achievement of strong GPRA 1 and GPRA 2 results and reaching these targets?

What evidence is available that supports the conclusions?

Are program activities cost efficient?

8

Components of Program Evaluation

To Determine the Level Program Activities Are Being Implemented (Prerequisite to ALL Evaluation Practices)

“With a level of quality, and all of the time!” “With a level of quality, most of the time ” “With a level of quality, some of the time …” “We really didn’t have the time ”

9

Components of Program Evaluation

Why study level of implementation? It can help answer the questions:

1. Did our project fulfill its responsibilities in our application?

2. Did our project’s strategies make a difference in meeting our GPRA targets?

10

Components of Program Evaluation

To Identify Strengths and Weaknesses in Program Implementation and Program Effectiveness Through customer satisfaction surveys (plus and delta, all

services) Through observations (e.g., tutoring, instruction, placement) Through documentation (e.g., logs of recruitment, counseling,

tutoring) Through exit interviews Through research, by finding correlations between practices

and results (using student level data, discovering relationships between practices and achievement)

11

Program Evaluation’s Purpose

To Provide Recommendations for Change in Program, in Order to Improve Results. Examples: Tutoring targeted to specific skills

(teachers/mentors/students) Mentors with regular and structured meetings, specific

supports Individualized Education Plans with specific learning

objectives and learning plans Structured recruiting plan

12

Measurement of Strategy Implementation

HEP Directors Present Ideas… Antonia Luna, Texas State Technical College at

Harlingen Adan Suarez, Columbia Basin College Michelle Kelly, Boise State University Karla Peterkoski, Michigan State University Santos Manrique, Fort Scott Community College

13

Measurement of Strategy Implementation

Questions for the presenters, from the audience…

14

Summary

Good evaluation will tell you:

If program activities are delivering results. If program activities are being highly implemented. Program strengths and weaknesses. Suggestions for program improvement.

15

Program Evaluation

THANK YOU!In order for OME to better serve your needs, please provide an evaluation of today’s webinar! Ed will send the link…

Also, if you’d like to discuss your program’s evaluation process…

Ed MonaghanEdward.Monaghan@ed.gov

202.260.2823