irene f. goodman, ed.d., president, goodman research group, inc. denise callahan, director of...

Irene F. Goodman, Ed.D., President, Goodman Research Group, Inc.

Denise Callahan, Director of Scholarship Programs, The Ford Family

Foundation

NSPA Conference Connect HourOctober 15, 2014

STRATEGIES FOR EVALUATING YOUR

SCHOLARSHIP PROGRAM

Goodman Research Group, Inc.

Group discussion about the opportunities for and challenges of evaluating your scholarship program

What would you most like to know?What is most useful to you?How can evaluation help?

What you can start doing right away and what you can plan for in the long term

WHAT BROUGHT YOU TO THIS SESSION?


Will compare list generated with three main questions that scholarship providers have often posed in the past:

1. What impact are we making? 2. What is the success rate of our participants? 3. What happens to our Scholars down the road?

Other questions generated by today’s group that can be addressed by evaluation

EVALUATION: AN IMPORTANT AND OFTEN NEGLECTED PROGRAM

COMPONENT


Importance of evaluation

Considerations for evaluation

Needs assessment data are important Goal setting crucial Outcomes

Types of evaluation

Making links between activities and outcomes

TODAY’S TOPICS


The trouble with not knowing what you want is that you don’t know what you’ve got when you get it.

EVALUATION IS IMPORTANT BECAUSE….

Assists program staff in learning what is working and what isn’t

Decision making and performance management

Assists in obtaining funds

Provides accountability to board/funders/ other stakeholders

IMPORTANCE OF EVALUATION


Is it worth evaluating?

Maturity, coherence of program

If so, what kind of evaluation should be done?

Formative testing of program component ideas? Process evaluation or outcome evaluation or both? Short term outcomes or long term outcomes?

Funding realities

Feasibility

CONSIDERATIONS FOR EVALUATION


Type of Evaluation Questions Answered

NEEDS ASSESSMENT What is the problem? What do we need to do about it?

FORMATIVE EVALUATION As we’re developing our program, what changes can be made to make it better?

PROCESS EVALUATION(“The Black Box”)

Is the program happening? Are we doing what we said we would? How much of it do we do? Are we reaching our target audience? How can it be better?

OUTCOMES EVALUATION Is it working? What short-term changes are there as a result of using the materials/program?

IMPACT EVALUATION What longer-term changes or benefits occur? What is the overall project effectiveness?Goodman Research Group, Inc.

DIFFERENT TYPES OF EVALUATION

GOAL SETTING

“Looks like the ghost of unarticulated goals has come back to haunt us.”


FORMATIVE EVALUATION

No matter how far you have gone down the wrong road, turn back.


OUTCOMES EVALUATION

The proof of the pudding is in the eating.


Ask yourself: What do I want to find out? What would success look like?

How will I know? (Indicators)

Produce a list and be specificGet feedback and learn what else has been done

It’s okay that the questions you start with aren’t necessarily the questions you end with.

OUTCOMES QUESTIONS TO ASK


What impact do you wish to have?

What is realistic given your program activities?

Use “if/then” statements, e.g.:

“If we give $1,000 scholarships to 1,000 people then we expect X to happen.”

What is reasonable to expect in the short-term, in the mid-term, and in the long-term?

DEFINING IMPACT


OUTCOMES AND IMPACT


Short-term Intermediate Longer-term

Closer in timeEasier to measureMore attributable to program

More distant in timeHarder to measure

Less attributable to program

LOGIC MODEL

INPUTS ACTIVITIESSHORT-TERM

OUTCOMES

IMPACT(LONG-TERM)

OUTPUTS


Working on how to answer what you want to know

GROUP ACTIVITY


ELEMENTS & STRUCTUREOutcome Definitions:

Intended ImpactYour example:

Indicator Observable and measurable behaviors and condition

Your example:

Data source Sources of information about conditions being measured

Your example:

Applied to The specific group to be measured (all or a subset)

Your example:

Data interval When data will be collected Your example:

Target (Goal) The amount of impact desired

Your example:


LOGIC MODEL ELEMENTS AND STRUCTURE

Tracking your participants – start with adding baseline questions to application

Retrospective survey of alumni

Following participants prospectively and longitudinally (baseline and annually)

More rigorous: comparing scholars with finalists

FROM THE SIMPLE TO THE COMPLEX


Emails

Annual check-ins

Asking for preferred mode of communication

Birthday cards/postcards

Obtaining updated contact information

Using social media to reach out

TRACKING IS EXTREMELY IMPORTANT


Looking back to their experiences

Asking “retrospective pre” questions (When it has not been possible to get baseline data, these types of questions ask respondents to compare then to now.)

RETROSPECTIVE SURVEYS OF ALUMNI


Starting when they have just started their scholarship (baseline)

Expectations High school participation Stress level and barriers

Comparing annual or so how their perspectives change

Can be a one-time snapshot or, if you have the luxury, annually

FOLLOWING PARTICIPANTS PROSPECTIVELY AND

LONGITUDINALLY


This answers the counter-factual: How do scholars compare with similar students who did not receive your scholarship?

What might have happened/not happened if the students had not received the scholarship?

Must statistically control for group differences

Can also compare with state, regional, and national data

MORE RIGOROUS: COMPARING SCHOLARS WITH FINALISTS


CONTACT US! Irene F. GoodmanGoodman Research Group, Inc. www.grginc.com(617) [email protected]

Denise CallahanThe Ford Family Foundation www.tff f.org(541) 485-6211dcallahan@tff f.org

QUESTIONS OR COMMENTS?


http://www.grginc.com/

mailto:[email protected]

http://www.tfff.org/

mailto:[email protected]

irene f. goodman, ed.d., president, goodman research group, inc. denise callahan, director of...

Documents