Alliance for Graduate Education and Professoriate (AGEP)

Evaluating Retention Strategies: Some Thoughts On The Homework

Patricia CampbellCampbell-Kibler Associates, Inc.

February, 2007

Wanda’s Evaluation QuestionWanda’s Evaluation Question

Are your evaluation results:Are your evaluation results:

Credible: Do you trust them?Credible: Do you trust them?

Relevant: Do they give information you can Relevant: Do they give information you can use to either improve the strategies or to use to either improve the strategies or to make judgments about the strategies’ make judgments about the strategies’ worth? worth?

Timely: Do you get the results fast enough Timely: Do you get the results fast enough to be able to use them?to be able to use them?

Pat’s Evaluation AssumptionsPat’s Evaluation Assumptions

The core evaluation question is “What works The core evaluation question is “What works for whom in what context?”for whom in what context?”

““Black hole” evaluations are bad.Black hole” evaluations are bad.

If you aren’t going to use the data, don’t ask If you aren’t going to use the data, don’t ask for it.for it.

A bad measure of the right thing is better than A bad measure of the right thing is better than a good measure of the wrong thing.a good measure of the wrong thing.

Acknowledging WIIFM increases response Acknowledging WIIFM increases response rates.rates.

Process is a tool to help understand outcomes.Process is a tool to help understand outcomes.

Outcomes are at the core of accountability.Outcomes are at the core of accountability.

Your Evaluations of Retention Strategies

Two thirds of the respondents do some evaluation of their retention strategies, most frequently:

giving students surveys after each activitygetting verbal feedback from students formally and informally giving students annual surveysconducting student exit surveys.

One quarter track students’ progress to degree as part of their evaluations.

Ten percent track student participation in retention activities.

One project is linking student participation in activities to student progression through their academic programs.

Your Evaluations of Retention Strategies More Comprehensive Efforts

“We evaluate all AGEP activities. The evaluation has taken a variety of forms including tracking students, surveys, focus groups and collection of anecdotal information from students.”

“Students receive an evaluation form at the end of each activity. Evaluation results are compiled and used to plan subsequent activities. One campus holds an evaluation brunch each April to assess all of the activities; an external consultant facilitates. Program-wide results are being assessed, participation is being linked to students’ progression through their academic programs.”

Others study specific components related to retention such as graduate advising and special summer programs.

Some Thoughts on MeasurementSome Thoughts on Measurement

Don’t reinvent the wheel; where possible use existing Don’t reinvent the wheel; where possible use existing measuresmeasures

Share measures with other projects. Common Share measures with other projects. Common questions can be useful.questions can be useful.

Look for benchmark measures that are predictors of Look for benchmark measures that are predictors of your longer term goals.your longer term goals.

Remember self developed measures need some Remember self developed measures need some checking for validity and reliability.checking for validity and reliability.

A sample with a high response rate is better than a A sample with a high response rate is better than a population with a low one.population with a low one.

For Those Who Are Doing It Yourselves: For Those Who Are Doing It Yourselves: Some Web-based ResourcesSome Web-based Resources

OERL, the Online Evaluation Resource LibraryOERL, the Online Evaluation Resource Library http://oerl.sri.com/home.htmlhttp://oerl.sri.com/home.html

User Friendly Guide to Program EvaluationUser Friendly Guide to Program Evaluationhttp://www.nsf.gov/pubs/2002/nsf02057/start.htmhttp://www.nsf.gov/pubs/2002/nsf02057/start.htm

AGEP Collecting, Analyzing and Displaying DataAGEP Collecting, Analyzing and Displaying Datahttp://www.nsfagep.org/CollectingAnalyzingDisplayingData.pdfhttp://www.nsfagep.org/CollectingAnalyzingDisplayingData.pdf

American Evaluation AssociationAmerican Evaluation Association http://www.eval.org/resources.asphttp://www.eval.org/resources.asp

Center for the Advancement of Engineering EducationCenter for the Advancement of Engineering Education http://www.asee.org/acPapers/2005-1983_Final.pdfhttp://www.asee.org/acPapers/2005-1983_Final.pdf