volunteer michigan: an introduction to evaluation by robert w. kahle, ph.d. kahle research solutions...

Volunteer Michigan: An Introduction to Evaluation

By Robert W. Kahle, Ph.D.

Kahle Research Solutions

www.KahleResearch.comMarch 22, 2011

Bob Kahle, Kahle Research Solutions

◦Sociologist with a Ph.D. from Wayne State

◦Designer and analyst of the Mentor Michigan Census now in its eighth wave.

◦ Involved with multiple program level evaluations, including Volunteer Michigan, Prisoner Reentry Initiative in Detroit and Mentoring Programs.

◦ Integrate evaluation analysis with strategic planning facilitation so that new strategic plans are developed with metrics to measure plan achievement.

3/22/11Kahle Research Solutions Inc. 2

Be able to define evaluation and distinguish◦Formative (Needs Assessments, Process Evaluation)◦Summative (Outcome, Impact, Cost Benefit)

Understand an overview of four broad evaluation strategies

Understand prerequisites for conducting evaluation◦The logic model and theory of change◦Data collection capacity◦Program maturity, capacity and commitment

Be able to describe benefits and challenges of evaluation◦Understand what you miss if you do not evaluate


Evaluation research is the systematic application of social research procedures for assessing the conceptualization, design, implementation and utility of social intervention programs (Rossi and Freeman, 1989).



Evaluation is the the systematic acquisition and assessment of information to provide useful feedback to some object (Trochim, 2006).



Evaluation is the the systematic acquisition and assessment of information to provide useful feedback to some object (Trochim, 2006).

Volunteer program evaluation is the ongoing process of collecting and analyzing data to provide the organization with adequate knowledge on which to base programmatic and organizational decisions (HandsOn Network).


Process Evaluation seeks to answer questions like:◦What is the program experience for participants?

Were they satisfied with their experience? ◦Did the program recruit appropriate volunteers? How

Many? Who were they (demographics)? ◦Were volunteers screened, trained and supervised? ◦What changes need to be made to the program to

improve the experience for participants? To more effectively address the specific problems?

◦How does the program actually work (compared to how it was designed?)

◦How can it be improved?


Outcome Evaluation seeks to answer questions like: ◦ Did the program have the desired effect on the

participants and the problem you seek to address? ◦ Does the program work as intended? ◦ What changed as a result of these volunteer efforts? ◦ Can we observe and document the change? ◦ Did we change what we intended to change?

If so, by how much? Were there unintended changes or consequences?


What are some of the key process evaluation questions specific to your Volunteer Michigan efforts?

Please type one or two process questions into the chat box on your screen.


What are some key outcome evaluation questions specific to your Volunteer Michigan initiative?

Please type one or two outcome evaluation questions into the chat box on your screen.


1. Funders and other key constituents (board members, partners, community leaders) deserve (require) to know if and how the program works.

2. Good process information can help you improve your program and overall organization resulting in better service to target audience.

3. Lessons learned can be shared with others to help them strengthen their programs and organizations.


…accept that the status quo is good enough.

…be forced to rely on anecdotes and opinions, rather than data and facts, when promoting, marketing and raising funds your program.

…risk being perceived as “old-school”, unsophisticated.

…be forced to rely on gut instinct to make decisions and plans.

…complete the program or project and be neither better prepared nor smarter for the next one.

Kahle Research Solutions Inc. 3/22/11 13

Other than MCSC and VCM, who are your key constituents?

What do your key constituents want to know about your Volunteer MI efforts? Why?

Please type your responses into the chat box on your screen.


(Source: Trochim, 2006)



Emphases: Impartiality, accuracy, objectivity, validity.

Methods: Experimental designs, Randomized Control Trials (RCT), econometric, cost-benefit.



Emphases: Comprehensiveness, continuous process improvement, improved mgt. decision making, leadership. Methods: PERT = Program Evaluation Review Technique.Logical FrameworkUTOS = Units, Treatments, Observations, Settings



Emphases: Understand context, subjective human experience, culture, environment. Methods: Personal Interviews, Observation, Focus Groups, Grounded Theory



Emphases: Understand and enhance participant experience. Methods: Personal Interviews, focus groups, customer/participant satisfaction surveys, analyses of repeat participation, loyalty to program/organization


Please type your response into the chat box on your screen.


Volunteers Paid staff Person managing volunteer efforts Outside evaluation experts Organizational decision makers Community members Board members Clients or those served by your organization Students and interns


Partner with your local college or university Recruit skills-based volunteers Network with other nonprofits Continue to learn (read, research, take additional

training, etc.)


Why are you conducting an evaluation? What decisions do you want or need to make upon

completion of the evaluation? Which strategy might be best for your organization? The purpose determines ◦Audience◦Methods◦Data to be collected

Note: This is the most important part of the process!


Connect evaluation efforts with organizational planning.◦Link objectives from strategic plan to key metrics

Always put evaluation efforts in context.◦The level of evaluation and the resources allocated to

the project need to be in sync. Ballpark estimate: evaluation should be about 10-15% of

overall program resource allocation.


A stakeholder is anyone who has a vested interest in the success (or failure) of your program, service, organization, corporation.

“Those groups without whose support the organization would cease to exist” (as defined by the Boston University’s Work and Family Research Network.)


Is program history and model documented and recorded? Are resources in-place? Have all stakeholders been identified? Are program staff in-place (job descriptions, roles

identified)? Are program goals and objectives clearly specified? (Logic

model) Does the organization have the capacity to collect relevant

data? How is the data going to be collected? Are program leaders willing to listen, change as a result of

evaluation research?


A. We are on it!B. Feeling good about evaluation, but still a way to go!C. Have started, but everything else is more important!D. When will this session be over? I have real work to

do.

In the chat box on your screen, please type the letter associated with the phrase above that best describes your readiness for evaluation.


An explicit statement of: ◦Resources◦Actions◦Expected Outputs◦Expected Outcomes

Short term Longer term


A diagram demonstrating the materials and activities that will produce the desired programmatic results.

There area at least five elements of a logic model:

1. Inputs2. Processes or activities3. Outputs 4. Outcomes5. Impact

Logic models help you follow a logic chain so that if you add these inputs and engage in these activities you will get these outputs and outcomes


Inputs - Resources used to meet your volunteer program goals. Examples: time, money, and materials.

Activities - Actions that the program engages in to meet the needs of those they serve or their constituents.

Outputs - The “bean counting” results of your efforts. These are what you get after you put all your inputs and activities into the program. Examples: Number of community members served, dollars spent, number of volunteers, etc. They answer the “what” question.

Outcomes - Short, intermediate and long-term meaningful changes that occur during, or as a result of, your program. Outcomes answer the question “so what.?” They are changes to knowledge, skills, behaviors, values and conditions.


Similar to Logic Modeling but: ◦More emphasis on making assumptions explicit◦Starting with the end in mind◦Depicting causal pathways and all necessary

ingredients to achieve END RESULTS


Source: Anderson, A. The Community Builders Approach to a Theory of Change. The Aspen Institute.


Lack of money and staff resources Concern that spending resources on

evaluation diverts resources from service to community

Fear that the evaluation will be “negative” and force change

Fear of complexity of evaluation, data collection, stats

Lack of existing data collection capacity Inability to start because tasks seem

overwhelming and scary


www.cdc.gov/healthyouth/evaluation/resources.htm American Evaluation Association (www.eval.org) Kellogg Foundation (Logic Models) (www.WKKF.org) Theory of Change (www.AckKnowledge.org)

and (www.theoryofchange.org) Local Colleges, Universities and Community Colleges Search Institute (www.search-institute.org) Aspen Institute (www.aspeninstitute.org)


volunteer michigan: an introduction to evaluation by robert w. kahle, ph.d. kahle research solutions...

Documents

evaluation research

kahle research solutionswww

bob kahle

volunteer program evaluation

outcome evaluation

evaluation analysis

program experience

broad evaluation strategies