PROGRAM PROGRAM EVALUATIONEVALUATION

Carol DavisCarol Davis

Purpose of EvaluationPurpose of Evaluation

The purpose of evaluation is to The purpose of evaluation is to understand whether or not the understand whether or not the program is achieving the intended program is achieving the intended outcomes.outcomes.

Helps identify what works and does Helps identify what works and does not work.not work.

Today’s Evaluation Today’s Evaluation DiscussionDiscussion

1.1. PEEC Outcomes PEEC Outcomes (Indigenous (Indigenous Evaluation)Evaluation)

2.2. Evaluation PlanEvaluation Plan

3.3. Formative EvaluationFormative Evaluation

4.4. Progress EvaluationProgress Evaluation

5.5. Summative EvaluationSummative Evaluation

6.6. Write ReportWrite Report

PEEC: Indigenous Evaluation PEEC: Indigenous Evaluation FrameworkFramework

Indigenous EvaluationIndigenous EvaluationConsultantsConsultants

American Indian Spiritual Leaders, American Indian Spiritual Leaders, Cultural Leaders and Educators Cultural Leaders and Educators assisted in the preparation of this assisted in the preparation of this guide. It’s main function is for guide. It’s main function is for evaluation of Indian Education evaluation of Indian Education programs. programs.

Four CoreFour CoreValues EmergedValues Emerged

1. Tribes are a people of place1. Tribes are a people of place 2. Tribes recognize their gifts2. Tribes recognize their gifts 3. Tribes Honor family3. Tribes Honor family 4. Tribes respect sovereignty4. Tribes respect sovereignty

Indigenous EvaluationIndigenous EvaluationFraming PrinciplesFraming Principles

Tribal Colleges reflect tribal core values.Tribal Colleges reflect tribal core values. When we create programs, we are When we create programs, we are

creating a set of understandings.creating a set of understandings. These understandings are our These understandings are our

hypothesis of what we believe will hypothesis of what we believe will happen.happen.

Our evaluations need to tell our story.Our evaluations need to tell our story.

Guiding Principles Guiding Principles

An Indigenous framework can An Indigenous framework can incorporate broadly held values while incorporate broadly held values while also remaining flexible and responsive also remaining flexible and responsive to local traditions and cultures.to local traditions and cultures.

Responsive evaluation uses practices Responsive evaluation uses practices and methods from the field of and methods from the field of evaluation that fit our needs and evaluation that fit our needs and conditions.conditions.

CultureCulture

““[It] is culture that provides the tools [It] is culture that provides the tools for organizing and understanding our for organizing and understanding our worlds in communicable ways.”worlds in communicable ways.”

- Jerome Bruner-- Jerome Bruner- The The

Culture of Education Culture of Education

Traditional KnowledgeTraditional Knowledge

““Every act, element, plant, animal, Every act, element, plant, animal, and natural process is considered to and natural process is considered to have a moving spirit with which have a moving spirit with which humans continually communicate.”humans continually communicate.”

-Dr. Gregory Cajete- -Dr. Gregory Cajete-

Native Science: Natural Laws ofNative Science: Natural Laws of

Interdependence (2000)Interdependence (2000)

Evaluation PlanEvaluation Plan

The purpose of evaluation planning The purpose of evaluation planning is to assess the understanding by all is to assess the understanding by all stakeholders of…stakeholders of…

Project GoalsProject GoalsObjectivesObjectivesStrategies, andStrategies, andTimelinesTimelines

Evaluation Planning is GuideEvaluation Planning is Guidefor Formative and for Formative and

SummativeSummative When you “plan to tell your story,” When you “plan to tell your story,”

your formative and summative your formative and summative evaluation becomes more than evaluation becomes more than numbers.numbers.

You collect information. You do You collect information. You do surveys. You collect information over surveys. You collect information over time. You analyze results.time. You analyze results.

Evaluator RoleEvaluator Role

To assure that the program To assure that the program evaluation will meet the needs of the evaluation will meet the needs of the organization, reservation, and/or organization, reservation, and/or community,community, require that the require that the evaluator submit an evaluation evaluator submit an evaluation proposalproposal prior to beginning the prior to beginning the project.project.

Intended OutcomesIntended Outcomes

Outcomes create the desired change Outcomes create the desired change or impact we seek through this or impact we seek through this program.program.

Stakeholders should brainstorm the Stakeholders should brainstorm the intended outcomes and post them. intended outcomes and post them. That will assure that everyone is That will assure that everyone is working toward the same outcomes.working toward the same outcomes.

Examining the StoryExamining the Story

What do we want to know as the What do we want to know as the story unfolds? story unfolds?

Drafting questions/statements guide Drafting questions/statements guide the process of tracking the story.the process of tracking the story.

DataData

DataData Data Data Data Data Data Data DataData Data Data Data Data Data Data DataData Data Data Data Data Data Data DataData Data Data Data Data Data Data DataData Data Data Data Data Data Data DataData Data Data Data Data Data Data

ObjectivesObjectives

Objectives are Objectives are measurable.measurable.

OutputsOutputs Your outputs result from your Your outputs result from your

activities.activities.

Staff can assist in determining the Staff can assist in determining the outputs for each activity. outputs for each activity.

A process for “documenting” is A process for “documenting” is essential and necessary when writing essential and necessary when writing reports.reports.

A Program is a storyA Program is a story

PEEC may want more than numbers. PEEC may want more than numbers. What is PEECs legacy? You may What is PEECs legacy? You may want to find out how the students want to find out how the students rate their participation in your rate their participation in your program or how faculty view your program or how faculty view your services.services.

Survey MonkeySurvey Monkey

SurveyMonkey's online survey SurveyMonkey's online survey tool is easy and quick. Create tool is easy and quick. Create many types of surveys using many types of surveys using their methodologist-certified their methodologist-certified survey templates.survey templates.

http://www.surveymonkey.com/

Report Program Impact?Report Program Impact? VALUE ADDED: VALUE ADDED:

e.g., Do you have a student who is the e.g., Do you have a student who is the first to attend collegefirst to attend college in the family? in the family?

Do you have a student who comes Do you have a student who comes from a from a foster homefoster home who is going to who is going to become an engineer?become an engineer?

Is this the Is this the first tribal memberfirst tribal member who who aspires to become an engineer?aspires to become an engineer?

Quantitative/QualitativeQuantitative/Qualitative

Both methods can contribute to telling the Both methods can contribute to telling the story. You decide which is appropriate.story. You decide which is appropriate.

QuantitativeQuantitative can build Breadth (These are can build Breadth (These are your numbers, percentages, etc.).your numbers, percentages, etc.).

QualitativeQualitative can build depth (These are can build depth (These are your personal stories, focus groups, etc.).your personal stories, focus groups, etc.).

Qualitative DesignQualitative Design It is important for American Indian It is important for American Indian

tribes to include qualitative reporting tribes to include qualitative reporting methods.methods.

Truth emerges from consensus among Truth emerges from consensus among informed people.informed people.

Value framework gives meaning to Value framework gives meaning to numbers.numbers.

Formative Evaluation Formative Evaluation

Formative Evaluation is assessment Formative Evaluation is assessment of the ongoing projects and activitiesof the ongoing projects and activities

Evaluation should be conducted for Evaluation should be conducted for action-related reasons and the action-related reasons and the information should be used to decide information should be used to decide a course of action.a course of action.

Progress EvaluationProgress Evaluation

Determine whether or not the project Determine whether or not the project is being conducted as planned within is being conducted as planned within the established the established timelines.timelines.

Summative Summative EvaluationEvaluation

Summative Evaluation is conducted Summative Evaluation is conducted at or toward the end of the program. at or toward the end of the program. The purpose is to assess the projects The purpose is to assess the projects outcomes to determine its success.outcomes to determine its success.

Use of Results of Use of Results of Summative EvaluationSummative Evaluation

SustainabilitySustainability Disseminate your modelDisseminate your model Continue fundingContinue funding Continue on probation, Continue on probation,

and/orand/or DiscontinueDiscontinue

Writing the ReportWriting the Report

Gather and organize the documents, data, Gather and organize the documents, data, Survey Monkey results & other Survey Monkey results & other informationinformation

Review the report format, instructions, Review the report format, instructions, and due date.and due date.

Create a checklist to make sure you Create a checklist to make sure you include all of the information.include all of the information.

Answer basic question: who, what, when, Answer basic question: who, what, when, where, why, how, how many.where, why, how, how many.

Today’s Evaluation Today’s Evaluation DiscussionDiscussion

1.1. PEEC Outcomes PEEC Outcomes (Indigenous (Indigenous Evaluation)Evaluation)

2.2. Evaluation PlanEvaluation Plan

3.3. Formative EvaluationFormative Evaluation

4.4. Progress EvaluationProgress Evaluation

5.5. Summative EvaluationSummative Evaluation

6.6. Write ReportWrite Report

ReferencesReferences American Indian Higher Education Consortium. American Indian Higher Education Consortium.

2009. 2009. Indigenous Evaluation Framework.Indigenous Evaluation Framework. Boyer, Paul. 2006. Should Expediency Always Boyer, Paul. 2006. Should Expediency Always

Trump Tradition? AIHEC/NSF Project develops Trump Tradition? AIHEC/NSF Project develops indigenous evaluation methods. indigenous evaluation methods. Tribal College Tribal College Journal. Journal. Vol. 18 No. 2. Pp 13-15.Vol. 18 No. 2. Pp 13-15.

LaFrance, J., & Nichols, R. (2010). Reframing LaFrance, J., & Nichols, R. (2010). Reframing Evaluation:Defining Indigenous Eval. Evaluation:Defining Indigenous Eval. Framwork. Canadian Journal of Eval. 23(2), 13-Framwork. Canadian Journal of Eval. 23(2), 13-31.31.

NSF.Gov/Evaluation. March 3, 2008.NSF.Gov/Evaluation. March 3, 2008. www.Unitedway.org/outcomes March 3, 2008.March 3, 2008.

MIIGWECHMIIGWECH

(Thank You)(Thank You)