overview of chapters 11 – 13, & 17

Overview of Chapters 11 – 13, & 17

Clarifying the evaluation request and responsibilities

1. An organization has requested that you to complete an evaluation. What questions do you feel are important to ask before beginning the evaluation?

2. Are there times when you would decline a request for evaluation? Why?

3. What are some of the advantages and disadvantages in having an evaluation conduct by an external evaluator? By an internal evaluator?

(take 5 minutes with a partner to discuss)

Chapter 11

1. Identify the stakeholdersKey questions:Who are the sponsors and clients?Who are the managers and staff?Are there other interest groups, such as agencies, elected

officials, the public at large?

2. Identify the purpose for the evaluationKey questions: Why is the evaluation being requested?What question will it answer?How will the results be used? By whom?

Clarifying the Evaluation Request and Responsibilities

3. Determine if the Evaluation is AppropriateKey questions:How will the results be used?Will the results produce trivial information?Is the program too new?Is the evaluation purpose ethical?


Steps to follow: 1. Clarify the intended program model or theory

2. Examine the program in implementation to determine whether it matches the model and could achieve the goals or objectives

3. Explore different approaches to determine if they meet the stakeholders information needs

4. Agree on evaluation priorities and intended uses of the study


Internal vs. External evaluator

Internal: know the organization, history, and decision-making style and would be around to encourage use of evaluation results

External: bring a greater objectivity and sometimes specialized skill for a particular project or evaluation

Who will evaluate the program?

Setting Boundaries and Analyzing the Evaluation Context

1. After determining who the stakeholders are, how do you determine who should be involved in the evaluation process?

2. What should the evaluator consider in analyzing the political context in which an evaluation will occur?

(take 5 minutes to discuss with a partner)

Chapter 12

The evaluator should identify and communicate with each stakeholder group or its representative:

1. To learn about the groups concerns2. To understand how they will use the

evaluation results

This will help the evaluator weigh their input during the evaluation


Do evaluators need to be aware of Squeaking wheels &Powerful Stakeholders?

You need to bring about a “democratic dialogue” if possible, with a balance in the number of people involved and diversity within the group.


Greene (2005) identified four groups:1. People who have decision authority over

the program2. People who have direct responsibility for

the program3. People who are the intended beneficiaries

of the program


Scriven (2007) adds to the list:

Political supporters or opponents

Community Leaders

Public in General


Political Context:

1. Who will gain to gain/lose?2. Which individuals and groups have power in the

setting?3. How should the evaluator relate/communicate to the

groups?4. Will all groups cooperate?5. Which groups have a vested interest in the

outcomes?6. What safeguards should be incorporated into the

evaluation agreement?


Identifying and selecting the evaluation questions and criteria

1. What is the focus of evaluation questions?2. What are good sources for evaluation

questions?3. Who should be involved in the divergent

and convergent phases? Why?

Chapter 13

Evaluation Questions:

Provide focus to the evaluationSpecify the information the evaluation will

provideGuide choices for data collection, analysis,

and interpretation


Good Sources:

Existing Standards in the FieldResearch LiteratureContent ExpertsEvaluators Own Experience


Divergent: Cronbach (1982), “Opening one’s mind to be entertained at least briefly as prospects for investigation” (p. 210).

Fitzpatrick, et al. (2011), “throw a broad net and learn from many possible sources” (p. 316).


Convergent: Three reasons for reducing the range of variables.

1. Budget2. Manageability/Complexity3. Attention Span of Audience

The evaluator aims for the maximum bandwith.Cronbach (1982)

*Typically completed by an advisory group with representatives from different stakeholder groups.


1. What considerations are important in tailoring the reporting of evaluation results to audiences?

2. What are some of the ways results can be communicated to stakeholders?

3. What should be included in an evaluation report?

(take 5 minutes to discuss with a partner)

Chapter 17: Reporting Evaluation Results: Maximizing Use and Understanding

Different audiences have different information needs and knowledge levels.

Reading abilityFamiliarity with the programAttitude toward the programRole in decision makingFamiliarity with evaluation practices/methodsAttitude and interest in evaluationExperience in using evaluation results

Torres, Preskill, and Pointek (2005)


If possible (Patton, 2008) points out that evaluation data are sued more if the evaluator discusses and negotiates the format, style, and organization of reports with primary users.

Other Key Points: Avoid Jargon, Use simple direct language, Use examples, anecdotes,

be interesting…


Key Elements of an Evaluation Report (p. 470)I. Executive SummaryII. Introduction to the ReportIII. Focus of the EvaluationIV. Brief Overview of Evaluation Plan &

ProceduresV. Presentation of ResultsVI. Conclusion and RecommendationsVII. Minority Reports or RejoindersVIII. Appendices


overview of chapters 11 – 13, & 17

Documents

evaluation request

evaluation priorities

evaluation conduct

evaluation process

evaluation setting boundaries

evaluation context greene

evaluation context scriven

evaluation purpose ethical