designing useful evaluations that actually get used · 2012. 12. 28. · michael quinn patton june...
TRANSCRIPT
![Page 1: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/1.jpg)
Designing useful evaluations that actually get used
with Michael Quinn Patton
Michael Quinn Patton June 14, 2010
1
![Page 2: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/2.jpg)
Michael Quinn Patton June 14, 2010
1st edition 1978 2nd edition 1986 3rd edition 1997 4th edition 2008
2
![Page 3: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/3.jpg)
Michael Quinn Patton June 14, 2010
Research-based Approach
• Follow-up study of the use of 20 federal health evaluations
• Canadian research on evaluation use
• U-FE now widely used among professional evaluators
(see appendix slides) 3
![Page 4: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/4.jpg)
Factors Affecting Use
Table Exercise
Michael Quinn Patton June 14, 2010
4
![Page 5: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/5.jpg)
![Page 6: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/6.jpg)
Michael Quinn Patton June 14, 2010
Evaluation and
Research: Same or different?
And why?
6
![Page 7: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/7.jpg)
Michael Quinn Patton June 14, 2010
Foundational Premise:
Research and evaluation are different – and therefore…
Evaluated by different standards.
7
![Page 8: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/8.jpg)
Michael Quinn Patton June 14, 2010
Evaluation Standards
Utility – ensure relevance & use Feasibility – realistic, prudent,
diplomatic & frugal Propriety – ethical, legal, respectful Accuracy – technically adequate to
determine merit or worth For the full list of Standards: www.wmich.edu/evalctr/checklists/standardschecklist.htm
8
![Page 9: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/9.jpg)
Michael Quinn Patton June 14, 2010
Evaluation Models
U-FE is one among many….
9
![Page 10: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/10.jpg)
10 Michael Quinn Patton June 14, 2010
![Page 11: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/11.jpg)
Cultural context
• Emergence and meaning of the jester
Michael Quinn Patton June 14, 2010
11
![Page 12: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/12.jpg)
Michael Quinn Patton June 14, 2010
12 Michael Quinn Patton Claremont 2009 12
![Page 13: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/13.jpg)
Michael Quinn Patton June 14, 2010
13 Michael Quinn Patton Claremont 2009 13
![Page 14: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/14.jpg)
Michael Quinn Patton June 14, 2010
14 Michael Quinn Patton Claremont 2009 14
![Page 15: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/15.jpg)
Changing societal and political context
for evaluation
Michael Quinn Patton June 14, 2010
15
![Page 16: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/16.jpg)
Michael Quinn Patton June 14, 2010
16 Michael Quinn Patton Claremont 2009
![Page 17: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/17.jpg)
Lessons
What we’ve learned about designing useful evaluations that actually get used.
A case example…
Michael Quinn Patton June 14, 2010
17
![Page 18: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/18.jpg)
Michael Quinn Patton June 14, 2010
18
Example
Caribbean Agricultural Extension Project
U.S. AID European Development Bank Caribbean Development Bank University of the West Indies Big Ten Universities (MUCIA)
CARDI
![Page 19: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/19.jpg)
Michael Quinn Patton June 14, 2010
19
Overview
• 10 year project in 3 phases • Phase 1, January 1980 to December 1982 • Mid-term and end-of-phase evaluations
planned and budgeted • 10 Caribbean countries
![Page 20: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/20.jpg)
Michael Quinn Patton June 14, 2010
20
Utilization Question
When will evaluation results be needed to contribute to the summative decision about whether to fund Phase 2?
When do you think?
![Page 21: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/21.jpg)
Michael Quinn Patton June 14, 2010
21
When will evaluation results be needed to contribute to the summative decision about
whether to fund Phase 2? State Department budget to Congress
Caribbean Budget to State Department Barbados Regional Office to Wash DC and
Caribbean Region Office Agriculture program to Caribbean
![Page 22: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/22.jpg)
Michael Quinn Patton June 14, 2010
22
Evaluation Questions
• Are outstanding extension agents making a significant contribution to improved farming for small farmers?
Answer “no”: End project Answer “yes”: Consider next question… • If so, is it worth training more such
agricultural extension agents?
![Page 23: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/23.jpg)
Michael Quinn Patton June 14, 2010
23
An ancient example
• Group exercise:
What lessons about making evaluation useful do you extract from this example?
![Page 24: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/24.jpg)
Michael Quinn Patton June 14, 2010
24
Lessons?
![Page 25: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/25.jpg)
Michael Quinn Patton June 14, 2010
Utilization-Focused Evaluation (U-FE)
A decision-making framework for enhancing the utility and actual use of evaluations.
25
![Page 26: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/26.jpg)
Michael Quinn Patton June 14, 2010
U-FE begins with the premise that evaluations should be judged by their utility and actual use. Therefore, evaluators should facilitate the evaluation process and design any evaluation with careful consideration of how everything that will be done, from beginning to end, will affect use.
26
![Page 27: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/27.jpg)
Michael Quinn Patton June 14, 2010
USE
• Take use seriously by evaluating use, the source of our own accountability and ongoing learning/professional development
• Different from dissemination • Different from producing reports • Groundwork laid and expectations set at
the beginning • Doesn’t happen naturally or automatically
27
![Page 28: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/28.jpg)
Michael Quinn Patton June 14, 2010
Goal of U-FE
Intended Use by
Intended Users
28
![Page 29: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/29.jpg)
Michael Quinn Patton June 14, 2010
Intended Evaluation Users
From… Audiences to… Stakeholders to… Primary Intended Users
Connotative differences?
29
![Page 30: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/30.jpg)
Michael Quinn Patton June 14, 2010
30
![Page 31: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/31.jpg)
Michael Quinn Patton June 14, 2010
31
Stakeholder Mapping High Interest/ Low Power
THE INVOLVED
High Interest/ High Power
THE PLAYERS
THE CROWD Low interest/ Low Power
Low Interest/ High Power
CONTEXT SETTERS
Low interest/ Low Power
The CROWD
![Page 32: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/32.jpg)
Michael Quinn Patton June 14, 2010
32
Low power stakeholders
High power stakeholders
High interest stakeholders
Support and enhance their capacity to be involved, especially when they may be affected by findings, as in the case of program participants. Their involvement increases the diversity of the evaluation.
High potential as primary intended users. These are often key "players" who are in a prime position to affect use, including using it themselves as well as drawing the attention of others.
Low interest stakeholders
Inform them about the evaluation and its findings. Controversy can quickly turn this amorphous "crowd" of general public stakeholders into a very interested mob.
Need to cultivate their interest and be alert in case they pose barriers to use through their disinterest. They are "context setters."
![Page 33: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/33.jpg)
33 Michael Quinn Patton June 14, 2010
![Page 34: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/34.jpg)
Michael Quinn Patton June 14, 2010
34
Research Base
Studies of evaluation use have identified primary user identification and involvement as a critical factor contributing to evaluation use.
![Page 35: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/35.jpg)
Michael Quinn Patton June 14, 2010
Goal of U-FE:
Intended Use by
Intended Users
35
![Page 36: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/36.jpg)
Michael Quinn Patton June 14, 2010
Different Evaluation Purposes
• For making judgments Commonly called summative evaluations:
• For improving programs Commonly called formative evaluations
• For ongoing development Sometimes called developmental evaluations
• For knowledge building Meta-evaluation, lessons learned, effective practices
36
![Page 37: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/37.jpg)
Additional uses
• Accountability
• Monitoring (M & E) integrated
Michael Quinn Patton June 14, 2010
37
![Page 38: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/38.jpg)
Michael Quinn Patton June 14, 2010
Tensions Different intended uses serve different
purposes and, typically, different intended users. Thus the need to FOCUS and manage tensions between and among
different purposes. 38
![Page 39: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/39.jpg)
Michael Quinn Patton June 14, 2010
Premises of Utilization Focused Evaluation
• No evaluation should go forward unless and until there are primary intended users who will use the information that can be produced
• Primary intended users are involved in the process
• Evaluation is part of initial program design - The primary intended users want information to help answer a question or questions.
• Evaluator’s role is to help intended users clarify their purpose and objectives.
• Make implications for use part of every decision throughout the evaluation – the driving force of the evaluation process. 39
![Page 40: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/40.jpg)
Michael Quinn Patton June 14, 2010
40
Use Distinctions
• Instrumental Use • Conceptual Use • Misuse
40
![Page 41: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/41.jpg)
Michael Quinn Patton June 14, 2010
41
Campbell's Law "The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.“ Campbell (1975)
41
![Page 42: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/42.jpg)
Michael Quinn Patton June 14, 2010
42
Beyond Misevaluation
Essential evaluator competencies • Beyond technical competence • Professional knowledge (e.g., standards
& guiding principles) • Interpersonal, communications,
facilitation, & political skills • Cultural competence
42
![Page 43: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/43.jpg)
Michael Quinn Patton June 14, 2010
43
Speaking Truth to Power
Eleanor Chelimsky
"Telling the truth to people who may not want to hear it is, after all, the chief purpose of evaluation"
(Chelimsky, 1995: 54)
43
![Page 44: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/44.jpg)
Michael Quinn Patton June 14, 2010
44
About this
book
Preview
this
book
Shake
Hands
with
the
Devil By
Roméo
Dallaire
This is a preview. The total pages displayed
will be limited. L
44
![Page 45: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/45.jpg)
SPEAKING TRUTH TO POWER
45 Michael Quinn Patton June 14, 2010
![Page 46: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/46.jpg)
Michael Quinn Patton June 14, 2010
46
Evaluation Use Factors • Evaluation & politics • Evaluation & leadership • Evaluation & organizational
culture • Evaluation & systems change • Evaluation & the social science of
change 46
![Page 47: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/47.jpg)
Michael Quinn Patton June 14, 2010
New Direction
Systems thinking & complexity science as frameworks for
conceptualizing interventions (replacing linear logic models)
Developmental Evaluation
47
![Page 48: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/48.jpg)
Michael Quinn Patton June 14, 2010
48
“A Leader's Framework for Decision Making” by David J. Snowden and Mary E. Boone, Harvard Business Review,
November, 2007: Wise executives tailor their approach to fit
the complexity of the circumstances they face.
![Page 49: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/49.jpg)
Michael Quinn Patton June 14, 2010
49 Michae
![Page 50: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/50.jpg)
Michael Quinn Patton June 14, 2010
50
Wise evaluators tailor their approach to fit the complexity of the circumstances they face
![Page 51: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/51.jpg)
Michael Quinn Patton June 14, 2010
The Challenge: Matching the evaluation
design to the evaluation’s purpose, resources, and timeline
to optimize use. Focus: Intended use by
intended users
51
![Page 52: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/52.jpg)
Michael Quinn Patton June 14, 2010
Leaders as Primary Intended Users
Reality-Testing, Results-Oriented, Learning-Focused
Leadership
52
![Page 53: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/53.jpg)
Michael Quinn Patton June 14, 2010
Four Leadership Functions
1. Create and support a results-oriented, reality-testing, learning-focused culture.
53
![Page 54: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/54.jpg)
Reality-testing
Michael Quinn Patton June 14, 2010
54
![Page 55: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/55.jpg)
Results-oriented
Michael Quinn Patton June 14, 2010
55
![Page 56: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/56.jpg)
Michael Quinn Patton June 14, 2010
56
![Page 57: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/57.jpg)
Michael Quinn Patton June 14, 2010
The importance of
Organizational Culture in
determining use
57
![Page 58: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/58.jpg)
Michael Quinn Patton June 14, 2010
Situation Analysis
What are the critical organizational culture factors
that affect evaluation use?
• Enablers/facilitating factors • Barriers to use
58
![Page 59: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/59.jpg)
Michael Quinn Patton June 14, 2010
Leadership Functions
1. Create and nurture a results-oriented, reality-testing culture.
2. Lead in deciding what outcomes to commit to and hold yourselves accountable for.
59
![Page 60: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/60.jpg)
Michael Quinn Patton June 14, 2010
Outcomes are
an expression of Values:
E-valu-ation
60
![Page 61: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/61.jpg)
Michael Quinn Patton June 14, 2010
1. Create and nurture a results-oriented, reality-testing culture.
2. Lead in deciding what outcomes to commit to and hold yourselves accountable for.
3. Make measurement of outcomes thoughtful, meaningful and credible.
61
![Page 62: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/62.jpg)
Michael Quinn Patton June 14, 2010
62
Use Distinctions
Findings Use and
Process Use
62
![Page 63: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/63.jpg)
Michael Quinn Patton June 14, 2010
Process Use Process use refers to and is indicated by individual
changes in thinking and behavior, and program or organizational changes in procedures and culture, that occur among those involved in evaluation as a result of the learning that occurs during the evaluation process. Evidence of process use is represented by the following kind of statement after an evaluation: "The impact on our program came not so much from the findings but from going through the thinking process that the evaluation required."
63
![Page 64: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/64.jpg)
Michael Quinn Patton June 14, 2010
Process Uses
• Enhancing shared understandings • Focusing programs: What gets measured gets
done • Supporting and reinforcing the program intervention, e.g., feedback for learning • Capacity-building for those involved, deepening evaluative thinking • Program and organizational development, e.g., evaluability assessments
64
![Page 65: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/65.jpg)
Michael Quinn Patton June 14, 2010
65
![Page 66: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/66.jpg)
Michael Quinn Patton June 14, 2010
New Direction
Infusing evaluative thinking as a primary type of process use.
IDRC example
Capacity-building as an evaluation focus of process use.
66
![Page 67: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/67.jpg)
Michael Quinn Patton June 14, 2010
The importance of
Organizational Culture in
supporting use & learning
67
![Page 68: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/68.jpg)
• Evaluation capacity-building as a priority to support use
Michael Quinn Patton June 14, 2010
68
![Page 69: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/69.jpg)
Basic evaluation literacy • Know the evaluation standards • Know how to apply the standards in the
actual conduct of evaluations • Understand different potential uses and
their implications methodologically and procedurally
• Understand how to identify and work with primary intended users
• Have evaluators with essential skils Michael Quinn Patton
June 14, 2010 69
![Page 70: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/70.jpg)
Michael Quinn Patton June 14, 2010
1. Create and nurture a results-oriented, reality-testing culture.
2. Lead in deciding what outcomes to commit to and hold yourselves accountable for.
3. Make measurement of outcomes thoughtful, meaningful and credible.
4. Use the results -- and model for others serious use of results.
70
![Page 71: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/71.jpg)
Michael Quinn Patton June 14, 2010
Use Scenarios
• Decisions and actions • Priority uses • Realistic timelines to increase
utility and actual use • Interpretation engagement • Recommendation options
71
![Page 72: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/72.jpg)
Michael Quinn Patton June 14, 2010
Walking the Talk: Reinforcing
a Reality-testing,
Results-oriented, Learning-committed
organizational culture
72
![Page 73: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/73.jpg)
Michael Quinn Patton June 14, 2010
73
Developmental Evaluation:
Applying Complexity Concepts to Enhance Innovation and Use
Michael Quinn Patton
![Page 74: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/74.jpg)
74 Michael Quinn Patton June 14, 2010
![Page 75: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/75.jpg)
Michael Quinn Patton June 14, 2010
75
And the beat goes on…
Evaluation as an ever-evolving field
![Page 76: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/76.jpg)
Michael Quinn Patton June 14, 2010
Reference Utilization-Focused Evaluation, 4th ed, Michael Quinn Patton Sage Publications, 2008. http://www.sagepub.com/booksProdDesc.nav?
prodId=Book229324
76
![Page 77: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/77.jpg)
Appendix
These are slides as resources in answering questions and going deeper into some issues. Time may not permit covering these during the Institute workshop.
Michael Quinn Patton June 14, 2010
77
![Page 78: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/78.jpg)
Michael Quinn Patton June 14, 2010
78
Research on U-FE use among evaluators:
As part of a review of developments over the first ten years of the American Evaluation Association (AEA), Preskill and Caracelli (1997) conducted a survey of members of AEA's Topical Interest Group on Use. They found that 85% rated as extremely or greatly important "identifying and prioritizing intended users of the evaluation" (p. 216).
![Page 79: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/79.jpg)
Michael Quinn Patton June 14, 2010
79
The only item eliciting higher agreement (90%) was the importance of
"planning for use at the beginning of the evaluation."
Preskill and Caracelli also found that 80% of survey respondents agreed that evaluators should take responsibility for involving stakeholders in the evaluation processes
![Page 80: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/80.jpg)
Michael Quinn Patton June 14, 2010
80
Fleischer (2007) asked the same question on a replication survey of American Evaluation Association members in 2006 and found that 98% agreed with this assertion. In rating the importance of eight different evaluation approaches, "user-focused" evaluation was rated highest. Stakeholder involvement in evaluations has become accepted practice among evaluation professionals.
![Page 81: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/81.jpg)
Michael Quinn Patton June 14, 2010
81
Summary Lessons on Useful Evaluation
1. Clearly identify primary intended users 2. Clearly identify primary intended uses
Goal: Intended use by intended users 3. Negotiate FOCUS -- get agreement on criteria 4. Establish a clear ACTION framework 5. Distinguish empirical questions from value questions 6. Select methods appropriate to the question 7. Facilitate actual use of the findings
![Page 82: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/82.jpg)
Michael Quinn Patton June 14, 2010
82
Classic S-shaped curve
![Page 83: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/83.jpg)
Rogers’ adopter categories • Innovators: Innovators are the first individuals to adopt
an innovation. Innovators are willing to take risks, and have closest contact to scientific sources and interaction with other innovators.
• Early Adopters: This is second fastest category of individuals who adopt an innovation. These individuals have the highest degree of opinion leadership among the other adopter categories. Early adopters are typically younger in age, have a higher social status, advanced education, and are more socially forward than late adopters.
Michael Quinn Patton June 14, 2010
83
![Page 84: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/84.jpg)
Adopter categories
• Early Majority: Individuals in this category adopt an innovation after a varying degree of time. This time of adoption is significantly longer than the innovators and early adopters. Early Majority tend to be slower in the adoption process, have above average social status, contact with early adopters, and show some opinion leadership
Michael Quinn Patton June 14, 2010
84
![Page 85: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/85.jpg)
Adopter categories • Late Majority: Individuals in this category will adopt an
innovation after the average member of the society. Late Majority are typically skeptical about an innovation, have below average social status, in contact with others in late majority and very little opinion leadership.
• Laggards: Individuals in this category are the last to adopt an innovation. Unlike some of the previous categories, individuals in this category show little to no opinion leadership. These individuals typically have an aversion to change-agents and tend to be advanced in age. Laggards typically tend to be focused on “traditions”, and very little to no opinion leadership.
Michael Quinn Patton June 14, 2010
85
![Page 86: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/86.jpg)
HYMN to
EVALUATION USE
Sung to the tune of Auld Lang Syne
![Page 87: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/87.jpg)
May all e-valu-a-tions done Be useful as they should They tell us how to separate What is poor from what is good.
We gather data near and far To see what we can learn The findings help us to decide What to keep and what to burn.
![Page 88: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/88.jpg)
There comes a time for each of us When doubts may give us pause We wonder what results will show Will the world see naught but flaws
But be assured there's naught to fear If learn-ing is what you seek Let outcomes guide your every move Listen to the data speak.
![Page 89: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/89.jpg)
Michael Quinn Patton June 14, 2010
89
KNOWLEDGE TRIANGUALTED LEARNING FRAMEWORK
ACTION
BELIEFS/ VALUES KNOWLEDGE
![Page 90: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/90.jpg)
Michael Quinn Patton June 14, 2010
90
Premises • Action flows from some combination of beliefs/values
and knowledge. • Informed action has a strong knowledge base. • By testing and evaluating beliefs, we build knowledge,
thereby making our actions more informed and empirically based.
• People have varying predilections that lean more toward one of these dimensions than others, e.g., more action-oriented, more values-oriented, or more knowledge-oriented.
• All three styles are valuable and needed for sustainable program, organizational and/or community development.
![Page 91: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/91.jpg)
Michael Quinn Patton June 14, 2010
91
• In short, ACTIONS flow from some combination of IDEAS (THEORY), BELIEFS(MISSION/VISION/VALUES) and KNOWLEDGE (EVIDENCE).
• A Learning Organization, over time, moves more action to a strong knowledge base.
![Page 92: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/92.jpg)
Michael Quinn Patton June 14, 2010
92
Images of Alignment Between Actions, Beliefs and Knowledge Non-aligned: Aligned Inconsistencies between Consistency between actions, beliefs and knowledge actions, beliefs and knowledge
Alignment and consistency support meaningful, high-impact and
Sustainable results.
![Page 93: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/93.jpg)
Michael Quinn Patton June 14, 2010
Utilization-Focused
Methods Decisions
93
![Page 94: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/94.jpg)
Michael Quinn Patton June 14, 2010
94
The Debate About Randomized Controls in
Evaluation:
• The Gold Standard Question
![Page 95: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/95.jpg)
Michael Quinn Patton June 14, 2010
New Direction
The morphing of the paradigms debate (qualitative vs quantitative) into the Gold Standard debate (randomized control trials as the alleged "gold standard" for impact evaluation)
95
![Page 96: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/96.jpg)
Michael Quinn Patton June 14, 2010
96
• No single design or method is universally “strongest” • Multiple ways of establishing causality • Dangerous to privilege one method, creates perverse incentives
![Page 97: Designing useful evaluations that actually get used · 2012. 12. 28. · Michael Quinn Patton June 14, 2010 32 Low power stakeholders High power stakeholders High interest stakeholders](https://reader036.vdocuments.mx/reader036/viewer/2022062404/614360f16b2ee0265c020258/html5/thumbnails/97.jpg)
Michael Quinn Patton June 14, 2010
GOLD STANDARD:
METHODOLOGICAL APPROPRIATENESS
not
Methodological orthodoxy or rigidity
97