essential competencies for program evaluators
TRANSCRIPT
![Page 1: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/1.jpg)
Essential Competencies for Program Evaluators Jean A. King and Laurie Stevahn
![Page 2: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/2.jpg)
Let’s be clear about competencies. . .
![Page 3: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/3.jpg)
There is a difference
between
EVALUATION
and
EVALUATOR
![Page 4: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/4.jpg)
QUICK INTRODUCTIONS
Who is in this session today?
![Page 5: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/5.jpg)
Questions to answer
•Where do you currently live?
•How long have you been an evaluator?
•What fields do you work in?
• What is your interest in evaluator competencies? (Why attend this session?)
![Page 6: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/6.jpg)
Agenda for our 2-hour session
1. Identifying competencies intuitively
2. Personal beliefs about evaluator competencies
3. The original decade-old ECPE
4. The status of competencies around the world
5. Assessing your competencies– What next?
![Page 7: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/7.jpg)
IDENTIFYING COMPETENCIES
People create these things!
![Page 8: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/8.jpg)
Identifying competencies intuitively
What are the knowledge, skills and
dispositions that a program evaluator
needs to successfully complete the evaluation?
![Page 9: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/9.jpg)
Vignette exercise
For your assigned vignette, identify the knowledge, skills, and dispositions/attitudes the evaluator needed to conduct
the study
![Page 10: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/10.jpg)
Evaluator “competencies”
Knowledge Skills Attitudes/Dispositions
?
?
?
![Page 11: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/11.jpg)
Evaluator relationships
•A relationship exists among • The evaluator • The client • Program staff and other evaluation
stakeholders
•Primary responsibility for two tasks: • Making evaluation decisions • Implementing the evaluation
![Page 12: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/12.jpg)
Who plans the evaluation?
Who conducts it?
Two basic questions
![Page 13: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/13.jpg)
Evaluator relationships
•Primary responsibility for making and implementing evaluation decisions may lie • With the evaluator • With someone else in the setting • Jointly with both
• The relationship may shift during the study
![Page 14: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/14.jpg)
Interactive participation quotient (IPQ)
LOW
HIGH Evaluator
Program leaders,
staff,
community
Invo
lvem
ent i
n
deci
sion
mak
ing
and
impl
emen
tatio
n
Participant-
directed
Collaborative
Evaluator-
directed
ZONES
![Page 15: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/15.jpg)
COMPETENCY STATEMENTS
What do you believe?
![Page 16: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/16.jpg)
Statement #1
Technical knowledge and skills are the most important category of evaluator
competencies.
![Page 17: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/17.jpg)
Statement #2
The competencies that
program evaluators need
depend in large part on specific evaluation settings.
![Page 18: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/18.jpg)
Statement #3
Professional training in program evaluation is essential to build
evaluator competencies.
![Page 19: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/19.jpg)
Question:
To what extent do personal beliefs matter in a discussion of evaluator competencies?
![Page 20: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/20.jpg)
BTW, not everyone
thinks
professionalization
is a good idea
•Negative view of neoliberalism
•Perceived hegemony of dominant players
![Page 21: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/21.jpg)
THE ORIGINAL ESSENTIAL COMPETENCIES FOR PROGRAM EVALUATORS
Presenting. . .
![Page 22: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/22.jpg)
Once upon a time. . . It began with an idea in a graduate course
![Page 23: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/23.jpg)
How were the ECPE developed?
• “What does the literature tell us?”
• A pilot study using the initial set of competencies, AJE publication (2001)
• A revision process including a crosswalk
• ECPE published in AJE (2005)
• Articles in CJPE (2005) and AJE (2006)
• CES’s Credentialed Evaluator program
• CJPE special issue on evaluator competencies; New Directions issue on evaluation professionalization
![Page 24: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/24.jpg)
Professional
Practice
Systematic
Inquiry
Situational
Analysis
Reflective
Practice
Project
Management
Interpersonal
Competence
Essential Competencies
for Program Evaluators
![Page 25: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/25.jpg)
![Page 26: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/26.jpg)
1. Professional practice
Fundamental norms and values of evaluation practice
• Applies evaluation standards
• Works ethically
• Respects all stakeholders
• Considers the general and public welfare
• Contributes to the knowledge base
![Page 27: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/27.jpg)
2. Systematic inquiry (1/2)
Technical aspects of evaluation practice • Understands knowledge base
• Knows quantitative and qualitative methods
• Knows mixed methods
• Conducts literature reviews
• Specifies program theory
• Frames evaluation questions
• Designs the evaluation PLUS. . .
![Page 28: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/28.jpg)
2. Systematic inquiry (2/2)
• Collects and interprets data
• Makes judgments and recommendations
• Provides rationales for decisions
• Reports procedures and results
• Notes evaluation strengths and limitations
• Conducts meta-evaluations
![Page 29: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/29.jpg)
3. Situation analysis (1/2)
Unique interests, issues, and contextual circumstances of evaluation • Describes program • Determines program evaluability • Identifies stakeholders’ interests • Serves intended users’ informational
needs • Addresses conflicts • Attends to evaluation use PLUS. . .
![Page 30: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/30.jpg)
3. Situation analysis (2/2)
• Examines organizational and political context
• Attends to organizational change
• Respects site and client uniquenesses
• Remains open to input
• Modifies evaluation as needed
![Page 31: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/31.jpg)
4. Project management (1/2)
“Nuts and bolts” of evaluation work
• Responds to RFPs
• Negotiates with clients
• Writes formal agreements
• Communicates with clients
• Budgets evaluation
• Justifies cost PLUS. . .
![Page 32: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/32.jpg)
4. Project management (2/2)
• Identifies needed resources
• Uses technology as appropriate
• Supervises and trains others
• Conducts evaluation in a nondisruptive manner
• Presents work in timely manner
![Page 33: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/33.jpg)
5. Reflective practice
One’s own evaluation expertise and need for growth
• Knows self
• Reflects on practice
• Pursues professional development in both evaluation and content areas
• Builds professional relationships
![Page 34: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/34.jpg)
6. Interpersonal competence
“People skills” necessary for evaluation practice • Uses written and verbal/listening
communication skills
• Uses negotiation skills
• Uses conflict resolution skills
• Facilitates constructive interpersonal interaction
• Demonstrates cross-cultural competence
![Page 35: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/35.jpg)
References Stevahn, L., King, J. A., Ghere, G., & Minnema, J. (2005a). Establishing essential competencies for program evaluators. American Journal of Evaluation, 26(1), 43-59.
Wilcox, Y., & King, J. A. (2014). A professional grounding and history of the development and formal use of evaluator competencies. Canadian Journal of Program Evaluation, 28(3), 1-28. King, J. A., & Stevahn, L. (2015). Competencies for program evaluators in light of adaptive action: What? So what? Now what? In J. W. Altschuld & M. Engle (Eds.), Accreditation, certification, and credentialing: Relevant concerns for US evaluators. New Directions for Evaluation, 145, 21-37.
![Page 36: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/36.jpg)
CURRENT STATUS OF COMPETENCIES DEVELOPMENT AND USE AROUND THE WORLD
What recent developments are triggering conversation?
![Page 37: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/37.jpg)
The pace of competency activity around the world has
increased rapidly
![Page 38: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/38.jpg)
Some competency applications
1. Canada’s Credentialed Evaluator (CE) program
2. AEA’s Credentialing Task Force (2015)
3. Aotearoa New Zealand (cultural focus)
4. South Africa (managerial use)
5. Japan and Thailand (educational evaluators)
6. International Development Evaluation Association (IDEAS)
7. European Evaluation Society- Voluntary Evaluator Peer Review (VEPR)
![Page 39: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/39.jpg)
AES Evaluator’s Professional Learning Competency Framework domains of competence
1. Evaluative Attitude and Professional Practice
2. Evaluation Theory
3. Culture, Stakeholders and Context
4. Research Methods and Systematic Inquiry
5. Project Management
6. Interpersonal Skills
7. Evaluation Activities
![Page 40: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/40.jpg)
Distinctions among terms
•Knowledge, skills, attitudes/dispositions
•Competencies
•Competence
![Page 41: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/41.jpg)
Definitions Term Meaning
Knowledge, skills, attitudes/
dispositions
From curriculum and training fields- What you know, can do, and your affect
Competencies
A “set of related knowledge, skills, and attitudes that enable an individual to effectively perform the activities of a given occupation or job function to the standards expected in employment” (Richey, Fields, & Foxon, 2001, p. 31)
Competence
“Habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values and reflection in daily practice for the benefit of the individual and community being served” (Epstein & Hundert, 2002, p. 226)
![Page 42: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/42.jpg)
Issues to consider
Conceptual • Validity evidence exists
• Importance of context (Wilcox, 2012 study)
• Relationship to improved practice?
Practical • People are using competencies, regardless
• Need to update existing sets
![Page 43: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/43.jpg)
Possible next steps- Practically
•Update existing competencies
•Create professional learning communities?
• Explore credentialing options—or not?
![Page 44: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/44.jpg)
Possible next steps- Conceptually
•Create a formal approach to developing and validating competencies
• Teach evaluation using the competencies
• Study the effect and effectiveness of competencies in a variety of settings
![Page 45: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/45.jpg)
APPLYING THE COMPETENCIES
Let’s think about the ECPE together
![Page 46: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/46.jpg)
Exercise/discussion
1. Which competency categories (or individual competencies) apply across all IPQ zones?
2. Which will work differently depending on the zone? (Please explain)
![Page 47: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/47.jpg)
SELF-ASSESSMENT
What’s next for you?
![Page 48: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/48.jpg)
ECPE self-assessment (What?)
Reflecting on your own practice, how do you rate your current knowledge, skills, and dispositions to effectively carry out program evaluations?
Professional
Practice
Systematic
Inquiry
Situational
Analysis
Reflective
Practice
Project
Management
Interpersonal
Competence
![Page 49: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/49.jpg)
ECPE rating scale
Entry /
Novice
Proficient/
Skilled Mastery/
Expert
![Page 50: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/50.jpg)
ECPE self-assessment (So what?)
Professional
Practice
Systematic
Inquiry
Situational
Analysis
Reflective
Practice
Project
Management
Interpersonal
Competence
•Learned?
•Affirmed?
•Challenged?
![Page 51: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/51.jpg)
Now what?
What is your professional
development plan?
![Page 52: Essential Competencies for Program Evaluators](https://reader034.vdocuments.mx/reader034/viewer/2022042707/586987451a28ab81568bf169/html5/thumbnails/52.jpg)
Feel free to contact us: