dpme evaluations - south african qualifications authority presentation.pdf · evaluation: what is...
TRANSCRIPT
NQF Impact Workshop,
Midrand
Antonio Hercules (DPME)
09 February 2016
DPME Evaluations
Outline
1. Evaluation: what is it? [2 minutes!] 2. NES and Types of Evaluation. what these are [6
mins!] 3. NQF Impact Study vs. NES Impact Evaluation.
the study and impact evaluation [5 minutes!] 4. 2017/18 NES Implementation Evaluation
current NES evaluation [5 minutes!] 5. Presentation Key messages
2
Evaluation – easy word, specific meaning
As a field, its value and its shortcomings 1. Field: about 30-40 years, imprecise but improving, specifically deals with policy/programme/project as the object, specialised, mixed methods, sensitive to bias
2. Value: offers “objective” external and “expert” internal approach options. great window for renewal/improvement, highly flexible, decades of practice strongly evidence-based
3
Evaluation – easy word, specific meaning
3. Shortcomings Complex Duration Financial investment Must have buy-in Needs good and committed managers/political principals to
take forward Evaluation is different from “normal”
evaluative reflection in policy analysis/research
4
National Evaluation System and Evaluation
Six (6) guidelines (for policy/prog./proj.) 1. Diagnostic*: … “identification”
“research” – info. foundations 2. Design*: “formulation” 3. Implementation: performance
(results), everything short of impact 4. Impact: using quantitative RCT
approach conventionally 5. Economic*: …economic benefits 6. Synthesis*: …synthesis of pre-existing
evaluations
5
NQF Impact Study vs. NES Impact Evaluation
1. NQF Impact Study has value in its own right 2. Key question: would Study pass in the NES? 3. Main reasons: Largely impact evaluation TOR, but methodology does not
deliver accepted evaluation design More of a “synthesis” evaluation pulling together existing
bodies of knowledge -> produces overview, trends, insights, but doesn’t subject underlying evaluation design of supporting reports to data verification/validation testing
Does not appear to comply with accepted large system impact evaluation design (correlations, regressions, etc.)
Still very useful, and well-timed for NQF Act policy implementation evaluation (NES – NEP 2016/2017)
6
Typical Impact Evaluation Approach
7
• Policy • Goals • Results
1 Results Frame-work
• Theory of Change
• Logframe • Operational
Strategy • “Nesting”
2 Logic
• Literature • Interviews • Focus
Groups • Survey • Databases
3 Evidence
• Policy Narrative
• Evidence • Analysis • Results • Conclusions
4 Assess-ment
• Sound • Specific • Based on
evidence
5 Recom-mend-ations
2017/18 NES Implementation Evaluation
Purpose: to assess the implementation of the NQF Act relative to its goal(s) and objectives in the period of review, including its associated policies and regulations. Successes and challenges in its implementation will be identified, and recommendations offered regarding improvements to implementation of the NQF Act in the future.
8
1. (Relevance and Appropriateness) To what extent is the theory of change (intervention logic) of the NQF Act adequately robust, including its main underlying assumptions?
2. (Effectiveness) To what extent has the implementation of the NQF Act been effective in achieving its policy goal(s), objectives and intended outcomes?
3. (Efficiency) To what extent has the implementation of the NQF Act been efficient?
4. (Impact) What is the emerging impact of the NQF, if any? 5. What needs to be done to improve the implementation
of the NQF Act, and which aspects of the NQF Act need to be reviewed?
9
2017/18 NES Implementation Evaluation
Sufficient teaching
time
Nutritious meals delivered to
schools on time
Nutritious meals eaten
Improved health and nutritional status of South African primary school children
Better educational performance
Improved levels of
primary school attendance
Improved concentration
in class
Activities
Local business prospers
Department of Education pays
for food delivered
Outputs
Children have adequate nutrition
Outcomes Impact
Assumptions
Local producers provide the
food/ingredients at fair price
Inputs
Funds
Local SMMEs
appointed to render
services
Decentralised purchasing
system Fair
procurement Payments to producers/suppliers
made on time
Food prepared locally and safely
Immediate Intermediate
Schools have adequate
food storage facilities
National School Nutrition Programme TOC
10
TOC example: Aspen Institute (2003)
12
Thanks! Antonio Hercules: Director: Evaluations & Research.
Head: Evaluation and Research, DPME. Ian Goldman. [email protected]
www.thepresidency-dpme.gov.za