flexible evaluation

Professional Certificate in Policy

Implementation and Evaluation

Ian “Ren” Rennie

FLEXIBLE EVALUATION

• Needs

• Evaluability Assessment

• Process

• Impact Analysis (summative evaluation)

• Efficiency Analysis

– Cost-benefit Analysis

– Cost-effectiveness Analysis

Types of Evaluation

• Too Complicated – Implications may not be presented in a way that non

researchers can understand

• Results can sometimes be counter-intuitive – Results sometimes contradict deeply held beliefs

• Vested interest in a program • Who evaluates government policies? • Answer: often the government evaluates itself.

Why evaluations may be ignored

• Often based on perception of immediate and personal risk or on the measurability of success

• Global warming (not an immediate threat)

• Lead in consumer goods (immediate threat)

• Endangered Species Act (not a personal threat, not easy to measure effects)

• Does support for environmental policy change if the policy effects prices of consumer goods (cars, gas, paper, lumber) or is the public willing to pay higher prices to protect the environment.

Political evaluation beyond effectiveness or efficiency

“Being Dismal in Policy Analysis or Learning to Love Policymaking”

• Max Neiman (2000)

• Modern democratic policymaking is an ugly process, but….

• Brings in lots of different perspectives and ideas

• Forces compromise and deliberation.

• Complex systemic policy in a rapidly changing world might necessitate an adaptive approach to evaluation

• Where adaptation is considered it is essential that key stakeholders are able to agree four key potential variations:

• Indicators: which indicators require changing? • Populations: which target population changes need

accommodating? • Thresholds: how much change to the original

indicators is required? • Timelines: how long will it take to achieve these

thresholds?

Theories of change in evaluation

Conclusions

• Evaluation is very expensive, demanding and complex

• Evaluation is a political process: need for clarity about why you do it

• Good evaluation always carries the risk of exposing failure

• Therefore evaluation is an emotional process • Evaluation needs to be acceptable to the

organisation and its stakeholders ...

Conclusions 2

• Plan and decide which model of evaluation is appropriate

• Identify who will carry out the evaluation and for what purpose

• Do not overload the evaluation process • Evaluation can give credibility and enhance learning • Informal evaluation will take place whether you plan

it or not.

• file:///C:/Users/Hunter/Downloads/policy_essentials_cost_benefit_analysis_final_7-9-2012.pdf

• http://www.behaviouralinsights.co.uk/sites/default/files/BIT%20Publication%20EAST_FA_WEB.pdf

• https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/62529/TLA-1906126.pdf

• https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/190984/Magenta_Book_quality_in_policy_impact_evaluation__QPIE_.pdf

• http://www.equalityhumanrights.com/sites/default/files/documents/PSD/equality_impact_assessment_guidance_quick-start_guide.pdf

References

CMI LEVEL 5 QUALIFICATION

• To ensure you understand the criteria by which the two CMI assignments will be assessed

• To outline the two assignment tasks

• To enable you to reflect and decide (in due course) the best way for you to take forward the two assignments.

Aim of session

• CMI – Chartered Management Institute

• ICPS – International Centre for Parliamentary Studies

• ASIC – Accreditation Service for International Schools and

Colleges

• CATS – Credit Accumulation and Transfer Scheme.

Acronyms

Generic Principles

• Demonstrate:

– WHAT you know/ did / needs to be done(knowledge and skills)

– HOW you did it / how it should be done (methodology and demonstrating understanding)

– OUTCOME you are aiming for/ have achieved / will achieve.

CMI Criteria

• In addition to the specific assessment criteria in each unit, the learner’s work must be:

– accurate, current and authentic

– relevant in depth and breadth and must also show the learner’s: • clear grasp of concepts

• ability to link theory to practice, and

• ability to communicate clearly in the relevant discipline at the expected level for the qualification.

• Plagiarism

• Signed statement of Authenticity

• “I formally declare that the work contained in this written assignment is all my own, and that it has been researched and written by me based solely on my professional experience, the content of the training seminar provided by ICPS, and the tutorial provided by ICPS on understanding and completing CMI assignments”.

Plagiarism & Authenticity

Structure: An Example

• Title page [name, declaration of authenticity, word count] • Contents and appendices • Acknowledgements • Glossary of terms and abbreviations • Statement of terms of reference • Executive summary • Introduction and background • Main body of the investigation / project [use CMI criteria as sub-

headings] • Conclusions • Recommendations • Appendices • Bibliography.

Some Thoughts

To get started it may help to:

• Write down all ideas, then structure later

Then:

• Check ideas cover all criteria

• Edit repeatedly!

• Process

• Think about the assessor

• Key “command” words

• Appendices.

More Thoughts

3,000 word report (excluding appendices)

• The scenario for this assignment is that your organisation has been asked to produce a report on best practice for an initiative to improve policy implementation and evaluation (in your selected country).

Unit 5005v1: Personal Development as a Manager and Leader

1.1 Determine organisational stakeholders and their expectations

1.2 Discuss methods of meeting stakeholder expectations or requirements

1.3 Identify methods of communicating stakeholders’ requirements with team members

1.4 Explain processes for updating information on stakeholders.

Assignment One (CMI Criteria) (Be able to understand stakeholders requirements)

2.1 Discuss the meaning of quality to an organisation

2.2 Identify and apply organisational quality policies and procedures

2.3 Determine how to encourage staff to contribute ideas to improving quality

2.4 Conduct a quality audit and make recommendations for improvement.

Assignment One (CMI Criteria) (Be able to define and determine quality)

3.1 Discuss the concept of and need for continuous improvement

3.2 Assess work activities and identify areas for improvement

3.3 Encourage staff to contribute ideas for continual improvement.

Assignment One (CMI Criteria) (Be able to understand and encourage continuous improvement)

• 3,000 word report (excluding appendices) on

• The scenario for the assignment is that your organisation is taking part in an initiative aimed at improving policy delivery and the evaluation of policy. You have been tasked to produce an action plan to project manage the taking forward of a new approach to an aspect of policy implementation or policy evaluation.

Unit 5006v1: Conducting a Management Project

• Clear timetable and milestones

• Clear outcomes and deliverables

• Accountability

• Good communication with stakeholders

• Be “joined up” internally and externally

• Resources identified and allocated

• Risk analysis

• Contingency planning.

Lessons from Project Work

1.1 Determine a management area for investigation that has an implication for a work-related area

1.2 Identify the aim, scope and objective of the project

1.3 Justify the aim and objective of the project ...

Assignment Two: (CMI Criteria) 1 (Be able to identify and justify a management project)

2.1 Identify sources of data and information for the project

2.2 Analyse the data and information for options or alternatives that meet the project aim

2.3 Determine an option or alternative that meets the project aim ...

Assignment Two (CMI Criteria) 2 (Be able to conduct research using sources and analyse data and options)

3.1 Evaluate the research to make conclusions

3.2 Recommend a course of action to meet the project aim

3.3 Assess the impact of the project recommendations ...

Assignment Two (CMI Criteria) 3 (Be able to make conclusions and recommendations that achieve the project aim)

4.1 Determine the medium to be used to present the results of the project

4.2 Produce the results of the investigative project

4.3 Evaluate the impact of the investigative project.

Assignment Two (CMI Criteria) 2 (Be able to develop and review the results of the investigative project)

• Once you have reflected and decided you definitely will do the assignments – please register via CMIassignments@parlicentre.org,

• Registration should be as soon as you are sure you will do the assignments, ideally within 4 months but absolutely within 6 months (CMI will not accept registrations more than six months after the course)

• After registration, ICPS will send you the final version of the task (CMI periodically update the criteria) with a deadline of four months to complete both assignments.

• Aim to produce drafts that you can send for feedback as soon as possible.

Deadlines and Registration

QUESTIONS AND CLOSE