module 8: monitoring, evaluation and learning – for increased impact and improvement of the iea...
TRANSCRIPT
Module 8: Monitoring, evaluation and learning – for increased impact and improvement of the IEA process
Sessions at a Glance
• Session 1: Introduction to Module
• Session 2: Definitions and Context
• Session 3: Developing a Monitoring and Evaluation Plan
• Session 4: Framework, Attributes and Measures
• Session 5: Planning a Self-Assessment
• Session 6: Improvement Opportunities
Exercise: Previous Experience with Monitoring and Evaluation (30 minutes)
Discuss with your neighbour (10 minutes)
• When have you been involved in an evaluation/monitoring process?
• What were some of the keys to success? • What were some of the challenges?
Discuss in plenary (20 minutes)
• Your stories, key elements of success and key challenges in plenary.
Monitoring Defined
• Monitoring is a planned, systematic process that closely follows a course of activities, and compares what is happening with what is expected to happen.
• Monitoring the IEA process makes sure the assessment meets its goals, while working within the scope of allocated resources (i.e., time, financial, human, informational and technical).
Evaluation Defined
• Evaluation assesses an achievement against preset criteria. Evaluations can have a variety of purposes, and follow distinct methodologies (process, outcome, performance, etc.).
• Evaluation of the IEA process determines the extent to which achievements (outputs, outcomes and impacts) are comparable with the originally intended purpose, and what lessons can be learned for the next environmental assessment and management cycle.
The evaluation of the process is, first and foremost, a capacity-development opportunity.
Compare Monitoring and Evaluation
MonitoringMonitoring1. Collecting data
2.2. Sense of progressSense of progress
3. Present time
4.4. Attention to detailsAttention to details
5. Inspires motivation
6.6. Occurs continuouslyOccurs continuously
7. Requires management skills
8.8. Asks: What needs to Asks: What needs to happen now to achieve happen now to achieve our goalour goal
EvaluationEvaluation1. Assessing data
2.2. Sense of achievementSense of achievement
3. Past / future time
4.4. Attention to bigger pictureAttention to bigger picture
5. Inspires creativity
6.6. Occurs intermittentlyOccurs intermittently
7. Requires leadership skills
8.8. Asks: Have we achieved Asks: Have we achieved our goal? Can we do better our goal? Can we do better next timenext time?
Learning Defined
• An emotional and/or cognitive transformation taking place during information collection and information processing, bringing about behaviour change or change in the ability to act differently.
Key Questions for Developing an Evaluation Plan
1. What is the purpose of the evaluation? 2. Who will use the evaluation results?3. Who will do the evaluation?4. What evaluation framework is practical?5. What needs to be monitored and evaluated?6. What are the steps to develop a self-
assessment matrix?7. How can I use the evaluation to enhance a
learning culture that keeps improving my IEA process?
Stages of National IEA Process
GEO as a Reporting Process
Plan - Do - Review Model
Why Monitoring and Evaluation?
• Lessons learned from IEAs help improve policy making processes, policies and state of environment.
• The IEA process builds capacity for periodic policy revision through monitoring and evaluation.
• Capacity is built among individuals and organizations.
Conceptual Understanding of the National IEA Process,
With Links to Ecosystem Health and Human Well-Being
Example: Ozone-related Treaties
• Continuous improvement of ozone-related international treaties was due to research-based evidence.
• This involved close monitoring of ozone concentrations in the stratosphere, simulation of possible scenarios, and revision of international agreements.
• Monitoring took into account processes, products and impacts.
Example: SoE Monitoring in India
• Capacity building and preparation of the SoE report are the main activities of the SoE Monitoring Program.
• National Host Institutions (NHI) and State Host Institutions (SHI) lead the program.
• A two-stage monitoring and evaluation program is in place.
SoE Monitoring in India
• Stage 1: – NHI performance evaluation (done by the
Ministry) is indicated by the number of states that have progressed in their SoE programs and published final SoE reports.
• Stage 2: – SHI performance as measured through the
number of SoE reports prepared.– The success of NHIs depend on SHI success,
and SHIs depend on NHIs for funding.
Coupling Science and Policy
• Communication between the cycle of scientific data collection and processing, and the cycle of policy making is facilitated by a GEO-type process.
• It can take decades to develop the right set of policies after the first evidence of an environmental issue.
Example: Lake Balaton, Hungary
• The first scientific warnings of eutrophication in were published in the early 1970s.
• The process of policy response included:1. unusual scientific observation;
2. scientific debate;
3. acceptance by the policy-makers;
4. policy development;
5. policy enforcement; and
6. a long time period required before the cumulative impact was apparent.
Example: United Kingdom
• The UK government has been publishing environmental indicators since 1996.
• The UK sustainable development strategy in 1999 significantly incorporated indicators.
• Headline indicators are used to communicate progress.
• An interdepartmental management board coordinates indicator development.
• Indicators were more successful in communicating progress than policy review.
Example: Ministry of Environment, Hungary
• Over a decade of published environmental and sustainable development indicators.
• Trends are distributed to a wide audience of technical experts, policy-makers and the general public.
• Regularly updated information has been a factor in the reports’ policy impacts.
Discussion: Additional Examples
• Do you know of other examples of how research-based evidence can guide policy making and lead to improvements?
• What are some examples in your country at the national level?
Sessions at a Glance
• Session 1: Introduction to Module• Session 2: Definitions and Context• Session 3: Developing a Monitoring and
Evaluation Plan• Session 4: Framework, Attributes and Measures• Session 5: Planning a Self-Assessment• Session 6: Improvement Opportunities
Foundation Steps for Effective Monitoring and Evaluation
1. Identifying your purpose.
2. Identifying your primary users.
3. Deciding whether internal or external evaluators best serve your purpose.
1. Types of Purpose1. Judgment: Sets clear criteria and
standards to judge performance. Can increase credibility of the GEO-process.
2. Improvement: Open-ended evaluation that measures change over time. Often applied to cyclical activities, like the GEO Process.
3. Knowledge-creation: Identifies emerging knowledge and insights. Increases the saliency of the GEO process.
2. Users of a GEO-type Evaluation
People who:
• can revise the GEO process: have the mandate, knowledge and skills; and
• want to revise the GEO process: have a vested interest in influencing the design and implementation of the GEO process.
Primary Users
The primary users of the evaluation may include:
• IEA core team (includes policy-makers);
• policy and decision-makers who are primary users of the IEA; and
• the evaluation team (internal and/or external).
Exercise: Purpose for Monitoring and Evaluation
In small groups, discuss the following:
1. Why do you need a plan for monitoring and evaluation of your IEA process and impact at the beginning of your planning process?
2. Why is improvement-oriented evaluation relevant to your IEA process?
3. Using the information above, what would you state as the purpose of your evaluation?
Exercise: Continued
4. Who are potential users?– What are their interests in using the
evaluation results?– Do they have a mandate to revise the GEO?
Group discussion (30 minutes), plenary presentations (15 minutes)
3. Evaluators
Evaluators may include:
• A small internal evaluation task force (including the IEA core team, which is recommended).
• External evaluators (consultants and internal evaluators of another IEA).
• A combination of internal and external parties.
Plenary Discussion: Identifying Evaluators
(40 minutes)
In small groups, for 20 minutes,
• Brainstorm key criteria for selecting evaluators; and
• Identify types of evaluators and possible names.
In plenary, discuss your results (20 minutes).
Plenary Discussion Continued:
What are some other parameters to consider for a monitoring and evaluation plan to be a success in my context?
Sessions at a Glance
• Session 1: Introduction to Module• Session 2: Definitions and Context• Session 3: Steps to Developing a Monitoring and
Evaluation Plan• Session 4: Framework, Attributes and Measures• Session 5: Planning a Self-Assessment• Session 6: Improvement Opportunities
Corresponding Attributes of Effective Evaluations
Conceptual Understanding of the National IEA Process,
With Links to Ecosystem Health and Human Well-Being
Framework for Monitoring and Evaluating
the National IEA Process
Monitoring the timely Completion of Key Activities and Outputs
Possible Measures for Effective Knowledge Management
Discussion: Effective Knowledge Management
(30 minutes)
In small groups, consider the following (15 minutes):
1. Can you think of additional important measures of effective knowledge management?
2. What do you think are reasonable targets for the measures you identified?
Possible Measures for Effective Opportunity Management
Discussion: Effective Opportunity Management
(30 minutes)
In small groups, consider the following (15 minutes):
1. Can you think of important measures of effective opportunity management not included in the table above?
2. What do you think are reasonable targets for the measures you identified?
Measures for Effective Relationship Management
Behaviour TypeBehaviour Type1. Receiving (individual,
organization)2. Seeking (individual,
organization)3. Acting Upon (individual,
organization, institution)
4. Demanding (individual, organization, institution)
Behaviour ExhibitedBehaviour Exhibited1. Reports, e-mails,
listserv2. Target users seek new
information. 3. Technical expertise is
sought to revise policies.
4. Specific needs, such as monitoring data for the next IEA cycle.
Identifying relationships based on behaviour
Possible Measures for Effective Relationship Management
Discussion: Measuring Relationship Management
(30 minutes)
In small groups, consider the following (15 minutes):
• Can you think of any other important measures of effective relationship management?
2. What reasonable targets would you recommend for various measures?
Measures for Improvements in Policy and Policy Making
• Measurement should relate to the change statement you identified in your impact strategy.
• Measurement should also track other observed improvements in policies and policy processes.
Sessions at a Glance
• Session 1: Introduction to Module• Session 2: Definitions and Context• Session 3: Steps to Developing a Monitoring and
Evaluation Plan• Session 4: Framework, Attributes and Measures• Session 5: Planning a Self-Assessment• Session 6: Improvement Opportunities
Planning a Self Assessment
Step 1Step 1Identify major issues and monitoring questions, and develop specific measures.
Step 2Step 2Identify sources of data and data collection methods.
Step 3Step 3Set priorities and frequency of monitoring
Step 1. Identify Major Issues and Monitoring Step 1. Identify Major Issues and Monitoring
Questions, and Develop Specific MeasuresQuestions, and Develop Specific Measures
OutcomesOutcomes• Your Change Statement • Effective Relationship Management
Activities and OutputsActivities and Outputs• Effective Knowledge Management• Effective Opportunity Management• Timely completion of activities and outputs
Outcome-based Measures: Possible Organization of Your Self-Assessment MatrixKey Issues/Questions Specific Measures
and TargetData Source Data
Collection Method
Your Change StatementHave the desired improvements in policies and policy processes that you identified in your impact strategy been realized? What other improvements in policies and policy processes have you observed during and following your national IEA process?
Effective Relationship Management What changes in the thinking and actions of policy-makers and decision-makers (and other important relationships) have you observed?
See Table 4 for guidance
Activity and Output-based Measures: Possible Organization of Your Self-Assessment MatrixStage of the
IEAKey Issues / Questions
Specific Measures and Targets
Data Source
Data Collection Method
Stage 1Inception
Timely completion of activities and outputs
See Table 7 for guidance
Effective Knowledge and Opportunity Management
See Table 5 and 6 for guidance
Stage 2Institutional Setup
Timely completion of activities and outputs
Effective Knowledge and Opportunity Management
Stage 3Scoping and Design
Timely completion of activities and outputs
Effective Knowledge and Opportunity Management
Stage 4Planning
Timely completion of activities and outputs
Effective Knowledge and Opportunity Management
Stage 5Implementation of IEA
Timely completion of activities and outputs
Effective Knowledge and Opportunity Management
Stage 6Communication and Outreach
Timely completion of activities and outputs
Effective Knowledge and Opportunity Management
Stage 7Evaluation
Timely completion of activities and outputs
Effective Knowledge and Opportunity Management
Step 2: Identify Sources of Data and Data Step 2: Identify Sources of Data and Data Collection MethodsCollection Methods
Collecting Data for Monitoring Effective Relationship Management• Requires that changes in behavior be
identified and mapped: – these incremental changes will lead towards
the decisions or changes you are seeking.
• Can be a time intensive process, so it is important to set up simple ways to monitor your strategy against those measures.
• Can set up a small contacts database with a journaling function (see next slide).
Collecting Data for Monitoring Effective Relationship Management
Step 3: Set priorities and Frequency of Step 3: Set priorities and Frequency of Monitoring and EvaluationMonitoring and Evaluation
• Ensure that critical indicators are monitored.
• Establish the frequency of monitoring for each indicator.Process indicators are needed throughout the
IEA.Progress indicators are needed less
frequently, but long after the IEA is complete.
Exercise: Preparing a Self-Assessment
1. Form working groups of 4–5 persons.2. Complete steps 1–3 for preparing a self-assessment
matrix.– Step 1: Step 1: Identify major issues and monitoring questions, and
develop specific measures. For activity and output-based measures, assign different stages to different groups and compile in plenary.
– Step 2: Step 2: Identify sources of data and data collection methods.– Step 3: Step 3: Set priorities and frequency of monitoring
3. Plenary Session to compile the results and discuss them.
Sessions at a Glance
• Session 1: Introduction to Module• Session 2: Definitions and Context• Session 3: Steps to Developing a Monitoring and
Evaluation Plan• Session 4: Framework, Attributes and Measures• Session 5: Planning a Self-Assessment• Session 6: Improvement Opportunities
Improvement Opportunities
• Thinking of an IEA as a capacity-building process significantly increases its impact.
• The more that monitoring and evaluation is treated as an organizational learning opportunity (versus a value judgment), the more effectively the IEA supports improvements in policy making.
Exercise: Learning
• In pairs, for 10 minutes:
Think of the last time you learned something new. What was it, how did you learn it and what did you do differently as a result?
What is learning?
• Learning is more than knowledge creation;
• Learning is demonstrated by behaviour change; and
• Information processing, in addition to information collection, is of paramount importance.
Learning Cycle
Exercise: Learning
A. Write what comes to mind based on the following four questions (5 minutes):
1. What did you hear during the GEO module training course (e.g., Stage 1) that you already knew?
2. What new information and insight did you gain?
3. How are you going to use this new insight?
4. How else and when could you use this new information? How could you improve policy making with this new insight?
Exercise: Learning
• Discuss your findings with your neighbors (5 minutes).
• In plenary, discuss the insights you have gained from this exercise. – How did the group discussion help you to
recognize improvement opportunities for the national IEA process and ways to have impact, such as causing changes in policy making? (15 minutes.)
A Few Prerequisites for Learning
• motivation, which often is the urgency to solve a problem, or act with the support of new knowledge;
• trust to discuss values, assumptions and ideas without repercussions;
• mandate and opportunity to apply the new knowledge; and
• shared understanding of the importance of learning (not only what to learn but also how to learn).
Learning Cycle with GEO-Specific Questions
Organizational Learning Cycle
The Role of Learning in Improving the IEA
• Formulate your change statement (Module 3).• Identify measures for your change statement and
other supporting measures for key outcomes and activities/outputs (Module 8, Tables 8 and 9)
• Examine performance against making the desired changes and summarize results
• Formulate recommendations and lessons learned.• Integrate (feedback) recommendations and lessons
learned to improve the next planning cycle and other decisions.
How Can We Use Learning Opportunities? – The Monitoring Meeting
• Learning opportunities naturally present themselves at the beginning and end of each IEA stage and each IEA cycle. These are the times when you need to reflect and articulate lessons learned to improve the next course of action.
• Given the limited time available, we suggest that your core IEA team organize regular, mid-stage and/or stage-end monitoring and evaluation meetings to serve two purposes: – 1. Monitor progress toward and capture lessons learned to
improve the next IEA stage and the next IEA cycle. – 2. Cultivate a learning, improvement-oriented approach
throughout the whole IEA process.
Questions for Monitoring Meetings
1. What were the most revealing lessons for you in this stage?
2. How can you use this knowledge, attitude to improve the next stages process, outputs or targeted changes?
3. Have you achieved what we planned for in this stage? If yes, what factors helped? If not, what factors hindered. How can we reach the desired goal?
4. Are there any unexpected results, emerging phenomena, trends or questions you want to discuss?
5. How did this stage contribute to the perception of saliency, legitimacy and credibility of the national IEA process?
Suggestions for General and Stage-Specific Questions and Exercises for Monitoring Progress and Promoting Learning at Specific GEO StagesGeneral questions for monitoring meetings
GEO Stage
Major activity Prevailing learning condition
Stage-specific activities for monitoring meetings (see Box 7)
(1) What were the most revealing lessons for you in this stage?
(2) How can you use this knowledge, attitude to improve the next stages, process, outputs or targeted changes?
(3) Have you achieved what we planned for in this stage? If yes, what factors helped? If not, what factors hindered? How can you reach the desired goal?
(4) Are there any unexpected results, emerging phenomena, trends or questions you want to discuss?
(5) How did this stage contribute to the perception of saliency, legitimacy and credibility of the national IEA process?
1 Inception
2 Institutional setup
Mandate Articulate, confirm mandate
3 Scoping and design
Trust Nurture trust Force-field analysis
4 Planning Information processing
Create opportunities for collective information processing
5 Implementation
Shared understanding of learning
Remind participants of how they learn best Keep, start, stop doing
6 Communication
Motivation Revisit motivation Carousel discussion
7 Evaluation Application Focus on lessons learned for application Samoan circle
Exercise: Designing a monitoring meeting
• In groups of 4 or 5, prepare an agenda for half-day monitoring meeting for any stage of your IEA process (30 min). Address the following:– What specific progress measures will be part of your
meeting?– What learning opportunities might be identified for the
stage of the IEA process you have selected?
• In plenary, discuss the results from two groups (15 min).