Monitoring and Evaluation

ECO-UNESCO’s experience

Elaine NevinNational Director

ECO-UNESCO

NYCI Conference 2011

A National Environmental Education and Youth Organisation. Est. in 1986

environmental education and environmental work with young people.

Affiliated to the World Federation of UNESCO Clubs, Centres and Associations (WFUCA)

Who we are?

Our MissionTo channel young peoples passion into environmental protection and conservation and to promote the personal development of young people.

Our Aims…

•Promote environmental awareness, knowledge and understanding in young people

•Promote environmental protection & conservation

•Promote the personal development of young people

•Promote UNESCO ideals

Environmental Youth programmes

ECO-UNESCO ClubsYoung Environmentalist Awards ProgrammeEnvironmental WorkshopsYouth Forums, events and activitiesECO Choices programmeYouth for Sustainable Development programme

Education and Training programmes

Short Trainings – Teachers, Youth Leaders & Community WorkerFETAC accredited centre

Sustainable Development L5ECO-Community Development L3

Educational PublicationsVarious interactive resources (Print/Media)

What we do…

Representation & Advocacy

What we do…

The Scale of our Work

•Over 9500 people annually•National Basis / YEA all-Ireland programme•A team of 10 staff members

• 2 government work placements at any one time • support 6 graduate interns• over 100 volunteers

What we want to know…M&E

How many people/organisations/ schools are accessing our programmes and services at any one time and over a period?

What are the demographics of these people - age, gender, geographical spread etc?

What is the retention rate of our programmes (for longer term direct contact programmes) – do we have a high fall off rate?

What we want to know… What have people gained from participating in

an ECO-UNESCO workshop, programme or training? - knowledge, skills developed, attitudes, behaviour change, how much have they enjoyed it etc.

What is the impact of our programme on the participant?

(impact from a range of perspectives – due to the nature of our work)

What we want to know… Are we delivering programmes that address a

need?

As an organisation are we achieving our goals, targets and expected outcomes?

Are we delivering quality programmes?

Why do we monitor & evaluate? Out of our own desire to develop quality

programmes; review, learn, modify & improve

Out of our own need to ensure our delivery to greater numbers of young people

To ensure we are doing the job the way we should be doing it, with the people we should we doing it with, achieving the results addressing the needs of young people with an impact on our participants

Are we responding to needs of young people?

Is our M & E funder driven?

Some of our tools of Monitoring & Evaluation

Performance Management System – Planning, developing, monitoring and appraisal

Before and after questionnaires on knowledge, attitude and behavior (baseline/endline)- for programmes

Analysis of programme participant feedback on evaluation forms

Specific programme evaluations e.g. YEA evaluations

Other tools – Youth worker self-evaluation forms post youth

programme/training /workshops people counter to count numbers of people visiting

the Greenhouse etc.

Samples of some of our tools ofMonitoring & Evaluation measure

PLANNING (Qualitative/Quantitative) Annual Implementation Plan

MONITORING (Quantitative/Quantitative) Direct Participant Analysis Information Direct Participant Feedback on Learning & on Satisfaction Evaluation forms as part of programmes e.g. YEA

Through YEA/Clubs/YSD, analyse what, how many and the impact of environmental action projects;

EVALUATION Reviews carried out Impact Assessment

SAMPLE 1: PLANNING - Annual Implementation Plan – Organisational Plan for a year

Programme - Strategic Aims

Objective Actions Projected Output

KPIs (Key Performance Indicator)

Outcome Target Group

Time Frame/ Deadline

Result / (Progress)

Review on a quarterly basis

SAMPLE 2: MONITORING -Participant Analysis – Demographics etc.

Monthly/annual programme -MONITORINGProgramme & Service Outputs

October 2011

YSD; 6; 14%

YEA; 9; 21%

EYC; 8; 19%TRAIN; 7; 17%

YOUTH; 12;

29%

YSD

YEA

EYC

TRAIN

YOUTH

Programme & Services OutputAugust - comparative analysis

0

2

4

6

8

10

12

14

P r ogr amme

2009

2010

2011

         Age range, gender breakdown & geographical spread of participants      

PROG CODE

DATE NAME OF EVENT

<10 10-14 15-17 18-21 22-24 >25

TOTAL PARTICI-PANTS

NEW PARTICIP-ANTS

CONTACT HOURS

Day Mth Year

M F M F M F M F M F M F    

3

SAMPLE 3: MONITORING - Participant Feedback on Satisfaction & Learning

ECO-UNESCO measures the following:• Content relevance• Process (quality of facilitation, presentation and group

work/activities)• Resource materials• Learning

We analyse reactions and comments to enhance quality of future work

Drimnagh Castle Oct 09 to Jan 100%

2%

8%

44%

46%

1

2

3

4

5

46% - Excellent44% - Very Good8% - Good2% - Fair0% - Poor

SAMPLE 3: MONITORING - Participant Feedback on learningYouth Leader Training Course

Participants carry out Self Assessment of Learning in terms of knowledge, skills & attitudes – before & after each activity/ programme

Before CourseSelf-assessment of Your understanding,

knowledge and Skills Related to:

After Course

1 2 3 4 5

Q1. What Environmental Education (Education for Sustainable Development) is 1 2 3 4 5

1 2 3 4 5

Q2. How to integrate interactive Environmental Education activities into your daily work

1 2 3 4 5

1 2 3 4 5

Q3. How to encourage critical thinking in young people through Environmental Education

1 2 3 4 5

1 2 3 4 5Q4. How to foster Environmental Awareness in Young People through Environmental Education

1 2 3 4 5

1 2 3 4 5Q5. How to carry out an Environmental Action Project (Energy Audit) with Young People 1 2 3 4 5

1 = No understanding or skills 3 = Some understanding or skills5 = A lot of understanding or skills

SAMPLE 4: Impact Assessment - PILOT INITIATIVE 2010-2011

In 2010, we wanted to look at how we could measure the impact of our programmes; reviewed and decided to implement an Impact Assessment mechanism

Impact Assessment Workshops (3 to 6 months post delivery) Workshop used to measure two of our programmes:

Youth for Sustainable Development Peer Education programme FETAC Level 5 Sustainable Development programme

The method/workshop measures in two ways: In relation to what use and in what ways the learning has been

applied in practice by the participant/learners themselves after completing the ECO-UNESCO programme

In a broader context how much the ECO-UNESCO programme has impacted on them and is of use to society in general and on their communities?

ECO-UNESCO took the role of facilitator Former participants become the impact assessors

Sample Impact Assessment Grid

Story 2Name: Julia Lane Barnes“My update since I finished ECO-UNESCO’s FETAC Level 5 Sustainable Development course is that I am involved with working with a local interfaith youth group with an emphasis on the common theme of the inner spiritual garden. We use lots of the methodology that I learned doing ECO-UNESCO’s FETAC Level 5 Sustainable Development course. Our long term aim is to make a link between the restoration of a local walled garden and going to Mount Sinai with the Makhad Trust to help mend the walled gardens of the Bedouim.”

Story 1Name: Frida Besong “Thanks to my participation in ECO-UNESCO’s FETAC Level 5 Sustainable Development course, my interest was aroused on Sustainable Development issues and I finally wrote my M.A thesis on International cooperation regarding the management of water in Lake Chad and other natural resources in the Lake Chad Basin. At this stage I have applied for a P.H.D In Education for Sustainable Development and I am waiting for the outcome”

Story 3Name: Helen “I helped organize the Greenhouse gig fashion show (recycled materials), which was about organization – there was loads of organization in it. And it was about making people aware about climate change & development, so that was Peer Education.” (about 50 to 60 Peers)

Sample Impact Assessment Stories

Challenges Time - Small organisation with limited capacity – want to do the

job and make sure we are doing the job well – but need to balance this with getting the job done – can be a challenge.

Buy in from all involved - depending on people to collect data and information and to input it into a system (Salesforce CRM)

Making sure what you are monitoring and evaluating is what you need to be monitoring and evaluating – using the information you get

M & E needs to focus on Quality as much as Quantity.

We gradually implemented the M & E system – but it is under constant review

Specialist organisation with diverse funding streams with diverse requirements in terms of outcomes and outputs

Future plans…2011 onwardsReview current M & E system – ensure it is useful

and informing our work in the way it should be

Ensure it is in line with and collecting necessary information for the implementation of National Quality Standards Framework - use this as a tool.

For further information please contact

ECO-UNESCO'the greenhouse'

17 St., Andrew St., Dublin 2

T: 00 353 (0) 1 6625491F: 00 353 (0) 1 6625493

E: info@ecounesco.ieW: www.ecounesco.ie

 ECO-UNESCO – Conserving the Environment; Empowering Young People

ECO-UNESCOThe Greenhouse, 17 Saint Andrew Street, Dublin 2,

tel +353 1 6625491 www.ecounesco.ie email: info@ecounesco.ie