creating robust monitoring systems dr susan denman head of research, monitoring & evaluation...
TRANSCRIPT
Creating Robust Monitoring Systems
Dr Susan DenmanHead of Research, Monitoring & Evaluation
Open Days Workshops “Managing Regional Development”
28-29 September 2004Brussels
Creating Robust Monitoring Systems
How can we ensure that we get high quality data which are useful for programme management and evaluation purposes?
www.wefo.wales.gov.uk
Structure of Presentation
• Welsh context
• Definitions
• Issues at design and planning stages
• Targeting projects for quality
• Issues at data collection and entry stages
• Reporting
www.wefo.wales.gov.uk
www.wefo.wales.gov.uk
The Welsh Context (1)
• Three main programmes:Objective 1
Objective 2 Objective 3
• Four Community Initiatives: INTERREG IIIA, URBAN II, LEADER+ and
EQUAL
www.wefo.wales.gov.uk
The Welsh Context (2)
• Total European Funding of €2,180 million
• Total value of €4,675 million (including private sector match funding)
Largest programme is Objective 1
Main aims:
• to create 43,000 net additional jobs
• to contribute to raising the Gross Domestic Product (GDP)
• reduce economic inactivity
www.wefo.wales.gov.uk
ED&T Group of Welsh Assembly Government
WEFO
ERDFINTERREG
Project Management Policy and Strategy
ESFEAGGF
FIFGLEADER+
EQUAL
LCDURBAN
Finance and Corporate Services
Paying Authority
Finance Payments Article 4 Article 10 IT
RM&EPolicy
BranchCommunic
ationsAssemblyBusiness
RM&E Research, Monitoring & Evaluation Branch
Monitoring and Reporting
4 staff: 1 researcher,3 administrators
Responsibilities:PMC Papers
Annual Monitoring Business PlansAnnual Implementation Reports
Local Partnership Reports
Evaluation
1 researcher,2 research and
evaluation specialists
Responsibilities:Data Quality
Programme Level EvaluationM&E advice and guidance
Head of Branch
Corporate
1 administrator, 1 researcher
Responsibilities:Training
DisseminationSynthesis Papers
Ex-Post EvaluationClosure Reports
Definitions
Monitoring is the regular and systematic examination of the resources, activities and results to inform management decisions
Audit is the verification of the legality and regularity of the implementation of resources.
Evaluation is a broad concept concerning the judgement made on the value of the initiative, which can encompass the needs which have to be met and the effects produced by it. Evaluation concerns not only outcomes but the processes by which they have been produced.
www.wefo.wales.gov.uk
Design and Planning
• Consider monitoring and evaluation at outset• start with ‘what’ not ‘how’• involve users from start• choose indicators with care• ensure hierarchy of indicators is coherent• build capacity
www.wefo.wales.gov.uk
Indicators
Helps if they are:• clearly linked with goal, objectives or targets• cover monitoring and evaluation needs • measurable on a regular basis using reliable
methods• provide clear and simple information• measurable against baselines• congruent with other monitoring indicator
systems
www.wefo.wales.gov.uk
Data Provided by Projects
Projects should:• understand purpose of centrally collected
data• collect other useful data• have access to training, support and
guidance• be clear on definitions• be checked for quality of their plans
www.wefo.wales.gov.uk
Project Sponsor
Gateway
Project development
Submission of application
Project appraisal
projectapproval
Delivery and review
Payments Audit
Closure
WEFO guidance
Advice on prioritisation from Thematic Advisory
Groups
WEFO advise
WEFOappraise
Advice from thematic groups, cross-cutting theme advisors and local partnerships
WEFO monitor
Advice from local partnerships and project officers
Assistance from local partnerships and project officers
Support and aftercare from local
partnerships and project officers
The Project Lifecycle and Monitoringwww.wefo.wales.gov.uk
Data Collection and Storage
• timing and methods• dealing with inaccuracies, gaps, double
counting and under-reporting • designing a good IT system • testing and evaluating the whole monitoring
system• management checks of data entry• liaison with global grants/grant schemes
www.wefo.wales.gov.uk
Use of data
• involve users in report design• co-ordinate release of data• ensure users understand data limitations • ensure that definitions are known and
understood by all• analyse the data collected• help with data interpretation
www.wefo.wales.gov.uk