TOWARDS RESULTS BASED MONITORING AND EVALUATION SYSTEM: LEARNING FROM FAILURES OF THE MONITORING AND EVALUATION SYSTEM OF SCHOOL OPERATIONAL ASSISTANCE (BOS) PROGRAM IN INDONESIA

UMI HANIK Co-founder and Chair for Department of Networking and Advocacy Indonesian Development Evaluation Community (InDEC); Currently working as Monitoring and Evaluation (M&E) Specialist for Poverty Reduction Support Facility (PRSF) an Australian Aid Initiative project in Indonesia managed by GRM International on behalf of the Australian Government

!

*) Disclaimer: The paper is shared mainly for knowledge sharing and academic purposes. None of the concepts, approaches, or statements in this paper should be taken as any official viewpoint of neither the Government of Indonesia nor any other agency mentioned in the paper.

Paper presentation for 6th International Evaluation Conference 2014 Malaysian Evaluation Society (MES) Park Royal Hotel, Kuala Lumpur, 24-28 March 2014

BACKGROUND q According to law, Indonesian citizens have the right to

education and the Government has an obligation to finance basic education without charging fees

q  In 2005, the government introduce School Operational Assistance or Bantuan Operasional Sekolah (BOS) to provide operational funds directly to schools based on a per-student formula

q BOS objective was to improve access to quality education for all children of ages 7 to 15 in Indonesia by strengthening school-based management (SBM) and community participation

q  The Government of Indonesia with the World Bank support work to increase the effectiveness of the BOS M&E system

OBJECTIVE

The paper will outline the BOS M&E System, to explore the failures and learnt on how to perform RBM system in real world evaluation for the social assistance program in education sector

CONTRIBUTION TO FIELD

The paper is expected to provide the value adds of practical knowledge of learning from failures on the Results-based M&E System in real world evaluation for the social assistance program or education issues in countries.

METHOD q The paper is

summarized and rewrite from BOS KITA M&E Capacity Development project’s proceedings 2009-2012

q Assessment focused on three aspects of the M&E activities.

The system, approach, and procedure for M&E of BOS program

The institutional (regulatory framework and institutional mandates)

the organizational context (organizational structures and functions) under which the system take place

LEARNING FROM FAILURES

M&E RESOURCES q Only one person was employed for each

level and seems didn’t meet the requirement to secure quality RBM implementation

q The staff was also working for a unit within Education Office, which potentially brings up conflict of interest

q Staff did not have appropriate knowledge and skills to carry out the tasks

COMMUNITY PARTICIPATION q The community as an external component of BOS

Program on M&E is limited

q The community does not familiar with any system of M&E within BOS Program

q School Committee (in particular those of government schools) has become a non target impact of BOS Program.

q Committee holds limited access to carry out budget monitoring, the major activity of Committee

M&E INSTRUMENT q the nature of monitoring is a management audit q most respondents did not have common

understanding on the final goal q  information gaps and inconsistency between BOS and M&E

guideline

q Not sure which BOS Management is accountable to the achievement of participation of Local Government

q The instrument to collect the relevant data was not there, q  the report available only to prepare the administrative

requirement.

QUALITY OF M&E IMPLEMENTATION q  Coordination among BOS Management has not been addressed

proportionally q  All BOS Managements targeted schools as the object of M&E activities

which potentially brings up duplication q  BOS Management tends to focus its M&E activities on outputs monitoring

at school level q  BOS Management staff was very skeptical toward the utility of monitoring

for broader education purposes q  They were not informed nor have knowledge on how useful is monitoring

results to decision makers q  There was not any follow up actions made as recommended in the report q  The program continue and funds are disbursed, disregard the quality of

monitoring q  There is not any reward for conducting good monitoring or preparing good

monitoring report. q  Monitoring report did not clearly show quality of the Program q  The report does not show any further analyses of each score for each

indicator to find out the causes – why the score is low or high, what factors influence achievement of a particular indicator.

IMPROVED RBM SYSTEM

CAPACITY BUILDING STRATEGY q Focus on building capacity of BOS Management to

continue conduct RBM activities;

q  Improve and strengthen monitoring instruments that ensure monitoring results provides benefits to the Central and Local Governments to develop their education program, and refine tune the system as necessary;

q Select at least two Provincial BOS Managements as the active partners of Capacity Building team, in addition to Central BOS Management;

IMPROVED SET OF OUTCOME AND INDICATORS q  Refine the tune of BOS M&E system is significant to ensure clarity in

the goal and intermediate goals of BOS Program.

q  Adopting cause and effect logic that highlights interrelation between Problem and Causes with Impact and Effects (or Outcomes) may

helps BOS defining its ultimate goal and intermediate goals

q  clear goals and indicators of achievement brings advantages for BOS Management in conducting quality monitoring and preparing quality report.

q  Monitoring will not focus only on outputs, but include outcomes to measure impact or changes that BOS Program would like to achieve.

q  Report will no longer present only figures, but will highlight changes that the Program has achieved, how they are achieved and why the expected changes do not take place.

q  In addition, report will present situation to quality education which is very useful for decision makers developing further interventions.

CLEAR ROLES AND RESPONSIBILITY q  Each BOS Management authority and responsibility should clearly be

defined according to the level. q  The authority and responsibility should not overlap one to another. q  Each BOS Management should support one to another,

Level! Authority! Responsibility!Central! •  Socialization at Provincial level

•  Monitor quality implementation at Provincial level

•  Monitor achievement of all goals!

•  Ensure quality implementation at Central and Province levels

•  Report on consolidated achievement of all goals at national level

!Provincial !

•  Socialization at District/City level •  Monitor quality implementation at

District/City !

•  Ensure quality implementation at Province and District/City levels

•  Report on consolidated achievement of all goals at Provincial level

!District City !

•  Monitor quality implementation at school level

•  Monitor fulfillment of children’s rights !

•  Ensure quality implementation at District/City and school levels

•  Report on consolidated achievement of all goals at District/City

•  Report on the agreed dimension of change !

School !

•  Implement quality activities at school level

•  Monitor fulfillment of children’s rights!

•  Ensuring quality implementation at school level •  Report on the progress of activities •  Report of the agreed dimension of change !

IMPROVED SYSTEM AND INSTRUMENT ü M&E activities should not give another burden to the

government official or BOS management

ü  Improved RBM system should consider the capacity of the government officials or BOS management to manage the improved RBM system in a daily basis. ü  covering simple monitoring form ü  BOS Management could use instantly ü  handy instrument that support data collection, monitoring

and supervision, reporting, evaluation, and the utilization ü  include an automate and user-friendly excel-based

application ü  generating BOS progress and update on results

from the schools up to national level easily and instantly

ü  feed BOS decision makers with instant and sound information on BOS results.

Ø  With one simple excel sheet application the BOS management will be able to : Ø  use the monitoring form, Ø  data input-process, and Ø  preview the results and visualization,

instantly!! Ø  With such instrument and application will

lessen the burden to carry out the M&E in a daily basis

YOU WILL GET THIS, INSTANTLY!!! Instant analysis at district/city/school level

SEE HOW IT WORKS Let’s open the excel sheet application now ;)

CONCLUSION q  RBM in real world evaluation

ü  Applicability and simplicity is the key to successful RBM ü  Applicable RBM system will guarantee the buy in from the

government or the expected actors ü  User-friendly application is very much needed to support the

applicable RBM implementation ü  Capacity of the M&E resources to perform or to carry out the

system in a daily basis is significant ü  Ensure continue resource allocation ü  Intensive nurturing and coaching for two years full on initial

implementation to change old behavior and to ensure the applicability

ü  Participative approach for the engagement and continuous learning to ensure compliance