quarterly performance report - united states agency for ...pdf.usaid.gov/pdf_docs/pa00n399.pdf ·...

July 30, 2014 This publication was produced for review by the United States Agency for International Development. It was prepared by DevTech Systems, Inc. under Contract Number AID-520-C-13-00001.

Quarterly Performance Report

April 1, 2014 – June 30, 2014

Monitoring & Evaluation Program

Quarterly Performance Report:

April 1, 2014 – June 30, 2014

Submitted to: Josefina Martinez, Contracting Officer’s Representative

USAID/Guatemala

Submitted by: DevTech Systems, Inc.

Contract AID-520-C-13-00001

DISCLAIMER:

This report is made possible by the support of the American People through the United State Agency for International

Development (USAID). The contents of this report are the sole responsibility of DevTech and do not necessarily

reflect the views of USAID or the United States Government

USAID/Guatemala Monitoring & Evaluation Program

Quarterly Performance Report: April-June 2014

AID-520-C-13-00001

Prepared by DevTech Systems, Inc., IARNA, ASIES and Geotecnológica.

i

TABLE OF CONTENTS

EXECUTIVE SUMMARY ...................................................................................................... 1

I. INTRODUCTION ............................................................................................................... 5

II. MONITORING AND EVALUATION PROGRAM RESULTS ............................................. 6

Result 1. Statistical Tools for Baseline, Mid- and Final-term Data Collection Developed .................. 6

Result 2. Local Capacity in Monitoring and Evaluation, including Data Collection of National and sub-National indicators, increased ............................................................................................................ 7

Result 3: Clearinghouse, Management Information System (MIS), Geographic Information System (GIS), Environmental Compliance and other M&E Services Developed and Delivered. ................. 11

Result 4: Mission Programs and Project Evaluations Conducted and Results Reported. ............... 15

Result 5. Monitoring and Evaluation Services for other USAID/Guatemala Offices, Conducted Upon Request............................................................................................................................................. 18

Gender Considerations ..................................................................................................................... 19

Coordination, Collaboration and Information Sharing ...................................................................... 20

III. SUMMARY AND CONCLUSION ................................................................................... 21

ANNEX A. SUCCESS STORY ........................................................................................... 23



AID-520-C-13-00001


ii

ACRONYMS

ASIES Asociación de Investigación y Estudios Sociales

CBA Cost Benefit Analysis

CDCS Country Development Cooperation Strategy

CEA Cost-Effectiveness Analysis

CEDICAFÉ ANACAFÉ’s Coffee Research Center

CNCG Climate, Nature and Communities in Guatemala

CONASAN National Food Security Council

CONAP National Council of Protected Areas

COP Chief of Party

COR Contracting Officer’s Representative

DIPLAN Directorate of Planning

DGO Democracy & Governance Office

DQA Data Quality Assessments

ECAM Environmental Region of Central American and Mexico

EGO Economic Growth Office

EMP Environmental Mitigation Plan

EMPR Environmental Mitigation Plan and Report

ENCA The Central National School of Agriculture

ENCOVI Household Expenditure Survey

ENSMI Maternal and Infant Health Survey

FTF Feed the Future

FSN Food Security and Nutrition

FUNDAECO Foundation for Ecodevelopment and Conservation

GCC Global Climate Change

GEO Geotecnológica de Centroamerica, S.A.

GEMS Global Environmental Management System

GIS Geographic Information System

GOG Government of Guatemala

HEO Health & Education Office

IARNA Instituto de Agricultura, Recursos Naturales y Ambiente, Landívar University

INE National Statistical Institute

INGUAT National Tourism Institute

IP Implementing Partner

MAGA Ministry of Agriculture

M&E Monitoring and Evaluation

MBR Mayan Biosphere Reserve

MDG United Nations Millennium Development Goals

MEO Mission Environmental Officer

MEP Monitoring and Evaluation Program

MIDES Social Development Ministry

MINIFIN Ministry of Public Finance

MIS Management Information System



AID-520-C-13-00001


iii

MOU Memorandum of Understanding

NGO Non-Governmental Organization

OCSE/SAN Sector Coordination Office for Food Security and Nutrition Statistics

PAT Poverty Assessment Tool

PMP Performance Monitoring Plan

PNDRI Política Nacional de Desarrollo Rural Integral

PRS Policy Regulatory Support Program

RA Rainforest Alliance

REA Regional Environmental Adviser

REO Regional Environmental Officer

RVC Rural Value Chain

RVCP Rural Value Chain Program

SEGEPLAN Secretariat for Planning and Programming/Office of the President

SESAN Secretariat for National Food Security and Nutrition

SIMON GIA Monitoring System of the Institutional Support Group

SINASAN National System for Food Security

URL Rafael Landívar University

USAID United States Agency for International Development

USG United States Government

WEAI Women’s Empowerment in Agriculture Index

WHIP Western Highlands Integrated Project



AID-520-C-13-00001


iv

This page is intentionally left blank.



AID-520-C-13-00001


1

EXECUTIVE SUMMARY

The Monitoring and Evaluation Program (MEP) is designed to support USAID/Guatemala’s

Economic Growth Office (EGO) by providing the necessary information to facilitate informed

program management decisions to improve the effectiveness of development assistance

interventions, and to shape longer-term strategic direction and decision making, in particular

with regard to the implementation of the Feed the Future (FTF) and Global Climate Change

(GCC) initiatives. The Program is working to develop a Monitoring and Evaluation (M&E)

mechanism that provides continuous performance monitoring, verification and impact evaluation

of program results, including environmental compliance.

Led by DevTech Systems, Inc. (DevTech), the prime contractor, the Program has a three-

pronged strategy which includes monitoring and evaluating EGO programs, measuring United

States Government (USG) “whole-of-government” progress towards EGO programs, and

measuring country-level progress towards M&E of EGO priorities. DevTech works in

partnership with three Guatemalan organizations representing a university (the Instituto de

Agricultura, Recursos Naturales y Ambiente, Landivar University, IARNA), a private research

association (the Asociación de Investigación y Estudios Sociales, ASIES), and a private company

(Geotecnológica, GEO). Together the four organizations form the MEP Team.

Overview of Achievements by Result for the Quarter

Result 1: Statistical tools for Baseline, Mid- and Final-term Data Collection Developed

MEP has developed and put into practice the conceptual framework of M&E. This has been the

basis to provide institutional context for the performance evaluation of the Rainforest Alliance’s

Climate, Nature and Communities in Guatemala (RA’s CNCG) program. The evaluation takes

into account elements of the Mission’s Cooperating Country Development Strategy (CDCS) and

national public conservation policies. MEP also has maintained its efforts to collect baseline data

on the different programs, especially Rural Value Chains (RVC), with the aim that the evaluation

systems are able to use standard indicators to evaluate performance as well as the impact that the

projects can have in the mid- and long-term. The beneficiaries’ directory has been jointly

designed with implementing partners for this purpose. The directory was conceived both as an

internal project management tool and as the basis for designing surveys which specify useful,

detailed information for the evaluations required by USAID. MEP has submitted the Poverty

Assessment Tool (PAT +), designed in 2013, to a revision by the national and regional scientific

community. This is important if the instrument is to be distributed for generalized application in

other situations. Finally, MEP has made a revision of the methodology and design of the Western

Highlands Integration Project (WHIP) baseline in its zone of influence, which was developed by

the MEASURE Evaluation. The aim of this was to propose an implementation plan for the mid-

and final-term evaluations in the coming years.



AID-520-C-13-00001


2

Result 2: Local Capacity in Monitoring and Evaluation, including Data Collection of

National and Sub-National Indicators Increased

MEP is in full swing providing technical assistance to the Governmental authorities responsible

for executing the Zero Hunger Pact, where Monitoring and Evaluation systems are relevant. In

this sense, after concluding the general diagnosis of the National Information System for Food

Security (SIINSAN), the Secretariat for National Food Security and Nutrition (SESAN) has

requested that follow-up is given to a proposal working towards ordered recording of data on

food security and nutrition. This database will facilitate analysis and also allow the system to

become one of the primary sources of statistical information for the Sector Coordination Office

for Food Security and Nutrition Statistics (OCSE SAN) in the future. MEP continues to assist the

Ministry of Agriculture (MAGA) in the design and implementation of the monitoring and

evaluation system. The design of the system has been approved by MAGA at this time, and it is

currently in the informatics phase. Finally, OCSE SAN is up and running as planned and has had

two meetings with official delegates from 20 Government entities as well as MEP and the Rafael

Landívar University (URL) through IARNA. At present, plans are in place to strengthen OSCE

through a grant from MEP.

Result 3: Clearinghouse, Management Information System (MIS), Geographic Information

System (GIS), Environmental Compliance and other M&E Services Developed and

Delivered

MEP is advancing its Clearinghouse services on each of the three levels: GIS, M&E, and

Environmental Compliance. The Environmental Compliance module was successfully brought

online during the quarter, and is now being used by the Mission. The module was approved by

the Regional Environmental Adviser (REA) in El Salvador. One important achievement for

USAID Guatemala and MEP is that this module will be used as an example and base to develop

Monitor Plus, which will be the official system for environmental compliance for the Latin

American region in the future. MEP acquired the new ArcGIS OnLine platform, which will

allow this module to provide a wider range of services. At the moment, MEP is migrating all its

data over to the new platform, which is already online for use. Finally, there is the storyboard

which conceptualizes the M&E module. This will shortly be validated and approved by USAID,

which will allow the development of the module to be concluded. The new beta version is

expected to be launched at the end of the year.

Result 4: Mission Programs and Project Evaluations & Studies Conducted and Results

Reported

MEP is in the concluding stage of data collection for the construction of baseline data for the

performance evaluation of the CNCG Program executed by RA. It is expected that RA will

present the final report on baseline data at the start of the next quarter. Also, within this result,

and as a part of the technical assistance to ANACAFÉ for the M&E system for Coffee Rust, a

Memorandum of Understanding (MOU) was signed between the involved parties (ANACAFÉ,

RUTA, and MEP). This sets the stage to complete the grant that MEP will award to CADICAFÉ



AID-520-C-13-00001


3

to strengthen the early warning system for Coffee Rust in Guatemala. Furthermore, within this

component and in coordination with the Technical Secretary for the Rural Development Cabinet,

indicators have been designed and proposed for the follow-up and evaluation system for

territorial implementation of the multi-sectorial policies that make up the Política Nacional de

Desarrollo Rural Integral (PNDRI). The field phase was concluded and a draft report on the

environmental audit of the RVC Projects of ANACAFÉ and AGEXPORT has been presented for

USAID review. The work was developed by the Cadmus Group and supervised by MEP and its

environmental compliance team. Finally, the “Rapid Assessment” of USAID’s Policy Regulatory

Support (PRS) Program began. The evaluation team was formed and approval was given for the

implementation plan. It is expected that the results will be presented at the start of the next

quarter.

Result 5: Monitoring and Evaluation Services for other USAID/Guatemala offices, Upon

Request

During the reporting period, MEP reviewed aspects of the Citizen Security and Community

Resilience indices produced by the University of Kentucky under MEP. The findings will be

consolidated into presentation by MEP to USAID’s Democracy and Governance Office (DGO)

to share results and an alternative methodology to develop a Citizen Security Index. The report

is currently being prepared and will be presented shortly.



AID-520-C-13-00001


4




AID-520-C-13-00001


5

I. INTRODUCTION

Activities carried out this quarter reflected the delivery of expected technical outputs under each

result. MEP has continued to ensure it develops all its work in a coordinated manner with

USAID, subcontractors and implementing partners. Close coordination with MEP’s Contract

Officer’s Representative (COR) has also facilitated smooth technical and administrative program

implementation.

In the final stretch approaching the end of the base period and the end of the 2014 calendar year,

MEP is making important efforts in all components and complying with the delivery of the

following results set out for each component: i) Development of statistical tools, ii)

Strengthening local capacity, especially in counterpart Government agencies, iii) Development

of Clearinghouse services in the modules of environmental compliance, GIS, and M&E, iv)

Commissioned evaluations and studies, and v) Sale of M&E services to the offices for

Democracy and Governance and Health and Education.

The Quarterly Report is organized by Result with reference to the Work Plan for each area, and

includes: (1) Activities, achievements, and deliverables; (2) Opportunities, problems, delays, and

proposed solutions; (3) Key meetings; and (4) Activities planned for the next quarter. The

technical report is accompanied by the quarterly financial report on invoiced expenditures by line

item and projected expenditures for the next quarter, submitted under separate cover.



AID-520-C-13-00001


6

II. MONITORING AND EVALUATION PROGRAM RESULTS

Result 1. Statistical Tools for Baseline, Mid- and Final-term Data Collection Developed

During the life of the Program, the team will work with USAID’s Economic Growth Office

(EGO) and implementing partners (IPs) in Guatemala to engage in baseline data collection at the

community and municipal level and establish targets. This result also includes activities to

support the generation of statistical information related to the objectives of the USAID Mission

in Guatemala, particularly those related to the Feed the Future (FTF) and Global Climate Change

(GCC) initiatives.

Activities, achievements, and deliverables1:

During the quarter, MEP has validated the results of the Conceptual Framework study of

Monitoring and Evaluation (M&E) through its performance evaluation of Rainforest Alliance’s

(RA) Climate, Nature and Communities in Guatemala (CNCG) Project. This was in response to

evaluation questions from the baseline development phase. In its conceptual framework of M&E,

MEP refers to the objectives and high-level indicators of USAID/Guatemala’s Country

Development Cooperation Strategy (CDCS). This allows for greater coherence in the analysis of

the assessed Program, showing its relationship with the complete cooperation strategy of USAID

in Guatemala.

After consulting with the local scientific community about the use of statistical information and

national surveys, it was suitable to make a comparison of the Poverty Assessment tool (PAT +)

developed by MEP with ENCOVI 2006 (Household Expenditure Survey). Initially, the PAT +

was developed using the more recent ENCOVI 2011 instrument. Since ENCOVI 2006 has

almost universal acceptance from the scientific community, this comparison would allow the

scientific community to validate the model and credibility of the PAT + developed by MEP over

the past year. The idea is to distribute the tool for public and private use as soon as it is validated

by the scientific community.

During the quarter, MEP developed methodology to improve the Beneficiary Directory of the

Rural Value Chain Project (RVCP). This is basically a census through which exact information

can be gleaned, such as: how many beneficiaries there are, how many homes there are, and the

characteristics of the households and people. A form has already been designed for this purpose,

which has three main sections: 1) Location of the household, 2) Household characteristics, and 3)

Characteristics of household members. Within these three sections are basic questions to obtain

all the necessary information. Furthermore, through the results obtained, surveys can be designed

1 Some tasks from Result 1 are reported in Result 2 as they are more in line with strengthening local M&E capacity.



AID-520-C-13-00001


7

with greater precision which will allow data to be obtained for more complex indicators such as

employment. This same methodology can be applied to other value chain projects such as MÁS

FRIJOL, and “Taking an innovative cooperative approach to food security and trade,” for

example.

MEP has begun the revision of the mid-term evaluation methodology of the Western Highlands

Integrated Project (WHIP). The baseline report presented by the MEASURE Evaluation will be

used to do this. Databases and sample designs have been requested for each area of study in the

baseline.

Opportunities, problems, delays, and proposed solutions:

MEP will obtain precise knowledge of the location of the Climate, Nature, and Communities

in Guatemala (CNCG) Program, within the context of the USAID’s CDCS. This will allow

for better contextualization of the performance evaluation developed by MEP in the initial

baseline phase.

Validate the PAT + instrument developed by MEP in 2013 with ENVOCI 2006. This will

give the tool greater credibility from the scientific community as a measurement instrument

for poverty levels and estimated income from the consumption aggregates.

MEP has prepared a methodological proposal for developing a Beneficiary Directory for the

agricultural Rural Value Chain (RVC) projects of ANACAFÉ and AGEXPORT. However,

the budgetary viability of developing the directory is still being analyzed.

There have been some delays in obtaining databases generated by the MEASURE

Evaluation, as well as with the sample designs. Therefore, the methodological and cost

proposals for the development of the medium-term analysis for WHIP have not yet been

completed. Steps have been taken to have these as soon as possible.

Activities planned for the next quarter:

MEP has planned an expert review on information capture and statistical tool development

from surveys to assess the following areas: PAT +, Beneficiaries Directory, and medium-

term analysis of WHIP in 2015.

Develop local support, including financial support, for advancing the Directory, as well as for

obtaining the databases generated by MEASURE in developing the baseline for evaluating

WHIP.

Result 2. Local Capacity in Monitoring and Evaluation, including Data Collection of National and sub-National indicators, increased

Under Result 2, MEP works to strengthen the institutional and statistical capacity of Government

of Guatemala (GOG) institutions and Guatemalan academic, nongovernmental, and private

research organizations to monitor food security and environmental/biodiversity progress and to



AID-520-C-13-00001


8

institutionalize sustained and systematic monitoring and evaluation beyond the life of Program

assistance. MEP also supports GOG efforts to measure Guatemala’s progress towards the United

Nations Millennium Development Goals (MDGs).

Activities, achievements, and deliverables:

MEP concluded the analysis of National System for Food Security (SIINSAN) and developed

the corresponding report. A proposal to strengthen SIINSAN was presented to the Secretariat for

National Food Security and Nutrition (SESAN) through the technical sub-secretary which will

generally consist of improving the structure of recording information. The aim of this is to

facilitate data analysis at different levels. In order to achieve this objective, MEP has proposed to

analyze the quality of data that comes from different sources and develop a ‘training the trainers’

program in different institutions. Another important recommendation of the proposal that Sector

Coordination Office for Food Security and Nutrition Statistics (OCSE SAN) should be the

official repository for this information system so that it begins a process of to standardize and

officialize the statistical information generated.

During this quarter MEP also initiated the submission of data on the different USAID food

security and nutrition programs to the Monitoring System of the Institutional Support Group

(SIMON GIA). This decision was made because indicators in question are directly related to

food security interventions. It was necessary to make a series of adjustments so that the

submitted information complied with the system’s restrictions. Negotiations are being made with

SESAN so that SIMON GIA is better adjusted to the nature of the cooperation in general, not

just with USAID. MEP participated in the National Food Security Council (CONASAN) and

presented a report on the support MEP has provided to the Zero Hunger Pact as USAID is

temporarily presiding over the Institutional Support Group (GIA). This presentation was made

jointly with the representative of the Spanish Cooperation (cooperación española).

The startup meeting of OCSE SAN was held on April 11, 2014 with the participation of

delegates from 20 institutions that make up OCSE. Dr. Hugo Melgar attended the meeting via

Skype, and directed the conference “The Importance of Statistics in food security and nutrition”

from McGil University in Canada. The second OCSE meeting was held with the same

participants, and the agenda included the overarching themes for the development of the Office.

Both meetings took place in the MEP offices. Also during the quarter, INE management showed

interest in reinforcing OCSE SAN through strategic actions that can be institutionalized in the

INE and can strengthen the objectives of the National Statistical System (SEN).

The management and board of directors of INE agreed on the following strengthening activities:

Equipping and strengthening the OCSE SAN

Developing national statistical capacities in food security and nutrition

Improving and promoting sustainability in statistical systems



AID-520-C-13-00001


9

In line with the previous points, a cooperation agreement was developed and approved to provide

technical and financial assistance to the Instituto Nacional de Estadística (INE) in order to

“strengthen, modernize, standardize, and integrate the different processes or methodologies of

recovery, production, analysis, and distribution of statistics on food security and nutrition.” This

assistance is offered in the context of the institutional development of the OCSE SAN, which

seeks to integrate and consolidate the decentralized National Statistical System (SEN) to

guarantee the production of information and statistical data on the country that is timely,

pertinent, reliable, and of high quality.

During the quarter, MEP continued developing the M&E system for Ministry of Agriculture

(MAGA). This was initially planned for the Logical Framework of the Strategic Plan which the

Ministry developed in 2013 with the support of USAID’s Policy Regulatory Support (PRS)

Program. This would include the primary programs of the Vice-Ministry of Rural Economic

Development (VIDER) and the National System for Rural Extension (SNER). However, during

the development of the consultancy the Minister decided to widen it to the whole Ministry, as the

process was of particular interest to the Ministry in its utilization across various units. This

participative process has allowed MAGA’s planning specialists to adapt the system even before

putting it into operation. Furthermore, work was begun on the computerization of the system by

the Ministry’s IT unit. An analysis was made of the possibility that MEP would support the

MAGA by contracting of a programmer to design and program the system Dashboards.

Finally, in agreement with the Ministry of Agriculture, MEP decided to no longer represent the

MAGA in the Board of Directors of the Central National School of Agriculture (ENCA). The

tasks set out within the Board were achieved during the quarter; therefore it was appropriate to

leave that role. These tasks are summarized as follows:

Selection of Director of ENCA following international standard procedures. This was done

with the support of the whole Board

Approval of the evaluation matrix of management indicators for the high-level management

of the School

Legal and equity transfer of the Agricultural Training Schools from MAGA to ENCA

Undertaking of necessary legal analyses to achieve the ratification of the Organic Law of

ENCA and political lobby to ensure that the Ministry of Public Finance (MINIFIN) assigns

the school budgetary resources in accordance with local law.


Upon completing the SIINSAN diagnostic for the SESAN, MEP expects an official request

of the Secretary of Food Security and Nutrition for wider-reaching support on the basis of the

findings. This would be carried out with support from the MEP grants program. In general

terms, this support will consist of:

o Field verification of data quality and the presentation of a proposal for improved data

collection prior to use in decision making.



AID-520-C-13-00001


10

o Train staff carrying out the work to gather and process said information, not just those

who work on SESAN, but in all of the institutions that carry out monitoring and

evaluation of food security and nutrition activities

o Systematize and document work done in this regard, primarily by SESAN and in

general of the Sistema Nacional de Seguridad Alimentaria y Nutricional.

o Propose and develop a record of information within the SIINSAN which allows for

easy management and facilitates analysis at the different levels required.

o Facilitate the coordination between SESAN and INE, so that the SIINSAN becomes

one of the sources of information and production of statistics on food security and

nutrition for the OCSE SAN. In other words, OCSE is to be the official repository of

SIINSAN information.

The Board of Directors of INE is interested in supporting statistical strengthening of food

security actions through the Sector Coordinating Office created for this purpose. INE has

already approved a revised grant agreement for the Sector Coordinating Office. This process

will strengthen the joint work of MEP and INE, which guarantees the institutionalizing of

grant results.

Having the design of the MAGA’s Monitoring and Evaluation system has allowed the

Ministry to comply with the conditions of resource leverage from the European Union in

order to achieve the reinforcement of the family agriculture Program. Work has begun to

jointly define the baseline indicators for this program on a national level. A follow-up system

will be developed by MEP.

The achievements established for ENCA after MEP’s participation as MAGA’s

representative in the Board of Directors of ENCA are strategic and do not currently require

such close support from Program.

Key meetings:

SESAN, MEP, ASEIES, USAID with Germán González, Otto Velásquez, and Sergio

Hugo González, among others

Two meetings of the OCSE SAN with the participation of delegates from the 20

institutions represented in the OCSE

Various coordination meetings with MAGA’s counterpart technical team in Directorate of

Planning (DIPLAN)

Various meetings with the Office of the Ministry of MAGA and with the Board of ENCA


Develop a grant-funded project with SESAN to reinforce the SIINSAN

Begin the process of strengthening the OCSE SAN through the grants program, including

technical assistance and equipment

Conclude the Design and Development of Software to support the Monitoring and

Evaluation System for MAGA and leave the complete system in operation to the

Ministry’s satisfaction



AID-520-C-13-00001


11

Maintain USAID support to the Ministry of Agriculture in reinforcing the system of

agricultural and forestry education in the country, from outside of the Board of Directors

of ENCA

Result 3: Clearinghouse, Management Information System (MIS), Geographic Information System (GIS), Environmental Compliance and other M&E Services Developed and Delivered.

Under Result 3, MEP is designing and creating a Clearinghouse information system to support

M&E services and management of information on the status and progress of USAID/Guatemala

programs. This web-based information system is aimed to provide comprehensive information

on the status and progress of USAID activities (3.1); a Geographic Information System (3.2); a

mechanism for Mission-wide environmental compliance (3.3); and M&E services including

development or updates of Performance Management Plans and Data Quality Assessments (3.4).

This component is mainly implemented by MEP’s subcontractor, Geotecnológica.

Activities, achievement, and deliverables:

Monitoring and Evaluation (M&E) Module

During the quarter, MEP developed the conceptual design based on an extensive revision of

documentation on USAID’s monitoring and evaluation procedures, including: “USAID M&E

Policy”, “USAID PMP Toolkit”, “ADS 202”, and “ADS 203”, among others. The functionality,

architecture, and learning points from the M&E module of the MONITOR system in Colombia

were also evaluated and analyzed. Also included in the conceptual design were elements

identified after reviewing the available documentation on the AIDTracker Plus system, which

USAID is currently piloting as a computerized M&E platform to be implemented at a global

level in the future. The analysis of systems and determination of requirements for the M&E

module was completed from feedback received by USAID/Guatemala officials during various

working meetings to improve understanding of the equipment in the development of a

Clearinghouse about the functionality requirements for the M&E module.

Geographical Information System (GIS) Module

During this quarter, MEP worked on migrating the content of the GIS Clearinghouse module to

the ArcGIS OnLine platform recently acquired by the program. This platform migration has been

a technical challenge for the team, due to profound technological changes with the new platform,

which offers new functionality features for end users.

At USAID’s request, maps of the municipal coverage of the programs that form part of the

Western Highlands Integrated Program (WHIP) were generated. These maps will be

incorporated into the GIS Clearinghouse module and MEP plans to incorporate them into a

communications module that will be developed to widen the Clearinghouse’s functionality. Also

at the request of USAID, the team began the design of a sub-module of “Communication” for the

Clearinghouse. This sub-module will be connected with the GIS module in order to create



AID-520-C-13-00001


12

information pages on the projects driven by the mission and which will be made accessible to the

media and to other end users selected by the mission.

Initiated by MEP, and with information from the Rural Value Chains Project (RVCP)

implemented by AGEXPORT, a project coverage map was generated at beneficiary home level

using geo-referenced information produced by INCAP. This map has been added to the library of

interactive digital maps for the GIS Clearinghouse module. During the quarter, the GIS Module

also generated coverage maps of RVCP activities implemented by ANACAFÉ and AGEXPORT

as an input to the sample design and planning of field activities for the environmental audit of

these projects which is being carried out by The Cadmus Group and supervised by MEP.

Finally, work continued on the task of gathering and processing information on the geographical

coverage of the EGO programs. This information was gathered beginning with the baseline study

on beneficiaries’ homes carried out by MEP. This furthered MEP’s existing coordination with

the USAID GeoCenter in Washington on institutional guidelines to be followed in the GIS

module.

Environmental Compliance Module

During the quarter, MEP validated the Clearinghouse environmental compliance module. During

this period, the environmental compliance team completed the presentation and validation before

USAID’s Regional Environmental Adviser (REA) for the Central America and Mexico (ECAM)

region. Based on this activity, the REA issued a series of recommendations to improve the

automated system for environmental compliance. The team worked on incorporating these

recommendations and proceeded to present the tool to the REA again, who subsequently gave

their approval. The REA was also trained to use the system and approval was given to the first

Environmental Mitigation and Plan and Report (EMPR) of the Foundation for Ecodevelopment

and Conservation (FUNDAECO) water management project through the new online system.

Also with the aim of ensuring that the environmental compliance module was suitable for the

many activities implemented by USAID in the country, the compilation and development of the

impact and mitigation library of USAID-supported projects for the Clearinghouse environmental

module began.

At the request of the REA, MEP staff traveled to Ecuador to establish a working relationship of

coordination with the GEMS (Global Environmental Management Support) technical team

which is currently developing the future regional IT platform for environmental compliance,

MONITOR+. During this activity, the design of USAID Guatemala’s environmental compliance

module was presented, and all parties agreed that the new regional system that is being

developed will use the procedures and structure of the library of environmental mitigation

measures designed by MEP for USAID’s mission in Guatemala.

During this period, a training in-service was given on the use of the Clearinghouse

environmental module for the semi-automated generation of EMPRs for staff in three new

programs: “Sustainable water management in the Cuchumatanes”, implemented by

FUNDAECO; “MasFrijol”, implemented by Michigan State University; and “Taking an

innovative cooperative approach to food security and trade”, implements by NCBA (CLUSA).



AID-520-C-13-00001


13

As a part of this activity, implementing partners were given a general introduction both to

USAID’s environmental regulations and to the management of the Clearinghouse environmental

module.

As previously mentioned, final EMPR approval was given for the program implemented by

FUNDAECO during this period. This was achieved through the Clearinghouse. This support will

continue over the following months until approval has been given for all the EMPRs of new and

approved projects so that work can begin on the process of environmental compliance reporting

through the Clearinghouse.

Environmental compliance Audit for RVCP

During the quarter, MEP and its environmental compliance team supported the planning of the

environmental audit of the Rural Value Chains projects of AGEXPORT and ANACAFÉ. Terms

of reference were formulated for contracting a suitable consulting business responsible for

auditing, and assistance and input were provided in the sample design for the valuation. It also

participated in the evaluation of technical offers received, provided support to the company

selected in the revision of documentation, and assisted in the planning of field activities. Finally,

MEP participated in the field visits completed, providing logistical, technical, and contextual

support to the international auditors.

The audit was carried out during the months of May and June 2014. Working sites were selected

from the RVC Projects in the Departments of Quiché, Totonicapán, Quetzaltenango, San

Marcos, and Huehuetenango. The evidence-gathering phase of the environmental audit of the

Rural Value Chains projects was completed and the (draft) report has been presented for the

environmental audit of the RVC Projects. The work was carried out by the Cadmus Group (a

consulting firm), and the report is currently being revised by USAID Guatemala.


Having access to a version of the conceptual design of the M&E module that presents the

storyboard will allow MEP and USAID Guatemala to adjust the conceptual model and the

model design before it is fully developed.

Having a sub-module of “Communication” for the Clearinghouse within the GIS module will

surely have a practical use beyond what was originally planned. This sub-module will be

connected with the GIS module to create information pages on the projects driven by the

mission which will be made accessible to the media and to other users chosen by the mission.

Monitor G and its environmental compliance module has served as a conceptual and design

base for Monitor Plus. This is relevant to USAID Guatemala, as it guarantees that the

environmental compliance system developed by MEP will be highly compatible with the

future regional platform, Monitor Plus in functionality and structure.

One of the secondary objectives of conducting the environmental audit with a specialized

firm such as Cadmus Group was to leave local capacity in place for audits of this type. MEP

and its local environmental compliance team are now trained in the methodology and



AID-520-C-13-00001


14

implementation of this type of exercise, which will allow the mission to support itself in

these local partners in the future.

Key meetings:

Various meetings with the Clearinghouse technical team of MEP, the MEP COR, an expert

in M&E computer systems and DevTech Home Office’s Adriana Harvey, in order to discuss

and gather basic information for the design, functionality, and technical characteristics of the

M&E module.

Various meetings during the quarter with the Clearinghouse development team (MEP),

implementing partners (AGEXPORT, ANACAFÉ, RA), IARNA, and USAID officials in

order to: (a) request, gather, and clean geographical data on cover at the community or

populated area level of the EGO programs; (b) share information in the GIS Clearinghouse

module; (c) discuss the technical implications for the IPs for the generation geographical data

on the program and MEP support; and (d) make a preliminary analysis of requirements for

the sub-module of Communication per USAID requirements.

Validation meeting for the environmental compliance module by the REA and training in its

use (El Salvador, April 7-9).

Meetings at Sun Mountain regarding the design of the regional system for environmental

compliance for Latin America (Ecuador, April 28 – May 2).

Various induction and user training meetings for the environmental module (REA, Mission

Environmental Officer (MEO), CORs/AORs, and IPs of four projects) to drive the use of the

tool.

Various meetings of the MEP Environmental Compliance team, the MEP Chief of Party

(COP), USAID/Guatemala officials, staff from AGEXPORT and ANACAFÉ, and officials

from the consulting firm Cadmus Group, in order to plan the scope, logistics, and

methodology for the Environmental Audit and support the activities planned.


Present the storyboard for the first version of the M&E Module conceptual model to USAID

Guatemala for validation and feedback.

Incorporate USAID suggestions into the conceptual design of the M&E Clearinghouse

module.

Continue evaluating the M&E module of the MONITOR system in Colombia to adapt and

reuse components, design elements, and lessons learned.

With the assistance of DevTech Home Office, obtain more information about the

functionality of the AIDTracker Plus system, in order to improve and refine the design of the

M&E Clearinghouse module.

Continue gathering information on cover of the EGO Programs in order to improve the

existing versions of their maps in the GIS module.

Include new programs in the GIS module in accordance with USAID priorities.

Give assistance to new programs (CLUSA, MasFrijol) on procedures, best practices, and

technical specifications for geo-referencing their activities and beneficiaries.



AID-520-C-13-00001


15

Continue coordinating technical aspects with GeoCenter at USAID Washington and, if

possible, join into the pilot Web-GIS program currently being driven by the GeoCenter.

Increase availability of information on the GIS module, broadening its analytical capacities.

Improve the detail and quality of cartographic information on the coverage of USAID

Guatemala programs available in the GIS Clearinghouse module.

Result 4: Mission Programs and Project Evaluations Conducted and Results Reported.

MEP works with USAID/Guatemala to help meet the requirements of the new USAID

Evaluation Policy. The Program designs and conducts impact evaluations in order to measure the

change in key development indicators attributable to USAID interventions, undertakes Cost-

Benefit Analysis (CBA) and Cost-Effectiveness Analysis (CEA) to help USAID analyze what it

is buying with its resources and how funds should be allocated and, at USAID’s request, carry

out assessments that form part of EGO’s results-oriented program of planning and

implementation activities.

Activities, achievements, and deliverables:

CNCG Performance Evaluation

MEP and its evaluation team have designed and gathered practically all of the data for the

baseline of performance evaluation of the CNCG of Rainforest Alliance. Surveys, perception

studies and all other necessary actions have been carried out to respond to the five evaluation

questions set out at the start. The activities carried out in terms of conceptual and methodological

design to establish the baseline of the CNCG can be summarized as follows:

Proposal to apply the principles of change theory to the impact assessment of USAID-driven

actions to ensure the sustainability of biologically important areas of Guatemala.

Definition of the Fragmentation Index for Forest Ecosystems in order to determine to what

extent USAID-driven actions contribute to the sustainable management of forests and to the

reduction of deforestation.

Construction of the Index for Community Adaptation to Climate Change in order to

determine the baseline of vulnerability of the systems near CNCG beneficiaries.

Verification of actions driven by RA in order to prepare the diagnostics and reinforcement

plans for ten nongovernmental organizations so that at the end of CNCG they are capable of

receiving grants from USAID and other donors.

Design the perception study in order to determine to what extent gender has been taken into

account during the implementation of the CNCG.

Coffee Rust Early Warning System

The technical assistance that MEP provides to ANACAFÉ has completed its first phase. The

final pending item is training on the use of the field data collecting equipment. MEP completed

the procedure for the delivery of the grant. This concluded with the signing of a Memorandum of

Understanding between ANACAFÉ, RUTA, and MEP. USAID’s Mission Director served as a

witness to this event. This was performed in ANACAFÉ’s facilities and the event was an all-

around success. The Minister of Agriculture participated in the event, and the subject of Coffee



AID-520-C-13-00001


16

Rust gained the country’s attention due to the importance of this crop and the economic and

social risks represented by the disease, especially for small producers.

Maya Biosphere Reserve

Some changes were presented by USAID during the quarter in that products originally

contracted should have a focus that is more general and less focused on the prevailing political

actors, both in the USA and in Guatemala. The products still have the objective of telling the

story of the cooperation between USAID and the GOG in terms of the development and

conservation of the Maya Biosphere Reserve after 20 years of working together. The work was

broadened in the sense of having Spanish versions of both the Brochure and the Booklet. The

video format was defined as a set of short YouTube-type videos.

Rural Development Monitoring System

During the quarter, support was provided by the Technical Secretary of the Rural Development

Cabinet. The conceptual and methodological design was presented for the monitoring system of

sectoral policies that make up the Política Nacional de Desarrollo Rural Integral (PNDRI). Work

was carried out on the proposal for the indicators of component that makes up the system:

policy’s connection to intended results, closing gaps in key processes identified for the various

sectoral policies, the level of geographic integration of the directly related Government bodies

(MAGA and the Social Development Ministry, (MIDES) initially), and the effectiveness of the

management model. The technical team, made up of six officials from the Executive Secretary of

the Rural Development Cabinet, received training given by MEP in designing and developing

indicators. The follow-up system was analyzed as well as the possibility of a grant award that

would develop and strengthen the Secretary’s follow-up.

Environmental Audit for the RVCP (ANACAFÉ and AGEXPORT)

During the quarter, MEP completed the field work for the environmental audit of the Rural

Value Chains Project (RVCP) carried out by ANACAFÉ and AGEXPORT. After following

proper procurement procedure, Cadmus Group was contracted for this evaluation. MEP formed a

counterpart team that carried out general supervision of the work, particularly in the field phase

of data collection. The work concluded without any major obstacles, in spite of the serious

weather problems. At present, Cadmus has presented the final draft of the audit and it is being

revised by USAID. It is expected that this work will conclude as soon as USAID’s observations

are received. The process will end with a presentation of the results by Cadmus’ team leader.

Rapid Assessment for PRS

During the quarter, MEP started work on prospective-type Rapid Assessment for the Policy

Regulatory Support Program (PRS). The evaluation team was formed and presented the

implementation plan for the evaluation, which generally included the following results: specific

recommendations for the next phase of PRS, specific strategies that guarantee sustainability in

processes, and some general recommendations for the transition period between this program

and the next one, in particular considering that a new electoral year is beginning as well as a

change of government in the new few months. The results of the evaluation are expected within

the anticipated timeframe.



AID-520-C-13-00001


17

Opportunities, problems, delays and proposed solutions:

MEP has suitable analysis tools to give institutional context to the evaluations requested by

USAID from various programs. This is the case for MEP’s performance evaluation of

CNCG (implemented by Rainforest Alliance). This evaluation allows the evaluated program

to be viewed in the context of national policy, and more importantly, in the context of the

USAID’s CDCS.

The political backing received for the signed Memorandum of Understanding for the

execution of the Coffee Rust Monitoring System with ANACAFÉ guarantees its

institutionalization and sustainability and possible extension to other countries in Central

America.

There have been various delays in getting definitive guidance for the work developed by

MEP to formulate the USAID and GOG success story about the Mayan Biosphere Reserve

(MBR). These delays were finally overcome and materials will be distributed shortly that tell

the story of the cooperative work between USAID and GOG in developing and conserving

the MBR. This is a good opportunity to transfer the experience and materials to the CNCG

for utilization in programs that support the MBR.

Having an M&E methodology for multisectoral policies for Rural Development is a

challenge and an excellent opportunity not just for the GOG, but also for the Mission, in that

the country strategy is based to a large extent on multisectoral policies, as is the case with the

Feed the Future and Global Climate Change initiatives. It is anticipated that a very simple

monitoring system could be developed to support the Executive Secretary for the Rural

Development Cabinet. In this sense, it is important to reduce sophistication, taking into

account that the system operation starts at the local community level as identified at the start

of the PNDRI implementation strategy.

MEP has developed local capacity for carrying out environmental audit actions of USAID

programs in the country. It also has the capacity to follow up on EMPRs from other

Programs in the Mission. This is one of the MEP objectives, to use the expertise of foreign

firms to develop actions that can be managed at a high quality by local organizations.

The prospective nature of the Rapid Assessment for PRS introduces new lessons learned for

the methodological design of evaluations, which can be utilized by USAID’s different

partners, including government agencies.

Key Meetings:

Meetings of the MEP-IARNA team to finalize the schedule of activities for the development

of the baseline for CNCG and develop methodologies for each of the evaluation questions

that form part of its Implementation Plan.

Various meetings with the sub-management of ANACAFÉ and the Director of its Coffee

Research Center (CEDICAFÉ), to coordinate the technical and administrative details of the

grant that MEP will provide to ANACAFÉ in support of the Coffee Rust M&E system.

The MEP technical team, consultants, and officials involved in environmental issues met to

define the specifications with the new guidance that consultants should integrate important



AID-520-C-13-00001


18

feedback while producing materials that communicate the success of the USAID-GOG

cooperation in developing and conserving the MBR.

Various meetings of the MEP technical team with officials from the Technical Sub-Secretary

of the Rural Development Cabinet and the Executive Secretary of the Presidency, in order to

discuss advances in the conceptualization, design, and proposal of indicators that form part of

the monitoring and evaluation system for multisectoral policies.

Various meetings between the MEP technical team, the Rapid Assessment of PRS team

leader, officials in charge of the project on USAID’s behalf, and PRS officials, in order to

communicate the general direction of the evaluation and share experiences and results

generated by the project.


Presentation of baseline Report that forms part of the first phase of the performance

evaluation MEP is making of RA’s CNCG Program.

Finalize an approved grant agreement with CEDICAFÉ of ANACAFÉ to strengthen their

M&E system for Coffee Rust.

Finish the three products that communicate the success story of the USAID-GOG

cooperation in developing and conserving the MBR.

Completion of the consultancy supporting the Executive Secretary of the Rural Development

Cabinet, which conceptualizes, designs, and puts into operation the M&E system of

multisectoral policies for Rural Development.

Presentation of the report on the environmental audit of the Rural Value Chains Project of

ANACAFÉ and AGEXPORT.

Completion of the Rapid Assessment of the PRS and presentation of the report with

conclusions and recommendations to USAID

Result 5. Monitoring and Evaluation Services for other USAID/Guatemala Offices, Conducted Upon Request

Through Result 5, USAID Mission offices have the opportunity to buy into Program M&E

services such as baseline, midterm and final performance and impact evaluations, engage in geo-

mapping of health/nutrition and governance projects, engage in statistical capacity building of

GOG institutions and other key stakeholders in health/nutrition and governance, and/or provide

other M&E services.

Activities, achievement, and deliverables:

Through IARNA, MEP concluded the analysis of the Citizen Security Index and prepared a final

report of results. The index was constructed by analyzing data from community surveys and

followed a practical method of factor selection that did not include basic theoretical assumptions;

the first attempt which was based on theoretical premises failed. A presentation was made to

USAID’s EGO team. The presentation complemented the Community Resilience Index report

that was already concluded and presented to USAID in the previous quarter.



AID-520-C-13-00001


19

With the news of a future evaluation of the Alianzas program for education and health, MEP has

begun contacting specialists in this area for potential consultancies. MEP has also been in

communication with ASIES, who have local specialty on these subjects, so that they can form a

local team and plan the logistics that the evaluation efforts will require.


The problems encountered in the preparation of the Citizen Security Index, following the

theoretical method, allowed MEP to explore other analysis techniques. A new proposal for the

construction of the Index will be submitted shortly to the DGO. It will also be shared with the

scientific community for validation purposes.

Key meetings:

The MEP technical team met with USAID officials on at least two occasions. These meetings

were held primarily with management staff from the Democracy and Governance Office (DGO)

in order to know the results of the study.

Activities planned for the following quarter:

Presentation of the final report for both indexes: Community Resilience and Citizen Security.

Analysis and development of the proposal to carry out an evaluation of the Alianzas program

for education and health.

Gender Considerations

MEP has included a gender focus in all its activities; this focus is particularly important in the

development of statistical tools. MEP worked to maintain gender disaggregation within all the

statistical tools it developed so that it will be possible to detect and analyze gender differences

from the data collected. Gender has also been an area of discussion as MEP begins to evaluate

the CNCG program. Within the Evaluation Implementation Plan of MEP, a general gender focus

has been included as well as a specific gender focus in one of the evaluation questions which

refers directly to gender issues.

MEP has also been consulting with USAID and EGO implementing partners on areas where

specific technical support is needed; one area identified was the need to address gender gaps

encountered in a recent USAID Agriculture and Environment study. USAID has asked MEP to

propose realistic and coherent solutions that implementing partners can address within their

current program structure. MEP may also, in the future, provide support to implementing

partners to address gender issues that were identified through the Women’s Empowerment in

Agriculture Index (WEAI).



AID-520-C-13-00001


20

Coordination, Collaboration and Information Sharing

MEP closely coordinated with all relevant partners across its activities. Collaboration is a key

step to ensuring that M&E tools that are developed are relevant to the needs of USAID,

implementing partners and Government of Guatemala (GOG) counterparts.

Under Result 1, MEP held multiple meetings with USAID/EGO implementing partners (IP) in

order to gain a full understanding of the Conceptual Framework of the monitoring and evaluation

system developed by MEP and which is derived from USAID Guatemala’s CDCS as well as its

indicators and results framework. MEP also held a range of meetings with IPs to discuss the need

of having a beneficiary directory for the RVC Programs implemented by ANACAFE and

AGEXPORT. MEP has also help multiple meetings with RA in order to coordinate and share

information about the baseline for the performance evaluation that MEP is carrying out under

result 4.

Under Result 2, MEP met with INE, SESAN and MAGA to discuss USAID/MEP technical

support related to strengthening M&E systems that the Program is planning to implement and to

inform them that these correspond to the specific needs of GOG agencies.

Under Result 3, MEP continues to meet with USAID officers from the three technical offices

(Health and Education Office - HEO, DGO, and EGO), as well as the Mission and Regional

Environmental Officers, to discuss requirements and gain feedback on key design issues related

to the Management Information System (MIS) Clearinghouse. Multiple meetings have been

carried out among implementing partners to share findings on the Data Quality Assessments

(DQA) that MEP carried out, with 67 indicators from the agricultural and environmental

programs.

Under result 4, in addition to what was already mentioned, multiple meetings have taken place to

share information on results from work on MBR communication materials and Rapid

Assessment of the PRS Program. Other meetings with GOG addressed progress made in terms of

follow up and evaluation of sectorial policies that support rural development.



AID-520-C-13-00001


21

III. SUMMARY AND CONCLUSION

During this quarter the MEP achieved significant results across the Program components. These

represent important contributions to the Mission’s M&E actions, particularly those of the

Economic Growth Office. This has been possible through combination of specialized technical

expertise from Guatemalan subcontractors together with DevTech’s leadership and management.

Moreover, the program is very well-positioned with implementing partners and counterpart GOG

agencies. Part of the positive reception from the IPs and the GOG is due to the fact that

USAID/Guatemala has provided on-going support in-country.

In accordance with Program design, concrete results should be completed in all the components

towards the end of the second year of work. This is especially important in Clearinghouse

services that will become a support tool for not only USAID and its implementing partners, but

also the GOG. Given the linkages and requirements to understand user needs, the next phase of

the Clearinghouse will require inputs from the local team, as well as the continued strong support

of DevTech’s Home Office and the unconditional support that MEP has received from USAID.



AID-520-C-13-00001


22




AID-520-C-13-00001


23

ANNEX A. SUCCESS STORY



AID-520-C-13-00001


24




AID-520-C-13-00001


25



AID-520-C-13-00001


26

For more information, contact:

DevTech Systems, Inc. 1700 North Moore St.

Suite 1720 Arlington, Virginia 22209

703-312-6038 www.devtechsys.com

quarterly performance report - united states agency for ...pdf.usaid.gov/pdf_docs/pa00n399.pdf ·...

Documents