-
8/3/2019 Joint Humanitarian Evaluations:Opportunities and Challenges
1/15
Joint Humanitarian Evaluations:
Opportunities and Challenges
Scott Green
United Nations Office for the Co-ordination of Humanitarian Affairs (OCHA)
-
8/3/2019 Joint Humanitarian Evaluations:Opportunities and Challenges
2/15
Drivers for Joint Humanitarian
Evaluations ALNAP research finds joint evaluations to be of higher quality than singleagency.Delivering As One Report:
Stresses importance of closer partnerships between the United Nations andNGOsIdentifies need for more accountability to both affected people and donorsEvaluation defined as a major driver for building system-wide coherence.Panel calls for humanitarian system getting more adept at using jointevaluations; or regular independent assessments of the performance of thehumanitarian system in responding to humanitarian emergencies
Humanitarian system becoming more adept at commissioning and using joint evaluations; OCHA playing a key role in moving the process forward.
-
8/3/2019 Joint Humanitarian Evaluations:Opportunities and Challenges
3/15
IASC Real-Time Evaluations
Approach initially piloted (2007 2009) in Pakistan, Mozambique andMyanmar.
2009 IASC Review endorses approach for scale up and regularimplementation across the system
Approach now routinely implemented across all major humanitarian crisesthrough use of an automatic trigger mechanism: applies in cases whereaffected population exceeds 1 million and appeal exceeds US$50 million.
OCHA serves as lead managing agent; IASC RTEs are jointly funded bymember agencies.
-
8/3/2019 Joint Humanitarian Evaluations:Opportunities and Challenges
4/15
Key Characteristics
IASC RTEs are rapid assessments which make use of participatorymethodologies; identify operational bottlenecks around co-ordination issuesand seek out the views of affected people.
Management StructureGlobal-level policy and Steering Group
Ad Hoc Management CommitteesIn Country Advisory Groups
-
8/3/2019 Joint Humanitarian Evaluations:Opportunities and Challenges
5/15
IASC RTEs: Key Characteristics
Teams deployed no later than 3 months after disaster
Content based on an agreed global IASC RTE assessment framework and
processes are implemented based on an agreed set of standard operatingprocedures
Real-time feedback used to support remedial action planning for immediateimplementation; action plans developed with assigned roles andaccountabilities for follow up.
Major IASC RTEs: Haiti (2010 & 2011) Pakistan (2010 & 2011) Kenya(2011)
-
8/3/2019 Joint Humanitarian Evaluations:Opportunities and Challenges
6/15
Important lessons from IASC RTE
experienceCreate the necessary political space for joint evaluation work: it isimportant to have clear sense of the purpose and use for joint evaluationwork and to make it a routine part of the culture
Have an agreed assessment framework in place. There needs to be astrong concept of what is being used as a baseline for the assessment.
Maintain a narrow scope to be most useful. Focus on issues of broad crosscutting concern rather than in depth assessments by sectors
Ensure good communication throughout the process.
-
8/3/2019 Joint Humanitarian Evaluations:Opportunities and Challenges
7/15
Ongoing Challenges
Ensuring RTE results get used at both strategic and operationallevels
Ensuring real-time deployments: significant demand for IASC RTEsto be deployed earlier for greater impact.
How to assess joint humanitarian impact: system still unable todeliver multiple-agency impact assessments. What value-added if any?
-
8/3/2019 Joint Humanitarian Evaluations:Opportunities and Challenges
8/15
JHIE ConsultationsTo assess the potential for future inter-agency impact evaluations,
OCHA led a series of consultations:Objective was to define feasible approaches to undertaking jointimpact assessments with possibility of having some pilot evaluations(group interviews held around issues of scope, focus, purpose, useand methodologies)Consultations were held with:The affected population in 15 communities in Sudan, Bangladeshand Haiti (including women, men, children, disabled, ethnic groups),mainly in focus groupsLocal government and local NGOs in the same countriesNational governments and international humanitarian actors in Haitiand Bangladesh67 international humanitarian actors, donors and evaluators in sevenmeetings in New York, Rome, Geneva, London and Washington
-
8/3/2019 Joint Humanitarian Evaluations:Opportunities and Challenges
9/15
MethodologySystematic attempt to consultwith national governments andwith disaster-affected peopleduring the design phase of amajor evaluative exercise
A new approach to evaluationdesign is warranted because of the scale of and resourcesrequired for JHIE, but also tomodel a participatory approachto designing impact evaluations
Evidence that it is necessaryand useful to consult with theaffected population duringevaluation design.
-
8/3/2019 Joint Humanitarian Evaluations:Opportunities and Challenges
10/15
Results and Implications: Possibility
of Pilot JHIEs95 percent of respondents were supportive and 75 percent were stronglysupportive.Differences in what JHIEs should look like not about their relevance
Main advantages seem to be the potential to offer a more comprehensivepicture of impact and look at areas that cannot be covered by single agencyevaluations
ImplicationsThere is a strong basis to proceed with some pilot JHIEs in the future
-
8/3/2019 Joint Humanitarian Evaluations:Opportunities and Challenges
11/15
Results & Implications: Questions
of Purpose, Focus and UseStrongest support was for generalizable knowledge and accountability toaffected populationsMain focus should be on changes in the quality of life of affectedpopulationsFocus may need to differ between sudden onset natural disasters andcomplex emergenciesConcern expressed over limited use of current system-wide humanitarianevaluations at country level (e.g. IA RTEs and other system-wideevaluations such as TEC)
Strong support for national governments to become primary users
-
8/3/2019 Joint Humanitarian Evaluations:Opportunities and Challenges
12/15
Results & Implications: Questions
of Purpose, Focus and UseImplications
Focus on making JHIEs useful first at the national level by buildingknowledge which is useful for national and local governments innatural disaster situations; get better at engaging with
governments on evaluation issuesPilot JHIEs should not be used in support of lesson learning for on-going programme modification or for meeting upward accountabilityneeds;Consider a pilot JHIE with primary purpose of supportingaccountability to affected populations
Pilots should avoid mixing purposes as most evaluations currentlytry to do: one option would be to have different JHIE pilots bedesigned around different purposes.
-
8/3/2019 Joint Humanitarian Evaluations:Opportunities and Challenges
13/15
Results & Implications: Meaning of
impact in humanitarian contextLittle interest in developing acommon definition of humanitarianimpactBoth short and longer-term changes
are relevantCommon indicators would be usefulbased on classification of phases:relief, transition and recovery
Implication:JHIE steering committees will need todecide on a case-by-case basis whathumanitarian impact means. Thereshould be an effort to focus on bothshort and longer-term changes, using
common indicators.
-
8/3/2019 Joint Humanitarian Evaluations:Opportunities and Challenges
14/15
Results & Implications: Questions
of evaluation methodologyStrong emphasis on the need for consultations with communities ahead of time; do not always assume communities will want to be involvedEvaluation design should build in learning for affected populations. Strongemphasis on follow up and feedback mechanisms at the community levelNo support for experimental design and only minority opinion supporteduse of formal quantitative population surveysStrong majority supported use of qualitativemethodologies. Participatory impact assessmenttools likely to be preferable.
ImplicationsCurrent fly -in fly-out model of Humanitarian
Evaluation not suitableLonger time frames needed for research andmultiple visits to communitiesGoal free evaluation might offer a better modelthan theory-based combined with QED
-
8/3/2019 Joint Humanitarian Evaluations:Opportunities and Challenges
15/15
Results and Implications: linkagesto broader M&E efforts andmanagement arrangements
Consultations emphasized need fordevelopment of a common set of indicatorswhich could be tracked from baselineConsultations supported a two-tiermanagement structure with a steeringcommittee made up of key stakeholders and
a management groupCommunities expressed doubts in Haiti thatagencies could effectively work together toassess joint impact
Implications:JHIE should build a small set of quantitativeand qualitative indicators in key interventionareas that can be tracked throughout theJHIE processDevelop JHIEs as part of a broader effort toenhance M&E efforts and a commonprogramme cycle
PreliminaryScenarioDefinition Flash
Appeal
Revision ofFlash Appeal
Coordinatedsectoralassessments
Contingency Planning Preparedness
Multi-clusterRapid Assessment
RTE / Monitoring
Recovery Assessment
ImpactMonitoring &Learning
HumanitarianDashboardUpdates
Performance monitoring &RTEs
Evaluation
Cluster-level monitoring andreporting