wicked problems peacebuilding evaluation ethics
TRANSCRIPT
WICKEDPROBLEMS:PEACEBUILDINGEVALUATIONETHICSDeterminingWhatisGoodandRight
DevelopedbyReinaC.Neufeldt,PhD
forthePeacebuildingEvaluationConsortium
August2016
Thisreportwasmadepossibleby
theCarnegieCorporationofNewYork
AboutthePeacebuildingEvaluationConsortiumThePeacebuildingEvaluationConsortium(PEC)isaprojectofAllianceforPeacebuilding(AfP)inpartnershipwithCDACollaborativeLearningProjects,MercyCorpsandSearchforCommonGround(SFCG).TheprojectisfundedbytheCarnegieCorporationofNewYork(CCNY)andisfield-wideefforttoaddresstheuniquechallengestomeasuringandlearningfrompeacebuildingprograms.ThePECconvenesdonors,scholars,policymakers,localandinternationalpractitioners,andevaluationexpertsinanunprecedentedopendialogue,exchange,andjointlearning.Itseekstoaddresstherootcausesofweakevaluationpracticesanddisincentivesforbetterlearningbyfosteringfield-widechange through threestrategicand reinforcing initiatives:1)DevelopingMethodologicalRigor;2) Improving theCultureofEvaluationandSharedLearning;and3)FosteringtheUseofEvidencetoInformPeacebuildingPolicy.
AbouttheAuthorDr.ReinaNeufeldtisanAssistantProfessorinthePeaceandConflictStudiesProgramatConradGrebelUniversityCollege, University of Waterloo. She has engaged in program design, monitoring, evaluation and learning inpeacebuildingandconflictresolutionforoverfifteenyears.Herpublicationsinmonitoringandevaluationinclude‘“Frameworkers”and“Circlers”–ExploringAssumptionsinImpactAssessment’(AdvancingConflictTransformation:TheBerghofHandbookII,2011),“Interfaithdialogue:assessingtheoriesofchange”(PeaceandChange,36,2011),aswellastheco-authoredReflectivePeacebuilding:APlanning,MonitoringandLearningToolkit(CRSandtheKrocInstitute, 2007) and “Building blocks for peacebuilding impact evaluation” (Journal of Peacebuilding andDevelopment,2,2005). HerrecentbookEthicsforPeacebuilders:APracticalGuide (Rowman&Littlefield,2016)providesamorecarefulexplorationofethicsandpeacebuildingforpractitioners.
ReinahasanMAinSocialPsychology(YorkUniversity)andaPhDinInternationalRelations(AmericanUniversity);she is trained in both qualitative and quantitative researchmethods. Reina’s current research focuses on fieldlearningandexplorestheroleofreflectivepractice in improvedpeacebuilding.She isalsoamemberofthePECAdvisoryBoard.
Author’s Note: Thank you to Diana Chigas and Colleen Duggan for insightful comments, careful reading andthoughtful engagement in our discussions of ethics in peacebuilding evaluation; thanks also to the larger PECadvisoryboardforactivelywrestlingwithquestionsofwhatconstitutesgoodinpeacebuildinginanongoingway.Finally,thankstoBrianAdiengeandSarahMcLaughlinattheAllianceforPeacebuildingfortheireditingworkandtoJazmineWalovitchatCDACollaborativeLearningProjectsforthefinallayoutofthedocument.
TableofContents1.Introduction.............................................................................................................................................12.Values,EthicsandEvaluation...................................................................................................................2
3.TheWhat:Determining“Good”and“Right”Peacebuilding....................................................................2
3.1GoodEnds:Consequences.................................................................................................................3
3.2RightMeans.......................................................................................................................................4
3.3VirtuesandGoodCharacter...............................................................................................................5
3.4Relationality.......................................................................................................................................6
3.5SkillstoWorkOn:IdentifyingMoralValuesinEvaluation.................................................................7
4.TheWho:WhoDecidesWhatisGoodandRight?...................................................................................7
5.TheHow:HowDoWeEnsurePeacebuildingEvaluationsAreDoneEthically?.......................................9
5.1HowFindingsAreGenerated.............................................................................................................9
5.2HowanEvaluatorConductsHer-orHimself?..................................................................................10
5.3HowSensitivetheEvaluationProcessistotheConflictEnvironment.............................................10
6.Conclusion:CreatingSpaceforMoralValues........................................................................................11
7.AdditionalResources..............................................................................................................................12
8.APPENDIX:EthicsQuestionsduringanEvaluationCycle.......................................................................15
8.1.QuestionsontheWhat(What’sGoodandRight)..........................................................................15
8.2QuestionsRegardingtheWho(WhoDecides).................................................................................16
8.3QuestionsofHowtoConductanEvaluationEthically.....................................................................17
1
1.Introduction
Anexchangelikethisiscommoninorganizationsandcanbefrustrating.Theprojectmanagerfeelslikeheorsheisbeingaskedfordatathatwillnothelpimproveimplementationinthecommunity,althoughitmightbeimportantformanagementandthereforenecessary.Theadministratoristryingtogatherdatatohelpjustifyorganizationaldecisions,andlikelyfeelingpressurefromboardmembersandfunderstodemonstrateimpact.
Thetensioninthedialogueinvolvesnotonlycompetinginstitutionaldemandsbutalsocompetingmoralclaims–that is, there are competing claims about what is good and right in peacebuilding as well as peacebuildingevaluation.Themoralclaims,however,arecloakedinlanguagethatmakesthemappearasiftheyarevalue-freedecisions,whichmakesthemhardtodetect.Acommonvalue-freeassertiongoeslikethis:ifwecollectthedatathenwewillknowwhattodo.Thisideaofbeingabletocollectdatathatwillsimplytelluswhattodoispopular,and is rooted ina schoolof thought thatespousesvalue-free social scienceas theway toanswermanagementquestions.
However,whenweaskwhatconstitutesgoodresultsoraskquestionsaboutwhatwasdeterminedasimportantinordertogatherthedata,wecatchaglimpseofthemoralvaluesuponwhichclaimsarebeingmade.Forexample,inthedialogueabove,weseemoralclaimsthatevaluationsaregoodandworthwhileif:
§ theygeneratedataonimpactforscaling-up,§ thepeopleinvolvedintheinitiativelearnfrom,andmakeadaptationsbasedontheevaluationdataand
process,§ theyhelptheprojectofficermakefield-leveldecisions,§ theyhelpadministratorsmakedecisionsaboutallocatingfuturefunding,or§ theyjustifytheprogramtoexternalfunders.
Thesevariousclaimsoftenexistside-by-sideandcancomplement,orbeintensionwitheachotherwhenchoosingevaluationapproachesandmethodologies.
Alsonestledwithinthisexchangearemoralclaimsaboutwhatwillconstitutegoodendswithinthepeacebuildinginitiative. That is,whattheprogramshouldbeevaluatedupon inordertodetermine itsworthormerit. Theseclaimsaresignaledbywordslikeimpact,effective,scaling-up,changethingsforthebetter,responsive,andbythecharacteristicsofbeingcourageous,dedicatedorpassionate.
“We want to know that our indicators are collecting data that will determine our impact,” the senioradministratorsays.“Wereallywanttoknowifourpeacebuildingmodelwaseffectivesowecanscale-up.”
“Right,”saystheprojectofficer.“And,ourinitiativeisreallyaboutmakingadifferenceinpeople’slivesinthe
community,sowedon’twanttojustshowimpact,wewanttohaveimpactandunderstandhowthathappened
sowecontinuetochangethingsforthebetter.Theyouthandwomen’sgroupleadersweareworkingwithare
terrific – they are really courageous, passionate and dedicated to building peace in their communities. The
situationisvolatilebutwearerespondingasbestwecan.”
“Great! Wewillalsoneedtousethat impact informationtoallocateourtimeandfundinginthisprogram
areainthefuture,anddevelopanewproposalforwhentheprojectwrapsup,”respondsthesenioradministrator.
“Okay,”repliestheprojectofficer,worryingaboutwhetherornotthereisenoughoccurringintheparticipatoryassessment thatwill constitute “impact” in theadministrator’s eyes –or inanexternal funder’s eyes –andwhetherornottheircurrentideasabouthowtoutilizeevaluationdatawithintheprojectwillbeunderminedbythesenioradministrator’sdemands.
2
Thepurposeofthisbriefingpaperistosupportevaluatorsinproducinggoodevaluationsbyhelpingtothinkaboutvalues and ethics consciously and carefully. To do this, we’ll first look at the relationship between ethics andevaluation.Wewillthenlookatthebigpictureofwhatconstitutes“goodpeacebuilding”orpeacebuilding“doneright.”Wewillalsoexploreethical issues involved inwhodetermines thevaluesandcriteriabywhichwe judgeprograms.And,finallywewillexploreethicalissuesinvolvedinhowevaluationsareconducted–anareaforwhichtheretendstobemoredevelopedguidanceavailable.Thepaperconcludeswithadditionalresourcematerialforfurtherexploration.
2.Values,EthicsandEvaluation
Evaluationinvolvesmakingvaluejudgementsaboutwhatisgoodandright.Wecollectempiricaldata,analyzeitandtheninterpretitinordertomakejudgementsabout“theworth,meritorvalueofsomething.”1AsDeborahFournier notes in her definition of evaluation for the Encyclopedia of Evaluation, “It is the value feature thatdistinguishesevaluationfromothertypesofinquiry,suchasbasicscienceresearch,clinicalepidemiology…”andsoforth.2Thevaluefeatureofevaluationmakesit,inpart,ethicaldeliberation–weighingandchoosingamongstwhatwillbeconsideredgoodandright,orworthyandofvalue,inagivenpeacebuildinginitiative.
Whenwethinkaboutevaluationtraining,however,ourskilldevelopmenttendstofocusontechniques,methodsandthebusinessaspectsofevaluation.3 Forexample,writingornegotiatingthetermsofreference,developingfocusedlinesofinquiry,gatheringdataandanalyzingitusingparticularmethodologies.Theseareimportantand,aswill be noted below, also part of an ethical evaluator’s practice. However, the skills for thinking about andweighingtheethicaldimensionsofevaluation–suchashowtoidentifyandweighthevalue-basedelementsofanevaluation–arefrequentlymissingevenwhenrecognizedasimportant.4
Our lack of attention to ethics often feeds into problems in evaluations. Peacebuilding initiatives occur inenvironmentswithconflictedparties,divergentinterestsandneeds,whichpeacebuildinginitiativesareattemptingtochange.Evaluationsinvolveknowledgegenerationandaffectresourcedistributionamongststakeholdersandsostakeholdersareveryconcernedabouthow“worth”isdeterminedandhowitwillreflectonthem.5Evaluationsarecommissionedbypartieswhowanttofundworkthatmakesadifferencefromtheirperspective(e.g.greatestimpact).Thosebeingevaluatedhaveastakeintheevaluationandtheconflictcontext–implementerswanttodogoodworkandgetfundstocontinue,whilecommunitymemberswantqualityprogramsthatsupporttheirvisionfor their community. When there aredivergent values, they can contribute to conflict and thepoliticizationofevaluations–particularlyifthosevaluedifferencescoincidewithconflictcleavages.We,asevaluators,maythenunintentionallycontributetoconflictandunderminethepeacebuilding initiativeweareevaluating ifwearenotattendingtomoralvaluesaspartofconflictandevaluation.Butwhataredifferentwaysofunderstandingwhatisgoodandright?
3.TheWhat:Determining“Good”and“Right”Peacebuilding 1Mathison,"ValueJudgment,”444.2ibid.,140-1.Thosewhoarguethatempiricaldatacanbegatheredinordertoinformrationaldecisionsinavalue-freewayarerootedincontemporarysocialsciencethinkingfromMaxWeberthroughtoAmericanPsychologistDonaldT.Campbellandtotoday’s“evidence-based”managementsystems;foranoverviewseeQuinnPatton,UtilizationFocusedEvaluation.3Ethics,whentaught,tendstoberelativelyshortoranadd-on.Forexample,theAmericanEvaluationAssociationhasdevelopeda1.5-2hourtrainingonitsethicsguidelines,whichisavailableathttp://www.eval.org/p/cm/ld/fid=105.Thereareexceptions,suchasBushandDuggan,“AddressingEthicalandPoliticalChallengesinEvaluation.”4Forexample,theAfricanEvaluationAssociation’s“AfricanEvaluationGuidelines-StandardsandNorms”documentnotestheimportanceofvaluesidentificationinasub-pointunderthediscussionoftheprincipleofutility.SeeAfricanEvaluationAssociation,“AfricanEvaluationGuidelines-StandardsandNorms,”5.5MichaelQuinnPattonhasacleardiscussionofthepoliticsofevaluationandhowtomanagethepoliticsofevaluationinChapter14ofUtilizationFocusedEvaluation.
3
Wetendtoassumeweorthosedesigningthetermsofreferenceknowautomaticallywhatisgoodandright,andthatwe—evaluators,evaluands,andotherstakeholders—allagreeuponit.However,thisisoftennotthecase.Isuspectoneofthereasonsthatpeopledon’tlikeevaluationsisbecausetheyfeeltheevaluationsarejudgingthemby criteria they don’t think are appropriate and are not “good” or “right”— in other words, they feel like theevaluatordoesnotunderstandorknowtheirrespectiveneedsandvalues.Howcanwedobetter?
Evaluation involvesaskingbigquestions:“Wasthis initiativetherightthingtodo?Didgoodpeacebuildingoccurhere?Whatchanged?Whatistheevidencethatgoodpeacebuildingoccurred?”Thefirsttwoquestionsinthissetdemandethicalthinking.
Whenwe think ethically, wemake judgements aboutmoral claims; it is the process of reasoning through andassessingwhatmakesforgoodendsandrightmeansgiventhelegitimateneedsandexpectationsofourselvesandothers.6Thisischallengingwork.
Thereis,however,alongtraditionofmoralphilosophythatpresentsmoraltheoriesthatcanhelpevaluatorsexplorearangeofvaluesthatcanbeusedtodeterminewhatisgoodandrightinpeacebuilding,andfromwhichcriteriaandstandardstojudgeempiricaldatacanbedrawn.Inthesectionbelow,we’llbrieflylookatbigideasfromfourmoraltheory families to help evaluators identify different moral values that might be prioritized by stakeholders inevaluations:goodends,rightmeans,embodyingexcellenceaspeacebuilders(andevaluators)andenactingcaringrelationships(aspeacebuildersandevaluators).Thislistisnotacompletelistofallpossiblevaluesofgood,butastartingpointforimportantconsiderations.
3.1GoodEnds:ConsequencesPeacebuildinginitiativesusuallypursuepeaceasagoodend–thismayincludestoppingovertviolence,addressingstructuralviolence,andbuildingpeacewithjustice.Theseare“bigends”orgoalsthatreflectthemoralvaluesthatguidewhywedopeacebuildingwork.
Feeding into the big ends are more specific moral values about ends and desired consequences that framepeacebuildingevaluations.Thesemoralvaluesappearinthewaythetermsofreferencepresentthepurposeoftheevaluationaswellasitsobjectivesandthelinesofinquirythatidentifyhowaninitiativeshouldbejudged.Termsof referenceusuallyemphasize thecriteriaandstandards thatalignwith thevaluesof thegroup funding theevaluation.
Forexample,termsofreferenceforpeacebuildingevaluationsoftendrawontheOECD-DACcriteria,whichdefinesgoodpeacebuildingintermsofits:
1. relevancetodrivingfactorsofconflict(asdeterminedbyaconflictanalysis),2. effectiveness vis-à-vis intended objectives and the degree to which results were achieved (outputs,
outcomesandtheconnectiontoimpacts),3. impact intermsof intendedandunintendedpositiveornegativeeffectsonthewiderenvironmentand
conflictcontext,4. sustainability or the “continuation of benefits” when funding is discontinued (can include things like
communityownershipandresilienceinsettingsofconflict),and5. efficiencyintermsofhoweconomicallytheresourcesthatareputintoaninitiativeproduceresults.7
TwoadditionalOECD-DACcriteriathatarementioned,butnotonequalfootingastheprecedingfive,arecoherence,andcoordinationamongstintervenorsorfunderstoproducemoreeffectiveresults.
6Weston,A21stCenturyEthicalToolbox,86.ForacarefulexplorationofvaluesinevaluationseeSchwandt,"TheLandscapeofValuesinEvaluation:ChartedTerrainandUnexploredTerritory."7OECD,"EvaluatingPeacebuildingActivitiesinSettingsofConflictandFragility:ImprovingLearningforResults,”65-71.
4
Thereisastrong,unifyingmoralclaimbehindthesecriteria:peacebuildingprogramsorprojectsaregoodiftheyachievethegreatestgoodforthemostnumberofpeopleorproduceatleastasgreatagoodasanyothercourseofactionwouldproduce. Ourbestaction is therefore theone thatachieves thebestend. Thismoral claim isparticularlyimportantforthecriteriaofimpactandtheeffectsonthewiderenvironmentandcontext,butalsokeytothedefinitionsofrelevance,effectiveness,efficiencyandsustainability.
Intheexchangebetweenthemanagerandtheadministrator(above),theadministratorisemphasizingthattherightthingto do is the one that produces the greatest impact – thegreatestgoodforthemostnumber,soitcanbe“scaled-up”and produce even greater good for more people. This isends-based consequentialist moral thinking, and is intensionwith the program staffer’s thinking about processandpeople.Theefforttoscale-upcanproducethe“greatestgood”bybenefittingmorepeople,yetalsocanunderminethequalityandnatureofthechangeprocessforthoseintheinitialproject(andstillproducethe“greatestgood”becausemore people are involved who receive some benefit). Inconsequentialistthinkingsomeharmsalongtheway–likeover-stretchedstaffor fieldpersonnelandpartners–areconsideredacceptableaslongastheoutcomeisatleastasgoodasanyothercourseofaction.
The emphasis on ends is strongly represented inpeacebuilding evaluation criteria such as the OECD-DACcriteria.Whenweaskquestionsaboutendsthenweweighthe harms and the benefits and look to determine if thebenefits togetheroutweigh theharmsvis-à-visourcriteria(e.g. was our impact overall positive? Was our initiative,overall,relevant?).
There are, however, other ways to think about whatconstitutes good peacebuilding (e.g. principles, character,relationships),whicharesometimesintensionwithends-basedthinking.And,asnotedabove,thesedifferencescanalignwithstakeholdergroupsandcontributetomisunderstandingandconflictwithinanevaluation.
3.2RightMeansWhileends-basedthinkingmaydominateevaluationdiscourse,itisneverthelesscommonforpeacebuilderstothinkaboutthewayinwhichwedoourworkasimportant,particularlyimportantintermsofmodellingwhatwehopetoachieve.Pursuing“peacebypeacefulmeans”isthereforeacommonphraseinthefield–aswellasthetitleofabook.8
Ourmeansandtheprinciples thatguideouractionscanbeseenasgood inandof themselves. Forexample,ImmanuelKantarguedweshouldalwaystreatpeopleasendsandneverasameanstoanend.Anyactionfollowingthisprinciplewasgoodinandof itselfregardlessofwhathappenedasaresult.Peoplemightrecognizethisasaformulationof“thegoldenrule”ofreciprocitythatappearsinmanyreligioustraditions.9Respecteveryone’shumanrightsisanoutgrowthofthisimperativeofKant’s.Othermeansthatareemphasizedinpeacebuildingevaluation 8Galtung,PeacebyPeacefulMeans:PeaceandConflict,DevelopmentandCivilization.9Kant,GroundworkfortheMetaphysicsofMorals.
EXAMPLE
Evaluations of “Quick Impact” Projects (QIPs)reflectconsequentialistreasoning.QIPsareoften
promoted in post-accord environments because
theyprovidepositive“peacedividend”benefitsto
many, such as rebuilt schools, which may help
stabilize a community as a base for other
interventions in a short period of time. These
projects are therefore typically evaluated based
onthespeedandscaleofthebenefits (e.g.how
manyvillagesreachedinashortperiodoftime).
Given this reasoning, they have been seen as
producing a better overall consequence (more
villages reached) than slow, intensive
participatory processes that might rebuild a
schoolinacommunityoveratwo-yearperiodand
generategreater communityownership,butdue
to its intensive nature does not reach as many
people as quickly. Note, however, if the time-
horizonofconsequencechanges,theassessment
of benefits and harms shifts and so QIP
evaluations may initially be positive but then
become negative if short-term benefits did not
producelong-termbenefits.
5
guidanceistheprinciplethatactionsmust“donoharm”ormustpromotenon-discriminationasabasisforinclusivesocieties.10
Ifwe–evaluatorsandthosebeingevaluated–decidetofocusonthe“rightmeans”thataredefinedbyourprinciplesandmost important indeterminingwhetherapeacebuilding initiativewasgood,weexamine theways that thepeacebuildinginitiativewasimplemented.Didtheprocess“donoharm”?Didittreatpeoplewellthroughout–alwaysasendsandneverasmeanstoadesiredend?Diditmodelhumanrightsprinciples?Wastheinitiative
participatoryandinclusive?Wasitgender-sensitive?Didtheapproachembodynonviolentprinciples?Thisisquiteadifferentsetofquestionsthandiditproducethegreatestgoodforthemostnumberofpeoplebecauseourevaluationquestions includeassessmentsofproblematicactions thatmighthaveaffectedonlyaminorityofpeople,orgroupsofpeoplewhohavebeenseenas less importantoverall (e.g.gender-based violence has often been overlooked inpeacebuildingprojectsthatfocusondemobilizingmilitants).When we shift to thinking about the right means andprinciples,thenwealterourevaluationfocustoincludetheprocessbywhichaninitiativepursueditsends.11
Notethatattimes,someprinciplesmaybeatoddsor inconflict.Forexample,valuingindividualsandindividualrightsovercommunitiesandcommunityrightscanbeproblematic.Inthesesituations,theevaluatormaybemakingjudgementsbasedondisputedvalues,whichisdiscussedfurtherinsectionfour(below).
Therearetwomoreperspectivesonhowtojudgewhatconstitutesgoodthatarelesscommonthanthefirsttwo.Lesscommon,however,doesnotmeanthattheydonothavemerit,particularlywhenthinkingaboutpeacebuilding.Oneistothinkaboutthepurposeofpeacebuildingasdefinedbythepursuitofpersonalexcellence.Asecondistothinkaboutthepurposeofpeacebuildingasbasedonrelationships–thatis,thinkingaboutapersonbeingapersononlythroughandwithotherpeoplewithinarelationalnetwork,asoccursinUbuntuorcareethicsthinking.12
3.3VirtuesandGoodCharacterIfwejudgepeacebuildingbyitspromotionofpersonalexcellence,thenwereframeourevaluationfromfocusingon activities that produced communal peaceoutcomes to looking at howpeacebuilding initiatives supportedcharacterdevelopmentandexcellencewithinsettingsofconflict.Thismightbeturnedintooutcomesandoutputs,but ifdonemechanically itwilldefeat thecentral idea thatexcellence involvesone’scharacter. So, rather than
10Forexample,Anderson,DoNoHarm:SupportingCapacitiesforPeacethroughAid.SeealsoOECD,"EvaluatingPeacebuildingActivitiesinSettingsofConflictandFragility:ImprovingLearningforResults."11Therearesomeevaluationmethodsthatwereenvisionedtodrawmoreattentiontomeansandprocessinevaluation,suchasempowermentevaluationortransformativeevaluation,whichcanbehelpful.SeeforexampleFetterman,Kaftwarian,andWandersman,Empowerment
Evaluation,orMertens,TransformativeResearchandEvaluation.12ForanintroductiontoethicsofcareseeHeld,"TheEthicsofCare."ForanoverviewofUbuntuethicsseeforexampleMunyakaandMotlhabi,"UbuntuandItsSocio-MoralSignificance."
EXAMPLE
Evaluation questions that focus on particular
principles, suchaswhetherornotgenderequity
was achieved in a community-based youth arts
project, may look at levels of gender parity or
increases in participation by young women in
peacebuilding activities. Evaluations may also
look at project staffing and whether or not the
principle of gender equity was evident in the
processofimplementation.
6
emphasizetheendorthinkaboutgoodvirtuesasameanstoanend,virtuesthinkingemphasizesalifewell-livedandseescollectiveflourishingemergeaspart-and-parceloflivingliveswell.
Personal excellence is the focusof virtueethics, and ispresentwithina varietyof religious traditions, includingChristianity, BuddhismandTaoismaswell as inAristotle’swork. Theparticular virtues that are important in a givencontextandculturemayvary.Forinstance,inthedialogueat the start of this paper, theprogrammanger commentsthat thewomenandyouth involved in theprojectpossessparticular character traits that she sees as excellent: theparticipantsarepassionateanddedicated.Therearelikelyother virtues that are also important in this context thatwouldbepartofalifewell-lived.Ifweweretothinkaboutplacinganintentionalfocusoncharacterdevelopmentinourevaluations,wewouldlookforvirtuesattheimpactslevel–whichmayseemcounter-intuitivebecausewetendtothinkofpersonalchangeasanecessarystepforotherchangestooccur,andthereforealower-leveloutput.13
3.4RelationalityAnotherwaytothinkaboutjudginggoodinpeacebuildinginvolvesseeingreconfiguredrelationshipsasmoralgoodsthemselves.14Conflicttransformationapproachestendtoassumethisgood.Relationalethicsfocusesonlisteningtovoicesofthosewithwhomone’slifeisintertwined(nearandfar)andrespondingtothoseneedsinwaysthatarereceivedbythepersonbeingcaredfor(toavoidpatronizingbenevolence).ThisorientationisevidentinUbuntuethics,whichwerepartofthephilosophyunderpinningtheSouthAfricanTruthandReconciliationCommission.Inthistypeofassessment,ourhumanityisintertwinedandsowebecomegoodonlyasweactwithandrespondtoother’shumanity.
Ifwejudgepeacebuildingbyrelationality,thenwejudgeitbytheways inwhichpeoplearecaredfor inandbythepeacebuilding initiative – those working in the initiative,thosereceivingthe initiativeaswellasthose inthe largercirclesaroundtheinitiative.Weretheirneedsrespondedto?Weretheirvoicesheard?Reframingourjudgementofgood to assess relationality requires the evaluation beconceptualizedwithinarelationalcontextaswell.Inotherwords,relationshipsarenotassessedintermsoftheirsocialcapital–whichwouldmake themmeans toanend–butratherasgoodsinandofthemselves.
13ThisisalsoanemphasisoftheReflectingonPeacePracticeProject,whichprioritizes“greatestgood”thinkingintermsofsystemic-levelchange,seeAndersonandOlson,ConfrontingWar:CriticalLessonsforPeacePractitioners.14Thisapproachisbasedonanethicofcare,andalsofiguresprominentlyinUbuntuethics.
EXAMPLE
Youth sports and peacebuilding projects often
containastrongfocusoncharacterdevelopment.
Thisinvolvesdevelopingleadershipskills,strength
ofcharacter,respect,trust,empathyandsoforth.
Thistypeofprojectmodelbuildsontheideathat
characterexcellence isa necessarypartof living
well. Evaluations then correspondingly assess
charactervirtues.Evaluationsmayalsoassessthe
character virtues of those implementing the
projectsandwhetherornotgoodcharacterwas
modelledinimplementation.
EXAMPLE
Restorativejusticeprogramsfocusonbuildingor
restoringrelationshipsbetweenthoseconsidered
victims and offenders. Youth victim-offender
programs, for example, work in communities to
restore relationships between youth who have
committed crimes and those affected by the
crimes.Inthesetypesofprograms,restorationof
relationships are highly valued goods and
evaluationofthese initiativesnecessarily lookto
assessthenatureandqualityoftherelationships
built.
7
3.5SkillstoWorkOn:IdentifyingMoralValuesinEvaluationExpandingourthinkingaboutwhatisgoodandrightinpeacebuildinghelpsustorecognizethatwhenwegointoevaluationswearenecessarilymakingjudgementsaboutwhatisgoodamongarangeofdifferentconceptionsofthe“good”andthe“right”—whichincludemanymorethanthefourperspectivespresentedabove.Asevaluatorsareaskedtomakejudgementsofworthandmerit,wecanbemoreexplicitandcarefulinidentifyingwhatisbeingvalued,toenhancethelikelihoodthattheevaluationitselfisgoodandright.
Importantskillsforevaluators,then,includeidentifyingmoralvaluesembeddedindocumentsliketheTermsofReference(TOR)foranevaluation,andmanagingdiscussionofthevalueswithstakeholderstoensurethatpeopleagreewiththedifferentmoralvaluesthataregoingtobeusedtojudgethepeacebuildinginitiative.Forexample,ifanevaluatoridentifiedthevaluesintheconversationabove,heorshewouldquicklyseethattherewasatensionbetweenwhatwasvaluedasgoodbytheadministrator(impact,scaling-up)andwhatwasvaluedasgoodbytheprogram staffer (doing good work, supporting the community). The community itself is not involved in thisconversation,andlikelywouldpossessadditionalvalues.Theevaluatordoesn’tneedtopre-judgethevalues,butrathertohelpstakeholdersidentifyandagreeonwhatismostimportant,worthyandmeritoriousthroughdialogue,inordertoconductanevaluationthatwillbeuseful,goodandcontributetolearning(andperhapsevenflourishing).
Thefollowingtext-boxcontainsquestionstoasktohelpanevaluatorthinkaboutmoralvaluesandwhatconstitutesgoodandrightinpeacebuilding.ThesequestionsaremappedontoanevaluationcycleinAppendix1,whichincludesthecontractingandentrypointoftheevaluation,evaluationdesign,datacollection,analysisandreportingstages.15
TEXTBOX3.5QUESTIONSTOASKABOUTWHATCONSTITUTESGOODANDRIGHTPEACEBUILDING
Herearesomegeneralquestionstohelpevaluatorsthinkaboutthegoodandrightinpeacebuilding:
§ HowarepeoplejudgingwhatisconsideredgoodandrightintheTORorinconversations?Whatisvaluedasmorallygood(e.g.ends,means,virtues,relationships)?
§ Doallofthestakeholdersagreewiththevaluesbeingusedtojudgetheinitiative?
§ Aretheretensionsbetweenvaluesthatdifferentstakeholderspossess?Ifso,whatarethey?
4.TheWho:WhoDecidesWhatisGoodandRight?Inpeacebuildingprojectsorprogramsthatarefundedbyanexternaldonor,therearetypicallythreecommunitiesofstakeholderswhoareinterestedinthequestionofwhatconstitutesgoodandrightpeacebuilding.16First,thereisthecommunityinconflict,inwhichthepeacebuildinginitiativeoccursthatisdirectlyvested-inandaffected-bythepeacebuilding.Second,thereistheimplementingorganization,whichmightbelocally-basedorinternationally-based.Andthird,thereisthefundingorganizationorthefundingcommunity.Anevaluatoristypicallyconnecteddirectlytothesecondand/orthirdcommunitiesbyvirtueoftheircontractagreement,andonlyindirectlytothefirstcommunity(unlessitisthecommunityitselfthatiscommissioningtheevaluation).
15ThankstoColleenDugganforthissuggestion.IextendDugganandBush’sethicalquestionsintheAppendixillustrationsfordifferentconsiderationsofthegoodandrightinpeacebuildingevaluation.Formoredetailedexploration,particularlyaroundthehowquestions,seeBushandDuggan"EvaluationinConflictZones:MethodologicalandEthicalChallenges,”5-25;BushandDuggan,“AddressingEthicalandPoliticalChallengesinEvaluation”;and,DugganandBush,"TheEthicalTippingPointsofEvaluatorsinConflictZones,"485-506.16Fast,Neufeldt,andSchirch,"TowardEthicallyGroundedConflictInterventions:ReevaluatingChallengesinthe21stCentury,"185-207.
8
So,whatroleshouldeachofthesegroupshaveindeterminingthegoodandtheright?Is itfairandrightforanoutsidefundertohavethefinalsayinwhatis“goodpeacebuilding?”Isitfairandrightfortheaffectedcommunityto have the final say in what is “good peacebuilding?” Likewise, is it fair and right if it’s the implementingorganization?Theseareimportantethicalquestionsthatevaluatorsmayfeelareoutsideoftheircontrol,especiallyifcoming-inonacontractthatseemstohavelimitedflexibility. Thisdynamicofpoweramongststakeholdersindetermining what is valued, however, can reinforce power differentials between funders and receivers, and isimportanttoconsiderandaddressifwereallydowanttobeethicalinourevaluationpractice.
TheAmericanEvaluationAssociationspeakstothebroaddomainofvaluesandtheresponsibilityofevaluatorstoconsidermultiplestakeholdergroups,includingsocietyasawhole,intheirGuidingPrinciplesforEvaluatorsunderthetopicheadingof“responsibilitiesforgeneralandpublicwelfare.”17TheAfricanEvaluationAssociation(AfrEA)similarly emphasizes the importance of recognizing and involving vulnerable groups as well as communityparticipants inevaluation.18 Theseare importantconsiderations,particularly toaddress themoralcritiquethatpeacebuildingisanewerformofcolonialdomination.
Howmightevaluatorsaddressconcernsaroundpowerdifferentialsethicallywithintheirowndomain?Oneconcretewaytodothisistoset-upareferencegroupfortheevaluationthatincludesallofthekeystakeholders.Forexample,MichaelQuinnPattonadvocatesareferencegroupmodelinUtilization-FocusedEvaluationinordertoensurethatthe evaluator is not “some stakeholder’s political puppet”.19 Patton’s “Evaluation Task Forces” involve majorstakeholdersinhelpingtomakedecisionsaboutthepurposeandfocusofanevaluationaswellasthemethodsused.Henotesthatifstructuredwell,suchagroupcanhelptoincreasetransparency,involvediverseperspectivesandvalues and increase openness to the evaluation process, among other things.20 There are other forms ofconsultationthatcanalsobeutilized(usingskillsthataresimilartothoseinvolvedinmediation).Onefinalnote,referencegroupsortaskforcesneedtobewell-facilitated,tobalanceinputs,andinvolvestakeholdersiftheyaretoworkwell.
Anotheroptionforevaluatorsatthecontractingstageistofindoutifthekeystakeholdersareopentohavingtheirprogramtheoryofchangeandthevaluesunderlyingitre-assessed.Inthisprocess,askingquestionslike“whatdoessuccessmean”or“howdoyouknowthingsareimproving”canidentifyvaluesthatpeopleholdinpeacebuilding,andhelp identifywaystoevaluatethem. Forexample,arestakeholderswillingtoconsiderthatgoodoutcomesmightmeanmoralcharacterdevelopmentratherthandevelopingsocialcapital.Reassessingvaluesmayalsohelpimplementersidentifywhetherornotatheoryofchangethatisinoperationisitselfaccidentlymarginalizingpeopleor groups, and can help make sure the evaluation process itself doesn’t compound problems (e.g. by usingcommunitiesorcommunitymembersasanendtostability).
Asevaluators thinkabouthow to structureevaluationsethically, thequestionofwhodecideskeyvalue-relatedquestionsarecritical.
17“AEAGuidingPrinciplesTrainingPackage,”AmericanEvaluationAssociation,accessedAugust1,2016,http://www.eval.org/p/cm/ld/fid=105.18AfricanEvaluationAssociation,"AfricanEvaluationGuidelines-StandardsandNorms."19QuinnPatton,Utilization-FocusedEvaluation,529.20Ibid.,537-9.
9
TEXTBOX4.1.QUESTIONTOASKABOUTWHOMAKESDECISIONS21
Herearesomegeneralquestionstoasktohelpevaluatorsthinkaboutethicsrelatedtothe“who”:
§ Whoarethemajorstakeholders?Howaretheyinvolvedindecision-making?
§ Arethemajorstakeholdersopentohavingtheirtheoryofchangereassessedbyabroadgroupofstakeholders?
§ Howcanwestructureaprocessthatcanidentifyandaddresssignificantmoralvaluedifferences?Addresspowerimbalances?Managecompetingclaimsandinterests?
§ Ifwesetupareferencegroup,whoshouldbeonit?Whatrolewillreferencegroupmembershave?Howwillpowerdifferentialsbemanagedinthereferencegroup?Shouldsomepeoplebeleftoffofthereferencegroupbecausetheyholdinordinatepower?
5.TheHow:HowDoWeEnsurePeacebuildingEvaluationsAreDoneEthically?
Abigconcern forevaluators isensuring theprocessbywhichwedoevaluations isethical. Beingethical inourconductisamajorfocusofethicsguidanceforpeacebuildingevaluators.Thissectionwillbethereforeshorterasthereareamplematerialsavailabletoexplorethesepointsingreaterdetail(seesuggestedresourcesattheend).However,concernsrelatedtoensuringwearedoingevaluationsethicallyaretypicallyraisedatthreelevels:1)howarefindingsgenerated?2)howdoesanevaluatorconducthimorherself?and,3)howsensitiveistheevaluationprocessitselfistotheconflictenvironment?
5.1HowFindingsAreGeneratedGatheringdataandanalyzingthefindings isthecentraltaskforevaluations. Inorderfortheevaluationtohaveintegrity,theprocessbywhichfindingsaregeneratedneedstohaveintegrity.Wecanthinkaboutthisintermsofthemethodsthatarechosentogatherdataandaskquestionslike:arethemethodsappropriateforthetaskathandand is the evaluator capable of implementing the appropriatemethods? Are there values implicit in the data-gatheringmethodologythatconflictwithvaluesheldbythestakeholders?
We can also think about the question of howwe collect data in terms of diligence and trustworthiness, askingwhetherevaluatorsarethoroughintheirwork(collectingandanalyzingdata),responsibleandup-frontwiththeevaluation’sstrengthsandlimits.Herequestionstoaskare:isthedatatrustworthy(collectedandanalyzedinwaysthat consider rigor and thoroughness that areappropriate to themethodology)? Am Imaking claimsbasedonsufficientdata?Whatarethelimitsofmydataandinferences?
Finally,weneedtothinkabouthowpeopleandrelationshipsarerespectedaspartofthedata-gatheringprocess.Herewecanaskquestionslike:doestheevaluationprocessrespectpeople?Doesitrespectrelationships?AmImakingsureIprotectpeople(interviewees,assistants,stakeholders)intheprocess(e.g.dopeoplefeelsafe?Isthereanypotentialcoercionoccurring?Isconfidentialitymaintained?)?AmItakingstepstoensurenoharmsoccur?AmIsharingmyfindingsbackwiththecommunitiesandstakeholdersinvolvedintheproject?
21Toseewaysinwhichquestionsabout“thewho”canmapontotheevaluationcycleseeAppendix1(Box2).
10
TEXTBOX5.1.QUESTIONSTOASKABOUTHOWANEVALUATIONISCONDUCTED22
Generalquestionstoconsidertoensureanevaluationisdoneethically:
§ Aretherevaluesimplicitinthedata-gatheringmethodologythatconflictwithvaluesheldbythestakeholders?Arethereothermethodsthatmightbemoreappropriate?
§ Isthedatatrustworthy?Arethelimitsofthedataandinferencesstated?
§ Doestheevaluationandpost-evaluationreportingprocessrespectpeople,relationshipsandtheculturalcontext?
§ Are people (interviewees, assistants, stakeholders) protected against coercion? Isconfidentialityensured?
§ Am I embodying important personal and professional character virtues (e.g. honesty,integrity,trustworthiness,competence)?WhoelsecanhelpmedeterminewhetherornotIamembodyingthesevirtues?
§ Howmighttheevaluationcontributetoharminthiscontext(e.g.worsenedethnic,political,socio-economicdivisions)?HaveIlimitedthepossibleharms?
§ HowcanIdotheevaluationinawaythatcontributestogood?
5.2HowanEvaluatorConductsHer-orHimself?Ethicsguidanceforevaluatorsuniformlyspeakstotheimportanceofanevaluator’sintegrity,whichinvolvebothcharacterexcellenceandasetofbehaviorsthatresultfromhavingintegrityorpropriety.Behaviorstypicallyincludebeing honest, transparent, trustworthy, competent (with respect to communication skills and evaluationmethodology), culturally sensitive, and declaring any personal conflicts of interest. 23 As an evaluator, the bigquestionstoaskinvolvethinkingthroughone’sownconduct–individuallyorwithmentorsandadvisors.AmIactingwithintegrity?AmIcontributingtotheoverallsocialenvironmentandpublicwelfare(oratleastnotcompromisingit)?
5.3HowSensitivetheEvaluationProcessistotheConflictEnvironmentThethirdlevelofconsiderationforensuringpeacebuildingevaluationsareethicalfocusesonthecontext.Conflictcontexts are volatile, peacebuilding addresses delicate issues and people are very vulnerable (physically andpsychologically).Thismeansitisveryeasytocontributetoharminandthroughtheevaluationprocess.Todogoodevaluation, therefore, requires considering theways inwhich the evaluation processwill do no harm by beingconflict-sensitive.24Thismeansbeingawareofwaysinwhichtheevaluationmayfeedintotensionsordivisionswithin the conflict context andmitigating them. Conflict sensitive framing tends to focuson reducingnegativeconsequences,howevermoral thinkingpushesus toalso consider thepositiveandalsoaskhow theevaluation
22SeeAppendix1(Box3)forhowallthreeofthesesetsofquestionscanbemappedontotheevaluationcycle.23Thislistwasgeneratedfrom“Ethics,”CanadianEvaluationAssociation,accessedAugust1,2016,http://www.evaluationcanada.ca/ethics;"AmericanEvaluationAssociationGuidingPrinciplesforEvaluators,"accessedAugust1,2016,http://www.eval.org/p/cm/ld/fid=51;AfricanEvaluationAssociation,AfricanEvaluationGuidelines-StandardsandNorms;Widmer,Landert,andBrachman“EvaluationStandards:ofSEVAL,theSwissEvaluationSociety(SEVALSTANDARDS)”;SociétéFrançaisedel’Évaluation,“Chartedel’evaluation:despolitiquespubliquesetdesprogrammespublics”;and,AustralasianEvaluationSociety,“CodeofEthics.”24See,forexample,OECD/DAC’sguidanceforhowtobeconflictsensitiveintheevaluationprocessinOECD,“EvaluatingPeacebuildingActivitiesinSettingsofConflictandFragility:ImprovingLearningforResults”.SeealsoJayawickrama,"’IfTheyCan'tDoAnyGood,TheyShouldn'tCome:'NorthernEvaluatorsinSouthernRealities,"26-41.Otherauthors,suchasEsserandVanderkampin“ComparableandYetContext-Sensitive?ImprovingEvaluationinViolentlyDividedSocietiesThroughMethodology,”42-56;andKennedy-Chouanein“DevelopingOECDDACGuidanceonEvaluatinginSettingsofViolentConflictandFragility,”110-115addressrelatedtopics.
11
processmightcontributetogood,suchasdesigninganevaluationitselfinawaythatcontributestoconstructiverelationshipbuildingbetweengroupsthatmistrusteachotherinaconflictsetting.
6.Conclusion:CreatingSpaceforMoralValuesTo conclude, we’ll go back to the beginning and our conversation between the program staff member andadministratorandaddanethicalevaluator:
“Thedatawilltelluswhattodo,”theadministratorsaysconfidently.
“Notsofast”theevaluatorinterjects,“Whatyouchoosetodohastodowithwhatyouvalue,what
you judgeasmattering,and thenhowyou interpret thedata inmakingyour choice. Let’shavea
conversationaboutwhatmatters–toyou,andtotheotherstakeholders.”
An ethical peacebuilding evaluator, just like an ethical evaluator more generally, develops reflexive skills inidentifyingand listeningtothevalue judgementsandclaimsthatstakeholdersaremaking. Werecognizeattheoutsetthatevaluationinvolvesmakingvaluejudgementsaboutthingsthatmattertopeople.Andtherearelayersofjudgmentsthatoccurinpeacebuildingevaluations,whichrequirecarefulandopenthinking,particularlyastheyoftenoccurinfragileordividedsettings.
Asaskilledandethicalevaluator,beingabletoaskimportantquestionsandcreatethespacetotalkaboutwhatis valued as good and right in peacebuilding is critically important. This isn’t a one-time event, but rather aperspectiveorsetofmusclesthataredevelopedwithuseovertime.Thequestionsexploredaboveaboutmoralvalueperspectives,whodecideswhatmattersinanevaluation,andhowtoimplementpeacebuildingevaluationsareintendedtobeahelpfulstartingplace.Itis,however,onlyabeginning.
12
6. ReferencesandAdditionalResourcesAmericanEvaluationAssociation.“AEAGuidingPrinciplesTrainingPackage.”AccessedAugust1,2016.
http://www.eval.org/p/cm/ld/fid=105.
Austin,Beatrix,MartinaFischer,andHansJ.Giessmann(eds.)AdvancingConflictTransformation:TheBerghof
Handbook.Secondedition.Leverkusen:BarbaraBudrichPublishers,2011.
Bush,Kenneth,andColleenDuggan.“AddressingEthicalandPoliticalChallengesinEvaluation.”PresentedattheEvaluationConclave,Kathmandu,February2013.
Esser,DanielE.,andEmilyE.Vanderkamp.“ComparableandYetContext-Sensitive?ImprovingEvaluationinViolentlyDividedSocietiesThroughMethodology.”JournalofPeacebuildingandDevelopment8,no.2(2013):42-56.
Fast,LarissaA.,andReinaC.Neufeldt.“EnvisioningSuccess:BuildingBlocksforStrategicandComprehensivePeacebuildingImpactEvaluation.”JournalofPeacebuildingandDevelopment2,no.2(2005): 24-41.
Fetterman,David,ShakehKaftwarian,andAbrahamWandersman.EmpowermentEvaluation.Secondedition.ThousandOaks,CA:Sage,2015.
Fournier,DeborahM.“Evaluation,”inEncyclopediaofEvaluation,editedbySandraMathison,161-162.ThousandOaks,CA:SAGEPublications,2005.
Galtung,Johan.PeacebyPeacefulMeans:PeaceandConflict,DevelopmentandCivilization.ThousandOaks,CA:SagePublications,1996.
Held,Virginia."TheEthicsofCare,"inTheOxfordHandbookofEthicalTheory,editedbyDavidCopp.NewYork,NY:OxfordUniversityPress,2006.
Kant,Immanuel.GroundworkfortheMetaphysicsofMorals.TranslatedbyArnulfZweig,editedbyThomasE.HillJr.andArnulfZweig.NewYork,NY:OxfordUniversityPress,2002.
Kennedy-Chouane,MeganGrace.“DevelopingOECDDACGuidanceonEvaluatinginSettingsofViolentConflictandFragility.”JournalofPeacebuildingandDevelopment8,no.2(2013):110-115.
Lederach,JohnPaul,ReinaNeufeldt,andHalCulbertson.ReflectivePeacebuilding:APlanning,Monitoringand
LearningToolkit.Mindanao:TheJoanB.KrocInstituteforInternationalPeaceStudies,UniversityofNotreDameandCatholicReliefServicesSoutheast,EastAsiaRegionalOffice,2007.
Mathison,Sandra(ed.)"ValueJudgment,"inEncyclopediaofEvaluation.ThousandOaks,CA:SAGEPublications,2005.
Mertens,DonnaM.TransformativeResearchandEvaluation.NewYork,NY:GuilfordPress,2007.
Munyaka,Mluleki,andMokgethiMotlhabi."UbuntuandItsSocio-MoralSignificance,"inAfricanEthics:AnAnthologyofComparativeandAppliedEthics,editedbyMunyaradziFelixMurove.Scottsville:UniversityofKwaZulu-NatalPress,2009.
Neufeldt,ReinaC.EthicsforPeacebuilders:APracticalGuide.Boulder,CO:Rowman&LittlefieldPublishers,2016.
Neufeldt,ReinaC.“Interfaithdialogue:assessingtheoriesofchange.”PeaceandChange36,no.3(2011):344–372.
OECD."EvaluatingPeacebuildingActivitiesinSettingsofConflictandFragility:ImprovingLearningforResults,"inDACGuidelinesandReferenceSeries.Paris:OECD,2012.
QuinnPatton,Michael.UtilizationFocusedEvaluation.Fourthedition.ThousandOaks,CA:Sage,2008.
Schwandt,ThomasA."TheLandscapeofValuesinEvaluation:ChartedTerrainandUnexploredTerritory,"inProgressandFutureDirectionsinEvaluation:PerspectivesonTheory,Practice,andMethods,editedbyDebraJ.RogandDeborahFournier.SanFrancisco,CA:JosseyBassPublishers,1997.
13
Weston,Anthony.A21stCenturyEthicalToolbox,thirdedition.NewYork,NY:OxfordUniversityPress,2013.
I.ProfessionalEvaluationAssociationGuidesonEthics
AfricanEvaluationAssociation(AfrEA)."AfricanEvaluationGuidelines-StandardsandNorms."AfrEA,2006/7.
AmericanEvaluationAssociation."AmericanEvaluationAssociationGuidingPrinciplesforEvaluators."AccessedAugust1,2016.http://www.eval.org/p/cm/ld/fid=51.
AustralasianEvaluationSociety(AES).“CodeofEthics.”Carlton:AES,2013.
CanadianEvaluationAssociation."Ethics."AccessedAugust1,2016.http://www.evaluationcanada.ca/ethics.
SociétéFrançaisedel’Évaluation.“Chartedel’evaluation:despolitiquespubliquesetdesprogrammespublics.”Paris:SociétéFrançaisedel’Évaluation,2006.
Widmer,Thomas,Landert,Charles,NicoleBrachman.“EvaluationStandards:ofSEVAL,theSwissEvaluationSociety(SEVALSTANDARDS).”Basel:TheSwissEvaluationSociety,2000.
II.ExplorationsofEthicalChallengesforEvaluatorsinConflictAreas
Bush,Kenneth,andColleenDuggan."EvaluationinConflictZones:MethodologicalandEthicalChallenges."JournalofPeacebuildingandDevelopment8,no.2(2013):5-25.
Bush,Kenneth,andColleenDuggan,eds.EvaluationintheExtreme:Research,ImpactandPoliticsinViolently
DividedSocieties.NewDelhi:SagePublicationsandInternationalDevelopmentResearchCentre(IDRC),2015.
Duggan,Colleen,andKennethBush."TheEthicalTippingPointsofEvaluatorsinConflictZones."AmericanJournal
ofEvaluation35,no.4(2014):485-506.
Jayawickrama,Janaka."'IfTheyCan'tDoAnyGood,TheyShouldn'tCome:'NorthernEvaluatorsinSouthernRealities."JournalofPeacebuildingandDevelopment8,no.2(2013):26-41.
Therearealsobriefdiscussionsofethicsrelatedtoevaluationin:
Church,Cheyanne,andMarkRogers.DesigningforResults:IntegratingMonitoringandEvaluationinConflict
TransformationPrograms.Washington,D.C.:SearchforCommonGround/UnitedStatesInstituteforPeace,2006.
Lederach,JohnPaul,ReinaNeufeldt,andHalCulbertson.ReflectivePeacebuilding:APlanning,MonitoringandLearningToolkit.Mindanao,PH:TheJoanB.KrocInstituteforInternationalPeaceStudiesandCatholicReliefServicesSEAPRO,2007.
III.EthicsandPeacebuildingMoreGenerally
Anderson,MaryB.DoNoHarm:SupportingCapacitiesforPeacethroughAid.Boulder,CO:LynneReinnerPublishers,1999.
Anderson,MaryB.,andLaraOlson.ConfrontingWar:CriticalLessonsforPeacePractitioners.Cambridge,MA:CDACollaborativeLearningProjects,2003.
Barry,Bruce,andRobertJ.Robinson."EthicsinConflictResolution:TheTiesThatBind."InternationalNegotiation7(2002):137-42.
Cohen,Cynthia.WorkingwithIntegrity:AGuidebookforPeacebuildersAskingEthicalQuestions.Waltham,MA:BrandeisUniversity,2001.
Fast,LarissaA.,ReinaC.Neufeldt,andLisaSchirch."TowardEthicallyGroundedConflictInterventions:ReevaluatingChallengesinthe21stCentury."InternationalNegotiation7,no.2(2002):185-207.
InternationalAlert."CodeofConduct:ConflictTransformationWork."London:InternationalAlert,1998.
14
Kelman,HerbertC.,andDonaldP.Warwick."TheEthicsofSocialIntervention:Goals,Means,andConsequences."InTheEthicsofSocialIntervention,editedbyGordonBermant,HerbertC.KelmanandDonaldP.Warwick,3-33.Washington,D.C.:HemispherePublishingCorporation,1978.
Laue,James,andGeraldCormick."TheEthicsofInterventioninCommunityDisputes."InEthicsofSocialIntervention,editedbyGordonBermant,HerbertC.KelmanandDonaldP.Warwick,205-32.Washington,D.C.:HemispherePublishingCorporation,1978.
Murithi,Tim.TheEthicsofPeacebuilding.Edinburgh:EdinburghUniversityPressLtd,2009.
Neufeldt,Reina.EthicsforPeacebuilders:APracticalGuide.Lanham,MD:Rowman&LittlefieldPublishers,2016.
Sawatsky,Jarem.JustpeaceEthics:AGuidetoRestorativeJusticeandPeacebuilding.Cambridge:TheLutterworthPress,2009.
Slim,Hugo."DealingwithMoralDilemmas."InPeacebuilding:AFieldGuide,editedbyLucReychlerandThaniaPaffenholz.Boulder,CO:LynneRiennerPublishers,2001.
Warfield,Wallace."IsIttheRightThingtoDo?APracticalFrameworkforEthicalDecisions."InAHandbookofInternationalPeacebuilding:IntotheEyeoftheStorm,editedbyJohnPaulLederachandJaniceMoomawJenner,213-23.SanFrancisco,CA:Jossey-Bass,2002.
15
8.Appendix:EthicsQuestionsduringanEvaluationCycle
8.1.QuestionsontheWhat(What’sGoodandRight)
Entry/Contracting
Evaluation
Design
Data
Collection
DataAnalysis&Interpretation
Communication
ofResults
Utililization
ofResults
§ Doestheevaluationdesignreflecttherangeofvaluesthatwillbeusedtojudgegoodandright?
§ Willtherebemethodslimitations?§ Aretheeffectsofcultureand
contexttakenintoconsiderationvis-à-visthegoodandrightinthedesign?
§ Arethedatacollectionmethodologiescapturingthedepthandrangeofconsiderationsofgoodandrightintheprojectbyitsdiversestakeholders(e.g.areyoumeasuringrelationshipsusinganends-basedsurvey?)
§ Aretheeffectsofcultureandcontexttakenintoconsiderationvis-à-visthegoodandright?
§ Doesanalysisandinterpretationtakeintoconsiderationtherangeofvaluesheldaboutgoodandrightbydiversestakeholders?(E.g.areyoudefaultingtoends-onlythinking?)
§ Doyourevaluationproducts(writtenandverbal)reflecttherangeofconsiderationsofgoodandrightbasedonthevaluesofitsstakeholders?
§ Aremoralvaluesexplicitlydiscussed?
§ Doyouanticipateanymisuseoffindingsiftherearediversevaluesincludedintheevaluation?Howcanyouaddressthis?
§ Arecommissionersopentodiscussingwhatisconsideredgoodandrightintheevaluation?(Isitanends-onlyfocus?Areconsiderationsofmeansincluded?Virtues?Relationships?)
§ Is“good”predeterminedbythosecommissioningtheevaluation?§ Doallstakeholdersagreeonthevaluesbeingusedtojudge?§ Aretheredifferencesbetweenwhatthecommissioningorganizationvalues
andwhatvulnerablepopulationsintheinitiativevalue?Ifso,whatarethey?
16
8.2QuestionsRegardingtheWho(WhoDecides)
• Whoarethemajorstakeholders?Howaretheyinvolvedindecision-making?(e.g.outsidersorinsiders?)
• Arethemajorstakeholdersopentohavingtheirtheoryofchangereassessedbyabroadgroupofstakeholders?(E.g.asking“whatwouldsuccesslookliketoyou?”)
• Aresomevoicesmoreorlessrepresentedinyourdatacollection?Isthisfairandappropriate?
• Istheevaluationstructuredtoidentifyandaddresssignificantpowerimbalancesandmoralvaluedifferencesbetweenstakeholders?
• Areyouusingareferencegroup?Doesitincludediversevoices?Howwillyoumanagepowerdifferentials?
• Doestheanalysisandinterpretationtakeintoconsiderationtherangeofstakeholdersviewsappropriately?
• Isthereundueinfluenceofsomeviews(e.g.thecommissionersoryourown)ontheanalysis?
• Doyouanticipateanymisuseoffindingsbymajorstakeholders?Howcanyouaddressthis?
• Canyoudomoretoensureevaluationfindingsareused?
• Inyourcommunications,doyoureflectconsiderationsofabroadgroupofstakeholders?
• Arevulnerablegroupsandcommunitygroupsincludedinthecommunication?
• Areresultsshared(includingmadepublicasappropriate)?
Entry/Contracting
Evaluation
Design
Data
Collection
DataAnalysis&Interpretation
Communication
ofResults
Utililization
ofResults
17
8.3QuestionsofHowtoConductanEvaluationEthically
Entry/Contracting
Evaluation
Design
Data
Collection
DataAnalysis&Interpretation
Communication
ofResults
Utililization
ofResults
§ Isthedatacollectedinatrustworthyway?
§ Doestheprocessofdatacollectionrespectpeopleandrelationships?Arepeopleprotectedfromcoercion?Isconfidentialityensured?Areallgroupsrepresented?
§ Areyouembodyingpersonalandprofessionalcharactervirtues?(e.g.trustworthy,havingintegrity)
§ Isthedatacollectionprocessconflictsensitive?
§ Isthedataprotectedandhandledcarefully?
§ Areyoucompetentintheevaluationmethodsproposed?
§ Arethedatacollectionmethodologiesinconflictwithothervaluesheldbystakeholders?
§ Isthedesignconflictsensitive?(Areyoureinforcinganystereotypes?Areyouavoidingissues?)
§ Haveyouconsideredcultureandcontext?
§ Whatarethelimitsofyourdesign?
§ Areyouactingwithintegrity(honest,transparent,trustworthy,competent)?Doyouhaveanyconflictsofinterest?
§ Cantheevaluationbedoneethically?Willthisevaluationcontributetothecommongood(doesitneedto)?
§ Willthecommissionerdiscussethicaldilemmas?§ Areyoupreparedtodecline?§ Haveyouconsidered“theWho”and“theWhat”questions?
§ Doyouanticipateanymisuseoffindings?Howcanyouaddressthis?
§ Canyoudomoretoensureevaluationfindingsareused?
§ Isthecommunicationstrategyconflictsensitive?
§ Arevulnerablegroupsandcommunitygroupsincludedinthecommunication?
§ Areresultsshared?§ Inyourcommunication,doyou
actwithintegrity(e.g.honest,transparent,includelimitations)
§ Isconfidentialitymaintained?§ Aretheclaimsbeingmadebaseduponsufficientdata?§ Whatarethelimitsofthedataandinferences?§ Doesyouranalysisconsidermultipleviewpoints?Isitcareful?
Areyoutransparentinyourprocess?§ Isconfidentialitymaintained?