Introducing a global MRM System in an Implementing Organization

An Experience ReportMarkus Kupper, Michael Fink

1

INTRODUCTION

Since the beginning of the new millennium, thepressure from various donors has increased toshow quantitative evidence that developmentaid creates value for money. With theemergence of comprehensive market develop-ment programs in the early 2000’s, the demandfor adaptive management of projects in dynamicmarkets increased. Under these premises, TheDonor Committee for Enterprise DevelopmentDCED established a working group which starteda process to define good practices for ResultsMeasurement. This effort resulted in the launchof a unique standard in 2008, applicable to awide range of projects, in the following namedthe DCED Standard.

This report details how Swisscontact successfullybuilt up a comprehensive system of resultsmeasurement aligned with the DCED Standardfor Results Measurement during the last years,allowing to learn, adapt and optimize projectinterventions and to present quantitative andqualitative achievements to its donors and thepublic.

THE DCED STANDARD FOR RESULTS MEASUREMENT

The DCED Standard is a popular framework thatadds value when working in complex, rapidlychanging environments.

The core elements are impact logics/resultschains for each intervention, quantitative andqualitative indicators for each step of the resultschain, measurement plans with suitableattribution strategies and measurementmethods for all indicators and approaches tocapture systemic changes in terms ofsustainability and scale. It stresses the use ofmeasured results in management decisionmaking.

The credibility of reported results depends onthe quality of results measurement. Projects canbe audited on compliance with the DCEDstandard based mainly on documentation, thecriteria are transparent.

2Swisscontact 2021

Katalyst

Katalyst is considered to be one of the mostsuccessful market development projects. During2002 and 2018 the Katalyst project benefited4,75 million SMEs and farmers with an incomeincrease of USD 724 million through access tobetter services, more advanced technology,higher quality inputs, and more effectiverepresentation.

The project was funded by different donors overtime with the main donor consortium alwaysconsisted of SDC and UKAid, joined at differenttimes by SIDA, CIDA, EKN and DANIDA.Throughout the duration of the project, theMinistry of Commerce (MoC) of theGovernment of Bangladesh partnered thesedonors and provided the project with consistentand valuable support.

https://www.swisscontact.org/en/projects/katalyst

3

A SHORT CHRONOLOGY: SWISSCONTACT ’S ROAD TO A GLOBAL MRM SYSTEM

In April 2000, Swisscontact started a BusinessDevelopment Services (BDS) project inBangladesh commissioned by SDC. The formerBritish UKAid joined SDC as donor and the projectwas converted into a Market SystemsDevelopment Program with Swisscontact andGIZ-IS as implementing consortium. This projectwas branded as “Katalyst” and ran from April2003 until March 2018.

Katalyst became a laboratory for designing andtesting approaches among others for monitoringand results measurement (MRM). In October2003, a concept for analysing markets andmeasuring results was adopted by the donors ofthe project. This was the starting point for thedevelopment and putting into practice of acomprehensive MRM-framework which reachedits maturity in Katalyst’s second phase (2008-2013). MRM consultants also engaged in theDCED Working Group for Results Measurementand contributed at various stages to ensure thetransfer of experiences from and to WorkingGroup. Of course, already in earlier years, resultswere measured in projects of Swisscontactaccording to their specific needs. Since the early1990’s, templates were in use for consolidatingoutputs and outcomes of the different projects.Katalyst showed that the DCED Standard wassuitable to consolidate not only outcomes butalso impacts of different interventions, sectorsand even approaches. This led to the decision bySwisscontact’s management in 2012, to set up asystem for Monitoring and Results Measurement(MRM) aligned with the DCED Standard, first forSwisscontact’s own projects within itsDevelopment Program, co-financed by SDC, thenfor mandates interested and ready to apply theMRM system, aggregating impacts on organi-sational level.

The MRM system was set-up in 2013 as aninvestment project of the Development Programof four years, the basic set-up was as following:projects were supported by regional MRMadvisors until they could run an appropriateMRM-system on their own.

All projects had to include the necessaryresources (internal MRM officers, tools,measurement activities, support by advisors andpossible audits) in their planning. The advisors inturn were supported by a coordinator at HeadOffice through trainings, regularly updated MRM-guidelines and a platform for virtual exchangesand tools (Community of Practice).

To date the MRM framework developed bySwisscontact has become general requirementfor project implementation by Swisscontact andis applied in almost all projects. Since 2013,statements with consolidation of quantitativedata on organization level are being created forSwisscontact’s annual report. With the years,more complete and reliable data could beintegrated. In 2018, a web-based tool wascreated, where projects can enter their data andautomatically a set of graphics displays differentinterpretations of the data, for internal con-trolling as well as for external reporting.

Swisscontact 2021

4

REQUIREMENTS TO SET UP A GLOBAL MRM SYSTEM

1. RESOURCES

Given the decentralised location and the numberof projects – around 100 – it was clear from thebeginning that implementation of a coherentMRM-system across all projects could not behandled by a Head Office team nor in a short time.Therefore, MRM advisors were recruited in eachregion coordinated by Head Office, and in eachproject, resources were assigned for MRM: as arule of thumb, one full-time position per onemillion Swiss Francs of annual project budget.Experience shows that a budget for MRM(including staff costs) within a range of 5 to 10% ofthe annual project volume is reasonable. Efficiencycan be increased by combining data collection andmeasurements across different interventions.Therefore, in large projects with manyinterventions the percentage of MRM costs isusually somewhat lower than in small projects.

2. STRATEGIC CLARITY ON OBJECTIVES

This points to a very important condition for acomprehensive MRM-system: strategic clarity onobjectives and approaches is needed on everylevel. Working groups elaborated the basic logics(theories of changes) for the various types ofprojects and then the common indicators. TheMRM advisors were either actively involved orcontinuously informed on this work: they had toknow about the underlying logics of the projects tobe able to support the project teams. The crucialstep in the MRM framework aligned with the DCEDStandard is to establish convincing results chainsfor each intervention and it is one of the verycrucial tasks of an MRM advisor/coach to guide theproject team towards a convincing logic.

3. ADVISORS WITH FACILITATIVE SKILLS

This points to the profile of the MRM-advisorswhich is quite demanding: they should be familiarwith and interested in social research, know aboutbasic statistics, data management and processing,able and motivated to provide consulting andtraining services, knowledgeable of approachesapplied by Swisscontact. An MRM advisor has to befamiliar with a systemic/facilitative projectapproach in different areas and must be capable tomoderate the elaboration of the elements by theproject team. This combination is rather rare andfrom the beginning therefore a continuous capacitydevelopment was foreseen consisting of regularexternal and internal trainings, virtual discussions,and a Community of Practice.

4. TECHNICAL KNOWLEDGE

A quick start of the organization-wide imple-mentation was possible due to the work done insome key projects such as Katalyst: examples,training programs, documentation, processes werereadily available with little adaptations. Thecommon technical knowledge was integrated ininternal MRM-guidelines which have been updatedregularly and are available in a didactical andmodular form in English, Spanish and French.Familiarization of new staff with Swisscontact’sMRM-system has been an ongoing task on regionallevel for local staff and at Head Office forinternational and head office staff. Nowadays, abriefing on MRM is an integral part of the on-boarding process of all new staff. Sound technicalknowledge will also help to take the right decisionson the measurement related level of rigour andeffort appropriate for each case. These decisionsare influenced by the demand from donors and theavailable resources. In general, a rigorousness toallow a steering of the project and to providecredible evidence of impact is strived for. Asystematic MRM does not only draw fromknowledge but does also produce knowledge.Particularly in complex and innovative projects,new insights are gained by a careful MRM.

Swisscontact 2021

5

SUCCESS FACTORS TO SET UP A GLOBAL MRM SYSTEM

1. TOP LEADERSHIP SUPPORT

The presentation of quantitative results inprojects such as Katalyst convinced the topmanagement of Swisscontact that the result ofSwisscontact’s work could be illustrated alsothrough key indicators. The management ofSwisscontact also recognized that in competitivebiddings the reference to a solid MRM thatguarantees quality project implementation andadaptive management is an asset. Furthermore,it was recognized that robust results provide avaluable input into a controlling system, wherevalue for money would be an important element,which in turn would be again a strong argumentto donors. The acknowledgment that an internalstandard for MRM was required and that theDCED Standard provided a practical frameworkresulted in a strong leadership support by theManagement Team of Swisscontact. It wasarticulated through internal directions and theallocation of resources in the annual budgets.

2. PROJECT MANAGERS BUY-IN

With the decision of the top management toinvest in the area of MRM, Project Managers hadto be convinced. In some projects, resourceswere their concern, particularly where donorswere not interested. Also, the time to prepareinterventions was sometimes felt to slow downimplementation activities. It occasionally alsorequired a change in management style, towardsdecisions based on results that involveimplementation as well as MRM-staff. To get theproject managers on board it was important tohighlight the benefits for steering and projectperformance by colleagues who already used thesystem. In all annual regional seminars, a sessionon MRM was integrated, where experiences withthe MRM framework were shared. Finally, thefocus was on rather young projects, whereprocesses and instruments were not yet rigidlyestablished.

3. DCED (PRE-)AUDITS

Important opportunities for learning were the pre-audits and full audits for compliance with theDCED Standard: primarily for the staff of theaudited projects but also for advisors and similarprojects. In most cases they were conducted byexternal auditors/consultants ensuring anindependent view and potentially fresh inputs.They are a useful tool to motivate project staff andmanagement also in rather small projects, whereit showed that even with limited resources a goodcompliance with the DCED Standard can beachieved.

Swisscontact 2021

6

VALUE AND BENEFITS OF A GLOBAL MRM SYSTEMS

1. ON PROJECT LEVEL

The more robust and timely available resultsallowed better decisions, and the joint elaborationof intervention logics and indicators with theimplementation team strengthened theparticipation and acceptance of decisions. Theproject management became more adaptive to thedynamic environment and could consider earlysigns of changes before a heavy external evaluationshowed the achievements of objectives of theproject. The complete documentation facilitatedthe re-construction of earlier decisions and madediscussions rather based on facts than onperceptions. The exchange of project logics andresults among projects benefitted new projects anddeepened the mutual learning process above all inchoosing a good approach.

2. FOR THE ORGANISATION

Communication of the achievements of projectsbecame easier and confidence on the robustnessof project results had increased. The aggregationof results allowed to show potential donors andthe interested public an overall value for money.Qualitative results and success stories could becomplemented by reliable figures. Thestandardized MRM-system helped to reduceinception processes such as training of personneland discussions about the right approaches, henceincreasing the efficiency of a project. Whereasother approaches to measurement might be morethorough but less practical, Swisscontact’sexperience confirms Voltaire’s statement that “theperfect is the enemy of the good”. A suitableknowledge management definitely helps toincrease adoption and learning and increases theefficiency of MRM within an organisation.

3. TOWARDS DONORS

At external monitoring and evaluations, theprovision of systematically collected resultshelped in the dialogue with the evaluators anddonors. Discussions were less based on “gutfeelings” or sympathies for certain topics butrather on jointly established logics, outcomes andimpact. Showing the ability to implement a robustMRM helped in the positioning among donorsand supported Swisscontact’s reputation as aresult oriented and able implementer. This wasunderpinned by Swisscontact projects achievinghigh scores in DCED audits, such as recently in2020 for our Promoting Private SectorEmployment (PPSE) programme in Kosovo: amedium-sized MSD project scoring 97% in MUSTand 94% in RECOMMENDED criteria.

Finally, robust results are important for donors toshow their constituencies reliable impact of theirprojects. The Swiss Federal Audit Office (SFAO)audited the use of federal funds by Swisscontactand stated in January 2020: “SC's approach tomanaging and measuring results is consistent.This was defined in internal guidelines and wasfollowed in the projects submitted for review. (…)For the projects and sub-projects analysed, theSFAO noted the reliability of the source data bothat SC project manager and local partner level. Itnoted the good traceability of the data used forreports throughout the information chain. Theinternal and external reports produced by SC withreference to projects, whether at the local,regional or global level, are consistent andtransparent.”

Swisscontact 2021

7

CONCLUSION

The development and implementation of an MRM

system aligned to the DCED Standard across virtually

all projects of Swisscontact has proven to be

beneficial: for positioning the organisation towards

donors, for mutual learning and for steering the

projects. In some projects it meant a change of the

management style from expert knowledge to

participative observation and discussion of results. It

remains a continuing task to train and motivate staff

to analyse and use the quantitative and qualitative

results in a systematic way.

Nowadays MRM is an integral part of project

management for Swisscontact with an MRM unit in

Switzerland as well as regional advisors and project-

based experts in the field. As new approaches, tools

and methodologies are continuously evolving, the

development and update of Swisscontact’s global

MRM system is an on-going process that continues

to respond to the dynamic environment we are

working in.

Swisscontact 2021