performance measurement framework for business development ... · performance measurement framework...

PerformanceMeasurement Frameworkfor BusinessDevelopment Services:

Progress Report on theSustainability and Cost-Effectiveness IndicatorsSurvey

Progress Report on the Sustainability and Cost-Effectiveness Indicators Survey

Performance Measurement Framework for BusinessDevelopment Services

USAID Microenterprise Best Practices Project Field Research

by

Mary McVay

October 2001

This work was supported by the U.S. Agency for International Development, Bureau forGlobal Programs, Center for Economic Growth and Agricultural Development, Office ofMicroenterprise Development, through funding to the Microenterprise Best Practices (MBP)Project, contract number PCE-C-00-96-90004-00.

Mary McVay is the manager of the Performance Measurement Framework Field Research, a multiyearresearch initiative to develop a practical and useful performance assessment system for business developmentservices (BDS) programs. She is also the coordinator of the Small Enterprise and Education Promotion (SEEP)BDS Working Group, which has developed an online guide to BDS (www.seepnetwork.org/bdsguide). Ms.McVay has more than 10 years of experience in microenterprise development in Africa, Asia, and the UnitedStates. She launched the first product development workshops for microentrepreneurs in Kenya with UnduguSociety of Kenya in 1989. As monitoring and evaluation officer with USAID/Kenya, she evaluatedmicroenterprise programs and designed a $25-million microenterprise development project with a subsectordevelopment component. With CARE and SEEP in 1996, she wrote one of the first frameworks for definingbusiness development services and launched CARE’s Manual for the Design and Implementation of SubsectorPrograms. Ms. McVay has authored several MBP publications, including “Performance MeasurementFramework for BDS: A Preliminary Framework,” “Virtual Conference on Performance Measurement ofBusiness Development Services for Micro and Small Enterprises,” and “Performance Measurement for BDS: ARevised Framework and Guide to the Preparation of Case Studies.”

i

TABLE OF CONTENTS

EXECUTIVE SUMMARY iii

CHAPTER ONEBACKGROUND AND PURPOSE 1LIMITATIONS OF THE PMF RESEARCH ..................................................................................2WHICH TYPE OF BDS PROGRAMS SHOULD USE THE PMF 2001?.........................................2THE GOALS OF ASSESSING SUSTAINABILITY AND COST-EFFECTIVENESS .............................4RESEARCH QUESTIONS..........................................................................................................5INDICATORS AND RATIONALE ...............................................................................................6

CHAPTER TWOSUMMARY OF RESULTS 11FINANCIAL SUSTAINABILITY...............................................................................................11COST-EFFECTIVENESS.........................................................................................................13

CHAPTER THREEMETHODOLOGICAL ISSUES AND RECOMMENDATIONS 15FINANCIAL SUSTAINABILITY...............................................................................................15

Organizational Sustainability.................................................................................15Service-Level Sustainability...................................................................................16

COST-EFFECTIVENESS.........................................................................................................18Cost Allocation.......................................................................................................18Cost per Person Acquiring, Using, and Benefiting from a BDS............................19Cost-Effectiveness Ratio........................................................................................19

CHAPTER FOURAREAS FOR FURTHER STUDY 21

ANNEX A: BDS PERFORMANCE MEASUREMENT RESEARCH:A QUICK ROAD MAP A-1

ii

Table

1 Sustainability and Cost-Effectiveness Indicators and Methodology iii

2 Sustainability Indicators, Proposed Methodologies, and Interpretation 7

3 Cost-Effectiveness Indicators, Proposed Methodologies, and Interpretation 10

4 Financial Sustainability 12

5 Cost-Effectiveness 13

iii

EXECUTIVE SUMMARY

This progress report is part of a global effort to develop a common performancemeasurement framework (PMF) for business development services (BDS) for smallenterprises (SEs). This participatory, consensus-building initiative has developed over threeyears, and is supported by the U.S. Agency for International Development(USAID)/Development Alternatives, Inc. (DAI) Microenterprise Best Practices Project, andlinked to the Committee of Donor Agencies for Small Enterprise Development. The PMFfield research, which took place from October 1999 to September 2001, was conducted bysix BDS provider organizations offering a range of services around the world, and wasfacilitated by three technical experts and an advisory committee.

The purpose of the PMF is to develop a common framework for practical, valid, and usefulperformance assessment (as opposed to evaluation and management information systems).The information is designed to be used to improve services, identify best practices programs,and improve fund allocation and management. The structure of the PMF reflects the goalsand structure of the market development approach to BDS. It attempts to assess:

§ Scale and outreach to underserved populations by assessing the overall BDS market;§ The sustainability and cost-effectiveness of BDS provider organizations; and§ Impact on SEs.

This report concerns the second goal above: the sustainability and cost-effectiveness of BDSprovider organizations. In the context of the PMF, sustainability refers to the financialviability of delivering services to SEs. The PMF considers the sustainability of the institutionsupplying the services, and then the profitability of the service itself. Cost-effectivenessrefers to the overall cost of the BDS program compared with the results achieved. It measuresthe benefits of investing public funds and the efficiency of the activities that support BDSdelivery. In its work, the PMF field-research team attempted to address questions about costallocation as well as the application and interpretation of the sustainability and cost-effectiveness indicators of the PMF.

The indicators selected and the general methodology recommended were as follows:

Table 1: Sustainability and Cost-Effectiveness Indicators and Methodology

Goal: Achieve Supplier Sustainability Indicator Methodology

BDS supplier financialsustainability

Financial sustainability = nondonor revenues / total expenses

BDS contribution margin BDS1 contribution margin = (SE revenues from BDS1 – BDS1direct expenses) / total expenses

BDS2 contribution margin = (SE revenues from BDS2 – BDS2direct expenses) / total expenses

BDS viability BDS1 viability = revenues from BDS1 / direct expenses for BDS1BDS2 viability = revenues from BDS2 / direct expenses for BDS2

iv

Goal: Improve Program Cost-EffectivenessIndicator Methodology

Annual programexpenses percustomer served

1) Conduct an impact survey and identify the fiscal year most closelycorresponding to the period of the impact survey. Calculate the annualprogram expenses for that fiscal year for the BDS initiative for which youare tracking impact.2) Calculate the ratio: annual program expenses / number of customersserved that year.

Annual programexpenses persupplier assisted

Annual program expenses / number of suppliers served that year(if you are the only direct provider in your program, the number ofsuppliers is one).

Annual programexpenses per firmthat “used” a BDS

1) Estimate the number of firms that used a service that year bymultiplying the percentage of firms reporting having “used” a service bythe total number of firms assisted that year.2) Divide the annual program expenses by the estimated number offirms that used the service (expenses / number of firms).

Annual programexpenses per firmthat reported“benefiting from” aBDS

1) Estimate the number of firms that benefited from a service that yearby multiplying the percentage of firms reporting having “benefited from” aservice by the total number of firms assisted that year.2) Divide the annual program expenses by the estimated number offirms that used the service (expenses / number of firms).

Simplified cost–benefit assessmentcomparing annualprogram costs withaggregate annualprogram benefits toentrepreneurs

1) Estimate the total profits for firms assisted that year that may beattributable to the program. To do this, first determine the estimatednumber of firms that attribute change to the service by multiplying thenumber of firms served that year by the percentage in the impact surveythat attribute profit change to the service. Then, multiply the averageannual increase in per-firm profits by the estimated number of firms thatattribute change to the service.2) Calculate the ratio of aggregate annual benefits (profits) to annualprogram expenses (benefits / cost). Express it in a ratio and inpercentage terms.

The PMF field-research participants reported the following program results using the PMF:

§ One of the six participating programs supports financially sustainable institutions, severalothers support institutions that are steadily increasing their financial sustainability, andtwo have been struggling to redirect their strategy from one of subsidy to one ofcommercially driven services.

§ In four of the six programs, the revenues for services are significantly greater than directcosts for delivering the services, indicating that the services themselves are financiallyprofitable.

§ Contribution margins among the programs are low, indicating the challenge of coveringhigh overhead costs.

§ No conclusions could be drawn from the cost-effectiveness data.

v

The PMF field-research team came to the following conclusions regarding the researchquestions about performance assessment technique, methodology, and interpretation:

§ In general, the indicators for sustainability are valid, practical, and useful.

§ It was helpful to have both contribution margin and service viability figures.

§ Allocation of overhead costs, direct services, costs, and appropriate revenue streams wasfacilitated by accounting systems that focus on cost and revenue allocation by programand by services for the purpose of profitability analysis.

§ Participants used a fairly standard definition of direct cost for services, which includedtechnical-staff fees and related expenses, materials, venue, and equipment.

§ Once the cost data were allocated appropriately, the calculation of cost-effectivenessindicators was simple.

§ The cost-effectiveness indicators that allocate cost per participant were unrevealing.

§ The simplified cost–benefit ratio is of questionable validity.

In addition, the PMF field research raised issues that remain to be addressed, including:

§ How realistic is it to expect that BDS facilitators and suppliers will have or put thesesystems in place?

§ What is the best strategy for overhead cost allocation?

§ Often, suppliers receive contracts from donors or other nongovernmental organizations(NGOs) to (a) provide services to SEs and (b) provide technical advice to their BDSprograms. Also, some suppliers depend heavily on revenue from voucher programs. Howshould revenues and expenses be treated in these different situations in order to calculatefairly sustainability and cost-effectiveness?

§ When supporting private-sector suppliers, how detailed must one get?

§ When calculating average sustainability ratios, how can these best be interpreted?

§ To what extent do the cost-per-person ratios reflect internal efficiency? Are there otherratios that would be more useful (for example, ratio of direct service staff toadministrative staff)?

The simplified cost–benefit ratio will be revisited when PMF field researchers report theirrepeat impact surveys and have more valid impact data to use. Finally, one participant raisedthe issue that sustainability should be considered at the market level rather than at theinstitutional level. What strategies might be explored to assess market-level sustainability?

Chapter One—Background and Purpose

1

CHAPTER ONEBACKGROUND AND PURPOSE

This progress report on the BDS sustainability and cost-effectiveness indicators is part ofa global initiative to identify valid, practical, and useful performance indicators forbusiness development services (BDS) programs. The indicators have been selected frombest practices studies and consensus-building global conferences and assembled into aperformance measurement framework (PMF). The PMF indicators reflect the followingcommon goals of most BDS programs:

§ Increasing outreach and reaching large numbers of people through the developmentof vibrant, private-sector BDS markets;

§ Delivering services through sustainable suppliers and cost-effective programs; and

§ Having a positive impact on client firms.

This PMF is not intended as a holistic BDS program evaluation or managementinformation system. Rather, it is a snapshot of performance indicators that could be usedto compare programs. As such, it could provide managers with useful data to improveservices to small enterprises (SEs), identify best practices, and assist in establishingprogram funding and performance guidelines.

The purpose of the PMF field research, which took place from October 1999 toSeptember 2001, is to explore the validity, practicality, and usefulness of such aframework. Six organizations that facilitate or provide a wide range of BDS to a widerange of clients in various countries are working together with three BDS experts and anadvisory committee of experts and donors to:

1) Develop common questionnaires and methodologies for the PMF 2001;

2) Apply these in their programs; and

3) Explore fundamental questions and issues regarding the goals, interpretation,application, and relevance of the PMF.

The third activity above was added as a result of dialogue about the PMF at the DonorCommittee on Small Enterprise Development conference in Hanoi, Vietnam, in April2000.

2

LIMITATIONS OF THE PMF RESEARCH

The research process was limited in scope, resources, and time frame in the followingways:

§ Given the short duration of the field tests, the effectiveness of the PMF in showingthe impact of BDS on markets and programs over time is unknown.

§ Many members of the research team applied the PMF only to particular services orgeographic areas of their programs.

§ The field-test organizations consisted of a small number of nongovernmentalorganizations (NGOs) in a small number of markets.

§ The PMF 2001 has not been tested with programs that were originally designedexplicitly to develop BDS markets.

§ Given the limited outreach of some of the field testers, it is not clear how effective theindicators will be in helping market development programs track the market-levelimpact of their interventions.

§ The PMF may not satisfy the needs of donors, who were only advisors to the PMFfield research.

§ The PMF follows the market development approach, which does not consider thefinal goods and services market in which SEs operate.

WHICH TYPE OF BDS PROGRAMS SHOULD USE THE PMF 2001?

The PMF was field-tested by a wide range of institutions that in many ways reflect thecurrent BDS field: they are programs in transition from a traditional approach to a BDSmarket development approach. One institution, for example, was transitioning from beinga supplier to serving as a facilitator of services. These programs:

§ Operate in three key global regions—Latin America, Asia, and Africa;§ Serve rural and urban businesses ranging in income and size from rural household

activities to medium-sized, urban manufacturing firms;§ Are managed by NGO suppliers and NGO facilitators;§ Work with both NGO and private-sector suppliers that range from sophisticated

business associations to individual rural entrepreneurs;§ Provide a range of training, technical, and marketing services; and§ Generally charge clients fees for services in order to recover costs or earn profits.

Thus, it is expected that the PMF 2001 would be widely applicable to the types ofprograms and services represented by the PMF field-research programs. However, asnoted above, the field research did not include programs explicitly designed to developBDS markets. The PMF 2001 has yet to be tested with programs such as the following:


3

§ Those that support many suppliers in one market to deliver many services,§ Embedded-services programs, and§ Voucher programs.

In addition, the PMF does not look at BDS product or service markets. The marketdevelopment guidelines and the corresponding structure of the PMF do not reflect manyof the goals and strategies of subsector development or systems development programs.1

The broad goal of developing BDS markets is to develop vibrant and competitive,primarily private-sector markets of relevant, differentiated services consumed by a broadrange and significant proportion of small businesses. One key element of developingBDS markets is ensuring the sustainability of service delivery from both an institutionaland a financial point of view. From an institutional perspective, the market developmentapproach recommends separation of financially viable, commercial delivery activitiesfrom broad market development activities for which, in general, it is difficult to recovercosts. For example, small, private-sector machine manufacturers may be able to sellimproved water pumps at a profit, but they may not be able to understand research anddevelopment of the pump technology itself. Thus, research and development may be anappropriate role for a nonprofit “facilitator” organization, while the commercial machine“supplier” focuses on the manufacture and distribution of water pumps. It is alsoenvisioned that the facilitator roles are temporary, whereas the supplier roles are intendedto be sustainable over the long run. It is the sustainability of the BDS supply activitiesthat the PMF intends to assess. Nevertheless, the facilitation activities also need to beassessed. Although they are not expected to be sustainable, they are expected to be cost-effective; that is, to deliver significant results given the public investment made. ThePMF, therefore, examines the sustainability of BDS suppliers and the cost-effectivenessof the BDS facilitation activities of BDS programs.

Finally, whether or not it is a goal of the program to become financially sustainable, it isimportant to assess financial performance, because it reflects the extent to which servicesare subsidized and may be crowding other, private-sector suppliers out of the market.

PMF RESOURCES AND DOCUMENTS

A documented set of resources was used in the PMF field-research effort.2 Theseinclude:

1) Background documents that describe the process of developing the PMF prior tothe PMF field research;

1 Dawson, Jonathan, Eric Hyman, Sunita Kapila, and Donald Mead, “Methodologies for the Design and

Delivery of High Impact Business Development Services (BDS) for Small Producers,” draft,International Development Research Council (IDRC), January 2001.

2 For information on resource documents, see Annex A.

4

2) Detailed guides, including questionnaires and reports, that the PMF field-researchteam tested; and

3) Interim reports that summarize the field-testing of each section of the PMF, thefindings, the conclusions, and recommendations.

The guides, published as part of the PMF field research, may be found on theMicroenterprise Innovation Project Web site (www.mip.org) and in the online SEEPGuide to Business Development Services and Resources(www.seepnetwork.org/bdsguide.html).

THE GOALS OF ASSESSING SUSTAINABILITY AND COST-EFFECTIVENESS

The broad goals of sustainability and cost-effectiveness focus on the performance of theinstitutions that supply or provide BDS to micro-, small, and medium-sized enterprises.The two goals are related but differ substantially. In the context of the PMF, thefollowing are working definitions:

§ Sustainability: the ability of an institution to offer services over the long term; thefinancial profitability of a BDS supplier; and/or the financial viability of a particularBDS. The terms “cost recovery,” “sustainability,” and “profitability” measure thesame thing but express different results. In other words, one program may cover only40 percent of its costs, another may achieve financial sustainability, and still anothermay generate profits, but one can use the same ratio of expenses to revenues tomeasure these different outcomes.

§ Cost-effectiveness: program expenses compared with program results or outcomes,such as the number of people served, the number of people who benefit from theservice, and so on. The PMF set of cost-effectiveness indicators also includes asimplified cost–benefit ratio.

§ Cost versus benefit compares program costs with program benefits (for example, theprofits generated in client businesses as a result of the service).

Sustainability focuses on whether a service will be available to businesses in the long run.The assumption is that financial sustainability is the best indicator of whether aninstitution will be able to offer services over the long term. In addition, by assessingfinancial sustainability, the framework attempts to assess the subsidy level of the serviceand, by inference, its potential for crowding other services out of the market. Finally, it isassumed that most suppliers, particularly private-sector suppliers, are concerned withfinancial sustainability or profitability.

The PMF looks at two types of sustainability: (1) the sustainability of the service itself:To what extent are SEs able and willing to pay for the cost of the service? How profitableis the service? and (2) the sustainability of the BDS supplier: To what extent is thesupplier institution (whether for-profit or NGO) profitable or able to cover its costs?


5

For many BDS practitioners and donors, sustainability is a sufficient indicator of success.If SEs are willing to pay for services and the services can be supplied profitably, then theservices have become part of the market, and that equals success. However, manypractitioners and donors are equally concerned about the impact of services, and whetherthe impact is being delivered for a reasonable cost. The PMF attempts to address thisquestion with its cost-effectiveness indicators. This set of indicators considers totalprogram costs and compares them with program outcomes and benefits, although the setby no means produces a thorough cost–benefit analysis. Rather, the cost-effectivenessindicators proposed here are simplified indicators that may be appropriate proxyindicators for true cost–benefit analysis, which is generally too complex for mostpractitioners to carry out.

Increasingly, there are institutional separations between BDS suppliers—organizationsthat directly serve SEs and hope to become sustainable—and facilitators—nonprofitorganizations that support suppliers by providing training, market research, and otherservices. In programs with this structure, sustainability is usually a goal of the supplierbut not of the facilitator. Nonetheless, cost-effectiveness analyzes the facilitator’sfinancial performance compared with the results achieved by the program.

The application and interpretation of the sustainability indicators and the cost-effectiveness indicators depend on the application and interpretation of other parts of thePMF. The financial sustainability indicator is intended to be interpreted in the context ofthe other performance indicators, particularly program scale and impact. The cost-effectiveness indicators depend on the collection of impact data and are thus also to beinterpreted in the broader context of the PMF. These indicators contribute to the PMF anunderstanding of the institutional performance of the BDS organizations involved insupplying a service: are the service and the supplier financially sustainable? What is thecost of the program relative to the results achieved?

RESEARCH QUESTIONS

The PMF initiative is examining particular issues in each performance category as thefield-research team tests the application of the sustainability and cost-effectivenessindicators. In general, the PMF field research examined whether these indicators are validand practical, how they may be interpreted and applied, and whether they lendthemselves to fair comparisons of program performance. With regard to sustainabilityand cost-effectiveness in particular, the following are the key research questions the field-research team is exploring:

Sustainability:

1) How easy is it for BDS programs to allocate operational costs and revenues tospecific services, programs, or SE clients? How standard and fair can theseallocation categories be?

6

2) Is cost recovery a good proxy indicator for impact?

Cost-effectiveness:

1) How easy is it for BDS programs to allocate cumulative program costs,particularly overhead, to specific services, programs, or SE clients? Howstandard and fair can these allocation categories be?

2) How valid is the simplified cost-effectiveness analysis based on the impactdata collected using the PMF methodology?

3) How valid and useful are the simplest cost-effectiveness indicators: cost perfirm (customer) assisted and cost per supplier assisted?

Thus, the major areas of concern for both sets of indicators are cost allocation and howthe indicators may be interpreted and applied.

INDICATORS AND RATIONALE

The following two tables summarize the sustainability and cost-effectiveness indicators,the corresponding methodologies that the PMF field-research team tested, and how theindicators should be interpreted. All indicators rely on internal financial figures, financialdata from BDS supplier partners, and data from the PMF field-research impact andmarket development surveys. No new surveys were carried out, although someorganizations collected data from the BDS suppliers they support. Also, the PMF fieldtesters worked closely with their accounting departments to gather and report accuratedata.


7

Table 2: Sustainability Indicators, Proposed Methodologies, and Interpretation

Goal: Achieve Supplier SustainabilityIndicator Proposed Methodologies Interpretation

BDS supplierfinancialsustainability

Financial sustainability = nondonorrevenues / total expenses

The financial sustainability ratioanswers the question: Lastyear, what percentage of totalexpenses of a BDS supplierwas covered by nongrantrevenue?

BDS contributionmargin

BDS1 contribution margin =(SE revenues from BDS1 – BDS1 direct expenses) /total expenses

BDS2 contribution margin =(SE revenues from BDS2 – BDS2 direct expenses) /total expenses

And so on.

The contribution margin of eachBDS answers the question:Last year, what percentage oftotal expenses was covered bythis particular BDS? Whencomparing different BDS, itanswers the question, WhichBDS is more profitable for myinstitution?

BDS viability BDS1 viability = revenues from BDS1 /direct expenses for BDS1

BDS2 viability = revenues from BDS2 /direct expenses for BDS2

And so on.

This ratio answers the question,How profitable is a particularBDS? Last year, whatpercentage of the directexpenses to deliver a BDS wascovered by the revenuesgenerated from that BDS?

Rationale: As one advisor aptly put it, in a time when BDS practitioners are attemptingto support private-sector BDS markets, it is best to use existing business ratios rather thaninvent complex, donor-related ratios. So, rather than attempt to calculate a sustainabilityratio that excludes specific donor-related costs, the organization-level indicator simplyasks for the profitability of the institution—the bottom line, including all overhead,donor-related costs, depreciation, and so on. 3 These costs are then compared withnondonor revenue, which should primarily be revenue from SEs, with occasionalconsulting income from other commercially driven suppliers.

On the service level, the framework asks for two indicators: service contribution marginand service viability. The former is most helpful for BDS program managers, as it helpsassess the extent to which different services are helping to achieve sustainability orprofitability. However, this indicator is not very comparable across organizations,because it does not address the profitability of the service. Rather, it asks what percentageof overhead is being covered by profits from the service. The service viability indicator,which compares the expenses of a particular service with its revenues, reports the service

3 If you feel strongly that your true institutional sustainability is better reflected by calculating a

sustainability ratio that excludes “donor-related costs” as you define them, you may also report this ratioas you usually calculate it, providing details on which expenses are included and which are excluded.This would be in addition to reporting the sustainability ratio described here.

8

profitability, excluding overhead and indirect costs. This ratio is helpful for looking at aBDS outside the context of its organizational home. Although it is difficult to standardizethe method of allocating “direct” costs, when used in conjunction with the aboveindicators, which capture total costs, the viability indicator should be useful indetermining whether a service is financially viable. The other sustainability indicators,meanwhile, reflect whether the service is profitable within the institution offering it.

Also noteworthy is determining which types of institutions should assess thesustainability ratios. There are generally four types of BDS providers or suppliers thatmay treat these indicators differently:4

§ Supplier: an NGO or government agency that supplies BDS directly to SEs, andwhose main activity is supplying BDS to SEs. In this case, the sustainability ratiosapply to the entire institution.

§ Supplier plus: an NGO or government agency that supplies BDS directly to SEs butalso supplies other services, to SEs or other clients. In this case, the sustainabilityratios apply to the supplier’s BDS program, but an appropriate allocation of overheadcosts for the support provided by the supplier should be included in the calculation oftotal expenses.

§ Facilitator: an NGO or government agency that supports other NGOs or private-sector BDS suppliers. In this case, the sustainability ratios apply to the partnerssupplying the BDS.

§ Facilitator/supplier: Finally, many organizations do both—support BDS suppliersand serve SEs directly. In this situation, the sustainability ratios will apply to the mainBDS organization and its partners. Separate calculations will apply for eachinstitution.

In addition, most programs report sustainability data for several services, and some reportsustainability data for several institutions. In these situations, service- and institutional-level data are essential for program managers. However, for reporting at the programlevel using the PMF, it is necessary to calculate averages. The following averages andguidelines are recommended:

§ For organizations working with more than one supplier, report the number ofinstitutions that are more than 100 percent sustainable compared with the overallnumber assisted (for example, two of four are viable), and then report the averagesustainability ratio. (Calculate the sustainability ratios for all programs and thenaverage the ratios.) If the facilitator supports numerous suppliers (more than 20, forexample), it may take a random sample of suppliers to report.

4 The PMF uses the term “supplier” to mean any institution that provides a BDS directly to an SE.

Although it is essentially the same as a “provider,” the PMF team prefers the more private-sector–oriented term.


9

§ For suppliers supplying more than one service, calculate the service-levelsustainability ratios per service at each institution. Then, average these for eachsupplier. Then, average the ratio for all suppliers. Alternatively, if the suppliers allprovide the same service, the average viability ratios can be calculated acrossinstitutions.

The proposed cost-effectiveness objectives and indicators are as follows.5 Unlike thesustainability indicators, which are private-sector–oriented, cost-effectiveness indicatorsare focused on analyzing the investment of public funds, and determining the value ofbenefits generated by this investment. They ask the question: Was the money spent todevelop and deliver this service worthwhile to the businesses?

Rationale: Significant numbers of practitioners and BDS experts argue that sustainabilityalone is an insufficient indicator of program performance, and that sustainability is notalways a good indicator for impact. Rather, in markets with few suppliers and fewservices available, entrepreneurs will pay high sums for services that may not, in the end,benefit their businesses. An example often cited is the large number of parents who sendtheir children to technical schools even in labor markets where most graduates oftechnical schools do not find jobs. In response, the PMF includes attempts at assessingimpact while analyzing impact compared with costs. Because it is meant to be practical,the types of impact indicators and the types of analysis must be simple. A thorough,technical cost–benefit analysis—comparing costs and benefits over time—was notviewed as feasible for most BDS organizations. Instead, the PMF has devised asimplified cost–benefit indicator and a series of cost-effectiveness indicators—indicatorsthat compare annual program costs with annual program benefits and outcomes. It isproposed that, collected over time, these indicators will form a valid, practical, and usefulproxy for in-depth cost–benefit analysis that could be performed on a more occasionalbasis.

The indicators look at total program costs for the year most closely corresponding to theyear the BDS organization conducted an impact assessment. Then, the program costs arecompared with the impact indicators: number of people acquiring, using, and benefitingfrom a service, and the profit increase that businesses attribute to the service. As with thesustainability ratios, the procedure for calculating the ratios varies depending on theinstitutional setup of the BDS program.

5 On the draft PMF that was agreed on at the PMF virtual conference, some suggested adding the provider

income indicator “total program cost per increase in provider revenue,” which has been eliminated. Fororganizations that are direct suppliers or providers, the calculation is not complicated, but there is roomfor error, and the indicator adds little value. For example, if one year a program generates 20 cents forevery dollar invested in program costs and the next year only 5 cents, what does this really tell us?

10

Table 3: Cost-Effectiveness Indicators, Proposed Methodologies, and Interpretation

Goal: Improve Program Cost-EffectivenessIndicator Proposed Methodology Interpretation

Annual programexpenses percustomer served

1) Conduct an impact survey and identifythe fiscal year most closely correspondingto the period of the impact survey. Calculatethe annual program expenses for that fiscalyear for the BDS initiative for which you aretracking impact.2) Calculate the ratio: annual programexpenses / number of customers servedthat year.

This figure represents the totalprogram cost of delivering a serviceto one client. How much does it costto serve one business? This figure iseasy to calculate, but used alonewould favor programs that servemany firms, but that may not have asignificant impact on those firms.

Annual programexpenses persupplier assisted

Annual program expenses / number ofsuppliers served that year(if you are the only direct provider in yourprogram, the number of suppliers is one).

This indicator illustrates the cost ofassisting each supplier. It should belooked at in the context of theprogram's effectiveness indeveloping the BDS market. This isan attempt to look at the relative costof that market development strategy.

Annual programexpenses perfirm that “used” aBDS

1) Estimate the number of firms that used aservice that year by multiplying thepercentage of firms reporting having “used”a service by the total number of firmsassisted that year.2) Divide the annual program expenses bythe estimated number of firms that used theservice (expenses / number of firms).

This figure tells us the annualprogram cost for each client whoapplied the service in his or herbusiness. It illustrates the cost ofproviding a useful service to a client.This number would normally exceedthe cost of serving a client.

Annual programexpenses perfirm thatreported“benefiting from”a BDS

1) Estimate the number of firms thatbenefited from a service that year bymultiplying the percentage of firms reportinghaving “benefited from” a service by thetotal number of firms assisted that year.2) Divide the annual program expenses bythe estimated number of firms that used theservice (expenses / number of firms).

This figure illustrates the annualprogram costs of supplying abeneficial service to a client. Thisnumber attempts to get at the cost ofhaving a monetary impact on aclient’s business, although it doesnot capture the depth of the impact.

Simplified cost–benefitassessmentcomparingannual programcosts withaggregateannual programbenefits toentrepreneurs

1) Estimate the total profits for firmsassisted that year that may be attributableto the program. To do this, first determinethe estimated number of firms that attributechange to the service by multiplying thenumber of firms served that year by thepercentage in the impact survey thatattribute profit change to the service. Then,multiply the average annual increase in per-firm profits by the estimated number of firmsthat attribute change to the service.2) Calculate the ratio of aggregate annualbenefits (profits) to annual programexpenses (benefits / cost). Express it in aratio and in percentage terms.

This is a simplified cost–benefitanalysis. It attempts to answer thefollowing question: In the mostrecent year evaluated, for everydollar spent on programdevelopment and operations, howmuch impact, in terms of profits toassisted clients, has the programgenerated? What was the socialreturn on the program investment forthe most recent year the programwas evaluated? For example, a ratioof 2, or 200 percent, would suggestthat every program dollar spentresulted in $2 of profits that theassisted clients reported as beingattributable to the use of the service.

Chapter Two—Summary of Results

11

CHAPTER TWOSUMMARY OF RESULTS

This chapter reports the financial sustainability and cost-effectiveness data gathered bythe PMF field-research team, and interprets the data from a comparative perspective. Thesix organizations participating on the PMF field-research team are:

§ CECI/MARD is an NGO agricultural development program in rural Nepal. It is afacilitator of agricultural training (supplied for free by community-based NGOs),marketing businesses (farmer cooperatives), and veterinary services (supplied for afee by individual entrepreneurs).

§ FAIDA is an NGO facilitator in Tanzania with support from SNV, the DutchVoluntary Organization. FAIDA supports management training services delivered bycommercially driven suppliers. The supplier reporting sustainability ratios for thePMF field research, FAIDA Ltd., is transitioning from subsidized to commercialBDS.

§ IEDI, an urban NGO in Nepal traditionally supported by the German developmentagency GTZ, is a business development organization transitioning from subsidized tocommercially driven BDS training services. IEDI is a supplier-plus organization;training to SEs is one of several of its programs.

§ SEEDS, a rural community development NGO in Sri Lanka, has a separate enterprisedevelopment department, further divided into microfinance and BDS. The BDSservices assessed here include technical training and management training. Clientspay a fee for the technical training, whereas the management training is supplied tomicrofinance clients at a charge to their microfinance institutions.

§ Swisscontact Peru is an international NGO that facilitates commercially drivenbusiness service centers that supply training services.

§ Swisscontact Philippines is an international NGO that facilitates commercially drivenbusiness service centers. The center participating in the PMF field research, OroSeCen, is a business association that supplies entrepreneurship development trainingand facilitates participation in trade fairs and exhibitions.

FINANCIAL SUSTAINABILITY

All organizations on the PMF field-research team are attempting to achieve some level ofcost recovery or sustainability, but are struggling to achieve break-even. Oneorganization (CECI/MARD) supports some profitable suppliers, but key components of

12

the program are delivered by NGOs that do not charge fees for services and, at themoment, do not strive for financial sustainability. Swisscontact in the Philippines and inPeru support several profitable suppliers and several who are still working towardprofitability. In general, the least sustainable programs are those NGOs supplyingservices directly that are making a transition from subsidized to commercially drivenservice provision. Some of these organizations are making progress in improving costrecovery, while others are facing market competition and are encountering declining costrecovery as a result.

At the BDS service level, four of the six organizations report highly viable services inthat direct costs are substantially lower than nondonor revenue for the services. The lowcontribution margins, however, highlight the challenge of covering high overhead costs.Although exact figures do not appear in the table below because the table reportsaverages only, the organizations note that some of their services are more viable and havehigher contribution margins than others. In one case, the service that is popular is not themost profitable, which presents an interesting strategic dilemma in terms of where tofocus marketing and delivery efforts.

Table 4: Financial Sustainability

BDS Supplier FinancialSustainability:

Nondonor Revenue /Total Program Expenses

Organizationand BDSOffered

Type ofInstitution(s)

Year 1 Last Year

Average BDSContribution Margin:(Nondonor Revenues

from BDS – BDSDirect Expenses) /

Total Expenses

Average BDSViability:

NondonorRevenues from

BDS /Direct Expenses

for BDSIEDI;training,six services

NGO provider 47% 43% -10% 65%

Three NGOproviders,*threecooperatives

117%(co-op

average)

120%(co-op

average)

26%(co-op average)

128%(co-op average)

CECI/MARD;Ag-marketing:training,marketing, andveterinaryservices

20 to 40entrepreneursuppliers

143%(average)

153%(average)

74%(average)

153%(average)

SEEDS;technicaltraining,mgmt. training

NGO provider,division oflarger NGO

17% 22% -5%(average)

57%(average of twoservices: 21%,

93%)FAIDA;training, threeservices

Facilitator,three suppliers,one reporting

21% 8% 2% 315%

SwisscontactPhilippines;mgmt. trainingand trade fairs

Facilitator,one supplier

90% 88% 20% 531%

Chapter Two—Summary of Results

13

SwisscontactPeru; mgmt.training andconsulting

Facilitator,sevensuppliers,four reporting

40%(average)

69%(average)

25% 120%

*CECI/MARD’s NGO partners do not charge for services.

COST-EFFECTIVENESS

It is difficult to come to comparative conclusions from the cost-effectiveness dataobtained. The annual program expense per client spans a surprisingly small range ($18 to$57), given the wide range of program budgets and scale (see Table 5). The cost perclient “using” and then “benefiting from” a service covers a wider range, which reflectsthe extent to which clients of different programs use and benefit from a service. The costper supplier assisted varies widely with no particular correspondence to program design,number of suppliers assisted, and so on. Only two organizations were able to report acost–benefit ratio, and both report very high returns on investment, 338 percent and 495percent.

Table 5: Cost-Effectiveness

EstimatedCost–Benefit

Ratio

Organizationand BDS Offered

AnnualProgramExpenses

perCustomer

Served

AnnualProgramExpenses

perSupplierServed

AnnualProgramExpensesper Firm

That“Used” a

BDS

AnnualProgram

Expenses perFirm ThatReported

“Benefitingfrom” a BDS

Year 2

IEDI Rs. 1,401(US$18)

Rs. 406,318(US$5,288)

Rs. 1,540(US$21)

Rs. 1,919(US$26)

N/A

CECI/MARD US$44 US$4,454 US$56 US$63 3.38 (338%)SEEDS US$31 US$307,000 US$51 US$51 4.95 (495%)FAIDA(numbers pending)Swisscontact Philip. US$57 US$11,234 US$59 US$59 N/ASwisscontact Peru US$56 US$32,440 US$65 US$105 N/A

Chapter Three—Methodological Issues and Recommendations

15

CHAPTER THREEMETHODOLOGICAL ISSUES AND RECOMMENDATIONS

In general, the sustainability ratios were practical, valid, and useful, but the cost-effectiveness measures were not very valid, were impractical, or were not useful,depending on the indicator.

FINANCIAL SUSTAINABILITY

The supplier and service sustainability ratios were fairly easy to calculate and wereviewed generally as valid and useful for PMF field researchers. Organizationalsustainability was simpler to determine than service sustainability, and organizations thatoffer multiple programs or serve multiple client groups face more challenges than otherorganizations. One challenge in determining sustainability was the lack of detailedaccounting systems that facilitate profitability analysis or even cost analysis. Mostorganizations in the field research use accounting systems focused on donorrequirements; namely, expense allocation based on source of funds.

Organizational Sustainability

Most organizations were able to calculate organizational financial sustainability forthemselves or their partners, as appropriate, and felt that the information was valid anduseful in assessing progress toward financial viability, which was a goal for allparticipants, regardless of target population. BDS service viability ratios were morechallenging to calculate, but when methods could be found to allocate costs and revenues,they were valid indicators of comparative service viability. It turned out that having bothratios and comparing them was helpful as well. Some issues and recommendationsfollow.

§ Nondonor revenue and nondonor expenses: For organizations that serve multipleclients (SEs and, for example, donors that contract the organization to serve SEs),figuring out how to deal with revenues and expenses from institutional clients was achallenge. Following the general guideline to exclude donor revenues, oneorganization (Swisscontact Peru) took out revenues from institutional contracts butleft in the related costs. In contrast, another organization, SEEDS, reported increasedsustainability as a result of consulting services presumably provided to other NGOs.In the first case, services to paying SEs may have been financially sustainable, butservices to all SEs were not. In the second case, services to SEs were not sustainable,but the program did better with its consulting services.

Issue: How should overall organizational sustainability be considered in situations inwhich donor or NGO contracts are received for service to SEs and for consultingservices?

16

§ Overhead allocation for multiprogram organizations: Multiprogram organizationswithout accounting systems that allocate central costs according to program facechallenges making sustainability calculations. For example, SEEDS provides servicesin several sectors and offers both microfinance services and BDS. Although theorganization’s direct costs were easily tracked, the overhead allocation was estimatedand provided without substantive backup notes. Thus, it would be difficult to compareSEEDS’s financial sustainability with that of other organizations. At the same time,because SEEDS is institutionally linked to other programs that may not be attemptingto become financially viable, SEEDS also faces institutional sustainability issuesquite apart from the financial sustainability of the program. In one interesting case,SEEDS was receiving payment from its microfinance program for delivering BDS toloan clients.

Issue: How should overhead costs be allocated transparently and fairly inmultipurpose organizations? Recommendation: Organizations attempting to achievefinancial viability or analyze cost-effectiveness need accounting systems that allocateoverhead in a standard and transparent fashion. Issue: In the absence of suchaccounting systems, what can the PMF team recommend? Also, when does revenuefrom institutional clients count?

§ Relevance to NGO programs: In one case, CECI/MARD, the facilitatingorganization, supported NGOs to provide free services. A sustainability ratio couldnot be calculated, and the PMF field researcher involved questioned the validity ofapplying the measure of financial sustainability in this situation.

Service-Level Sustainability

The ratios for both service viability and contribution margin were viewed as valid anduseful, and several organizations found the comparison of the two figures helpful. Five ofthe six organizations used very similar definitions of direct costs, which in generalincluded the following:

§ Trainers’ fees and expenses (transportation), or an allocation of staff expenses;§ Materials and services: photocopying, books, food, inputs, transportation, and the

like;§ Venue and equipment rentals; and§ Promotion costs.

Despite the simple and common definition, three of the six organizations found directcosts challenging to calculate because of inappropriate accounting systems. Nodifferences arose in the validity, practicality, or usefulness of the viability andcontribution margin measures, although participants seemed to have an easier timeinterpreting the viability ratio. Some issues and recommendations follow, mostconcerning cost allocation:


17

§ Calculating direct expenses by service category: Calculating direct expenses waschallenging for three of the six organizations. For example, FAIDA Ltd., a small,private-sector supplier, geared its accounting system to facilitate tax-record keepingrather than analysis by service. Therefore, FAIDA Ltd. could not report its directcosts by service. The supplier was, however, able to estimate its direct costs becauseit normally invoices FAIDA for direct costs on a per-client basis for services FAIDAcontracts. Organizations supported by Swisscontact Peru were able to allocate costsby either service or customer (institutional versus SE), but not by both. Thus,Swisscontact was unable to report service-viability figures using SE revenue. As aresult, it calculated the viability ratios of all BDS delivered to SEs. SEEDS simplyreported that allocating costs according to services was challenging because itsaccounting system does not allocate costs to that level of detail. In contrast,Swisscontact Philippines allocated costs easily, because it had assisted its partnerswith appropriate financial systems, as noted in the following quote fromSwisscontact:

“Calculations were greatly facilitated by Swisscontact Philippines’monitoring and reporting system, which has been functionally installedwithin its program-assisted BDS providers. The [Swisscontact] financialmonitoring system allows for classification of costs into direct and indirectcosts, which facilitated direct extraction of data from the BDS suppliers’financial statements. This is complemented by quarterly external financialaudits.”

Despite the challenge of allocating overhead to the BDS program overall, IEDI alsocommented on the ease of calculating direct expenses by service because of theavailability of an appropriate accounting system. CECI/MARD was able to calculateservice-level sustainability because its different suppliers provide different servicesand specialize in those services.

Issues: With private-sector suppliers, to what extent is it necessary and feasible toanalyze profitability by service for the PMF? With NGO suppliers, what standardcost-allocation system can be applied, if any? Recommendation: These ratios areeasiest to calculate when accounting systems are geared toward profitability analysisand are detailed enough to allow for service-level analysis.

§ Defining direct costs: In one case, an organization (SEEDS) attempted to allocate awide range of personnel costs by service type, costs that existed in other organizationsbut were considered indirect costs.

Issue: Is the limited definition of direct costs valid for most services and mostorganizations?

§ Revenue allocation: In general, revenue allocation by service and SE client wassimpler than cost allocation, but two organizations reported challenges. Organizations

18

supported by Swisscontact Peru had the same problem with revenue allocation aswith cost allocation; that is, they could allocate revenues by payer or by service, butnot by both. Thus, Swisscontact was unable to isolate revenues from SEs by servicetype. Additionally, FAIDA did not keep accounts by revenue source, so the PMFresearchers literally went through receipts to calculate its SE revenue.

Issue: To what extent can we expect accounting systems to allocate revenueaccording to source and service type?

§ Gathering data from multiple suppliers: One organization, CECI/MARD, supportsmore than 40 suppliers. Collecting profitability figures from all of them posed alogistical challenge. So, CECI/MARD took a random sample. The sample was madeup of suppliers located fairly close to the CECI/MARD office because of securityconsiderations in traveling to the more remote areas served.

Issue: How valid is the sustainability ratio when calculated in this instance? Canfacilitators of many private-sector suppliers be expected to gather data from allsuppliers?

§ Average sustainability ratios: For facilitators supporting numerous suppliers, anaverage sustainability ratio was calculated.

Issue: Which averages are valid and which raise issues, and what are these issues?Would it be more useful instead to calculate the percentage of suppliers that aresustainable?

COST-EFFECTIVENESS

In general, the cost-effectiveness ratios were fairly simple to calculate but were not veryuseful. Most organizations on the PMF research team could not calculate the overallcost–benefit ratio because most had not completed the second impact survey and couldnot make substantive comments on its viability.

Cost Allocation

For all the cost-effectiveness ratios, organizations were asked to calculate the totalprogram cost to the primary implementing organization. If the organization was afacilitator, the costs of the direct suppliers were not taken into account. If an organizationwas a multiprogram or -sector organization, direct and overhead costs needed to beallocated to the program. Only multisector or multiprogram organizations faced achallenge in calculating overall program costs. For example, FAIDA supports threeprograms. Although its accounting system allocates direct costs by program,administrative, management, and monitoring and evaluation costs are not allocated byprogram. Rather than estimate an appropriate allocation, FAIDA calculated cost–benefit


19

ratios using total costs, which dramatically exaggerated its costs. In addition is theaforementioned issue with the SEEDS program regarding sustainability ratios; namely,its accounting system does not allocate indirect costs according to program but, rather, isoriented toward donor-fund accounting.

Issue: A prerequisite for using the PMF is having an accounting system geared towardcost analysis by program. Is this a realistic expectation?

Cost per Person Acquiring, Using, and Benefiting from a BDS

Once total program costs were determined, calculating the cost per person acquiring,using, and benefiting from a service was simple, using data from the impact survey.Many participants commented on the usefulness of this information, particularly forcomparisons with other programs. However, in looking at the other programs,comparisons are not particularly enlightening. Comparisons might be more appropriatewithin a country or even within a smaller geographic environment and with similarservices. One participant commented that the comparison of cost per person acquiring aservice versus the cost per person benefiting from the service was enlightening, but notedthat the percentage figures from the impact survey provide the same information.

Issue: Which is more valid and useful, cost per person acquiring a BDS or cost perperson benefiting from a BDS?

Cost–Benefit Ratio

If calculated in a valid manner, this ratio can be very useful. However, in general,participants questioned its validity because they questioned the validity of the impactdata. The ratio has become fairly simple to calculate, but the simplifications furtherreduce its validity. This ratio will be reconsidered when more impact data are available.

Chapter Four—Areas for Further Study

21

CHAPTER FOURAREAS FOR FURTHER STUDY

The PMF field research as conducted to date suggests the following areas for furtherdiscussion and research:

§ Successful calculation of the PMF relies on accounting systems that allocate revenuesby source and costs by program and by service to facilitate profitability analysis byprogram and by service. How realistic is it to expect that BDS facilitators andsuppliers will have or put these systems in place?

§ What is the best strategy for overhead cost allocation to BDS programs and servicesin multiprogram institutions or “supplier-plus” institutions?

§ Often, suppliers receive contracts from donors or other NGOs to (a) provide servicesto SEs, and (b) provide technical advice to their BDS programs. Also, some suppliersdepend heavily on revenue from voucher programs. How should revenues andexpenses be treated in these different situations in order to calculate fairlysustainability and cost-effectiveness?

§ When supporting private-sector suppliers, how detailed must one get? For example,do we need service-level viability indicators? Do we need data from every supplier?How realistic are those expectations? Is it more useful to know the percentage ofsuppliers assisted that are profitable, and the percentage that are still supplying after acertain amount of time?

§ When calculating average sustainability ratios, how can these best be interpreted?Which are more valid averages, and when is it better simply to look at the originalratios?

§ Is the definition of direct costs used in the PMF field research valid for most servicesand organizations?

§ To what extent do the cost-per-person ratios reflect internal efficiency? Are thereother ratios that would be more useful (for example, ratio of direct service staff toadministrative staff)?

§ To what extent do the sustainability and cost-effectiveness indicators andmethodology presented in this paper hold for market development programs? Howwould programs explicitly designed in line with the donor guidelines adapt theframework to test the indicators and methodologies further?

22

§ Finally, one participant raised the issue that sustainability should be considered at themarket level rather than at the institutional level. What strategies might be explored toassess market-level sustainability?

A-1

ANNEX A

BDS PERFORMANCE MEASUREMENT RESEARCH:A QUICK ROAD MAP

A-3

BDS Performance Measurement Research: A Quick Road Map

Research Paper Key Questions Go Here If You Want To...

• How did the PMF evolve? Who was • Understand the research and consensus- involved and how? building process that led up to the PMFk field research.

• What were some of the issues and research • Understand key issues that were resolved related to the PMF prior to the field research? prior to the PMF field research.

• Understand why some indicators were eliminated prior to the field-research phase.

• What is the purpose of assessing impact, outreach, • Better understand the rationale of each part sustainability, and cost-effectiveness? of the PMF and the corresponding indicators.

• How are the PMF indicators defined, and how should • Better understand the tools and techniques they be interpreted? that the PMF field-research team used.

• What tools and methodologies were used by the PMF • Find tools and methodologies that can be field-research team? adpated for application for further research.

• What were the main issues that the PMF field research • Better understand the findings and recom- was exploring and testing? mendations of the PMF field research.

• How did the selected indicators and methodologies • Understand how the guides might need to be help PMF field researchers better understand their adapted for application and further testing of programs’ performance? the PMF.

• To what extent did the indicators and methodologies • See additional illustrations of how some of the produce practical, valid, and useful performance PMF indicators might be applied in practice. information?

• What is the purpose of the PMF, and what was the • Understand the field-research purpose, purpose of the PMF field research? process, and results.

• What were the results of the PMF field research? • Find the latest version of the PMF.

• What are the limitations of the current PMF? How • Understand key issues in BDS performance might it be applied to performance assessment, and measurement, and remaining challenges in how might it be used for further research? developing practical, valid, and useful

performance indicators.

• What goals does the PMF 2001 measure?

• What are some examples of how field-research programs used the PMF?

• What further research strategies and topics does the PMF field research recommend?

The above documents can be accessed through the following Web sites: www.mip.org andwww.seepnetwork.org.

Progress Reports

• Impact Indicators Survey• Outreach and MarketDevelopment Indicators Survey•Sustainability and CostEffectiveness Indicators Survey

BackgroundDocuments for PMF

Field Test

Synthesis Report

• Technical Note on theResearch Findings of thePerformance MeasurementFramework Action Research

Survey Guidance