uadp ia training

8/7/2019 UADP IA training

1/22

INTERNATIONAL FINANCE CORPORATIONINTERNATIONAL FINANCE CORPORATION

Workshop on IMPACT

ASSESSMNENT ANDRESULTS EVALUATION:

Developed by Alexander Kobzev

May 24, 2003

IFC KIEV


2/22

Introduction: anticipated results of this training

Upon completion of the training Ukraine Agribusiness DevelopmentProject:

UADP team members would know how IFC monitoring andevaluation system is working

Appropriate team members would fill in monitoring tool correctlyand use information

Input of UADP team members will be incorporated in monitoringtool

Any other expectations?


3/22

TA MONITORING AND EVALUATION WAS DESIGNED TO MEETTWO KEY OBJECTIVES:

To evaluate project progress and learn from best practices

To demonstrate projects transparency and to promote results toPEPs donors and IFC worldwide


4/22

Project Design

Clearly definegoals andobjectives

(Keep in mindhow you willmeasure theirachievement )

SOM, IAM

Suggest ourapproach or meetspecific donorrequirements

SOM

Agree measurableresults of the project(through M&E)framework

- Verify goals andobjectives- Define actions, timing- Set targets

SOM, IAM, PM and keyteam members(if appropriate)

Baseline datacollection at thevery early stage

Project team orexternal source

Timely collectprogress data andadjust project plan(if appropriate)

Appointed teammember underguidance of PM andSOM

Critical steps

Responsibility

FundraisingProject

PlanningAction

Monitoring &Evaluation

REQUIRED STEPS AT VARIOUS STAGES OF PROJECT LIFE


5/22

FRAMEWORK FOR RESULTS MONITORING AND EVALUATION

Goal Objectives Activities Output Outcome Impact

What we areaimingto change

To increasefinancing to SMEthrough thedevelopment of

leasing as afinancial instrument

What should beimplementedto achieve the

desired change

To improve/increase leasingservice

To improveknowledge andattitude of potentialcustomers ofleasing service

To provide legalassistance...

Our physicalactions

To deliver 24trainings,2 bookletspublished, 5000

copies each

-Direct results fromthe Activities- Have effect

immediately or in ashort term periodduring the projecttimeframe

600 trainedemployees of financialinstitutions

75% of participantshad significantly

improvedunderstanding ofleasing transactions(measured by test)

- Anticipated effect- Can be observedduring the project life

and within 1-3 yearsafter completion- Measures theachievement ofObjectives

-Increased number offinancial institutionsproviding leasing service-At least 4 as a directoutcome of the project

- Desired final change- Has effect within 1-10 years after

completion, rarelyduring the projecttimeframe- Measures theachievement of Goal

Leasing companiesgrowing their business

More leasing finance isavailable to SMEs

Definition

Example


6/22

FIRST TOOL USED TO STRUCTURE PROJECTS AND TO SET TARGETSE.g. Technical Assistance for Sugd Agro Serv

ACTIVITIES: ACTIVITIES TARGETS:EXPECTED

OUTPUTS:OUTPUT TARGETS:

EXPECTED

OUTCOMES:OUTCOME TARGETS: EXPECTED IMPACT: IMPACT INDICATORS:

Priority A

Deliver trainings/

consultations to SAS

shareholders andfarmers in the broader

Sugd province

24 sessions of the basic

course (basic course =60

hours) and 8 sessions of

advanced course (advanced

course =100 hours)

a) crop production

b) sound environmental

practices, including irrigation

management

c) occupational health and

safety

At least 400 participants

have received basic

training and at least 100 -

advanced training in:

a) crop production

b) sound environmental AG

practicesc) occupational health and

safety

80% of participants have

passed test w ith a pass

mark of 75%

On all SAS shareholder farms

w ater table is lower than 2

meters at all locations by 2007

SAS area of salinized land does

not increase

Increased SAS area under active

use of biological control systems

Full adoption of integrated pest

management by SAS

Set up and maintain

demonstration field

Establish 25 hectares demo

site, involving other supplier

companies in the region and

governmental agencies

Provide field training

and advice to SAS

members

8 field days a y ear, (20 in

total) covering

a) laser levelling

b) irrigation

c) seeding

d) tillage

e) harvest management

Model is replicated in

other Tajik and CIS areas # of replications

SAS shareholders

and farmers in the

broader Sugd

province improve

their knowledge of

modern crop

production practices

Improvement in:

1.1 Salinity

management

1.2 Pest

management

1.3 Fertilizers use

1.4 Irrigation

1.5 Crop

diversification

1.6 Safety of

farming

At least 40 participants per

field day (~800 in total)

a) laser levelling

b) irrigationc) seeding

d) tillage

e) harvest management

Objective 1: To introduce modern agronomic pr actices

Whatare we actually doing? hatare the immediateresults of ouraction Whatshould be implemented?

All SAS hectares receive

accurate timimg, rate and

placement of fertilizers by 2006

30% reduction in water use per

SAS hectare by 2006

Flexibility in crop production is

resulting in profitable operations

each year

Decreased accident and mortality

rate

Poverty is reduced

through:

Crop productivity gradual

increase

Farms earn more income

and become more

profitable

Farmers debt burden is

servicable

Farmers earn more for

their f amilies

Crop yield per hectare

increases by 50% by the

year 2010

Average net incomeincreases to 100USD per

hectare

Loan defaults less tha

3%

Debt per hectare is

servisable

Income per farm w orker

increased to 20 USD per

month by 2007

Where itshould bring us atthe very end?


7/22

SECOND TABLE USED AS A PLANNING TOOL FOR MONITORINGE.g. Central Asia Leasing Project

II 2002 III 2002 IV 2002 I 2003 II 2003 III 2003

Develop and deliver

customized training in

each country to

government officials and

businesses

# of customized training modules

# of seminars for government officials

# of seminars for financial institutions

# of seminars for lessers accountants

# of seminars for lessees

Internal reports Quarterly Team assistasnt 9-12 (total)

40

50

20

40

4

2

0

1

1

Provide individual

consultation to lessers,

lessees, potential

lessees and lessers,

local commercial banks,

microf inance institutions

# of hours of consultation provided

# of consultations provided

Internal reports Quarterly Team ass is tasnt 1200

1000

55

II 2002 III 2002 IV 2002 I 2003 II 2003 III 2003

# of trained participants:

~300 governmental of ficials

~700 fin institutions rep's

~180 lessers' accountants

~1000 SME representatives

Internal reports Quarterly Team assistasnt

300

700

180

1000

30

0

40

20

# of organizations served Internal reports Quarterly Team assistasnt 600

(200 per

country)

16

II 2002 III 2002 IV 2002 I 2003 II 2003 III 2003

Governmental off icials

are more supportive to

leasing

Local leasing institutions report to

have support from local officials

Leasing institutions

survey

Annually Deputy project

manager

N/a N/a

Financial institutions

launch leasing or

improve leasing

operations

# of participating companies launching

leasing operations

# of participating companies improving

leasing operations


survey


manager

4

6

N/a

N/a

Lessers take all available

tax exemptions

% of participating companies improve

profitability


survey


manager

90% N/a

Private sector is more

active in attaining

finance through leasing

% of participating SME consider

choice between cr edit and leasing

% of participating SME apply for

leasing

SME survey

(comparison of

participating and

other comppanies)


manager

40%

10-20%

N/a

RESPONSIBILITYFREQUENCY

RESPONSIBILITY

TARGETACTIVITIES: OUTPUT INDICATORs

DATA SOURCE &

COLLECTION

METHODs

Objective 2: To improve/increase the knowledge and understanding of equipment l easing

TARGET

CURRENT

STATUSOUTCOME OUTCOME INDICATORs

DATA SOURCE &

COLLECTIONMETHODs

FREQUENCY

CURRENT

STATUS

OUTPUTS:

DATA

DATA

DATA

Potential lessees and

lessers, and local

officials understand

concept and

practicalities of leasing

OUTPUT INDICATORs

DATA SOURCE &

COLLECTION

METHODs

FREQUENCY RESPONSIBILITYCURRENT

STATUSTARGET


8/22

PROJECTS BECOME MORE EFFECTIVE

Project will train fourteen local Uzbek consultants over two separate rounds witheach round lasting some three weeks

The project team decided to allow consultants a period of six months to implementtheir training

In March 2002 an evaluation of the first round was conducted through personalinterviews with consultants and their clients

Result: redesign of program for 2nd round

Key findings: Consultants showed improved knowledge of key business specializations where they

previously had little or no exposure

During the six given months they progressed toward service diversification,

project/client management, but less than expected

Eg.: Pilot Training of Uzbek Consultants - Midpoint Evaluation


9/22

Results for Activity 1Results for Activity 1

Activity Specification:

Produce informational and

analytical materials;

Carry out consultations on

legislative issues;

Participate in working group

activities.

Activities Outputs

Issues addressed in produced

policy papers(concepts,proposals), the extent of papers

distribution;

Different groups/ministerial

departments consultedand issues

addressed;

Submitted proposals and

comments to working groups.Outcomes

Policymaking process is

facilitated;

Policymakers are betterinformed aboutagribusiness issues;

New strategies and

legislation are proposed

and developed by

policymakers.

Outcome Verifiable Indicators

Provided advice is used by

government;

New legislation specifically

incorporates our suggestions;

Requests from government for

information/advice or to

participate in working groups.

Impact

Policy environment is more

favorable for agribusinessdevelopment (needed

legislation is adopted)

Activity:

Conduct and disseminate

policy-oriented research and

carry out consultations with

policymakers


10/22



Provide consultations to banks,

insurance companies and other

financial intermediaries;

Promote interaction/be anintermediary between

farms/agribusinesses and financial

institutions

Organize and conduct trainingprograms.

Activities Outputs:

Number of clients served and

issued addressed;

Number of different companies

consulted;

Number of contacts between

farmers and financerscreated;

New concepts developed andpresented;

Number of business plans

developedand submitted tobanks.

Outcomes:

Better access to credit,

insurance secured;

Financial institutions

more knowledgeable

about agribusiness (andvice versa);

New financial products

are introduced, including

long-term credit.

Outcome Verifiable Indicators:

Expanded the number of banks,

financial institutions providing

credit, insurance;

Amount of credit, insurance provided

and extended amount of long-termcredit;

Increase the rate of repayment;

Percentage of crops insured;

Number of new financial products

introduced.

Impact:

Farms become more

profitable and efficient;

Stable market for long-term

rural credit and insurance isestablished.

Activity:

Promote credit and

insurance relations inthe agricultural sector


11/22

Results for Activity 3: Agronomic ActivitiesResults for Activity 3: Agronomic Activities


Provide consultations, organize

seminars on new crop varietiesand use and management of

agricultural inputs;

Set up and maintain

demonstration fields introducing

new crop varieties;

Be an intermediary between

input-supplier and producers.

Activities Outputs:

Farmers served through carried

out consultations and seminars;

Increased the number of farmers

introduced to new crops andnew technologies;

Training materials produced and

extent of their distribution to ag

producers;

Increased the access of input

suppliers to ag producers and

vice versa.

Outcomes:

New, more productive

crop varieties planted;

New techniquesfor

planting , inputmanagement adopted;

Better utilization of

agricultural inputs;

Farmers are better

educated on agronomic

practices and better

decision makers.


Number of new crops farmers

cultivate;

Number of new inputs used; more

economical/productive use ofexisting inputs

Extent of new technologies

farmers use in their use.

Impact:

Increase in yields, crop

productivity and in farm

income increases

Activity:

Introduce new agronomic

practices.


12/22



Publish bimonthly bulletin and

other publications;

Organize and conduct press events

(press clubs; meetings with themedia) and prepare press releases;

Develop and maintain contacts with

government, the media, NGOs,

other research institutions, etc.

Activities Outputs:

Publication produced, issues

presented to the public, and

number of subscribers;

Number of media representativesand experts participated in

events;

Number of interviews conducted

and number of different mediaoutlets that cooperate with the

Project;

Number of contacts established

and joint events conducted.

Outcomes:

Established project as a

reliable source of

information;

Provided regular andfree access to

information;

Facilitated contact

among government,

NGOs, media,

researchers.


Number of times articles are

referencedfrom Projects

publications;

Number of new subscribers, rate ofre-subscription;

Number of articles, broadcasts about

event.

Number of new requests for

interviews

Impact:

Increase awareness of

private agribusiness issues

and gain support for private

agribusiness.

Activity:Conduct an information

campaign through mediaevents, publications, and

public discussions.


13/22

OUTPUTS

CompletedActivities

Outputs

Conduct training onEffective Presentation

Provide 3 consultations to

Policy makers

Carry out study of ForestsSector in NW Russia

E.g. Nine UIP consultants haveimproved their presentation skills

Two draft amendments to Land Code

were proposed

At least 15 potential investors receivethe study report

Completed activities are not Outputs

- Direct results from the Activities

- Have effect immediately or in a short term period during the projecttimeframe


14/22

OUTCOMES

- Anticipated effect

- Can be observed during the project scope and within 1-3 years aftercompletion- Measure the achievement of Objectives

Outputs vs Outcomes Whats the difference?

E.g. How many people haveimproved their knowledge?

300 SME representativeswere trained on advantagesof leasing

How well they have appliedthe received knowledge

50 SME participated in the training haveapplied for lease financing

Outputs Outcomes


15/22

IMPACT

- Desired final change- Has effect within 1-10 years after completion, rarely during the projecttimeframe- Measures the achievement of Goal

Outcomes vs Impact Whats the difference?

E.g. What was implemented?

Accurate fertilizingCrop management

How the implemented changeshave affected the business?

Increase of yields per hectare,Revenue and profits

Outcomes Impact


16/22

TARGET

- Planned level of the indicator

BASELINE- Level of the indicator at the project starting point

WE SET TARGETS AND DEFINE BASELINE LEVELTO HAVE CLEAR EXPECTATIONS FOR THE PROGRESS


17/22

HOW TO WORK WITH THE TABLE?

What should change?

Write short verbal description of theexpected result in the first column

How you will measure?

Whats your target for it?

Choose indicator and set target levelfor it

EXPECTEDO TPUTS:OUTPUT

TARGETS:

Pla e s on leasing ma ket

of NW Russia imp ove

thei know ledge and skillsof application and

advantages of lease

100-150 manage s f om

at least 50 companies

t ained o consulted


18/22

COMPONENTS OF THE PROJECTTraining

Objectives

ActivitiesOur means oftraining

OutputsHow many andhow well peoplewere trained

OutcomesResults ofapplication ofreceivedknowledge(NB measureachievement ofObjective)

Potential indicators

- To introduce modern sustainable agriculturaltechnologies in XX oblast

- To raise local management skills in financialcontrolling

- # of trainings delivered by subject- # of training hours

- # of copies and key components of manualsdistributed

- Name key components of online training tools

- # of people trained from # of companies- % of participants completed test with a XX pass

mark

- Increase in # of hectares under biological controlsystems

- Improve in profitability of participating companies- % of participating SMEs apply for leasing- % of served companies improve internal documents

and CG practices

Comment

- Better avoid: To raise local management skills toogeneral objective, its better to be specific on the onthe subject of training

- Better avoid: # of training modules and othertraining materials developed additional measure,successfully conducted training assumes developedtraining materials

- Better avoid: Independent testing of the trainingmaterials additional costs & questionable value


19/22

COMPONENTS OF THE PROJECTTrain the Trainer

Objectives

ActivitiesOur means ofknowledgetransfer

Outputs

OutcomesHow many people

and how wellare trained bythe partners


- To raise local management skills in financialcontrolling trough building local training capacity

- To increase knowledge about CG of the nextgeneration of managers

- # of seminars/round tables other events forpartners

- # of trained partners- # of case studies, manuals, training modules

transferred

- # of partners delivering training over time- # of SME representatives trained by the partners

over time- % of participants completed test with a XX pass

mark

Comment

- Be specific on the on the subject of training

- Better avoid: Number of issued covered

- NB to identify target audience: e.g. business peoplevs students

- For internal information can be worthy to revisitpartners and check quality of teaching


20/22

COMPONENTS OF THE PROJECTLegislative Work

Objectives

Activities

Our means ofinfluencing thePolicy makers

OutputsWhat wepropose

OutcomesWhat wasadopted


- To improve legislative framework of XX sector

- # of meetings/round tables/other events held- # of presentation given during the- # of hours of consultation to government

- # of drafts/amendments proposed- Name of targeted laws to draft or amend

- # and names of approved drafts/amendments

Comment

Concern for further usability:

- # of issues identified or discussed- # of collaborative activities and joint activities- # of different departments consulted- # of other organizations requested our policy input


21/22

COMPONENTS OF THE PROJECTDirect Assistance to Companies Objectives and Scope Can Vary Greatly

Objectives

Activities

Outputs

Outcomes


- To support investment initiatives in particularsector

- To built up supply chain links for XXX company- To increase access to XXX market of particular

companies

- # of feasibility studies/business plans- # of potential partners evaluated- # of hours of consultation- # of deals consulted

- # of companies assisted with the tailored program- # of partnerships created

- Value of investment in progress and completed in #of companies

- Additional sales, market share gain- Reduced costs, additional profits

Comment

Concern for further usability:

- # of obstacles identified

- Level of satisfaction of investors/clients

- Independent valuation of research quality


22/22

COMPONENTS OF THE PROJECTPublic Relations

Objectives

ActivitiesPR means

OutputsExpected

publicity

OutcomesPublicity reachand effect


- To increase public awareness of equipment leasingissues

- # of round tables/conferences/ press conferencesheld

- # of press releases sent- Deadline for website launch and frequency of

update

- # of media appearances per month/quarter and intotal

- # of people reached from target audience permonth/quarter and in total

- # of project replications

Comment

- PR is useful to to get more clients/partners orreplicate the project. If there is no need in it then,we should consider usefulness of nation-widecampaigns

Concern for further usability:- # of new contacts established- #of requests from media to organize press events- # of times articles/press releases are referenced- # of new requests for interview

uadp ia training

Documents