concepts and application deutsche gesellschaft für technische zusammenarbeit (gtz) gmbh impact...

Concepts and Application

Deutsche Gesellschaft fürTechnische Zusammenarbeit (GTZ) GmbH

Impact Monitoring in Value Chain Promotion

Heike Höffler, GTZ Kenya

First sequence of thoughts, prepared for the Training Course

„Promoting Promoting Value Chains for Agribusiness Development in Africa“

held in Nairobi, 4th - 7th April 2005

2

Impact Monitoring: Concepts

and Application

Measuring Impacts in Value Chain Promotion1. Impact Hypotheses – chain of changes 2. Scope of influence 3. Benchmark Data, Milestones and Indicators4. Methods of Measuring Impact 5. Using Monitoring Information

2

A Concept for Impact Monitoring1. What to monitor? What are impacts?2. GTZ Impact Model3. Theory: Six steps to set up a monitoring system

1

Practice: PSDA Impact Monitoring System3

Heike Höffler Kenya

3


and Application


2


1



4

Monitoring is an ongoing activity!

• Projects and Programmes need to follow-up their progress

• Successes and failures need to be looked at while projects and programmes are running

• Monitoring is an internalised process of team communication, continuously undertaken while implementing,

whereas

• Evaluation is an act of stopping implementation to reflect past activities (but drawing from information from monitoring).

1


5

The World of Impact Monitoring

Impact Chain

Impact Model

Impact Indicators

Outputs

Outcome

Outreach

BenefitsPerformance Monitoring

Input Monitoring

Activity Monitoring

Output Monitoring

Implementation Monitoring

Impact Monitoring

Impact Assessment

Process Monitoring

Result-based MonitoringEvaluation


6


Impact Chain

Impact Model

Impact Indicators

Outputs

Outcome

Outreach

BenefitsPerformance Monitoring

Input Monitoring

Activity Monitoring

Output Monitoring


Impact Monitoring

Impact Assessment

Process Monitoring

Result-based MonitoringEvaluation


7


Performance Monitoring Impact Monitoring Impact Assessment

Input Monitoring

Activity Monitoring


Process Monitoring

Output Monitoring

Impact Chain

Impact Model

Impact Indicators

Result-based Monitoring

Outputs ImpactsBenefits

Outreach

Evaluation

Outcome


8

What to monitor in projects – Performance & Impact Monitoring

1 Performance

• Observing outputs against planned activities

• Providing information for project management

• Day-to-day activity

To trigger short-term adjustments in operation

Impact

• Focussing on effect of outputs: impacts!

• Observing the direct benefit of outputs

• Strategic steering of implementation

To self-evaluate whether activities contribute to objectives


9

Impacts are ...

... changes that have a causal - or at least a plausible - link to a project/programme

... a change of circumstances as a consequence of an intervention, it can be intended or unintended, positive or negative.

... there: from the first moment of intervention and they continue to occur all the time.

... rather the result of social interaction than a straight-forward interventions

... the result of complex interactions and thus, a complex matter to deal with!

1


10

Why concentrating on Impacts?

• Broad international discussion late 90ies

• Criticism about the efficiency of development cooperation

• Criticism about the Monitoring and Evaluation system of GTZ

1

• BMZ is a contractor, GTZ is an agent• New structure of the project documents• Changes in political dialogue• Changes in Report obligations

AURA


11

1

The GTZ Impact Model (I)

• Impact Orientation has become a principle of GTZ‘s corporate development.

• „quality at entry“ „quality at exit“

• „what has been done“ „what has changed“


ProjectUse of OutputDirect BenefitIndirect Benefit

Engel, P., The Social Organization of Innovation, 1997

Model of Interaction

1


GTZ Input

Use of Outputs

Poverty Alleviation

Direct Benefit(Goal)

Outputs

Indirect Benefit

Activities

The GTZ Impact Model (II)

Attribution?

1

Partner


Observation

!

14

Illustration: which of the 100 Bricks did we donate?

GTZ Input

Use of Outputs

Poverty Alleviation

Direct Benefit(Goal)

Outputs

Indirect Benefit

Activities

The GTZ Impact Model (II)

1

Partner

diff. Impact Levels


16

Increase in rural employment & income

Indirect Benefits

Direct Benefit (Goal)

Use of Output

Output

Activities

Inputs

Producers can access international markets

Market information is used to change production standards

Presentation of Research to stakeholders and publishing

Market research for a strategic agricultural product

Advisory Services in the Ministry of Agriculture

The GTZ Impact Model (III)

1

Attribution Gap

Pro-poor Rural Economomic Growth


17

• Impact Models form the essential methodological part in project design!

• Project reports are reduced to the information relevant for political intervention, i.e.

– Changes in risk

– Changes in assumptions

• and: IMPACTS! i.e.

– Which changes can be observed?

– Which impacts can be plausibly attributed to the project?

– Are there unintended impacts?

Consequences for GTZ Projects

1

Changes in Monitoring!


19

Theory: Six Steps (GTZ)

Step 1: Identify the System Boundaries

Step 2: Agree on Purpose and Procedures for Results-based Monitoring

Step 3: Agree on Results Hypotheses

Step 4: Review Indicators and Define Milestones

Step 5: Conduct Data Survey

Step 6: Using Monitoring Results

1


GTZ 2004: Result-based Monitoring: Guidelines for Technical Cooperation Projects and Programmes

20

Theory: Six Steps (Herweg/Steiner)

Step 1: Involvement of Stakeholders and Information Management

Step 2: Review of Problem Analysis

Step 3: Formulation of Impact Hypotheses

Step 4: Selection of impact Indicators

Step 5: Development and Application of Impact Monitoring Methods

Step 6: Impact Assessment & Follow-up

1


Herweg, K & Steiner, K. 2002: Impact Monitoring and Assessment, Vol. I & II

21

Theory: Seven Steps („MAPP“, GDI)

Step 1: Preparing a Life Line

Step 2: Preparing a Trend Analysis

Step 3: Cross-checking with other sources

Step 4: Compiling an Intervention List

Step 5: Developing the Influence Matrix (connecting trends and interventions)

Step 6: Developing the Impact Profile

Step 7: Attribution of Impacts to MDGs

1


DIE 2004: Briefing Paper: Impact Analysis of Development Cooperation is Feasible

22


and Application


2


1



23

Impact Models: A chain of changes

What do we expect to happen in value chain promotion?

2

SpecificInputproviders

Primaryproducers Traders

Final Con-sumers

Logisticscentres,Industry

SpecificInputs

Trans-formation

Trade ConsumptionProduction

C H A N G E !

Intervention; i.e. extension

Trans-formation

Trade ConsumptionSpecificInputs


24



Final Con-sumers


SpecificInputs

Trans-formation

Trade2

Defining the Impact Hypothesis (I)

Consumption

Use of Output Farmer group gets EuroGap certified

Output

Activities Training on EuropGap Compliance and Certification

Inputs Advisory Services for Extension Providers

Production

Direct Benefit (Goal) Small-scale producers access international markets

Farmer Groups are setting up an Internal Control System


25



Final Con-sumers


SpecificInputs

Trans-formation

Trade2

Defining the Impact Hypothesis (II)

Consumption

Use of Output

Output

Activities

Inputs

Production



26



Final Con-sumers


SpecificInputs

Trans-formation

Trade2

Defining the Impact Hypothesis (III)

Consumption

Use of Output

Output

Activities

Inputs

Production



27



Final Con-sumers


SpecificInputs

Trans-formation

Trade2

Defining the Impact Hypothesis (IV)

Consumption

Use of Output

Output

Activities

Inputs

Production



28

Defining the Impact Hypothesis (V)

2

Use of Output

Output

Activities

Inputs

Direct Benefit (Goal) • Each intervention undertaken along a value chain should follow an impact model;

• thus: each value chain to be promoted needs a concept of impact hypothesis.

• All impact hypotheses should be derived from the project goal;

• thus need to be derived from the impact indicators!


29

Impact Hypotheses – Food for Thought:

2

• Pre-formulation of impact chains can lead to mono-causal conclusions.

• Linear following of impact chains can fade out the complex interaction of interventions and impacts.

• The higher the impact level, the more a context oriented approach is needed, taking into account development trends.

Danger of

mono-causality !


30

Defining the Scope of Influence (I)

Specific single interventions

2



Final Con-sumers


SpecificInputs

Trans-formation



31

Defining the Scope of Influence (I)

Specific single interventions

2



Final Con-sumers


SpecificInputs

Trans-formation



32

Defining the Scope of Influence (II)

Chain promotion at all stages

2



Final Con-sumers


SpecificInputs

Trans-formation



33

m

acr

om

eso

mic

roDefining the Scope of Influence (III)

Full chain promotion, all stages, all levels

2



Final Con-sumers


SpecificInputs

Trans-formation


Subsector-specific Technical Agencies

Subsector-specific BDS providers

Associations

Local Government, Providers of Utilities / Infrastructure

National Government (Line Ministries) & Public Agencies


34

Impact Hypothesis - Conclusion

2



Final Con-sumers


SpecificInputs

Trans-formation


Subsector-specific Technical Agencies

Subsector-specific BDS providers

Associations

Local Government, Providers of Utilities / Infrastructure

National Government (Line Ministries) & Public Agencies


2

35

2

Benchmarks, Milestones & Indicators

Benchmark Data & Information

The state of the art at given point in time (i.e. programme start)

Milestones

Desired output at a given point in time for (programme) achievements

Indicators

Determinants for checking whether activities led to the expected output (performance) or expected implications (impact).


36

2

Benchmarks, Milestones & Indicators (I)

Indicators


0 t

Benchmark Data

Programme level

Value chain level

Milestones Progress Reports

Evaluation

37

SpecificInputs

Trans-formation

Trade

2

Benchmarks in Value Chains

ConsumptionProduction

Usage of Inputs

Usage of services

Outreach: How many are involved?

Employment

Gross margins

Regional distribution

Number of processors

Degree of vertical concentra-tion and integration

Market Concentra-tion

Entry Barriers

Price Volatility

Seasonality

Trade margins

Quality

Food Safety

Product Diversity


38

SpecificInputs

Trans-formation

Trade

From Benchmarks to Milestones


Output

Activities

Inputs

3 processing firms

Low process quality

High costs of processing

In 18 Month:

2 more firms established

At least 2 certified under ISO xxx

Cost-effectiveness of at least 2 firms improved by x %

Milestones serve as process indicators.

They are directly derived from programme indicators and the impact hypothesis.

2


39

SpecificInputs

Trans-formation

Trade

Benchmarks & Milestones in Value Chains


Try to build these „couples“ per stage in a chain that you are targeting with your project.

Not all Benchmarks need to be quantitative data; qualitative description also serves the purpose.

Market Analyses need to take the need for benchmarks into account (into the TORs!)

We might not be the first to undertake this – keep looking for partners in gathering data!

2


40

2

From Milestones to Indicators –

or was it the other way round?

Output

Activities

Inputs

Use of Output


Production


Several Milestones in a chain

Single Milestone

Programme Indicator(s)

Value chain Indicator(s)

41

SpecificInputs

Trans-formation

Trade

Benchmarks, Milestones & Indicators (II)


Output

Activities

Inputs

3 processing firms

Low process quality

High costs of procesing

In 18 Months:

2 more firms established

At least 2 certified under ISO xxx

Cost-effectiveness of at least 2 firms improved by x %

...

2


Use of Output


Impact Indicator, Component 3, Phase 2

The number of profitably operating private processors doubles (2002: 6)

43

Example: Employment

CustomersMarket 2

CustomersMarket 1

(Research Institute)

Retailers A

Retailers B

IndustrialSMEs

Importers

Large processing companies

3

2

1Primary Producers 1

SM Primary Producers 2

n = 400 n = 450n = 20

n = 6000

n = 40

n = 2400

n = 20

n = 200

3


The number of primary producers doubles.

44

100,00

SpecificInputs

Production Trans-formation

Trade Con-sumption

Example: Value Added



Final Con-sumers


Sales Prices / unit 60,0025,00

Cost / unit Income / unit

Distribution in %- of value-added- of income

23,002,00

27,008,00

30,0010,00

25 %10 %

35 %40 %

40 %50 %

3


The value added at producer level increases from 25% to 35%.

45

Example: Better Business Relations


SpecificInputs


Trade Con-sumption



Final Con-sumers

Key informant interviews show that wholesale traders are more satisfied with the quality/quanity/reliability of produce delivered by small scale farmers.

Interviews show that small cale farmers have improved relationship to traders and trust them.

46

2

Indicators in Value Chains

Some examples:

• Customer satisfaction

• Repeated customers

• Number of enterprises demanding a service

• Satisfaction with last service purchased

• Percentage of women-owned enterprises


„There is increasing interest in, and use of, private sector tools for performance and impact measurement;

i.e. consumer market research tools for measuring changes in markets and private sector business tools

for measuring service provider performance.“

47

Methods of Measuring Impacts

2


• Market Studies

• Key Informant Interviews

• Point of Leverage – comparisons (before – after)

• Participatory Monitoring

• Time-series

• etc...

SpecificInputs


Trade Con-sumption

Where to measure in the chain?

What data – qualitative or quantitative?

Which degree of participation?

48

Example: Spider Web Diagrammes

0

5

10

Product Quality

Consumer Satisfaction

Farm Income

Growth in Exports

Soil fertility

Use of Services

Beginnging

after 3 years

after 10 years

2


49

E-Val

2


GTZ tool to catch different perceptions of projects and programmes.

There are only subjective opinions to changes and impacts

Triangulation of opinions:

GTZ, partners, and target group

50

Using Impact Monitoring Systems

2


The objective of Impact Monitoring was ....

• Focussing on effect of outputs: impacts!

• Observing the direct benefit of outputs

• Strategic steering of implementation

To self-evaluate whether activities contribute to objectives

51


2


SteeringObserve direct Benefit Self-evaluation

Steering Committee

Reports

Annual Progress Reports

Participation!

Discussion with Stakeholders

Finetuning

Communication

Team Meetings

Partner Meetings

Other donors

Private Sector

52


2


More ideas – let‘s be creative!

Replanning

Communication of Results

Newsletter

Business Fora

Publications

Go- No-go- Decisions

Accountability

53

It‘s better to be approximately rightthan precisely wrong!

( John Maynard Keynes )

Please email your comments & questions to [email protected]


54


and Application


2


1



concepts and application deutsche gesellschaft für technische zusammenarbeit (gtz) gmbh impact...

Documents