case study as used in evaluation

7/27/2019 Case Study as Used in Evaluation

1/7

1

CASE STUDY

1 Why is this tool used in evaluation?The implementation of case study reviews of one or more actualexamples, in order to gain an in-depth knowledge of the topic and,if possible, to learn about the entire evaluation programme.

In complex situations, case studies are the preferred evaluationtool when how and why questions are being posed, They allowa detailed examination of the actual elements in line with theevaluation goals. The purpose of the case study is to provide apicture, which is often more reliable than the outputs from othertools in context of the scarcity of basic data (which is often thecase in country evaluations).

Figure 1 Case studys components

If case studies include the analysis of documents, statistical and

implementing data, they are mostly known as a field observationtool and a means to interview people directly involved in theprogramme, such as the officials and stakeholders.


2/7

2

2 How is a case study carried out?

3 What are the conditions for the use ofthis tool?

To ensure that a case study is credible and yields satisfactoryresults the geographic evaluations specific context needs to:

find an effective local partner, who should be neutral to thetopic under evaluation and competent both in the theme to bestudied and in evaluation techniques

plan for supervision procedures of the international and localworking teams

keep control of the selection of participants

have a fair distribution between interviews with officials andbeneficiary representatives

Figure 2 : How is a case study carried out?

Steps involved in case study implementation

3.1 How is the instance selection undertaken?

This selection is crucial because an incorrect basis for selecting aninstance can lead to a flawed evaluation outcome and can

jeopardise its generalisation.

The United States General Accounting Office suggests 3 possiblekeys for instance selection:

The tools conditions foruse

Instance selection

Analysis and interpretationof the results

Studys organisation andplanning

Data collection


Instance selection



Data collection


Instance selection



Data collection


3/7

3

Convenience

Purposive samples

Probability

Example of the countrys selection criteria for the Europeanassistance evaluation in the water sector:

1. Be included in the list of the countrys main beneficiaries of the

watsan European assistance2. Have a representative in every region of the world

3. Have water sanitation as a priority intervention sector

4. Not being influenced by the last evaluations conducted bythe European evaluation Unit

3.2 How is the case study organised andplanned?

A modus operandi defining how to carry out one or more casestudies is always useful.

Figure 3 : A modus operandi defining how to carry

out one or more case studies is always useful.

Among other advantages the carrying out a pilot case study:

validates the methodology with a field test

determines more precisely the categories of respondents and thebasic bibliography

completes the interview guidelines and the questionnaires

reviews the questions asked to the evaluators, criteria andindicators, in the light of what is available on-site

Data on thecontext Questions to

which the casestudy must

answer

Datacollection

methods

Evaluationcriteria

Studysmodus

operandi

Studysmodus

operandi

Framework forthe report, etc.

Actual elements ofmethodology (interview

guides, etc.)

Data on thecontext Questions to

which the casestudy must

answer

Datacollection

methods

Evaluationcriteria

Studysmodus

operandi

Studysmodus

operandi

Framework forthe report, etc.

Actual elements ofmethodology (interview

guides, etc.)


4/7

4

designs a standard report to be followed by the other evaluators

3.3 How are the data collected?

To ensure that the case study findings are reliable, a number offundamental elements should be carefully taken into account:

The data collection should include adequate longitudinal data,i.e. data covering a sufficiently long period of time in order to

avoid taking an anomalous situation as a reference point. The data collection should be based on a principle

systematically adopted in the evaluation: informationverification through triangulation.

The evaluator must ensure that the information collected is

thoroughly used, and that nothing important has beenoverlooked. It is essential to obtain as much information aspossible, especially when opinions differ among the peopleinterviewed.

By definition, the case study is open to any possiblediscoveries throughout the course of its implementation. Thus,the evaluator must know how to identify key features during

the case study implementation and focus on them, even if they

were not expected or scheduled in advance.

The field stage leads to first-hand observations of what isoccurring. The evaluator must also note them down carefully.

3.4 How are the results analysed andinterpreted?

This is the most challenging stage of the case study. Its goal is toanalyse the data that have been collected during the fieldwork andto link as far as possible the effects of the observed facts to theircauses. This analysis is difficult to conduct because it is lessstructured than at the conception and the collection stages.

The analysis overlaps with the data collection stage, and this is

particularly true for case studies in which: The data collection stage includes a pre-established hypothesis

test that may partly modify the study content during itsimplementation.

The study is large enough to allow the evaluator to review andrefine his criteria for the next data collection as a result of theinitial findings.


5/7

5

4 What are the preconditions for its use?

Figure 4 : the preconditions for its use

The time

spanPreparation: 15 to 20 working days

Field mission: 3 to 10 working days

Data analysis: 3 to 10 working days

Evaluators training (in case of multiple sites casestudies): 2/3 working days

Analysis and conclusions from multiple sites casestudies: 10 to 30 working days.

Human

resourcesQualified people who fully understand the problemsassociated with the evaluation, have sufficientexperience of interview techniques and speak thelanguage of the people interviewed.

Financial

resourcesA minimum budget of 15,000 should be fixed andallocated to the multiple case studies preparationstage.

A budget of at least 5,000 to 7,000 should beplanned for each case study, not including long-distance transportation.

5 What are the advantages and limitationsof the tool?

Figure 5 : The advantages and limitations of the tool

Advantages

Its richness, made possible by detailed

qualitative information and the context ofimplementation precisely described.

Its relatively straightforward use.Its flexibility, making continuous adaptation to

various situations possible.

Its implementation stage is compatible withthat of a country evaluation.

The opportunity to obtain and understandinformation at a sufficiently deep level. Thistool allows evaluators to become familiar with


6/7

6

the logic of action of the various actors.

Limitations

The difficulty of identifying the appropriatetargets.

The difficulty of identifying cases, settingboundaries and linking them to problems asbroad as those commonly addressed in acountry evaluation.

The difficulty arising from generalisation to aglobal level (for example, a country) of themesthat were studied at a local level.

The tools cost.

The fact that this tool rarely allows statisticalinterpretation of data.

The fact that this tool relies on the judgementof one or more evaluators can lead topartiality, even with the most careful use ofcase study methods.

6 Check-lists

Check-list for evaluators

Is the use of the case study tool in the evaluation backed upby adequate argumentation?

Is the choice of the case study application well-argued?

In the context of multiple sites case study, is the number ofcase studies justified?

Has the design methodology been properly elaborated?

In the context of multiple sites case studies, does themethodology assure consistent reports?

Has a pilot case study been scheduled?

Is the use of triangulation clarified in the methodology andincluded in the mission reports?

Have the sources of information (documentation, interview,monitoring data, direct observation) been included in the


7/7

7

mission reports?

Do the methodology and reports distinguish facts fromopinions?

Is the plan for the development of a chain of evidence well-argued in the mission report?

Does the iterative process, initiated at the collection stage,

carry on to the analysis stage, and support the chain ofevidence?

Were alternative explanations studied and rejected after a fullreview of the evidence?

Are the facts supporting the argumentation strong enough toguarantee systematic replication elsewhere?

Does the analysis include research into causality?

Are the techniques used for the analysis of multiple sites dataset out and argued?

Is the case study report sufficiently understandable andexplicit?

In the case of multiple case study has the team leader checkedthe relevance /consistency of the studies ?

Are the limitations of the impact of the study findingssufficiently well explained?

Check-list for managers

Is the use of the case study tool in the evaluation backed upby adequate argumentation?

Do the methodology and reports distinguish facts fromopinions?

Are the techniques used for the analysis of multiple sites dataexposed and argued?

Is the case study report sufficiently understandable andexplicit?

Are limitations of the impact of the study findings sufficientlywell explained?

case study as used in evaluation

Documents