REAL WORLD RESEARCHTHIRD EDITION

Chapter 8:

Designs for Particular

Purposes: Evaluation,

Action and Change

1©2011 John Wiley & Sons Ltd.

Some purposes of evaluation:

1 To find out if client needs are met

likely questions:• What should be the focus of a new

programme?• Are we reaching the target group?• Is what we provide actually what they need?

continued…

2©2011 John Wiley & Sons Ltd.

Some purposes of evaluation:- continued

2 To improve the programme

likely questions:• How can we make the programme better

(e.g. in meeting needs; or in its effectiveness; or in its efficiency)?

continued…

3©2011 John Wiley & Sons Ltd.

Some purposes of evaluation:- continued

3 To assess the outcomes of a programme

likely questions:• Is the programme effective (e.g. in reaching

planned goals)?• What happens to clients as a result of

following the programme?• Is it worth continuing (or expanding)?

continued…4©2011 John Wiley & Sons Ltd.

Some purposes of evaluation:- continued

4 To assess the outcomes of a programme

likely questions:• How do the costs of running the programme

compare with the benefits it provides?• Is it more (or less) efficient than other

programmes?

continued…

5©2011 John Wiley & Sons Ltd.

Some purposes of evaluation:- continued

5 Assessing relative costs and benefits

likely questions:• How do the costs of running the programme

compare with the benefits it provides?• Is it more (or less) efficient than other

programmes?

(from Robson, 2000, Table 1.1, p. 10)

6©2011 John Wiley & Sons Ltd.

Features of evaluation

Any evaluation should meet the following criteria:

1 Utility2 Feasibility3 Propriety4 Technical adequacy

7©2011 John Wiley & Sons Ltd.

Checklist for planning an evaluation

1 Reasons, purposes and motivations2 Value3 Interpretation4 Subject5 Evaluator(s)6 Methods7 Time8 Permissions and control9 Use

(adapted from Robson et al., 1988, p. 85)

8©2011 John Wiley & Sons Ltd.

Skills needed to carry out evaluations

• writing a proposal• clarifying purposes of the evaluation• identifying, organizing and working with an

evaluation team• choice of design and data collection

techniques• interviewing• questionnaire construction and use

continued…

9©2011 John Wiley & Sons Ltd.

Skills needed to carry out evaluations- continued

• management of complex information systems• data analysis• report writing, including making of

recommendations• fostering utilization of findings• sensitivity to political concerns

10©2011 John Wiley & Sons Ltd.

Needs assessment checklist

1 Identify key individuals to be involved in a steering group

2 Define the target population for the needs assessment3 Ensure their needs are the focus of the needs

assessment4 Communicate the aims of the needs assessment to

service providers5 Decide who will carry out the needs assessment 6 Consider whether additional assistance may be needed

continued …

11©2011 John Wiley & Sons Ltd.

Needs assessment checklist- continued

7 Estimate the cost and identify the source of funding for the needs assessment

8 Identify the appropriate overall approach to your needs assessment

9 Gather existing sources of information about the needs of your target population

10 Identify existing services in your area that are already available to meet the need

continued …

12©2011 John Wiley & Sons Ltd.

Needs assessment checklist- continued

11 Consider the ways in which you will obtain the views of your target population about their needs, and whether ethical approval is needed

12 Consider the ways in which you will obtain the views of service providers about the needs of the target population

13 Ensure that information is analysed and interpreted, and that conclusions are drawn

14 Consider how those who gathered the information can be involved in the analysis, and how the results can be relayed back to all those who contributed to the process

continued …

13©2011 John Wiley & Sons Ltd.

Needs assessment checklist- continued

15 Once you have identified the needs of your target population, prioritize them, consider all the options for meeting them, and develop an implementation plan

16 Once agreement is reached on the changes to make, consider how to monitor and evaluate them

17 Consider the most appropriate methodology for the evaluation

(adapted and abridged from Griesbach et al., 2004, p. 37)

14©2011 John Wiley & Sons Ltd.

Stages of action research

1 Define the inquiry 2 Describe the situation3 Collect evaluative data and analyse it 4 Review the data and look for contradictions

continued …

15©2011 John Wiley & Sons Ltd.

Stages of action research - continued

5 Tackle a contradiction by introducing change6 Monitor the change7 Analyse evaluative data about the change8 Review the change and decide what to do next

(adapted from Bassey, 1998, pp. 94–5)

16©2011 John Wiley & Sons Ltd.

Assumptions for those wishing to initiate change

1 Don’t assume that your version of what the change should be is the one that could or should be implemented

2 Change involves ambiguity, ambivalence and uncertainty

3 Some conflict and disagreement are not only inevitable but fundamental to change.

4 People need pressure to change 5 Effective change takes time

continued …

17©2011 John Wiley & Sons Ltd.

Assumptions for those wishing to initiate change - continued

6 Lack of implementation isn’t necessarily because of rejection or resistance

7 Don’t expect all, or even most, people or groups to change

8 You need a plan based on these assumptions and underpinned by a knowledge of the change process

9 Change is a frustrating, discouraging business - if you are not in a position to make the above assumptions don’t expect significant change, as far as implementation is concerned.

(adapted and abridged from Fullan, 1982, p. 91)

18©2011 John Wiley & Sons Ltd.