The role of evaluation in mental health and greenspace

Dr Ruth JepsonCo-Director, Centre for Population and Public Health Research, University of StirlingLead for Physical Activity, Diet and Health Research Programme

Outline of talk•T

he work at the Centre for Public Health and Population Health Research

•Why do evaluation?

•Points to consider in evaluation

•Some types of evaluation techniques and three examples

•Good sources of support/toolkits/ training available to projects wishing to evaluate what they do

Centre for Public Health & Population Health Research

Programme on physical activity and dietFocus on:•Promoting physical and healthy diet as part of everyday behaviour•Promoting physical activity through health professional referrals•Understanding the barriers to physical activity and healthy eating in different population groups (including ethnic minority groups)•Encouraging people to use the outdoors to increase their feelings of health and wellbeing (including walking and gardening)

Three relevant evaluation projects:•Walking groups via a GP practice•Health effects of community gardening•Indoor versus outdoor activities via Exercise Referral Schemes

Why do evaluation?

Evaluation can be powerful and exciting! (Hmm..)It can help you: •Improve services•Understand what works and what doesn’t •Demonstrate the difference that a project makes•Make decisions about the best use of funding•Have evidence for policy and decision-making

(Evaluation Support Scotland)

Can be your most important and influential tool for getting new or sustained funding - funders want evidence of what works

Who is evaluation for?

Evaluation Stakeholders

Policy makers

Funders

Planning & Performance Managers

Researchers

Service users

What sort of evaluation is valued?

Effectiveness; what works?

Accountability/value for money

Performance monitoring/targets Process evaluations

Knowledge building; research quality and utility

Service quality – access, experience, relevance to needs

What makes a ‘good’ evaluation?

Is objective and well executed

Influences decision making

Tells us if

it’s a good

investme

nt of our

funding

Contributes to the evidence

base

Helps us

understand

what

difference

we are

making

Shows the value of

what we’re doing

Before you start an evaluation

• Clarify the aims of your project

• Think about who you are targeting

• Think about how your project will effect change

• Identify what you want to achieve in the short, intermediate and long-term

• Decide how you will measure what you want to achieve [if possible used well validated measures – don’t attempt to make up your own]

• Think about what information you will need to collect [NEVER collect information you don’t intend to use!]

Types of evaluationThere are 4 broad types of evaluation:•process (which deals broadly with the processes involved in service delivery)•outcome (which determines whether aims have been met & how effective the service is)Often carried out at the same time or slightlystaggered

•impact (determining the wider implications of the service, often comparing with an areas where no service is provided)•economic (determining whether the service is cost effective)

Process evaluation

•Takes place during setting up and/or delivery of the project

•Provides guidance to those who are responsible for ensuring and improving the project’s quality

•Focuses on identifying barriers and facilitators to successful implementation/delivery, as well as assessing whether the key objectives have been met

•Can be used to refine/modify the delivery of the project

•Can be used to determine the effects of the project (intended and unintended)

‘We are too ready to assume that how a project should work is the way that the project does work’

Good process evaluation:1. Is carried out by people not involved in the project (makes it

more objective and people may answer more honestly)

2. Doesn’t make assumptions about how the project works and what it achieves – sometimes there are unintended consequences

3. Uses different types of data to assess the processesQualitative (talking to people) data can help identify if the project is running as intended; meeting the needs of the participants (staff and users); the experiences (positive and negative) of participants; the changes experienced (intended and unintended); does the project work as it should?

Quantitative (numbers) data should be collected to demonstrate that you are attracting the ‘right’ participants (called reach) Minimum data should include all the variables you think are relevant. For example: age, gender, postcode, ? health condition, ?physical activity level for all participants, performance data (e.g. how many referrals were made, activities carried out etc)

Outcome EvaluationAims to

answer questions such as:•D

oes the project work?•D

oes it provide the benefits to participants you wanted it to make?•H

ow do you measure success? [Where possible VERY IMPORTANT to use questionnaires etc. that have been validated and used in the same type of setting/population]

If possible should try and measure change when people start in the project, after a few months and after about one year.

Gold standard would be the randomised controlled trial where you have a control group – you can be more certain that any benefits that are seen are due to the project and not due to other reasons.

GOOD DATA MANAGEMENT SYSTEMS ARE VITAL FOR A USEFUL OUTCOME EVALUATION

To do a good outcome evaluation • Be realistic and clear about what your project is likely to achieve – make

sure you are measuring the right outcomes (e.g. improving mental health, not curing depression) –ask why you think your project would impact on these outcomes

• Collect core information on everyone who comes to the project (baseline)

• Consistently ask same questions to all participants

• Use proper validated tools to measure outcomes

• Don’t collect unnecessary data (just because you think it ‘might be interesting’)

• Follow up participants at medium and longer term (e.g. 6 months & 12 months) to see if change/benefit has occurred

• Demonstrate effectiveness by using experimental methods (not necessary for all projects) – aspirational but not impossible!

Musselburgh Health Walks for sedentary people and/or mental health problems

An evaluation by Roma Robertson, PhD student

Process evaluation

• Is it possible to run a programme of health walks 3 times/week?

• Do health practitioners (HPs) refer onto the service?

• Do sedentary patients/people with depression utilise the service?

• Can the model be improved? What positive and

negative impacts do people report?

Outcomes evaluation

• Is there any evidence of benefits to participants?

(used quantitative and qualitative data)

1. Is it possible to run a programme of health walks 3 times/week?

Yes: The programme of walks ran well for the planned 24 weeks. It was organised by CHANGES Community Health Project and supported by 8 volunteer walk leaders, many of whom had attended the well established wellbeing walks run by CHANGES over the years. However, without the volunteer walk leaders it would have been difficult to provide such frequent walks and at low cost.

2. Do health practitioners refer to the service? The initial plan was to recruit walk participants via GP consultations. Low recruitment numbers led to other alternative recruitment strategies being introduced (so No!)

Process evaluation

3. Do sedentary patients/people utilise the service?

Of the 19 who participated, 13 (68%) stayed until the end of the 12 weeks and 9 people continued to walk with the group beyond the 12 weeks they had agreed to. This suggests that a large proportion of participants valued the service.

4. Can the walks be improved?

There was a lot of very positive feedback.

When pressed for how the walks could be improved, walkers indicated they would like a greater variety of walks; and a quarter felt the walks were too short. One person found it embarrassing to meet at the health centre and one was worried in case someone they knew would see them on the walks. Seven people liked going for coffee afterwards; 5 disagreed or were undecided

Process evaluation

Outcomes evaluationI

s there any evidence of benefits to participants? •C

ollected data at baseline, 6 weeks and 6 months•U

sed validated measures of health outcomes (IMPORTANT!)•C

ollected data on mental wellbeing, physical activity, social networks, general health

Physical activity

Immediately after the intervention and 6 months later more people reported that they

took at least 20 minutes exercise on 3 days each week than before the study

Mental wellbeing

Scores for mental health improved after 12 weeks of walks, but the

improvement had reduced and was no longer statistically significant

6 months later

General health and other outcomes

There did not seem to be any improvement in the participants’

general health or for other outcomes

Effects of community gardening on health outcomesProject by Di Blackmore, PhD student

Aim to investigate the effect of community gardens on health and

related outcomes. The objectives of this research are to:

• explore a range of health effects for the individuals

• explore mechanisms by which the community gardening project affects health

•determine how/if the outcomes vary between the different community gardens and other variables such as the amount of time spent in the garden

Setting

4 community gardens in socially

deprived areas of Scotland

Methods Intend to recruit participants near the start of their gardening experience, and take baseline measures of stress level and physical health: blood pressure, body mass index, activity level and salivary cortisol.

In addition, participants will be asked to complete validated questionnaires that examine aspects of mental wellbeing, physical activity, quality adjusted life years, loneliness, community cohesion and social capital.

This data will be collected at baseline, some measures again at 6 weeks and all measures at 12 weeks.

Also collecting qualitative data to explore participants experiences of being involved in the projects, and how they felt they benefited from them

A feasibility study of Exercise Referral Scheme: indoor versus outdoor activities Led by Dr Larry Doi

Aim To

test the feasibility, acceptability and effectiveness of randomising patients to ERS in either indoor or outdoor activities. YES a randomised controlled trial!!

Research questions

1. What are the initial estimations of effectiveness of indoor versus outdoor activities on a range of health outcomes? [OE]

2. Are there are particular aspects of the outdoor exercise or indoor exercise, delivered via an exercise referral scheme, that confer specific health benefits? [OE]

3. Do the patients have strong preferences for setting of physical activity? [PE]

4. What are the main barriers and facilitators to implementing the outdoor intervention successfully? [PE]

5. What are the underlying mechanisms of action or change (why and how the outdoor and indoor activities have an effect on health outcomes) [PE]

6. Are there are any unintended or adverse consequences of participating in outdoor activities? [PE]

[OE] = outcomes evaluation; [PE] = process evaluation

A feasibility study of Exercise Referral Scheme: indoor versus outdoor activities Setting: Bathgate, West Lothian

Interventions 1) Indoor ERS (normal activities in leisure centre)2) Outdoor ERS Intervention to be developed but will roughly equate with duration and exercise intensify to the indoor intervention. Components may include: 1.Green gym 2.Led walks 3.Other outdoor activitiesDuration of the intervention period would be 12 weeks. Participants will be asked to only do activities in the arm of the trial to which they have been allocated. After the 12 weeks all participants will be able to continue with the physical activities of their choice.

All the outdoor activities (e.g. green gym, led walks) will remain in place for a minimum of 48 weeks (making the duration of both interventions a year in total). All participants will be followed up at one year.

Data collected routinely by the exercise referral team

•Weight loss BMI/ Abdominal Girth

•Physical activity General Practice Physical Activity Questionnaire

•Physical fitness Peak Flow

•Blood pressure Systolic BP/ Diastolic BP

•Mental health Hospital Anxiety and Depression Questionnaire

•General health The General Health Questionnaire (GHQ12)

•Patient satisfaction Satisfaction survey

•Adherence Electronic records on attendance at feedback sessions

This data is recorded electronically and is collected at several time

points (including at one year)

This ERS is in an ideal place to do high quality outcome evaluation because

they (as routine) collect data using validated tools and have a great data

management systems

Good Evaluation Resources

Evaluation Support Scotland

http://www.evaluationsupportscotland.org.uk/

BHF Exercise Referral Toolkit

http://www.bhfactive.org.uk/sites/Exercise-Referral-Toolkit/

Standard Evaluation Framework for physical activity interventions http://www.noo.org.uk/uploads/doc/vid_16722_SEF_PA.pdf

Learning, Evaluation and Planning (LEAP)

http://www.scdc.org.uk/what/LEAP/

Email: Ruth.jepson@stir.ac.uk