the alchemy project evaluation report report - the... · london | brussels | los angeles |new york...

Proprietary and Confidential

© Optimity Advisors, 2016

2

Matrix Knowledge formally joined the global consultancy group Optimity Advisors in September 2014. As its European arm, the newly combined business trades as Optimity Matrix to run the public policy arm of Optimity Advisors’ global operations. For more info go to: www.optimitymatrix.com. Optimity Matrix and Matrix Knowledge are trading names of TMKG Limited (registered in England and Wales under registration number 07722300) and its subsidiaries: Matrix Decisions Limited (registered in England and Wales under registration number 07610972); Matrix Insight Limited (registered in England and Wales under registration number 06000446); Matrix Evidence Limited (registered in England and Wales under registration number 07538753); Matrix Observations Limited (registered in England and Wales under registration number 05710927) and Matrix Knowledge Group International Inc. (registered in Maryland, USA under registration number D12395794). Disclaimer In keeping with its values of integrity and excellence, Optimity Matrix has taken reasonable professional care in the preparation of this document. Although Optimity Matrix has made reasonable efforts, it cannot guarantee absolute accuracy or completeness of information/data submitted, nor does it accept responsibility for recommendations that may have been omitted due to particular or exceptional conditions and circumstances. Confidentiality This document contains information that is proprietary to Optimity Matrix and may not be disclosed to third parties without prior agreement. Except where permitted under the provisions of confidentiality above, this document may not be reproduced, retained or stored beyond the period of validity, or transmitted in whole, or in part, without Optimity Matrix’s prior, written permission. © TMKG Ltd, 2016 Any enquiries about this project should be directed to

[email protected]

http://www.optimitymatrix.com/

mailto:[email protected]



3

Table of Contents

Plain English Summary ............................................................................................. 4

1. Introduction ...................................................................................................... 5 1.1. Background ......................................................................................................... 5

1.1.1. The purpose of this evaluation ........................................................................... 5 1.1.2. The background to the intervention evaluated .................................................. 5

1.2. Context of the intervention .................................................................................. 7 1.3. Intervention details and conceptual theory .......................................................... 8

2. Methodology .................................................................................................. 11 2.1. Evaluation logic model ....................................................................................... 11 2.2. Methods of data collection ................................................................................ 12

2.2.1. Quantitative methods ....................................................................................... 13 2.2.2. Qualitative methods .......................................................................................... 14

2.3. Methods of data analysis ................................................................................... 14 2.4. Limitations ........................................................................................................ 16

3. Results ............................................................................................................ 18 3.1. Quantitative data .............................................................................................. 18 3.2. Qualitative data ................................................................................................. 37 3.3. Economic analysis .............................................................................................. 39

3.3.1. Intervention cost ............................................................................................... 40 3.3.2. EQ-5D outcomes ............................................................................................... 41 3.3.3. Outcome star .................................................................................................... 44 3.3.4. Warwick-Edinburgh Mental Wellbeing Scale .................................................... 45 3.3.5. Interpretation .................................................................................................... 47

4. Discussion and conclusions ............................................................................. 48

5. Recommendations .......................................................................................... 50

6. Appendices ..................................................................................................... 51 6.1. Appendix 1: EQ5D .............................................................................................. 51 6.2. Appendix 2: Outcome star ................................................................................. 55 6.3. Appendix 3: Alchemy Project Evaluation ............................................................ 56 6.4. Appendix 4: The Warwick-Edinburgh Mental Well-being Scale ............................ 57



4

Plain English Summary

Introduction

An intensive contemporary dance programme was used in an innovative pilot project in 2013 as an

intervention for young people with psychosis in South London. ‘SeaBreeze’ was conducted by Dance

United in conjunction with South London and the Maudsley’s (SLaM) Early Intervention (EI) service.

Positive feedback from the EI clinical teams and service users led to a further test of the intervention with

participants drawn from the same population, to assess whether any identified impact was consistent

with the pilot results. An independent evaluation of both service impact and value for money was

commissioned to help build the case for local commissioning.

The intervention consisted of an intensive four-week programme, preceded by taster sessions during

which participants learned to perform a contemporary dance piece, while engaging in trust-building and

team-building exercises. Two groups of participants were drawn from SlaM’s EI teams and were recruited

following a thorough process by EI team members, with support and guidance from the intervention

team.

Methods

An evaluation was conducted using standardised tools that measure changes in participants’ mental

wellbeing and quality of life, focus group meetings with the intervention team and interviews with the EI

teams. A simple and limited value for money analysis was also conducted to assess the costs of the

benefits derived from the programme.

Results

The findings of the evaluation are limited by the lack of data for assessments beyond the end of the

intervention. The data available shows that, for both groups, there were improvements in participants’

self-belief, confidence and trust in others over the course of the intervetion. The intervention also

improved the quality of life of the participants in the timeframe measured (four weeks). Optimity was

unable to evaluate how sustained the effects of the intervention were over longer periods of time due to

lack of data for assessments.

The value for money analysis showed that, based on the gains observed, the intervention is unlikely to be

cost-effective unless benefits are sustained for a considerable period of time.

Recommendations

There is a need for a longer-term study to assess whether the intervention effects are sustainable in the

medium to long term. Such a study should use a larger sample size and a control group to ensure that the

findings are sufficiently powered to evidence any intervention effects, and allow for the attribution of the

effects to the intervention.



5

1. Introduction

1.1. Background

1.1.1. The purpose of this evaluation

In late 2013, an intervention, developed and tested by Dance United, was piloted in a mental health

setting for the first time. ‘SeaBreeze’ was designed to test the effectiveness of professional contemporary

dance training and performance on the mental wellbeing of young adults accessing treatment for

psychosis at an Early Intervention service. The methodology of the intervention was found to be

“…innovative, holistic and based on positive psychology rather than deficits, and one that fully

complements SLaM’s social inclusion and recovery policy.” The results, together with the positive

feedback from the EI clinical teams and service users, led to a need to further test the intervention with

participants drawn from the same population. The aims were to assess whether any identified impact was

consistent with the pilot results and build a strong case for local commission-ability through independent

evaluation of both service impact and value for money.

1.1.2. The background to the intervention evaluated

Psychosis is a mental health condition that causes people to perceive or interpret things differently from

those around them.1 The condition might also involve hallucinations or delusions. Typical treatment

involves a combination of antipsychotic medication, psychological therapies such as cognitive behavioural

therapy (CBT) and social support that might include education, training, employment or accommodation.

In the UK, the prevalence of psychotic disorders is 0.7% in the general population, and 0.2% in 16-24 year

olds.2 With limited evidence of the efficacy of antipsychotics in the latter population and a higher risk of

their side effects such as weight gain, metabolic effects and movement disorders, psychological

interventions, including family intervention, cognitive behavioural therapy (CBT) and arts therapies, have

been suggested as the preferred treatment approach for this group.3 Currently, Early Intervention in

Psychosis Services (EIS) provide those aged 14–35 years with a more intensive therapeutic service than is

obtainable with traditional community mental health services. These EI services are designed to intervene

1 Royal College of Psychiatry Mental Health Factsheet

http://www.rcpsych.ac.uk/healthadvice/parentsandyouthinfo/parentscarers/psychosis.aspx

2 Kirkbride JB, Errazuriz A, Croudace TJ, et al (2012). Incidence of schizophrenia and other psychoses in England, 1950–2009: A Systematic Review

and Meta-Analyses. PLoS One. 7(3):e31660

3 NICE (2013) CG 155: Psychosis and schizophrenia in children and young people: recognition and management



6

early, and deliver support and evidence-based interventions in a more relaxed environment for the first

three years after onset of psychosis4.

The incidence rate of psychosis in young people is higher in South London, with 55.8% more new cases of

schizophrenia and other psychoses each year, compared to England as a whole (49.4 cases per 100,000

persons per year in South London compared to 31.7 cases per 100,000 persons per year in England).5

However, these figures for South London are likely to be underestimates with other studies finding higher

incidence rates for psychosis6, thus emphasising the need for care in the area.

The South London and Maudsley National Health Service (NHS) Foundation Trust (SLaM), provides mental

health services in South London. It provides front-line EI for psychosis services via its various local teams

in Southwark (Southwark Team for Early Intervention in Psychosis - STEP), Lambeth (Lambeth Early Onset

- LEO) team, Lewisham (Lewisham Early Intervention Service – LEIS), and Croydon (Croydon Older Adults

Support Team – COAST). These services work with young adults in the first few years after the onset of

psychosis, when there is often the potential for a full recovery and return to health. The services focus on

the journeys of these young adults, offering treatment options for a maximum of three years before

discharging them to their GP. The treatments focus not only on the reduction of symptoms, but also on a

more holistic recovery with improved wellbeing, and return to education, training or employment.

In 2013, following the identification of problems faced by EI clients which included feeling isolated,

struggling with interpersonal relationships, issues with their body awareness and physical fitness which

have a negative impact on their overall levels of confidence, difficulty with maintaining energy and

optimism, and a tendency to over-focus on their condition and worry about the future7, a collaborative

programme was initiated. Senior personnel from the Institute of Psychiatry (IOP) at King’s College, London

and SLaM, having witnessed a performance by Dance United’s Academy participants, for whom

contemporary dance was used as an intervention, initiated a collaboration between Dance United, the

IoP, SLaM and two voluntary organisations (Bipolar UK and Rethink Mental Illness). This collaborative

programme pilot, SeaBreeze, used dance as an intervention for people with early psychosis. It was found

to be successful in its objectives of helping participants achieve greater satisfaction in interpersonal

relationships, improved embodied confidence and raised motivation, energy and optimism8. It also sought

to achieve some of its broader ambitions of reducing the stigma associated with accessing mental health

services and consequently reduce the co-morbidity of delayed diagnosis, highlighting the fact that modern

medications for psychosis do not impair movement functions, and drawing attention to recent radical

changes in the delivery of a care model from clinical management to holistic recovery.

4 Ibid.

5 Kirkbride JB, Errazuriz A, Croudace TJ, et al (2012). Incidence of schizophrenia and other psychoses in England, 1950–2009: A Systematic Review

and Meta-Analyses. PLoS One. 7(3):e31660

6 Campion (2013) Commissioning Support Factsheet: Prevalence, Causes and Treatment of People with Psychosis

7 Seabreeze: South London Mental Health Pilot Project Evaluation Report (2014)

8 Ibid.



7

Using this pilot as a baseline, the project team sought to prove that the effect of the intervention was not

random. To evidence the replication of the intervention results, two cohorts were selected by SLaM EIS

teams in 2015, using clear criteria and process and put through the intervention. The 2015 intervention

programme was conducted by SLaM, Dance United and Cultural Utilities Enterprises (CUE), and funded

by the Guy’s and St Thomas Charity and the Maudsley Charity. This report presents the findings of an

independent evaluation of the 2015 intervention programme.

1.2. Context of the intervention

In England, mental health and emotional wellbeing is an integral part of both the Public Health and Adult

Social Care Outcomes Frameworks. It is associated with a range of outcomes and applied to all areas of

health and care. Given the high level of mental health need, improving mental health and wellbeing makes

a vital contribution to achieving these general measures. The cross-government mental health outcomes

strategy for people of all ages, ‘No health without mental health’9, supports the vision for improving

mental health through evidence-based practical recommendations, as well as providing the framework

for improving outcomes. With the transfer of public health into local government in England there are

opportunities for arts in health, as part of community empowerment strategies and outcomes-based

commissioning, to reduce health and wider inequalities and help improve the lives of local communities.

The case for future commissioning of arts in health initiatives is underscored by the fact that sustained

investment in the arts results in significant economic savings even in the short term occurring in a wide

range of public sectors including health and criminal justice10.

Recently, the work in arts and health has been aimed at tackling key targets of national and international

health policy such as physical health, emotional wellbeing, drug and alcohol misuse, healthier lifestyles to

reduce obesity and heart disease, supporting families, healthy ageing and engaging individuals and

communities. The arts, culture and heritage can contribute to health promotion, social capital

development and community engagement. Providers of services have been encouraged to include arts as

part of offerings in health and social care settings. At SLaM, an arts strategy exists with which the Trust

develops the use of art in mental health, wellbeing and recovery. Some of the aims of the strategy include

improving patient experience through engagement in the arts, reducing ‘revolving door care’ by sustaining

people’s wellbeing and recovery, encouraging a culture of innovation, and increasing the knowledge and

evidence base of the effect of the arts on health and mental wellbeing.

Although the use of arts in healthcare has been growing in the UK for a long time, it is only recently that

significant robust research has been carried out to provide evidence of the claimed benefits. A review of

medical literature, exploring the relationship of arts and humanities with healthcare and the influence

9 Department of Health (2011) No health without mental health: a cross-government mental health outcomes strategy for people of all ages

10 Department of Health (2011). Mental health promotion and mental illness prevention: the economic case. London.



8

and effects of the arts on health, found clear, reliable evidence of clinical outcomes achieved through the

intervention of the arts including reduction in blood pressure, heart rate, length of hospital stay and

perception of pain11. It also showed the value of the arts in mental healthcare including improving clients’

communication skills, enabling them to express themselves, enhancing self-esteem, and bringing about

positive behavioural changes, as well as the differing impacts of different art forms on mental health

service users. A study carried out at Chelsea and Westminster Hospital also evaluated the effect of visual

and performing arts in healthcare and explored their effect on psychological, physiological and biological

outcomes of clinical significance12. It found that the integration of visual and performing arts into the

healthcare environment has a range of outcomes of clinical significance.

Recognition of the impact of such intervention on outcomes including mental wellbeing, social isolation

and physical health have informed their current use with these outcomes mandated by current local13 and

national policy14.

1.3. Intervention details and conceptual theory

EI services are aimed at early detection and treatment of symptoms of psychosis during the formative

years of the psychotic condition. Specialised multidisciplinary teams provide intensive case management

using a combination of low dose medication, cognitive behavioural therapy and integrated

psychotherapy.15 Clients are also offered the opportunity to participate in activities such as sports and

photography as well as support for their family and carers in partnership with a range of statutory and

non-statutory services. Results from evaluations show that EI services are cost effective in an evaluated

short term16 and gains achieved are lost when clients are moved to traditional community based mental

health teams17. However, the frequency of contact with the service tends to vary depending in individual

need, from weekly to monthly contact18, although the focused interactions at an early stage of the onset

of psychosis are considered a main feature of the success of such services19.

11 Staricoff (2004) Arts in health: a review of the medical literature.

12 Staricoff et. al. (2005). A Study of the Effects of Visual and Performing Arts in Health Care

13 South London and Maudsley NHS Foundation Trust (2014). Social Inclusion and Recovery (SIR) Strategy 2013–2018.

14 NHS England (2014). Our Ambition to reduce premature mortality. NHS, London.

15 Stafford et. al. (2013). Early interventions to prevent psychosis: systematic review and meta-analysis BMJ;346:f185

16 McCrone P, Craig TKJ, Power P, Garety PA (2010). Cost-effectiveness of an early intervention service for people with psychosis. Br J Psychiatry;

196: 377– 82

17 Gafoor R, Nitsch D, McCrone P, Craig TKJ, Garety PA, Power P, et al. (2010) Effect of early intervention on 5-year outcome in non-affective

psychosis. Br J Psychiatry; 196 : 372–6.

18 As reported by EI team members

19 Singh (2010). Early intervention in psychosis. The British Journal of Psychiatry, 196 (5) 343-345



9

The Alchemy Project intervention emphasised focused interactions with clients and was designed

specifically for people who had never danced before. It was delivered in a highly structured, safe and

supportive environment. The main intervention, which was preceded by four, two-hour taster sessions,

consisted of a four-week, full-time dance-based programme starting at 10am and ending at 4pm each

day, and included the provision of healthy meals. Participants learned how to dance, and also engaged in

trust-building and team-building exercises. The intervention culminated in the performance of El Camino,

an original dance composition choreographed by Dance Directors Carly Annable-Coop and Ellen

Steinmuller in collaboration with Darren Ellis. Post intervention dance sessions were also provided for

participants. The Project was managed by Gwen van Spijk from Cultural Utilities and Enterprises with

expert advice provided by Dr Matthew Taylor of the IoP.

Participants were recruited by SLaM EI staff with support from members of The Alchemy Project delivery

team. The recruitment process followed meetings between The Alchemy Project delivery team and EI

teams who were provided with recruitment packs which contained detailed information about the

intervention, a typical participant profile and the criteria (including clinical considerations) for selecting

prospective participants. The EI teams oversaw the recruitment process, working closely with The

Alchemy Project delivery team to jointly meet with prospective participants and explain the intervention.

Each participant signed a referral and consent form and was encouraged to attend the taster sessions at

which they could interact with dancers and ask any questions. Regular communication was maintained

throughout the intervention period with updates from the intervention team to EI care coordinators

about the progress of their clients.

A ‘theory of change’, was created to articulate how the activities and their key qualities could bring about

a number of measurable, intermediate outcomes leading to three, long-term outcomes.

1. Satisfying interpersonal relationships

Improved communication skills

Increased level of trust in others

Increased capacity to work as part of a team

2. Positive Functioning: improved embodied confidence

Improved stillness and bodily control

Increased capacity for symbolic expression

3. Positive Affect: raised motivation, energy and optimism

Improved resilience

Increased optimism

The logic model for the ‘theory of change’, which illustrates the relationship between the intervention

and the anticipated outcomes, is presented below.

Figure 1: The Alchemy Project ‘Theory of Change’



10

Two cohorts of young people participated in the project with the intervention provided in February and

June 2015. This report presents the results of an independent evaluation of the programme.



11

2. Methodology

This section presents the general research strategy or design process that underpinned this evaluation.

It includes the methods that were utilised in the evaluation of The Alchemy Project as well as a

description of the logic model being tested.

2.1. Evaluation logic model

Logic models (or ‘theories of change’) explicitly set out the hypotheses about the impact of an

intervention. They are deductive models of reasoning and can be used both formatively (during an

intervention to inform learning iteratively) and summatively (after an intervention is complete to report

on impact). In this instance, the model was used summatively. A summary of types of evaluation

considered by Optimity Advisors is presented below (Fig. 2). This evaluation was aimed and answering the

last two questions and therefore falls into the outcome/impact evaluation category.

Figure 2: Types of evaluation

The premise of The Alchemy Project intervention was that the activities and rigour associated with it, and

the discipline and commitment it requires, will lead to a number of measurable, intermediate outcomes

such as improved team working, trust and communication skills. These would in turn lead to long-term

outcomes of improved self-efficacy and confidence, and a return to Education, Training and Employment

(ETE).

The research questions set out for the evaluation were as follows:



12

Does the intervention improve Quality of Life (QoL) of participants? (Improved self-efficacy and

confidence)

Does the intervention improve participants’ interaction with the Early Intervention Service (EIS)?

(Improved interactions with the service)

Does the intervention enable progression of the participants to Education, Training and Employment

(ETE)? (Return to education, training and employment)

Is the intervention value for money? (Cost per benefit derived)

A logic model for the evaluation of these outcomes, and associated tools with which they would be

measured was designed. The model is presented below:

Figure 3: Evaluation logic model

2.2. Methods of data collection

The tools with which data was collected can be found in the Appendix. The methods with which the

evaluation was conducted are as follows:

Data was to be collected at various time points during the evaluation for each cohort, as follows:

Table 1: Data collection timetable

Time of collection Data collection tools

Baseline Participants - Outcome star, WEMWBS, EQ5D

EIS Staff - SES, Outcome star, interview

Four weeks Participants - Outcome star, WEMWBS, EQ5D

EIS Staff - Outcome star



13

Time of collection Data collection tools

Three months Participants - Outcome star, WEMWBS, EQ5D

Six months Participants - Outcome star, WEMWBS, EQ5D

EIS Staff - SES, interview

2.2.1. Quantitative methods

Data was collected from programme participants, the intervention delivery team and members of EI

teams using standardised quantitative tools. The tools selected provided data relevant to the summative

outcome and impact evaluations. For the outcome evaluation, the Warwick-Edinburgh Mental Wellbeing

Scale (WEMWBS) and the Outcome star, which assessed positive affect, personal optimism, team working

and level of trust were used. These tools had been used in the pilot and provided comparable data. For

the impact evaluation, the EQ-5D-5L was used to assess changes in the self-efficacy and confidence of the

participants.

WEMWBS: A self-reported tool for measuring mental wellbeing, using a 14-item scale with five response

categories, summed to provide a single score ranging from 14-70. The items are all worded positively and

cover both feeling and functioning aspects of mental wellbeing. For this evaluation, the WEMWBS was

used to assess outcomes (participants’ positive functioning, positive affect and improved optimism from

satisfying interpersonal relationships) at baseline, end of intervention (four weeks) and at three and six

months post-intervention.

Outcome star: The Outcome star provides an easy to use and visual way of measuring progress in mental

health settings. It is completed on a scale of one to 10 in five domains and was used to assess participants’

team work, increased trust levels, improved communication skills, improved body control, and increased

resilience. It was completed by participants, and for each participant by EI staff and the project delivery

team at baseline and end of intervention.

Service Engagement Scale (SES): The Service Engagement Scale asks 14 questions to measure service user

engagement, reported from a service provider perspective. It is a practical, reliable, and valid measure of

engagement with services that can be completed quickly. It assessed participant interaction with the EI

team and was completed by EI staff at baseline and six months.

EQ-5D-5L: The EQ-5D is a standardised user reported instrument for use as a measure of health outcomes.

It assessed improved self-efficacy and confidence as a measure of Quality of Life (QoL). It was completed

at baseline, end of intervention (four weeks), and at three and six months post-intervention.

Data collection templates: Bespoke data collection templates were prepared for the collection of project

related inputs (number, time spent and costs of staff/mentors), process (quantitative data on

recruitment) and outputs (participation, adherence and completion rates) data. This was completed

throughout by the intervention delivery team.



14

Value for money: An economic analysis was conducted. .

2.2.2. Qualitative methods

Qualitative data was collected from the project intervention team via focus group sessions conducted at

baseline and end of intervention for each cohort to assess the adequacy of participant referral and

recruitment and adherence to intervention. In addition, face-to-face interviews were conducted with EI

staff at baseline and six months to assess participant recruitment, participant engagement with EI staff,

and participants’ return to ETE.

2.3. Methods of data analysis

The analysis of the collected qualitative and quantitative data, as well as the value for money analysis,

consisted of the following:

Quantitative data analysis: Three instruments were used for the collection of quantitative data – the

Outcome star, the WEMWBS and the EQ-5D-5L. Data from the Outcome stars completed by participants,

EI and The Alchemy Project intervention staff were analysed using simple descriptive statistics and

presented pictorially. Data from the WEMWBS was analysed by individual and by cohort using simple

descriptive statistics and by paired sample and correlation statistics. The data was also analysed to show

the degrees of change for participants moving from a score band (described as low, medium or high) to

another score band. Data collected using the EQ-5D-5L was analysed using the standard EuroQol analysis

template, and presented descriptively as well as converted to quality adjusted life years (QALYs), a

standard measure for quality of life.

Qualitative data analysis: Data collected from the qualitative enquiries were summarised into a

framework to allow comparisons both within and between data sources, enabling Optimity to gain an in-

depth understanding of the key issues. Once the data had been organised and reduced into the

framework, analysis was conducted to establish the main emerging themes. The preliminary themes for

data analysis and the framework for the analysis of the data are presented below:



15

Figure 4: Qualitative analysis framework

Value for money analysis: In considering the approach to be taken, the research team sought to compare

the benefits of the intervention (derived from the EQ5D, Outcome stars and WEMWBS) to the cost of the

intervention for each cohort, to provide an understanding of value for money.

Ideally, this would take the form of a cost-benefit analysis, where benefits and costs are both expressed

in monetary units, enabling the calculation of a ratio of benefit to cost (e.g. £3 value created per £1 spent).

This is possible, even with outcomes such as mental wellbeing, as monetisation is just an attempt to

measure outcomes in common units, rather than calculating money saved. (For example, a three point

improvement in total OS score could be valued at £8,000 and a 10 point improvement in WEMWBS score

over a year could be valued at £10,000, but this simply reflects that the OS improvement is valued at 80%

of the WEMWBS improvement).

Regrettably, there is no available research on the monetary values of the OS or the WEMWBS to draw on.

Instead, for these measures, the improvement in scores is simply compared to the cost, to provide

information to support a subjective decision on value for money. The EQ-5D, however, is well established

as a metric from which quality-adjusted life year (QALY) gains can be drawn.20 QALYs are used by NICE,

among other bodies, as a way of appraising the cost-effectiveness of health interventions (known as cost-

utility analysis). One QALY equals one year of perfect health, and NICE has an informal ‘threshold’ of

between £20,000 and £30,000 per QALY, below which an intervention is more likely to be considered

cost-effective. 21 By presenting this cost-utility analysis alongside the outcomes of the OS and the

WEMWBS, the research team sought to draw an initial picture of outcomes versus cost for The Alchemy

20 https://www.nice.org.uk/article/pmg9/chapter/the-reference-case

21 This threshold is based on analysis around value for money and affordability for the NHS, and is intended for use as a guide to what is cost-

effective, rather than an explicit rule. It has also been subject to some contention in the literature, with some advocating for a lower and

others for a higher threshold. See, for example, https://www.nice.org.uk/news/blog/carrying-nice-over-the-threshold

https://www.nice.org.uk/article/pmg9/chapter/the-reference-case

https://www.nice.org.uk/news/blog/carrying-nice-over-the-threshold



16

Project. However, there are a number of limitations of this analysis. As such these results should be

considered a preliminary exploration of cost-effectiveness, with recommendations for future data

collection included.

2.4. Limitations

The aim of this project, and its evaluation is to provide further evidence of the effectiveness and value of

the intervention for the population served by EI services provided by SLaM. However, the achievement of

this aim was limited by a number of factors. These limitations included:

Bias in selection of participants

Although efforts were made to standardise the recruitment process by providing a recruitment package

and Alchemy Project staff supporting EI staff it is likely that there was some bias in the process. This is due

to the subjective application of the guidance, with staff recruiting mainly those they ‘felt’ would either

benefit from the programme, or who were available at the time. It is likely that other EI service clients

that were not selected might have derived benefit from the programme.

Sample size

The sample size is important to determine the impact of the intervention on a typical group of users. One

of the difficulties with a small sample – 22 across both cohorts in this case – is that the overall results are

very sensitive to changes in one or two people’s scores. As an example, take participant AP0112 in cohort

one. Their measured utility gain was 0.433, while the sum of the whole group’s utility gains was 0.428.

This means that without AP0112 in the group there would have been a slight utility loss overall. With a

larger sample size there would be a clearer picture of whether AP0112 is legitimately representative of a

subset of the population (if others had similar results), or an outlier whose results should not inform

overall conclusions as much as they do in a small sample. Additionally, a larger sample size may pick up

on a potential subset of the population, not represented by the participants in this study, who may

respond differently to the 22 considered here.

Lack of a control group

In the assessment of the evidence of the effectiveness of an intervention, it is difficult to determine

whether any observed outcome is directly attributable to the intervention, or a chance occurrence. The

use of a comparable control group allows for the attribution of outcomes to an intervention. The lack of

an intervention group for The Alchemy Project limits the attribution of the observed effects to the

intervention, especially for a population receiving that intervention, as well as standard care.

A control group was not used for this evaluation due to difficulties with identifying and recruiting one

from a comparable service. In addition, the lack of access to data from SLaM precluded the use of a

standard care dataset to use as the comparator.

Self-reported tools



17

There is potential for either exaggeration or underreporting of outcomes by the participants using self-

reported tools. However, this potential limitation was minimised by the use of standardised tools that

have been tested and validated for use in this population group.

Issues with data collection

The main limitation of this evaluation was the difficulty with collecting data when due. Planned data

collection was to have occurred at baseline, four weeks, three months and six months for each cohort.

However, data was only collected essentially as planned at baseline, and at four weeks with some delay.

Three and six month data for cohorts one and two were not collected despite efforts to ensure its

collection.

Issues with the data collection were primarily due to the difficulty experienced in getting relevant EI staff

to collect and send data from their clients. The busy and dynamic nature of their workloads was cited as

the reason for the failure to engage with data gathering. However, this limitation had a detrimental impact

on the ability to evidence some of the programme outcomes as medium term data such as return to ETE

was not available.



18

3. Results

3.1. Quantitative data

Participants’ recruitment and dropout

For cohort one, 15 participants were recruited for the intervention, of whom 12 completed the four-week

programme, with a completion rate of 80%. For cohort two, 18 participants were recruited for the

intervention, of whom 11 completed, with a completion rate of 61%. These compare to the pilot

‘SeaBreeze’ programme for which 18 participants were recruited, of whom 16 completed the four-week

programme,22 with a completion rate of 89%.

Recruitment rates for all three groups were similar for cohort two and the pilot group, with cohort one

having the fewest number of recruits. The time frame available for recruitment of this cohort was the

shortest of the three cohorts because it was a busy reporting period for the EI service and staff could not

be made available for selection and recruitment.

Table 2: Recruitment and completion rates

Detail Pilot Cohort one Cohort two

No. recruited

18 15 18

No. completed intervention

16 12 11

% completed

89% 80% 61%

Reasons for attrition 2 drop outs One completed the taster session and only one day in week one and decided not to continue with the project One only attended a taster session and did not return. One attended taster session only and was not allowed to return to do the project at this time – clinical decision

Four only attended a taster session and did not return. One had to travel on a family holiday. One was ill and had to drop out. One started a job and had to leave the project.

Warwick-Edinburgh Mental Wellbeing Scale Scores

22 Dance United (2014). SeaBreeze: South London Mental Health Pilot. www.dance-united.com



19

Individual scores

Individual WEMWBS scores for cohort one ranged from 26 to 56 with an average of 47.1 at baseline, and

41 to 67 with an average score of 55 at the end of the intervention. For cohort two, the scores ranged

from 38 to 64 with an average of 51.3 at baseline, and 45 to 69 with an average score of 55 at the end of

the intervention. These were comparable with scores from the pilot ‘SeaBreeze’ project for which scores

at baseline ranged from 31 to 67 with an average of 46.7 at baseline, and 37 to 70 with an average score

of 53.4 at the end of the intervention.

Figure 5: Cohort 1 individual WEMWBS scores

Table 3: Cohort 1 individual WEMWBS descriptive statistics

N Min Max Mean Std. Deviation 95% CF

Baseline 13 26 56 47.1 9.2 5.0

Week four 12 41 67 55.0 7.0 3.9

Valid N 12

Table 4: Cohort 2 individual WEMWBS descriptive statistics

N Min Max Mean Std. Deviation

95% CF

Baseline 14 38 64 51.3 7.4 3.9

Valid List ?

20

25

30

35

40

45

50

55

60

65

70

B A S E L I NE W E E K 4

INDIVIDUAL WELLBEING SCORES ( WEMWBS) AP0101

AP0102

AP0103

AP0104

AP0105

AP0106

AP0107

AP0109

AP0110

AP0111

AP0112

AP0113

AP0108



20

Week four 11 45 69 55.0 7.6 4.5

Valid N 11

Figure 6: Cohort 2 individual WEMWBS scores

Figure 7: Pilot individual WEMWBS scores

Valid List ?

30

35

40

45

50

55

60

65

70

75


INDIVIDUAL WELLBEING SCORES ( WEMWBS) AP0201

AP0202

AP0203

AP0204

AP0205

AP0206

AP0207

AP0208

AP0209

AP0210

AP0211

AP0212

AP0213

AP0214

AP0215

AP0216

AP0217

AP0218



21

Table 5: Pilot individual WEMWBS descriptive statistics

N Min Max Mean Std. Deviation

95% CF

Baseline 16 31 67 46.7 10.4 5.1

Week four 14 37 70 53.4 9.1 4.8

Valid N 14

Participants were grouped by their scores. A score of low wellbeing where the total score is less than 42,

moderate in the range of 42-58 and high for scores greater than 58. Cohort two participants showed

marked progression across groups compared to cohort one and the pilot groups, as none of the

participants were in the ‘low’ wellbeing group at four weeks as well as an increase in the proportion with

‘high’ wellbeing scores.

Valid List ?

20

30

40

50

60

70

80


INDIVIDUAL WELLBEING SCORES ( WEMWBS) P001

P002

P003

P004

P005

P006

P007

P008

P009

P010

P011

P012

P013

P015

P014

P016



22

Figure 8: Cohort 1 - proportion of participants by wellbeing group

Figure 9: Cohort 2 - proportion of participants by wellbeing group

15%8%

85%

58%

33%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Baseline Week4

Proportion of participants in each group before and after intervention

% Low wellbeing % Moderate wellbeing % High wellbeing

14%

64%

73%

21%27%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Baseline Week4

Proportion of participants in each group before and after intervention

% Low wellbeing % Moderate wellbeing % High wellbeing



23

Figure 10: Pilot group - proportion of participants by wellbeing group

Individual degrees of change

When participants were analysed with respect to their positions relative to the midpoint of the scale

(score 42) at baseline and at four weeks, in cohort one, two participants were below this point at baseline,

but only one was at four weeks. For cohort two, two participants were below the midpoint at baseline,

but none was below the line at four weeks. The pilot group was similar to cohort one, as three participants

were below the midpoint of the scale at baseline, but only one participant was below that point at four

weeks.



24

Figure 11: Cohort 1 - Individual degrees of change

30

35

40

45

50

55

60

65

70

75

25 30 35 40 45 50 55 60 65

We

ek

4

Baseline

Degrees of Change for Participants on WEMWBS

Mid-Point Line

Mid

-Poi

nt

Line



25

Figure 12: Cohort 2 - Individual degrees of change



26

Figure 13: Pilot group - Individual degrees of change

Group scores

Group average wellbeing scores increased across all groups from 47.1 to 55 for cohort one and 51.5 to 55

for cohort two. These scores compare favourably with the average results of the pilot, which increased

from 46.7 to 53.4. Group average incremental score for cohort two was slightly lower than for cohort one

and the pilot, but their baseline average group score was higher, and might suggest that the participants

had better wellbeing at baseline and consequently only improved marginally, relative to participants in

cohort one and the pilot.

Data from the paired sample and correlation statistics, also show that the effects elicited for all groups

were broadly comparable.

30

35

40

45

50

55

60

65

70

75

25 30 35 40 45 50 55 60 65

We

ek

4

Baseline

Degrees of Change for Participants on WEMWBS

Mid-Point Line

Mid

-Poi

nt L

ine



27

Figure 14: Cohort 1 – group wellbeing scores

Figure 15: Cohort 2 – group wellbeing scores

Figure 16: Pilot group - group wellbeing scores

0

10

20

30

40

50

60

70


AVERAGE WELLBEI NG SCORES

0

10

20

30

40

50

60

70


AVERAGE WELLBEING SCORES

0

10

20

30

40

50

60

70


AVERAGE WELLBEI NG SCORES



28

Figure 17: WEMWBS paired sample and correlation statistics

Outcome star Average Outcome star scores improved for all three groups from baseline. The data shows that ’Level of

trust’ was the domain showing the most consistent improvement across all the groups.

For the pilot group, ‘communication skills’ was the domain with the most marked improvement at four

weeks, while for cohorts one and two, it was ‘level of trust’.

Pilot

Descriptives N Min Max Mean Std. Deviation 95% CF

Baseline 16 31 67 46.7 10.4 5.1

Week4 14 37 70 53.4 9.1 4.8

Valid N (l istwise) 14

Paired Samples Statistics Mean N Std. Deviation Std. Error Mean

Baseline 43.9 14 7.5 2.0

Week4 53.4 14 9.1 2.4

Paired Samples Comparisons Mean N Std. Deviation Std. Error Mean t df Sig (2-tailed)

Baseline_Week4 -9.5 14 6.1 1.6 -5.8 13 0.000

Correlation Pearson Correlation Sig. (2-tailed) N

Baseline_Week4 0.743 0.002 14.0

Cohort1


Baseline 13 26 56 47.1 9.2 5.0

Week4 12 41 67 55.0 7.0 3.9



Baseline 46.5 12 9.4 2.7

Week4 55.0 12 7.0 2.0


Baseline_Week4 -8.5 12 11.3 3.3 -2.6 11 0.025


Baseline_Week4 0.0653 0.0246 12.0

Cohort2


Baseline 14 38 64 51.3 7.4 3.9

Week4 11 45 69 55.0 7.6 4.5



Baseline 50.2 11 7.2 2.2

Week4 55.0 11 7.6 2.3


Baseline_Week4 4.8 11 6.5 2.0 2.5 10 0.033


Baseline_Week4 0.618 0.033 7.0



29

Figure 18: Cohort 1 – average self-reported outcome star scores

Completed By

Outcomes Baseline Week4

Communication Skills 6.8 7.5

Resilience 6.8 7.1

Concentration and Focus 6.1 8.0

Level of trust in others 5.4 8.0

Working with others as part of a team7.2 9.2

Participants

0

1

2

3

4

5

6

7

8

9

10

CommunicationSkills

Resilience

Concentrationand Focus

Level of trust inothers

Working withothers as part of

a team

Baseline

Week4



30

Figure 19: Cohort 1 – average project team reported outcome star scores

Completed By



Resilience 4.9 8.2




Intervention Team

0

1

2

3

4

5

6

7

8

9

10

CommunicationSkills

Resilience




a team

Baseline

Week4



31

Figure 20: Cohort 1 – EI team reported outcome star scores

For cohort one scores, the participants on average, rated themselves higher on the domains assessed, when compared with the average scores given to them for the same domains by both the EI and Alchemy Project intervention teams. This is consistent with participants rating themselves high on self-assessments. This difference was more obvious for the baselines than for the four-week scores. The EI team also rated the participants higher than the project team for the same domains.

The project team and the participants scored the assessment for the four week Concentration and Focus domain the

same (eight) while the EI team scored those 6.5. It is unclear the reason for such differences in the scoring.



Resilience 5.5 6.5




0

1

2

3

4

5

6

7

8

9

10

CommunicationSkills

Resilience




a team

Baseline

Week4



32

Figure 21: Cohort 2 – average self-reported outcome star scores

Figure 22: Cohort 2 – average project team reported outcome star scores

Completed By



Resilience 7.6 8.9




Participants

0

1

2

3

4

5

6

7

8

9

10

CommunicationSkills

Resilience




a team

Baseline

Week4



33

Completed By



Resilience 4.9 9.1




Intervention Team

0

1

2

3

4

5

6

7

8

9

10

CommunicationSkills

Resilience




a team

Baseline

Week4



34

Figure 23: Cohort 2 – EI team reported outcome star scores

Figure 24: Cohort 2 – Jide Ashimi reported

Completed By



Resilience 5.2 7.4




EIS Team

0

1

2

3

4

5

6

7

8

9

10

CommunicationSkills

Resilience




a team

Baseline

Week4



35

For cohort two scores, participants on average similarly rated themselves higher on the domains assessed, when compared with the average scores given to them for the same domains by both the EI and Alchemy Project intervention teams. This difference was more obvious for the baselines than for the four-week scores. The EI team also rated the participants higher than the project team did for the same domains, with the scores by Jide Ashimi (an EI staff embedded with the project) comparable with those given by the intervention team.

Figure 25: Pilot - average self-reported outcome star scores

Completed By



Resilience 4.8 9.4




Jilde Ashmi

0

1

2

3

4

5

6

7

8

9

10

CommunicationSkills

Resilience




a team

Baseline

Week4



36

EQ5D scores

There were improvements in mean quality of life scores reported for both cohorts. However, although

both cohorts had similar baseline mean scores, cohort one showed greater mean improvement in quality

of life scores than cohort two. This difference appears to be consistent with the finding from the Outcome

stars and WEMWBS scores for which cohort one had lower baseline scores than cohort two, and showed

a more obvious improvement at four weeks.

For cohort one, baseline EQ-5D scores ranged from 0.567 to one, and at week four the range was 0.837

to one. The greatest improvement in scores was 0.43, the lowest was a decline of 0.163. Of the 12

participants who provided week four EQ-5D scores (out of 13), three suffered a decline in utility, five

remained the same and three saw an improvement. The total improvement in utility scores was 0.4280,

or 0.036 per person. However, as the EQ5D is self-reported, it is unclear what effect any exaggerated

scoring might have had on the results.

Figure 26: Cohort 1 EQ5D scores

Completed By



Resilience 3.0 6.0




Participants

0

1

2

3

4

5

6

7

8

9

10

CommunicationSkills

Resilience




a team

Baseline

Week4



37

Figure 27: Cohort 2 EQ5D scores

Service Engagement Scale

The SES was not used for the evaluation of the pilot. However, only baseline data was collected for both

cohorts one and two, due to the difficulty in collecting the data from EI staff, one of the key limitations of

the study. Consequently, the data is not presented.

3.2. Qualitative data

Qualitative data was collected from EI staff via face-to-face and telephone interviews, and from members

of The Alchemy Project team through focus group meetings. The qualitative enquiries sought to

determine their perceptions of the intervention, its impact on the participants, and challenges they faced

during the intervention and its evaluation. The key themes that emerged are presented below.

Interviews with EI staff



38

Face-to-face baseline interviews were conducted with seven EI staff responsible for cohort one

participants, while five interviews were conducted for cohort two. A total of eight interviews were

conducted post intervention, although three of the interviewed staff had clients in both cohorts. Using

the analytic framework described above, data from the interviewees was analysed thematically. Some of

the themes that emerged are as follows:

The relevance of the project for the participants

All the interviewed staff noted that the intervention was well received by their clients. A repeating theme

was that all their clients had improved social skills, greater confidence, and had started taking interest in

things other than their health status. Most stated that their clients had been socially isolated prior to the

intervention, but had since become more involved in activities around them.

One had been pessimistic about his client’s ability to participate in the programme, but was pleasantly

surprised to find that the client fully participated, and “…was attending sessions on time!”

One interviewee, however, noted that both clients she had on the programme had relapsed and been

admitted into acute care. She stated, though, that it was unlikely to be linked to their participation in the

programme.

All the interviewed staff noted that the full time nature of the intervention provided structure to their

clients, increased their levels of physical activity, with some reporting weight loss during the period.

The longer term impact of the project on the participants

Most of the interviewees noted that it was unclear what the longer-term effects of the intervention might

be. This was primarily due to the fact that they felt that the participants essentially went from being

occupied with activities five days a week for a month, to having a once weekly dance session, and this did

not provide adequate continuity for participants.

While some suggested that a post-intervention activity, more frequent than the once a week session in

place might help, others suggested that the difficulty and cost of such an on-going effort might make it

unlikely.

Difficulties experienced

The main difficulty experienced by the interviewees, was the requirement to help collect data from

participants in the midst of a busy schedule with multiple competing priorities. Some suggested that

although they valued the impact of the intervention on their clients, the burden of data collection

requirement might be better placed elsewhere.

Focus groups with The Alchemy Project team



39

Focus group meetings were held with members of The Alchemy Project team, at baseline and end of the

intervention for each cohort. Using the analytic framework described above, data from the meetings was

analysed thematically. Some of the themes that emerged are as follows:

Participant attributes

At baseline and end of intervention, the project team was asked to describe each cohort with attributes

pertinent to a dance company, and rate them on a scale of one to 10, with one the least rating and 10 the

best. For cohort one, the pertinent features were trust, social interaction, wellbeing, sense of self and

confidence. The ratings improved from two to nine for trust, four to 10 for social interaction, three to nine

for wellbeing, two to 10 for sense of self, and one to 10 for confidence. For cohort two, the features were

support for others, trust, confidence, willingness to learn, self-belief and mastering movement. The

ratings improved from seven to 10 for support for others, four to nine for trust, three to nine for

confidence, six to 10 for willingness to learn, two to 10 for self-belief and two to eight for mastering

movement.

Difficulties experienced

The team reported that the main difficulty they experienced was the short turnaround time for the

recruitment of participants and commencement of the intervention for cohort one. This was suggested

to be the cause of the low recruitment number for that cohort. This difficulty was, however, not

experienced with cohort two.

Things to improve on

For cohort one, the team considered that some of the things that could be improved upon included the

information provided for the EI staff about the intervention and the evaluation in order to better educate

them and foster increased acceptance. These elements were implemented for cohort two and the team

felt it helped improve the recruitment of participants.

3.3. Economic analysis

Three main outcome measures were collected from participants at the beginning and the end of the

intervention (after four weeks), for both cohorts. These comprised EQ-5D scores, Outcome star (OS)

scores, and Warwick-Edinburgh Mental Wellbeing Scale (WEMWBS) scores. All are compared to the cost

of the intervention for each cohort to provide an understanding of value for money.

Ideally, this would take the form of a cost-benefit analysis, where benefits and costs are both expressed

in monetary units, enabling the calculation of a ratio of benefit to cost (e.g. £3 value created per £1 spent).

This is possible, even with outcomes such as mental wellbeing, as monetisation is just an attempt to

measure outcomes in common units, rather than calculating money saved. (For example, a three point

improvement in total OS score could be valued at £8,000 and a 10 point improvement in WEMWBS score



40

over a year could be valued at £10,000, but this just reflects that the OS improvement is valued at 80% of

the WEMWBS improvement).

Unfortunately, there is no available research on the monetary values of the OS or the WEMWBS to draw

on. Instead, for these measures, the improvement in scores is simply compared to the cost, to provide

information to support a subjective decision on value for money. The EQ-5D, however, is well established

as a tool from which quality-adjusted life year (QALY) gains can be drawn.23 QALYs are used by NICE,

among other bodies, as a way of appraising the cost-effectiveness of health interventions (known as cost-

utility analysis). One QALY equals one year of perfect health, and NICE has an informal ‘threshold’ of

between £20,000 and £30,000 per QALY, below which an intervention is more likely to be considered

cost-effective.24

By presenting this cost-utility analysis alongside the outcomes of the OS and the WEMWBS an initial

picture of outcomes versus cost for the Alchemy Project can be seen. However, there are a number of

limitations of this analysis, including a small sample size, lack of a control group, short follow-up as well

as additional potential benefits not covered by these tools (such as productivity benefits and reduced

health service utilisation). As such these results should be considered a preliminary exploration of cost-

effectiveness, with recommendations for future data collection included.

3.3.1. Intervention cost

In order to provide an assessment of value for money, the cost of the intervention must be calculated.

These figures were provided by Alchemy, and the column referring to ‘actual’ costs was used for

calculations. Planning, set up and development costs were £11,807 for cohort one and £13,715 for cohort

two. (This difference results from the need to train a new member of staff for cohort two.25) The £43,000

project management and running costs were divided in two, representing £21,500 per intervention. This

gave a total cost for cohort one of £85,463; and for cohort two of £87,661. Documentation and evaluation

costs, as well as the cost of an ongoing dance group, were not considered part of the intervention cost.

Per person costs were also calculated. For this the total cost was divided by the number of people who

completed the programme and reported results (11 in cohort one26 and 11 in cohort two), rather than

23 https://www.nice.org.uk/article/pmg9/chapter/the-reference-case

24 This threshold is based on analysis around value for money and affordability for the NHS, and is intended for use as a guide to what is cost-

effective, rather than an explicit rule. It has also been subject to some contention in the literature, with some advocating for a lower and

others for a higher threshold. See, for example, https://www.nice.org.uk/news/blog/carrying-nice-over-the-threshold

25 While this means that future rollout of the intervention could be slightly cheaper if trained staff were already available, there is likely to be

some need to train staff in future, and so this cost has been included in the analysis.

26 Twelve participants reported results in cohort one, but one – participant AP0110, was excluded due to their self-reported baseline results

being significantly anomalous. This is discussed further in the EQ-5D section

https://www.nice.org.uk/article/pmg9/chapter/the-reference-case

https://www.nice.org.uk/news/blog/carrying-nice-over-the-threshold



41

the number starting at baseline (13 and 18 respectively). Cost per person was therefore £7,531 for cohort

one and £7,731 for cohort two.

Table 6: Intervention cost

Intervention cost Raw data For analysis

Cohort one Cohort two Total Cohort one Cohort two

Planning set up and development £11,807 £13,715 £11,807 £13,715

Delivery £49,531 £49,821 £49,531 £49,821

Ongoing dance group N/a

Project mgmt. & running costs £43,000 £21,500 £21,500

Documentation & evaluation N/a

Total £82,838 £85,036

Number completing programme 11 11

Cost per person £7,531 £7,731

3.3.2. EQ-5D outcomes

EQ-5D values were converted into utility scores using a conversion table obtained from EuroQol, who

developed the questionnaire.27 Utility scores are the quality of life aspect of QALYs, and they range from

zero (death) to one (full health). A utility score of one sustained for one year equals one QALY.

For cohort one, results were obtained for the 12 participants (out of 13). However, one of these results

(for participant AP0110) showed a baseline score of 0.097, i.e. very close to death, and a week four score

of 0.837, i.e. not far from perfect health. This was assumed to be inaccurate self-reporting and was

removed from calculations. As such, baseline EQ-5D scores ranged from 0.567 to one, and at week four

the range was 0.837 to one. The greatest improvement in scores was 0.43, the lowest was a decline of

0.163. Of the 12 participants who provided week four EQ-5D scores (out of 13), three suffered a decline

in utility, five remained the same and three saw an improvement. The total improvement in utility scores

was 0.4280, or 0.036 per person. However, that aside, from one other participant, AP0112, who saw a

0.433 utility improvement, there would have been a slight decline in utility scores. As such, the small

sample size and lack of a control group for this intervention make it difficult to make firm conclusions

based on these values.

Table 7: Utility gains for cohort 1

Participant ID Baseline Week four

State EQ-5D-5L index value State EQ-5D-5L index value Utility Gain

AP0101 11112 0.879 11121 0.837 -0.042

27 http://www.euroqol.org/about-eq-5d/valuation-of-eq-5d/eq-5d-5l-value-sets.html

http://www.euroqol.org/about-eq-5d/valuation-of-eq-5d/eq-5d-5l-value-sets.html



42

AP0102 11113 0.848 11113 0.848 0.000

AP0103 11111 1 11111 1 0.000

AP0104 11112 0.879 11111 1 0.121

AP0105 11111 1 11111 1 0.000

AP0106 11111 1 11111 1 0.000

AP0107 11212 0.837 11121 0.837 0.000

AP0108 11222 0.736 99999 #N/A

AP0109 11111 1 11121 0.837 -0.163

AP0110** 33444 0.097 11121 0.837 0.740**

AP0111 11112 0.879 11111 1 0.121

AP0112 23322 0.567 11111 1 0.433

AP0113 11112 0.879 11121 0.837 -0.042

Total 0.4280

Average 0.036

** Removed from calculations

For the 11 participants in cohort two who reported results, a more uniform slight utility increase was

observed, although the small sample size and lack of a control group prevent this from being interpreted

with a great deal of confidence. Baseline utility ranged from 0.606 to one, and at week four from 0.635 to

one. The total improvement was slightly larger for cohort two, at 0.6690, with an average of 0.06 per

person.

Table 8: Utility gains for cohort 2



AP0201 11111 1 11111 1 0.000

AP0202 12223 0.634 11121 0.837 0.203

AP0203 11123 0.75 11111 1 0.250

AP0204 11112 0.879 11114 0.635 -0.244

AP0205 11214 0.606 11233 0.696 0.090

AP0206 11222 0.736 11122 0.768 0.032

AP0207 11113 0.848 11111 1 0.152

AP0208 11111 1 99999 #N/A

AP0209 99999 #N/A 99999 #N/A

AP0210 11211 0.906 11211 0.906 0.000

AP0211 11123 0.75 31131 0.727 -0.023

AP0212 99999 #N/A 99999 #N/A

AP0213 12222 0.649 12121 0.737 0.088



43



AP0214 11112 0.879 99999 #N/A

AP0215 99999 #N/A 99999 #N/A

AP0216 99999 #N/A 99999 #N/A

AP0217 11112 0.879 11111 1 0.121

AP0218 11111 1 99999 #N/A

Total 0.6690

Average 0.061

In order to convert these scores into QALY gains how long this improvement was sustained for needs to

be known. Unfortunately, this is not possible given that the post-intervention data is only from one time

period (four weeks). As such, a range of scenarios is presented, and calculated costs and cost per QALY

for each: one month, three months, six months, one year, two years, five years and 10 years. These are

shown below:

Table 9: QALY gains and cost per QALY

Cohort one Cohort two

Total utility gain 0.428 0.669

Benefit sustained

for

Undiscount

ed QALY

gain

Discounte

d QALY

gain

ICER Undiscounted QALY

gain

Discounte

d QALY

gain

ICER

1 month 0.04 0.04 £2,322,56

1

0.06 0.06 £1,525,31

5

3 months 0.11 0.11 £774,187 0.17 0.17 £508,438

6 months 0.21 0.21 £387,093 0.33 0.33 £254,219

1 year 0.43 0.43 £193,547 0.67 0.67 £127,110

2 years 0.86 0.85 £97,494 1.34 1.33 £63,555

5 years 2.14 2.08 £39,870 3.35 3.25 £25,422

10 years 4.28 4.01 £20,677 6.69 6.26 £12,711

The ‘undiscounted QALY gain’ column multiplies the total utility gain by the time period (in years) to

calculate the QALY gain from each intervention. For instance, if it is assumed that the intervention benefit

is sustained for one month, the 0.428 utility gain in cohort one represents 0.04 QALYs gained. However,

if the intervention benefit were sustained for 10 years, this would represent 4.28 QALY amongst the

cohort.

The discounted QALY gains column introduces a concept called discounting, which is regularly used in

economic analysis. It represents the concept that a benefit in the future is not equal to the same benefit



44

now, by ‘discounting’ future benefits by a given amount each year.28 Following NICE guidance29, a discount

rate of 1.5% has been used. This means that for cohort one, were the benefit to be sustained for 10 years,

the discounted QALY gain is 4.01 QALYs, slightly less than the undiscounted 4.28.

The third column for each cohort calculated the ICER, or Incremental Cost-Effectiveness Ratio. This

represents the marginal intervention cost divided by discounted QALY gain, i.e. cost per QALY. The NICE

threshold policy for health interventions generally takes a cost per QALY of £20-30,000 or below to mean

an intervention is cost-effective. As the table above shows, this would only occur if the benefit was

sustained for several years. The years of benefit required at each threshold value were calculated for each

cohort and presented below:

Table 10: Years of benefit required for NICE cost-effectiveness thresholds

Threshold value Benefit required in years

Cohort one Cohort two

Using undiscounted

QALY gains

Using discounted

QALY gains

Using undiscounted

QALY gains

Using discounted

QALY gains

£20,000 9.7 10.4 6.2 6.6

£30,000 6.5 6.7 4.1 4.3

The table shows that, using discounted QALY gains as is the recommended approach, cohort one benefit

would have to be sustained for 10.4 years to achieve a £20,000 ICER, or 6.7 years for a £30,000 ICER.

Cohort two, with a slightly higher utility gain, would require 6.6 years of benefit using the £20,000

threshold, or 4.3 at the £30,000 threshold. Overall, the suggestion here is that the benefit would have to

be sustained for several years following a one-off intervention to be cost-effective in terms of utility gain,

which is unlikely to be the case.

However, given the focus of the OS tool on non-health outcomes, and the focus of the WEMWBS on

mental health outcomes that reaches beyond the one EQ-5D mental health question, there may be some

benefits captured by those tools that is not covered in the EQ-5D, and this is discussed below.

3.3.3. Outcome star

The Outcome star measures five areas of benefit on a one to 10 scale: communication skills, resilience,

concentration and focus, level of trust in others, and working with others as part of a team. It was

completed by participants, the intervention team, the EIS team, and, for cohort two participants and Jide

Ashimi. Results were averaged across all of these assessors to provide the average scores below.30

28 Costs should also be discounted if occurring beyond the first year of the intervention (which is not the case for this intervention)

29 https://www.nice.org.uk/article/pmg4/chapter/1%20introduction

30 As per the EQ-5D scores, participant AP0110 was removed from results



45

Table 11: Outcome star results

Category Cohort one Cohort two

Before After Improvement Before After Improvement

Communication Skills 5.6 6.9 1.2 5.3 8.3 3.0

Resilience 5.8 7.1 1.3 5.6 8.7 3.1

Concentration and Focus 5.5 7.5 2.0 5.6 8.3 2.7

Level of trust in others 5.4 7.6 2.1 5.2 8.2 3.0

Working with others as part of a

team 5.5 7.9 2.4 6.1 8.8 2.7

Total improvement 9.1 14.4

As the results show, an improvement occurred across each of the five categories for both cohorts. This

was more significant in cohort two, who saw most improvements in the range of two to three out of 10,

while cohort one saw improvements of around one to two points out of 10.

In the absence of any monetary valuation available for these results, or a similar cost-effectiveness

methodology to that used for EQ-5D results, the cost per improvement in OS score can only be presented.

The cost per person achieving a one point increase has been calculated. Results are shown below.

Table 12: Cost per improvement in OS score

Category Cohort one Cohort two

Improvement

Cost /

Improvement Improvement

Cost /

Improvement

Communication Skills 1.2 £6,106 3.0 £2,611

Resilience 1.3 £5,648 3.1 £2,504

Concentration and Focus 2.0 £3,693 2.7 £2,889

Level of trust in others 2.1 £3,574 3.0 £2,598

Working with others as part

of a team 2.4 £3,167 2.7 £2,858

Total 9.1 £828 14.4 £537

The table shows that, overall, it cost £828 for every point increase in OS score per person in cohort one,

and £537 in cohort two.

3.3.4. Warwick-Edinburgh Mental Wellbeing Scale

The Warwick-Edinburgh Mental Wellbeing scale measures a number of aspects of mental wellbeing, and

aggregates these into one score, ranging from 14 (the lowest value) to 70. Although the EQ-5D also

contains a question on mental health (in terms of depression and anxiety), the WEMWBS is a mental



46

health specific tool covering areas beyond that captured in the EQ-5D, suggesting that at least some

WEMWBS benefits are additional to EQ-5D improvements.

For this economic analysis, the cost per improvement in three areas was measured: the mean score, the

number of people achieving meaningful positive change, and the number moving from one category of

wellbeing (low, medium, high) to another. (Note that the number moving category counts those moving

up a category as +1 and moving down a category as -1.) These results are provided in the table below31:

Table 13: WEMWBS Results

Cohort one Cohort two B

efo

re

Aft

er

Imp

rove

men

t

Co

st/

imp

rove

men

t

Bef

ore

Aft

er

Imp

rove

men

t

Co

st/

imp

rove

men

t

Mean score 48.3 54.4 6 £1,255 50.2 55.0 4.8 £1,595

No. people with meaningful positive change 7 £11,834 7 £12,148

No. moving up a category (net) 3 £27,613 3 £28,345

There was a general improvement in both cohorts – six points for cohort one and 4.8 for cohort two.

Interestingly, the mean score after the intervention was the same for both cohorts, so the difference

represents the higher baseline values in cohort two. Using cost per person figures as for OS scores, the

cost per one point improvement per person was £1,255 for cohort one, and £1,595 for cohort two.

In addition, results were provided showing the number of people who achieved a ‘meaningful positive

change’ (defined as an increase greater than two points) was seven for both cohorts. Using the total

intervention cost, this represents £11,834 per person achieving meaningful positive change for cohort

one, and £12,148 for cohort two.

Finally, results were categorised into ’low‘, ’moderate‘ and ’high‘ wellbeing, using the same scale as the

pilot programme.32 Based on this, the numbers of people who moved from one category to another were

calculated. Cohort one saw one participant move from low to moderate wellbeing, three move from

moderate to high wellbeing, and one move down from moderate to low wellbeing. This was considered

for the analysis as a net gain of three people moving up a category. Similarly for cohort two, two people

moved from low to moderate, two from moderate to high, and one from high to moderate, also giving a

net gain of three. The cost for both cohorts per person moving up a category was, therefore, just under

£30,000.

31 As per EQ-5D & OS, participant AP0110 was removed from the cohort one results

32 Low wellbeing is a score <42, high wellbeing is a score >58, and moderate wellbeing is a score between the two



47

3.3.5. Interpretation

As indicated in the introduction to this section, caution should be exercised when interpreting this analysis

for a number of reasons. There were three key limitations – small sample size, lack of a control group and

a short follow-up. The small sample size (11 in each cohort) means that results are very sensitive to the

scores of individual participants which, when combined with the self-reported nature of the outcomes,

means it is difficult to judge how accurate outcomes results are. Similarly, the lack of a control group

means results cannot be compared against what would have happened without the intervention – some

participants who declined in mental health outcomes may have declined more without the intervention,

and vice versa. Finally, the short follow-up prevents understanding of how benefits are sustained over

time, which is crucial for making assessments of the value of a programme – if participants quickly return

to the same state they were in at baseline the benefits are limited.

What the results do show is that, based on the utility gain observed, the intervention is unlikely to be cost-

effective unless benefits are sustained for a considerable period of time. While the OS and WEMWBS

provide potentially complementary benefits, it is difficult to adjudicate the value of this in isolation,

especially given the limitations mentioned above. However, other potential benefits, which could add to

the intervention’s cost-effectiveness, were not measured: these include productivity benefits through

improved employment and training outcomes as well as health service utilisation (which could provide a

direct cost saving to the NHS).

In summary, the analysis would suggest that further follow-up, the rollout of the intervention to a larger

group and the addition of a well-matched control group is likely to greatly improve insight into the value

for money of The Alchemy Project.



48

4. Discussion and conclusions

The evaluation sought to assess whether the outcomes elicited in the pilot can be reproduced in

comparable cohorts, and to extend the evidence base by considering medium term outcomes. The

questions posed were:

Does the intervention improve Quality of Life (QoL) of participants? (measured by improved self-

efficacy and confidence)

The results showed that for both cohorts, there were improvements in self-efficacy and confidence

over the duration of the intervention. For cohort one, QALY scores ranged from 0.567 to one at

baseline, and at week four the range was 0.837 to one. The greatest improvement in scores for a

participant was 0.43, the lowest was a decline of 0.163. The total improvement in utility scores for

cohort one was 0.4280, or 0.036 per person. For cohort two, QALY scores ranged from 0.606 to one,

and at week four from 0.635 to one. The total improvement was slightly larger for cohort two, at

0.6690, and the average was 0.06 per person. These indicate good effect of the intervention on

participants in the short term. QoL, as a study that assessed the appropriateness of using the EQ–5D

to measure improvements in mental health (participants with psychosis), 33 showed a post-

intervention mean change of 0.029 to 0.117. Thus it can be stated that the intervention improved the

quality of life of the participants in the timeframe measured (four weeks of the intervention). Data is

not available to demonstrate if this improvement is sustained over time.

Does the intervention improve participants’ interaction with the Early Intervention Service?

(measured by improved interactions with the service)

The evaluation sought to assess whether there was any change in interactions with the EI service but

the data collected was insufficient. This was essentially due to the lack of post intervention data.

Consequently, this question cannot be answered by this evaluation.

Does the intervention enable progression of the participants to Education, Training and

Employment (ETE)? (measured by return to education, training and employment)

One of the anticipated programme impacts was a return of participants to education, training or

employment. This was to be evidenced by either a return to ETE at six months, or at the least

interactions with the service’s social interaction team. However, due to the lack of data, this

evaluation cannot answer this question.

Is the intervention value for money? (measured by cost per benefit derived)

A value for money analysis, as proposed in the original methodology, was not conducted because of

the limitations mentioned above. Based on the utility gain observed, however, the intervention is

33 Barton et al (2009). Measuring the benefits of treatment for psychosis: validity and responsiveness of the EQ–5D. The British Journal of

Psychiatry



49

unlikely to be cost-effective unless benefits are sustained for a considerable period of time. Other

potential benefits, which could add to the intervention’s cost-effectiveness, were also not measured:

these included productivity benefits through improved employment and training outcomes as well

as possible lower health service utilisation (which could provide a direct cost saving to the NHS).



50

5. Recommendations

This evaluation sought to assess whether the intervention improves the mental wellbeing and quality of

life of participants as evidenced by the pilot as well as its value for money.

Despite the limitations the results indicate that the programme results are positive in the short term and

replicated across the pilot and both cohorts. However, in order to further strengthen the evidence base

for using this intervention in the EI population, the following recommendations can be made:

There is a need for a longer-term study to assess whether the intervention effects are sustainable in

the medium to long term. This will also provide further evidence to support the value for money of

the intervention, especially from a wider societal perspective.

The use of a larger sample size and a control group in any further research will ensure that findings

will be sufficiently powered to evidence any intervention effects, and allow for the attribution of the

effects to the intervention.

As the literature and anecdotal evidence suggests that there are wider health benefits to this

intervention, further research should include an assessment of these wider health benefits such as

physical activity.

This evaluation noted the limited nature of the tools available for assessing some of the ‘softer’

effects of the intervention. Any further research should consider other tools, including qualitative

tools that might better assess such outcomes.

Considering the possible selection bias encountered in this study, any further evaluation of such an

intervention should include an assessment of the selection process, in order to determine whether

likely participants that might benefit from it are not excluded due to such bias.

A key limitation of this evaluation was the difficulty in accessing data. It is likely that the embedding

of an evaluation and tools used to collect data into routine EI work and data collection, will greatly

reduce the difficulties experienced with the current evaluation.

There is a need to ensure some continuity or gradual reduction of the pace of activities participants

experience over the course of the intervention. This softer landing has been suggested by EI staff as

a necessity to allow for a fostering of the cadence of the activities experienced during the

intervention.



51

6. Appendices

6.1. Appendix 1: EQ5D

Health Questionnaire

English version for the UK



52

Under each heading, please tick the ONE box that best describes your health

TODAY.

MOBILITY I have no problems in walking about

I have slight problems in walking about

I have moderate problems in walking about

I have severe problems in walking about

I am unable to walk about

SELF-CARE I have no problems washing or dressing myself

I have slight problems washing or dressing myself

I have moderate problems washing or dressing myself

I have severe problems washing or dressing myself

I am unable to wash or dress myself

USUAL ACTIVITIES (e.g. work, study, housework,

family or leisure activities) I have no problems doing my usual activities

I have slight problems doing my usual activities

I have moderate problems doing my usual activities

I have severe problems doing my usual activities

I am unable to do my usual activities

PAIN / DISCOMFORT I have no pain or discomfort

I have slight pain or discomfort

I have moderate pain or discomfort

I have severe pain or discomfort



53

I have extreme pain or discomfort

ANXIETY / DEPRESSION I am not anxious or depressed

I am slightly anxious or depressed

I am moderately anxious or depressed

I am severely anxious or depressed

I am extremely anxious or depressed



54

The worst health

you can imagine

We would like to know how good or bad your health is TODAY.

This scale is numbered from 0 to 100.

100 means the best health you can imagine.

0 means the worst health you can imagine.

Mark an X on the scale to indicate how your health is TODAY.

Now, please write the number you marked on the scale in the box

below.

The best health you

can imagine

YOUR HEALTH TODAY =

10

0

20

30

40

50

60

80

70

90

100

5

15

25

35

45

55

75

65

85

95



55

6.2. Appendix 2: Outcome star

OUTCOME STAR:

SCORE: 1-10 (1 being low and 10 being high) on the following

statements – Please Circle:

My communication skills: such as speaking with confidence and listening to others 1 2 3 4 5 6 7 8 9 10

My resilience: my capacity to overcome obstacles in order to achieve something 1 2 3 4 5 6 7 8 9 10

My capacity to maintain concentration and focus on what I am doing 1 2 3 4 5 6 7 8 9 10 My level of trust in others 1 2 3 4 5 6 7 8 9 10

Baseline assessment Participant: AP02 Completed by:



56

My capacity to work as a team 1 2 3 4 5 6 7 8 9 10

6.3. Appendix 3: Alchemy Project Evaluation

Service Engagement Scale (Tait et al. (2002)

Note: Items are rated 0 (not at all or rarely), 1 (sometimes), 2 (often), 3 (most of the time).

*Reverse scored (i.e. 0 = most of the time, 3 = not at all or rarely). Please circle relevant response.

Availability

1. The client seems to make it difficult to arrange appointments

2. When a visit is arranged, the client is available*

3. The client seems to avoid making appointments

Collaboration

4. If you offer advice, does the client usually resist it?

5. The client takes an active part in the setting of goals or treatment plans*

6. The client actively participates in managing his/her illness*

Help seeking

7. The client seeks help when assistance is needed*

8. The client finds it difficult to ask for help

9. The client seeks help to prevent a crisis*

10. The client does not actively seek help

Treatment adherence

11. The client agrees to take prescribed medication*

12. The client is clear about what medications he/she is taking and why*

13. The client refuses to co-operate with treatment

14. The client has difficulty in adhering to the prescribed medication

Baseline assessment Participant: AP02 Completed by:



57

6.4. Appendix 4: The Warwick-Edinburgh Mental Well-being Scale

Below are some statements about feelings and thoughts.

Please tick the box that best describes

your experience of each over the last

two weeks STATEMENTS

None of the time Rarely Some of the

time

Often All of the

time

I’ve been feeling optimistic about the

future

1 2 3 4 5

I’ve been feeling useful 1 2 3 4 5

I’ve been feeling relaxed 1 2 3 4 5

I’ve been feeling interested in other

people

1 2 3 4 5

I’ve had energy to spare 1 2 3 4 5

I’ve been dealing with problems well 1 2 3 4 5

I’ve been thinking clearly 1 2 3 4 5

I’ve been feeling good about myself 1 2 3 4 5

I’ve been feeling close to other people 1 2 3 4 5

I’ve been feeling confident 1 2 3 4 5

I’ve been able to make up my own mind

about things

1 2 3 4 5

I’ve been feeling loved 1 2 3 4 5

I’ve been interested in new things 1 2 3 4 5

I’ve been feeling cheerful 1 2 3 4 5

the alchemy project evaluation report report - the... · london | brussels | los angeles |new york...

Documents