@

How to robustly evaluate health

and wellbeing interventions

Welcome23rd April 2015

Dr Tony Zarola

The current position

• Importance of health and well-being established

• Lots of investment (time, money and action) to promote

and protect health and well-being

• Still lacking good quality evidence on whether and/or how

interventions work (or not)

Why should we bother?

• Quality assurance

• Understanding of intervention

• Impact on staff and service

• Duty of care

• Sharing learning

QUIDS

Challenge: Time

Challenge: Criteria

Challenge: Process

Challenge: Technical

Checklist

1. Aims of the evaluation

2. Gathering information

3. Formulating key questions

4. Designing the evaluation

5. Project resources

6. Organisational context

7. Communication strategy

8. Reflecting on practice

Evaluating PhysioPlus

• Team keen to demonstrate the broader value of the

intervention for Trust staff

• Sickness absence seen as a key driver/statistic

against which to assess success or value

• General statistics, e.g. number of patients seen and

patient experience, necessary but not sufficient

Evaluation process

• Longitudinal design – repeated assessments

• Surveys and some interviews where necessary

• Strongest design for assessing interventions

FIRST BASELINE

PRE TREATMENT SURVEY

POST SESSION SURVEYS

FOLLOW UP POST

TREATMENT SURVEY

What’s happening over time?

0

2

4

6

8

10

12

14

Time 1 Time 2 Time 3

Trend 1

Trend 2

Trend 3

Sample of evaluation criteria

Importance of pain confidence (PC)

PAIN CONFIDENCE

Burnout Sub Scales Low High Significant

Emotional Exhaustion 16.93 14.79 Sig. <0.05

Cynicism 12.07 10.25 Sig. <0.05

Professional Efficacy 32.03 33.52 Sig. <0.05

• Staff with higher levels of pain confidence lower

exhaustion and cynicism and higher professional

confidence

What’s happening: PC over time?

30

32

34

36

38

40

42

44

Baseline P1 P2 P3 P4 P5

Scal

e 0

-6

0

CHRONIC CASES –ADDITIONAL SUPPORT ?

Time of measurement

RECALIBRATION EFFECT

Overall

Yes79%

No21%

PhysioPlus Prevented The Need For Time Off

Staff quotes

“Without the level of treatment and support that I received from the therapists, I’m sure it would

have taken me a great deal longer to return to work.”

“I found it gives you a positive approach to actually managing the pain and that acute episode

really, It could have led to a couple of days off sick but it helped to prevent that”

Treatment confidence and anxiety

• Levels of confidence increase over time and across/between sessions

• Levels of treatment anxiety reduce over time and across/between sessions

Manager quotes

“They’re teaching staff how to manage their symptoms. They’re also

treating the symptoms and are reducing pain levels. This enables staff

to be at work rather than off sick.”

“It has kept staff at work and able to do their job effectively. As a result of the

advice and guidance about how to cope or deal with their condition, It has kept them

feeling psychologically very strong. ”

PhysioPlusSheffield Teaching Hospitals

Staff Physiotherapy Service

Service evaluation

David Craig April 2015

Brief history2007 First outline business case written with proposal to start staff physiotherapy

service. The main drivers were:

• concerns over level of staff absence due to musculoskeletal disorders

• recent publication by Dame Carol Black and Dr Steve Boorman

• increasing awareness of the importance of staff wellbeing.

2009 Service started in Sheffield PCT (3000 staff based in community settings

across Sheffield).

2011 Sheffield PCT Provider Services, including PhysioPlus transfer to Sheffield

Teaching Hospitals as part of Transforming Community Services (17,000

staff based mostly based in 2 campuses).

2011 Pilot service started in Obstetrics and Gynaecology Care Group.

2012 Pilot extended to Specialised Medicine Care Group.

2014 Service expanded and open to whole of STH.

Service principles:

• Self referral

• Fully confidential service, not a management reporting service

• Targeted at staff off sick with MSD, but not restricted to this group

• Responsive – target of 48 response time

• Paperless

• Separate from, but working closely with Occupational Health Services.

Evaluation – why?

• Various different patient experience questionnaires.

• Different outcome measures, e.g. EQ-5D, Oswestry.

• Analysis of sickness absence data.

Were not confident that any of these gave a true indication of the ‘range’ value of

the service.

Agreed to work with NHS Employers and Zeal Solutions to develop a

comprehensive evaluation tool, dimensions include:

• managing pain

• degree to which intervention has affected capacity to work

• confidence in clinician

• degree to which intervention has affected work attendance/absence

• burnout

• perceived level of organisational support

• the patient experience.

Evaluation – how?

• Need to know what you want to find out.

• Need a lot of data sets – over 100 matched sets collected for our evaluation.

• Significant implications for clinical team. Patients completed questionnaires prior

to seeing a physiotherapist at the start of treatment, after each treatment session

and three months after discharge.

• Needs to have a longitudinal element three month follow up in this case, an

element of attrition is inevitable.

• Needs to include different sampling questionnaire, semi structured interviews

etc.

• Needs to target different groups – patients, managers, staff working in service.

Evaluation - the benefits of doing it well

• Traditionally staff services are measured solely by their perceived impact on the

overall level of absence in the trust. This is too crude to be of real value, too

many external factors, some known some do not affect absence.

• Gaining a knowledge and understanding of our own service.

• Applying the knowledge gained to promote and attract further investment into the

service

• Informs future development in promotion and management of health and

wellbeing.

Principles of best practice

Design of an effective evaluation

1. Ensure the purpose of the evaluation is determined

2. Establish your evaluation criteria

3. Plan, prepare and document

4. Look for change

5. Long-term impact

Principles of best practice

Context of an effective evaluation

6. Consider the bigger picture

7. Senior management engagement

8. Building an evaluation capability

9. Dual focus on process and outcome

10.Good communication

Be realistic

Right tools for the job

How to robustly evaluate health

and wellbeing interventions

Thank You23rd April 2015

Dr Tony Zarola

mailto:Jennifer.Gardner@nhsemployers.org