evaluation of the shared service on-call after hours support project · evaluation findings client...

Attachment 7. CH Evaluation report

Evaluation of the Shared Service On-Call After Hours Support project

Prepared for The Australian Centre for Social Innovation

2 July 2018

Design. Evaluate. Evolve. Clear Horizon / i

Contact Details Clear Horizon Contact Client Contact Liz Bloom Leanne McPhee Consultant Senior Social Innovator Clear Horizon Consulting The Australian Centre for Social Innovation

129 Chestnut Street, Cremorne VIC 3121 Level 1, 279 Flinders Street, Adelaide SA 5000, Australia

Telephone: (03) 9425 7777 P: 08 7325 4994 E: [email protected] E: [email protected]

Document review and authorisation Version Date distributed Issued to Comments 1 Draft 2 July 2018 Leanne McPhee 2 6 July 2018 Leanne McPhee Feedback incorporated from version 1 3 11 July 2018 Leanne McPhee and

Alazne Alberdi Alvaro Final amendments

Lead author/evaluator Liz Bloom

Project Director Angelos Blackwood

Internal Contributors Edgar Daly

External Contributors

Last Saved 11/07/2018 12:27 pm

Clear Horizon Reference No. CH18_060

Design. Evaluate. Evolve. Clear Horizon / ii

Disclaimer This document has been produced with information supplied to Clear Horizon by The Australian Centre for Social Innovation and participants in the evaluation, including at the reflection workshop on the 27th June 2018. While we make every effort to ensure the accuracy of the information contained in this report, any judgements as to suitability of the information for the client’s purposes are the client’s responsibility. Clear Horizon extends no warranties and assumes no responsibility as to the suitability of this information or for the consequences of its use.

Design. Evaluate. Evolve. Clear Horizon / iii

Contents Executive Summary ............................................................................................................................ 1

1. Introduction .................................................................................................................................. 3

1.1. Background to the Shared Services for On-call After Hours Support project .................................................... 3

2. Approach to the evaluation ......................................................................................................... 4

2.1. The evaluation approach ....................................................................................................................................... 4

2.2. Purpose and audience .......................................................................................................................................... 4

2.3. Key evaluation questions ...................................................................................................................................... 4

2.4. Performance expectations .................................................................................................................................... 5

2.5. Methodology and limitations ................................................................................................................................. 6

3. Evaluation findings ...................................................................................................................... 7

3.1. Client outcomes ..................................................................................................................................................... 7

3.2. Staff outcomes ....................................................................................................................................................... 8

3.3. Contribution to organisational sustainability ..................................................................................................... 10

3.4. Implementation and collaboration ..................................................................................................................... 11

3.5. Key lessons so far ................................................................................................................................................ 13

Appendix 1: Theory of Change ........................................................................................................ 14

Appendix 2: Data collection tools ................................................................................................... 15

Support worker interview template ............................................................................................................................. 15

Manager and Team Leader interview guide ............................................................................................................... 17

Appendix 3: Reflection workshop agenda ...................................................................................... 19

Design. Evaluate. Evolve. Clear Horizon / iv

List of figures Figure 1. Theory of Change for the SSOAHS program ........................................................................ 14

List of tables Table 1. Summary of key learnings ....................................................................................................... 2

Table 2: Evaluation audience ................................................................................................................. 4

Table 3: Key aspects that could be improved ..................................................................................... 13

Table 4: Learnings about shared service delivery that could be transferred to other contexts (upscaling)................................................................................................................................................................ 13

Acronyms TACSI The Australian Centre for Social Innovation TL Team Leader SW Support worker NDIS National Disability Insurance Scheme NDS National Disability Services SSOAHS Shared Services On-call After Hours Support

Design. Evaluate. Evolve. Clear Horizon / 1

Executive Summary

Introduction to the project

The Australian Centre for Social Innovation (TACSI) are working with three disability support service providers (Community Living Australia, Lighthouse Disability and Lutheran Disability Services) to deliver the Shared Services for On-call After Hours Support project. The aim of the project is to work together to prototype a shared on-call system that leads to positive outcomes for staff and increases organisational sustainability.

Purpose

The purpose of the evaluation is to capture and assess the emergent outcomes and learnings from the service prototype to date. Clear Horizon has been working with TACSI to evaluate the first two of four live test cycles as the end of cycle 2 coincided with the reporting deadline for the project.

Methodology

The following methodology was completed for the evaluation:

1. Initial collection of data with 12 staff members

2. Initial analysis of data using thematic coding and cluster analysis

3. A reflection workshop to collect and analyse data

4. Synthesis of data against Key Evaluation Questions to develop findings

Evaluation findings

Client outcomes: The evaluation found it was too early to see changes to the level of support provided to clients resulting from the service. There is some optimism about the potential for the level of support to improve as the service continues into the future. This finding is consistent with the performance expectations for the evaluation.

Staff outcomes: The evaluation found that the level of support for support workers has generally been maintained. There is some optimism about the potential for the service to encourage the support workers to take more responsibility for rostering issues. However, it is too early to determine whether support workers are improving their ability to manage rostering, client support and exercise independent decision-making. There is evidence to show that the service has led to a reduction in stress for Managers and Team Leaders on on-call roster rotation.

Contribution to organisational sustainability: The evaluation found early signs that the prototype will contribute to organisational sustainability based on the potential for the service to alleviate the stress of Managers and Team Leaders on on-call roster rotation. The evaluation identified two additional ways the service could contribute to organisational sustainability including covering shifts in a more affordable way and by facilitating learning between organisations.


Implementation and collaboration: The evaluation found that the social innovation processes and methods have effectively supported close collaborations between participating organisations throughout the design and implementation to date.

Key learnings: The evaluation identified the following key learnings related to the service model and implementation.

Table 1. Summary of key learnings

Area Learning

Service model

• Need to ensure that the On-Call worker has adequate and timely contextual information in order to be able to provide support

• Ensure streamlined information and a user-friendly data base for the On-Call worker • Improve rostering coordination between Managers and On-Call workers • Make minor adjustments to the toolkit and system • On-Call worker must be skilled and adaptive • Clear communication around the role of the On-Call worker to the broader

stakeholders (support staff and clients) • Need to centralise the CMS and rostering systems

Implementation

• Time for investing in relationships was a key factor for success • The service model needs to be costed to determine viability • A diverse group of people need to be involved in the design and implementation, e.g.

Managers, On-Call workers and support workers

Summary of performance

The findings are consistent with the expected performance after cycle 2 of the live prototyping. There is evidence that the organisations are adapting and learning from the prototype and there are early signs to suggest the service is on track to reach its expected end of program outcomes, should the service continue for an appropriate period in the future. The positive outcomes for Team Leaders and Managers on on-call roster rotation identified in the evaluation surpasses the performance expectations for this stage of the prototype.


1. Introduction This report describes the approach to, and results of, the evaluation of the Shared Service On-Call After Hours Support (SSOAHS) project. The report is structured as follows:

• Section 1: Introduction, including background to the service model

• Section 2: Approach to the evaluation

• Section 3: Findings, including lessons learned

• Appendices:

o Appendix 1: Theory of Change for the prototype

o Appendix 2: Data collection tools

o Appendix 3: Reflection workshop agenda

1.1. Background to the Shared Services for On-call After Hours Support project

The Australian Centre for Social Innovation (TACSI) are working with three disability support service providers (Community Living Australia, Lighthouse Disability and Lutheran Disability Services) to deliver the Shared Services for On-call After Hours Support project. These service providers have an interest in learning 'what it takes' to do shared on-call after hours support to relieve workforce stress and improve outcomes for their support workers (primary users of the service), whilst maintaining the level of support that the service provides to clients (beneficiaries of the service). The key feature of the service is the addition of centralised On-Call workers, whose role is to provide after-hours support to support workers and clients.

To date TACSI have worked with the three service providers to:

1. Co-design the service concept, service blueprint, value proposition and Theory of Change

2. Analyse and compare call-logs and core policies, processes and approaches of each service provider

3. Develop and test prototyping service model and tools

4. Recruit and induct on-call staff (2 rounds)

5. Conduct paper prototyping, enacted prototyping and two of four live prototype cycles


2. Approach to the evaluation

2.1. The evaluation approach

The approach to this evaluation drew from a combination of Developmental Evaluation and more traditional evaluative approaches. Developmental Evaluation is a useful to guide evaluations in the social innovation space, allowing for flexibility and learnings to be incorporated throughout the evaluation period. The aspects of this evaluation which are consistent with a Developmental Evaluation approach included the incorporation of a mid-point reflection meeting, and taking a flexible approach to methodology, including by adjusting the data collection tools and Key Evaluation Questions as the prototype evolved.

2.2. Purpose and audience

The purpose of the evaluation is to provide an assessment of the progress and to capture emergent outcomes and learnings from the service prototype to date. Clear Horizon worked with TACSI to evaluate the first two of four testing cycles, which coincided with the project reporting deadline. The evaluation audience and information needs are in Table 1 below.

Table 2: Evaluation audience

Audience Information needs Funder: National Disability Services (NDS)

• Client and staff outcomes: To understand what learning and outcomes (if any) have been achieved for staff towards increasing client and staff satisfaction.

• Business model outcomes (organisational sustainability): How sustainable the innovation is.

• Implementation: To understand how the social innovation process and methods applied in the project have worked (or not) to engage, set up governance and develop shared service delivery solutions.

Service delivery partners

• Similar to NDS in regards to effectiveness.

• Learning from each other’s practice.

TACSI • Similar to NDS.

Clients • Learning: To understand if client’s needs are being met

2.3. Key evaluation questions

There are four Key Evaluation Questions (KEQs) that guide this evaluation. In forming the KEQs, particular emphasis was given to meeting the information needs of the funder. The KEQs and sub-questions fit into three areas of enquiry: client and staff outcomes, business model outcomes and implementation.


Client and staff outcomes

1. Does the prototype of the service model at least maintain the level of support for people with disabilities?

2. Does the prototype of the service model result in positive outcomes for staff?

a. To what extent are the On-Call workers and support workers better managing rostering, client support and decision-making?

b. To what extent is the prototype resulting in positive outcomes for Managers and Team Leaders?

Business model outcomes

3. Does the prototype of the service model contribute to the sustainability of the organisation?

a. What is the business model and how has it changed over time?

b. To what extent did the organisations involved learn and benefit from participation in the program?

c. What has been learnt around what it takes to do shared service delivery between the participating organisations?

d. What has been learnt about what it takes to do shared service delivery that can be transferred to other contexts?

Implementation

4. How well have the social innovation processes and methods worked to engage and foster learning and collaboration between participating organisations?

a. What were the strengths and weaknesses of project design and delivery?

b. What could be improved?

2.4. Performance expectations

At the time of evaluation, only Cycles 1 and 2 of the prototype were completed. Thus at this stage of project delivery, it is only expected that outputs and some intermediate outcomes will have been achieved from the Theory of Change (see Figure 1). To achieve the end of program outcomes, the project would need to run beyond the four test cycles for at least 12 months. For this evaluation the prototype will be judged as a success if it is on track to achieve the end of project outcomes; more specifically, if there is evidence that there is learning between organisations about feasible, desirable and viable practice leading to:

• the On-Call worker and support workers better managing rostering, client support and decision making

• Support workers beginning to demonstrate independent decision-making • the business model and back of the envelope analysis being developed


2.5. Methodology and limitations

Methodology

The following methodology was completed for the evaluation:

1. Developing the Evaluation Plan, including clarifying the audience, purpose, program logic and KEQs.

2. Initial collection of data, including: four interviews conducted by Clear Horizon with a total of five Managers and Team Leaders from all three organisations; four interviews with support workers from across the organisations conducted by their Managers; reflection logs completed by the On-Call workers after cycle 1 (n=1) and cycle 2 (n=3); informal reflections from TACSI (by teleconference). The data collection tools are presented in Appendix 2.

3. Initial analysis of data using thematic coding and cluster analysis.

4. A reflection workshop with key leadership from the three organisations, one On-Call test worker and three TACSI staff in attendance to validate existing data, collect additional data, and draft findings. The reflection workshop agenda is presented in Appendix 3.

5. Synthesis of evidence from reflection workshop and prior analysis against key evaluation questions to form the basis of the findings.

6. Development of the Evaluation Report.

Limitations to the evaluation

Limited range of views at the reflection workshop

Managers and one Chief Executive from the three organisations, as well as one On-Call worker and three staff from TACSI were present at the reflection workshop. Greater representation from support workers and On-Call workers at the reflection workshop would have allowed for support worker and client perspectives to be better understood and incorporated into the evaluation findings.

Breadth of stakeholder interview data

It was expected that the evaluation would include firsthand feedback from clients who had participated in the prototype through interviews with Managers. Additionally, it was planned that the evaluators would conduct phone interviews with support workers. However, data could not be collected in either way. Clients were unable to be interviewed as the On-Call worker only received direct calls from one organisation and the one client who called during Cycle 1 and 2 was unable to take part in an interview. Four support workers were interviewed by Managers within the organisations. While the evaluators developed the interview guide for these interviews, there was no opportunity to further explore the support workers responses to each question which led to a limited understanding of their perspective.

Short timelines

As is common in social innovation the timelines for the evaluation were relatively short, compared with more traditional projects design processes, which left limited time for data collection, analysis and synthesis. This affected the extent to which the report could explore the prototype’s outcomes.


3. Evaluation findings This section describes the findings of the evaluation, arranged under the following headings:

• Client outcomes

• Staff outcomes

• Contribution to organisational sustainability

• Implementation and collaboration

• Key lessons so far

The findings are presented in order of key evaluation questions as presented in Section 2.2 above, with the exception of Section 3.4 (‘Key lessons so far’). This section incorporates learning oriented sub-questions from KEQ 3 and 4 and have been separated from their initial key evaluation questions for presentation purposes. The findings include illustrative quotations from interviewees and workshop participants and the brackets following quotations designate the quote’s source.

3.1. Client outcomes

At the point of evaluation, the project team expected the level of support for clients to be maintained but that the service would need to run for a significantly longer period for increases in support provided to clients to occur. The evaluation found it was too early to discern changes in the level of support provided to clients. There is some optimism about the potential for the level of support to improve as the service continues in the future.

No impacts observed

All Managers and Team Leaders interviewed reported that they had not observed or heard of the service affecting the level of support provided to clients. Explanations included that the service has been ‘generally running pretty smoothly’ and that in most instances, clients would not be aware of any changes to the afterhours system of support. No reflection workshop participants provided any evidence that the level of support had changed. One participant noted that the level of support was unlikely to have changed, as two of the three organisations do not have clients directly phone the On-Call worker and the On-Call worker can still call Managers for additional support if necessary. Of the three support workers surveyed using a Likert scale, one rated the level of support provided to the client as ‘no change’, with the other two support workers rating the level of support as ‘slightly worsened’. When probed, these two support workers suggested that a slight decrease in the level of support for clients may occur as a result of the On-Call worker being less informed about staff and client context. However as the interview was not conducted by Clear Horizon this was not explored further and no specific instances of decreasing support to clients were recalled. Additionally, the support workers may not have been aware of measures put in place to ensure that the On-Call workers had access to additional support from Managers and Team Leaders, should they have needed this.

Optimism regarding the potential for support to improve in the future

Reflection workshop participants expressed optimism about the potential for the level of support to continue to improve once the On-Call workers are established in their position and adjust to their roles and responsibilities. An On-Call worker described one instance where a client was enthusiastic about the service when they became aware of the new on-call support structure.


“A client called to request care and was quite concerned about calling due to a past experience. I explained it was a new system and [they] were very excited” (Cycle 2 On-Call worker)

3.2. Staff outcomes

Outcomes for support workers

At this point of the evaluation, it was expected that the level of afterhours support provided to support workers would be largely maintained and that there would be some early signs that support workers were better managing rostering, client support and decision making as a result of the service. The evaluation found that the level of support was maintained with all four support workers interviewed reporting that they felt ‘moderately supported’ or’ very supported’ during their interactions with the On-Call worker. When interviewed, two Managers and Team Leaders expressed optimism about the potential for the service to encourage the support workers to take more responsibility for rostering issues, such as covering shifts when they cannot come to work. However, one Manager at the reflection workshop reported that it was difficult to discern support worker outcomes at this early stage.

Barriers and enablers

Enabling factors

The evaluation identified four promising early outcomes that are likely to support improved rostering, client support and decision-making among support workers:

• On-Call workers encouraging the support workers to find the information themselves. “I think staff often look for the easy way to get answers but it did push them to make more of an effort” (Support worker interviewee).

• On-Call workers having enough contextual information about the workers and clients to ensure that the support worker has confidence in the advice they are receiving.

• On-call workers taking a supportive and coaching role: “I feel that this is an invaluable service for the support workers, particularly having someone to connect with [and] discuss a situation and gain guidance. My experience has been to confirm actions already taken therefore instilling independent decision-making and reaffirming their initiative” (On-Call worker, Reflection Log).

• Support workers have an extra level of support and accountability from the On-Call worker.

Barriers

The evaluation also identified two potential issues that may hinder support worker outcomes, although these can be mitigated.

• The ability of the On-Call worker to provide support was hindered by their lack of contextual knowledge about staff and clients. Three support workers interviewed voiced this concern, with one support worker stating, “The person I talked to was helpful but didn’t really understand our…client. So the answers were very generic and not focused around the individual client’s needs” (Support worker interviewee). One Manager noted that the support provided to support workers could have slightly decreased due to the On-Call workers lack of knowledge about the clients and staff, though


they were optimistic that, as these workers were still learning their role, it would improve overtime. Participants at the reflection workshop also agreed that this would improve with time.

• Some support workers called unnecessarily out of fear of getting in trouble. An On-Call worker noted that support workers are scared of getting in trouble by making the wrong decision and therefore second guess themselves: “In my experience the SW rang to tell me what action they had taken and they needed confirmation that the action taken was the correct decision” (Cycle 2 On-Call worker, Reflection Log). This barrier is not specific to the prototype but may hinder support worker outcomes around independent-decision making if it were to continue.

Outcomes for Team Leaders and Managers on on-call roster rotation

At the point of the evaluation, it was expected that there would be signs that the prototype was leading to positive outcomes for Managers and Team Leaders. The evaluation found evidence that positive outcomes were occurring in the form of reduced stress for Managers and Team Leaders. All the Managers interviewed described the reduced stress for staff who had been on-call when the prototype was being implemented as the most significant change resulting from the service so far.

Barriers and enablers

The evaluation identified a number of enablers and barriers to positive outcomes for Team Leaders and Managers. These are listed below.

Enabling factors

• Managers and Team Leaders being able to have a break: “If you are on call pretty much 24/7 you have no down time and have high stress levels” (Manager/Team Leader interviewee).

• Increased work life balance: “It’s about giving that life back to us and giving us back to our kids. “It impacts on their personal life and their own mental health. My Saturday is my Saturday. Family would be woken up at 7:05am and now I don’t have to wake up until I want to wake up” (Manager/Team Leader interviewee).

• Team Leaders feeling they have been heard: “Being able to put another solution to team members that addresses some of these issues makes them feel heard” (Manager/Team Leader interviewee).

Barriers

• Having to manually divert the phone is not ideal. One Cycle 2 On-Call worker noted that this was an issue for some Team Leaders, especially on a Friday night when they had to turn off the divert at 10pm and be up by 7am to turn the divert on again. However, this was not reported as an issue by Managers or Team Leaders during their interviews. During the reflection workshop it was identified that this could be addressed through the right technology solutions should the service continue.


3.3. Contribution to organisational sustainability

Reduced stress may lead to better retention of Managers and Team Leaders

The evaluation found early signs that suggest the prototype will make a significant contribution to organisational sustainability based on the potential for the service to alleviate Manager and Team Leader stress. This is significant because of reports that the high level and stress and lack of work life balance has previously led to some Managers and Team Leaders leaving their roles.

As described by one Manager, prior to the prototype being tested, the on-call model was not a costed model and therefore Managers and Team Leaders would not be paid an hourly rate to be on call.

“Sometimes I’ll be playing a board game with my kids and have to walk away because someone calls with a behaviour” (Manager/Team Leader interviewee)

During their interviews, Managers/Team Leader described that under the old system they were stressed and that people resigned from their roles because it was, ‘all too much’.

“On call has been a thorn in our side. [It]… has been one of reasons why I lost team members in the past. For On-Call workers there’s a huge skill shortage. Lots of people burn out and hop between teams. Either they will go back to being a support worker or leave the sector entirely. The pay doesn’t match the work they put in” (Manager/Team Leader interviewee)

Since the beginning of the prototype, all Managers and Team Leaders interviewed expressed that things had improved for them, with comments that they now had better work life balance and reduced stress levels.

“The biggest thing is the number of phone calls the on-call people are getting at night. Now they’re able to have a break. From 10 phone calls you are now probably only getting one or zero- depending on the week and what is happening” (Manager/Team Leader interviewee)

As stated by one Manager this is significant as Team Leaders and Managers are essential to ensuring good quality assurance through their coaching and mentoring of support workers. Without work-life balance, they will likely not be able to perform at their best. One Manager stated that this transfer from a ‘charity model’ to a ‘costed model’ is more sustainable in the long term.

Potential future contributions to sustainability

In addition to the positive outcomes for Managers and Team Leaders, the evaluation found two additional ways the prototype may contribute to organisational sustainability. These are listed below:

• As stated by an On-Call worker and discussed in the reflection workshop, the implementation process has given the organisations opportunity to learn from each other’s organisational practices and systems and applying these learnings has the potential to improve each organisation practice.

• As stated by a Manager, people are now beginning to cover shifts by utilising the current staffing pool, rather than calling an agency that is more expensive. If this pattern continues it may provide a cost benefit in the future. This was also evident in the Call Logs completed by the On-Call workers.


Changes to the service model

Changes are being made to the service model as the prototype progresses based on feedback from participating organisations and the On-Call workers. The changes made to the service concept during testing cycles 1 and 2 include:

• Defining the hours the service will run by testing what the periods of high demand are.

• Modifying the On-call worker’s induction process to ensure they get the maximum contextual information (e.g. client and house information) and service specific detail (e.g. service model tools, organisations’ CMS and rostering system) in order to perform the On-Call role.

• Updating service delivery tools so that the on-call workers ability to coach staff to make decisions, particularly around rostering support, can be strengthened.

These changes are intended to improve the overall effectiveness and efficiency of the service model and are expected to enhance the outcomes including those listed above. However, it is not yet clear how the changes are impacting on the overall sustainability of the organisations.

3.4. Implementation and collaboration

The prototyping process has been participatory and iterative, incorporating regular planning and reflection meetings to test, learn from and adapt the service model as it is being prototyped. At the time of the evaluation there had been eight working sessions facilitated by TACSI with key members from the organisations and On-Call workers. The evaluation found that the social innovation processes and methods have effectively supported close collaborations between participating organisations throughout the design and implementation to date. The existing relationships between organisations and the buy-in at executive and leadership levels have been other key factors that have allowed for successful collaboration. The evaluation identified key challenges faced during implementation but that collaboration and a continued commitment from organisations has meant these have effectively been managed.

Opportunity for strategic thinking and collaboration appreciated

There is evidence to suggest that organisations valued the opportunity to engage in strategic level problem solving through the service design process. A reflection workshop participant said that the pace and detail of the process has allowed the leadership teams to think ‘longer-term and bigger picture’ and that opportunity to work on bigger picture issues was particularly significant given those working in the sector are generally time poor and more focussed on day-to-day operations. A Manager/Team Leader interviewee described the process of meeting and problem solving between organisations as a valuable opportunity to learn other ways to do things.

“When you’re neck deep in your work you find it difficult to examine things outside of your current experience. I’ve really learnt ways to do things differently … we’ve taken on board other agencies’ call structures.” (Manager/Team Leader interviewee)

Strong relationships maintained and strengthened within organisations

The evaluation found that the strong existing relationships between the organisations at the leadership level has been a significant enabling factor behind a strong collaboration and that these relationships have strengthened during the implementation process. The majority of Managers/Team Leaders


interviewed positively described the relationships and communication between organisations. Reflection workshop participants noted that these relationships have strengthened as the process has progressed.

“What we’ve learnt is that if you have trust anything is achievable really.” (Reflection Workshop Participant)

Strategic commitment and buy-in at leadership levels key

The majority of Managers/Team Leaders interviewed described the clear commitment, championing and leadership of executives within the organisations as significant in creating an authorising environment and ensuring the service could be effectively tested, describing their commitment as ‘paramount’ to ensuring that any potential lessons learnt are effectively addressed.

“Leadership buy in was an important factor to create the authorising environment, but this took time, particularly at the start.” (Reflection Workshop Participant)

A reflection workshop participant described the important role of senior Managers and Team Leaders in sharing knowledge and learnings and that having Manager and Team Leader input allowed for tougher and more detailed questions to be asked about the model and for critical feedback to be incorporated.

“Our CEOs already had a strong collaboration between each other but as senior Managers we’ve also been able to share knowledge and learnings and having collaboration at both levels makes the trust and collaboration even better.” (Reflection Workshop Participant)

Competing interests and associated complexity well managed

The evaluation found that including the diversity in client and business needs between the organisations was a challenging factor during service implementation, but that this has been well managed through the process. Managers and Team Leaders interviewed described the needs of the three organisations as competing and varied, and that this added complexity to the issues worked through during the social innovation processes. However, a respondent described the project management as effective in managing competing interests including that the quick responsiveness of the project delivery team allowed for complex needs to be addressed.

A Manager/Team Leader interviewee described the competitiveness of the NDIS as contributing to the overall sense of isolation within disability sector organisations, underscoring the significance of the collaboration. Responses from reflection workshop participants and Managers/Team Leaders interviewed showed that the overall process of collaboration exceeded expectations, with one Manager describing this collaborative venture as working better than others they have been involved in.

“There are a lot of conversations about collaboration [in the sector] but the reality of collaboration is quite different to the ideal, but in this case people are quite fascinated about how strong the collaboration is. There was a lot of expectation that it wouldn’t go as well as it has.” (Reflection Workshop Participant)

Time pressures are challenging

The evaluation found time constraints to be a challenging factor, particularly as those involved in the process were already quite time poor. However, most of these challenges were beyond the control of the project team. Reflection workshop participants and Managers/Team Leaders described the ongoing changes at the sector and organisational level as significant. A reflection workshop participant described an instance where a procedural delay meant that the training opportunities and induction process for


one of the On-Call workers was less than ideal. One Manager/Team Leader commented on the frequency and depth of working sessions, noting that the level of detail was often necessary but that they expected that good outcomes could have been achieved had there been less sessions.

3.5. Key lessons so far

The evaluation identified a number of opportunities and learnings both in relation to improving the current model and to take into consideration when doing shared service delivery in other contexts. These are summarised in Tables 3 and 4 below.

Table 3: Key aspects that could be improved

Area Learning

Service model

Improve: Ensure the On-Call workers are provided with more detailed rostering information (such as those available for back-up) so that that they can cover shifts efficiently.

Improve: Streamline the communication channel so that anticipated calls (e.g. behavioural, medical) can be shared prior to an on-call shift. Improve: Ensure the rostering coordination of the service between the organisation and On-Call workers so that all involved are clear about escalation processes. Improve: Make minor adjustments to current tools and systems, including to the shift handover process to allow for On-Call workers to communicate important information; and the re-coding defining quality standard for entries of the call log in a way that is clear to all users. Add: Establish feedback and development processes to improve capabilities of the On-Call workers and to ensure a high quality of support. Iterate: Provide On-Call workers with details of typical call scenarios and responses from service delivery to date.

Table 4: Learnings about shared service delivery that could be transferred to other contexts (upscaling)

Area Learning

Service model

Maintain: On-call workers that are skills and versatile to respond to the diversity of needs within and between organisations. Iterate: On entering the shared service it is critical that support staff have a clear understanding of what expectations they can have of the On-call worker and that the On-Call worker is aware of the specific needs of each organisation. Add: To support upscaling it will be important to design and test a customer management and rostering system that creates efficiency in the delivery of the service, reducing complexity at scale.

Implementation

Maintain: Critical to the design and implementation is the involvement of multiple stakeholders. In this case this refers to the three layers of Management, support workers and On-Call workers. Maintain: The time and approach to building trust between the three organisations and therefore the commitment to shared services. Add: Critical to the service model success is desirability, feasibility and viability. Viability was yet to be tested, with cost modelling currently being undertaken. Add: Establish an ongoing Monitoring, Evaluation and Learning framework Add: It is critical to establish the confidence of key decision makers and that they take ownership of the process in order to then engage those involved in using and delivering the service.


Appendix 1: Theory of Change Figure 1. Theory of Change for the SSOAHS program


Appendix 2: Data collection tools

Support worker interview template

Title: Shared Services for On-call After Hours Support - support worker perspectives

Introduction

Instructions for the interviewer

The Australian Centre for Social Innovation (TACSI) is working with three disability service organisations (Community Living Australia, Lighthouse Disability and Lutheran Disability services) to deliver the Shared Services for On-call After Hours Support (SSOAHS) prototype. The SSOAHS project aims to support a higher retention of Team Leaders and support workers, and develop a service where clients are satisfied with the support they have received.

This interview is to be conducted with support workers who have spoken with the new On-Call worker in the last three weeks, since the beginning of Cycle 2 of the SSOAHS prototype. Answers are to be recorded on the interview sheet below. If necessary, the interviewer can ask follow up or prompting questions to explore the support workers’ answer further. Feel free to attach additional sheets of paper if required.

The purpose of the interview is to capture support worker experiences of getting support from the new On-Call worker. Before beginning the interview, it is important that clients are read the consent information below and that verbal consent is received.

About the survey

Participation in this survey is optional and anonymous.

The purpose of this survey is to ask you to draw on your experience and perspectives about your recent experience speaking with the new On-Call worker. The information you provide will be used to inform learning and improvement for the program.

This interview is voluntary and anonymous, however due to the small amount of people being surveyed you may be able to be identified by information you provide in your answers. If you have any questions about the new on-call model or wish to withdraw your answers to the interview questions then please contact [email protected].

Do you agree to participate in this survey? Yes / No

These questions relate to your recent experience receiving support from the On-Call worker.


Question 1. Using the following multiple-choice criteria, how supported did you feel by the On-Call worker to make your own decisions about the appropriate action to take?

€ Very supported € Moderately supported € A little supported € Not at all supported € I don’t know € N/A

Question 2. Why did you select the above answer?

Question 3. Using the following multiple-choice criteria, do you believe the support given by the On-Call worker resulted in improved support for the client(s)?

€ The level of support has significantly improved € The level of support has moderately improved € No changes to the level of support € The level of support has slightly worsened € The level of support has significantly worsened € I don’t know € N/A

Question 4. Why did you select the above answer?

Question 5. Is there anything you like about the new on-call model?

Question 6. Is there anything you don’t like, or you think could be improved, about the new on-call model?

Question 7. Which aspect of the on-call service is most important to you?

Question 8. Do you have any further comments?


Manager and Team Leader interview guide

Preamble

Hello, my name is [interviewer name] from Clear Horizon Consulting. We are working with The Australian Centre for Social Innovation (TACSI for short), to help evaluate the emergency after hours, on-call prototype which is currently being tested with three disability organisations including [name organisation].

We’re looking to understand what’s been working well and what could be done differently and iterated for the next prototype cycle.

We’re keen to hear about your experience and perspectives on this so far.

The information you provide will be analysed together with other interviews and used to develop a short evaluation report. We won’t be using people’s names in the report and if there is anything you mention during the interview that later you want taken out, you can let us know and we’ll do that. If there are any comments you would not like to be associated with please let me know so that I can ensure confidentiality.

Is this ok with you?

Background / introduction

7. Can you briefly outline your role both in [interviewee’s organisation] and in the shared service prototype? (Probe to test interviewer’s understanding/assumptions around the prototype test) (For Team Leaders) Could you please talk about your role in the prototype?

8. How would you describe your experience so far?

Most significant change

9. What do you think is changing as a result of the prototype?

a. Of these changes, which is most important to you?

b. Why?

Outcomes for clients and support workers

The next set of questions are about reflecting on the benefits (actual or potential) to support workers and clients as a result of the prototype.

10. As a result of the prototype, have you observed or are you aware of any change in the ability of support workers:

a. To provide support to clients? (Please describe what you’ve observed)

b. To manage shift changes (where applicable)? (Please describe what you’ve observed)

c. To demonstrate independent decision-making? (Please describe what you’ve observed)


11. Have you observed or are you aware of any changes to client’s level of satisfaction with support they’ve received since the prototype started? (Please describe what you’ve observed)

What’s worked well and what can be done differently

The next set of questions relate to what is working well and what could be done differently for the next iteration of the on-call prototype.

12. What aspects of the shared service prototype have been working well?

13. What aspects can be done differently?

Lessons learned/improvements

14. What have you learned along the way and learned about the potential of shared service delivery:

a. related to the three organisations involved in the prototype?

b. in general?

15. Aside from what you have already told us, do you have any suggestions for the next iteration of the service prototype? (Probe: are these suggestions context specific or do they apply to integrated delivery in general?)

Closing comments

16. Do you have any other comments that you would like to make?


Appendix 3: Reflection workshop agenda

Time: 9:30am – 3pm Wednesday 27th June 2018

Location: TACSI Offices, 1/265-279 Flinders St, Adelaide SA 5000

Purpose: The purpose of the workshop is to collect data to fill data gaps, draw out findings against the Key Evaluation Questions and identify ways to improve the service.

Outcomes: By the end of day we want to develop shared findings to inform the evaluation report and identify ways to improve the service in the future.

Participants:

• CEO’s, Managers, Team Leaders from Lutheran Disability Services, Lighthouse Disability and Community Living Australia

• Three on call support people

• Leanne McPhee, Alazne Alberdi Alvaro and Kerry Jones (TACSI)

Facilitators: Liz Bloom and Edgar Daly, Clear Horizon

Time 9:45 Opening

• Meet and greet • Evaluation purpose and role of reflection workshop • Workshop expectations • Overview of workshop agenda

10:05 Clarifying the current service model

• Present current service model • Identify changes so far

10:45 Break 11:00 Present data and initial analysis

11:20 What data is missing?

• Participants to share their reflections

12:15 Lunch (including VOX POPs) 13:00

Review of data Evaluation findings

• Answering the KEQ’s as a group

14:00 Moving forward • Identifying opportunities for changes to the service

15:00 Close

evaluation of the shared service on-call after hours support project · evaluation findings client...

Documents