Building a culture of measurement in a large

not-for-profit organisation:

Findings from three and a half years of Results Based

Accountability implementationDesiree Nangle, Evaluation ManagerAnglicare WA2015

Agenda• Background

• RBA implementation in Anglicare WA

– Case Study: RBA on RBA

• Learnings and Challenges

• Where has it worked

• Future Directions

Anglicare WA works together with people, families and their communities to enhance their abilities to cope with their lives and relationships

From Surviving to Thriving

Surviving where life is a real battle

Coping where you are just getting by

Building where life isn’t bad

Thriving where life is going well

Planning FrameworkFive Year Strategic Plan

VisionCorporate Purpose

Beneficiary Performance IndicatorKey Result Areas

Three Year Business PlansStrategic Service Areas

Housing, Community, Finances, Relationships

CorporatePeople and Culture, Marketing, Fundraising, etc.

Operational PlansLocation, Team and Service Level

Results Based AccountabilityService Level

Where Results Based

Accountability fits

within Anglicare

WA’s Strategic

Direction

Why build a culture of measurement and

evaluation?Evaluation Culture Benefits to an

Organisation•Better information to support program decision making

•Greater capacity to evaluate the merit, cost effectiveness and cost capacity of programs

•Delivery of programs that are relevant and focused on delivery of outcomes to the community

Hanwright, J., & Makinson, S. (2008). Promoting Evaluation Culture: The development and implementation of an evaluation strategy in the Queensland Department of Education, Training and the Arts. Evaluation Journal of Australia, 8(1), 20-25.

Why build a culture of measurement and evaluation?

• Increased capability to reprioritise in a climate of static funding

• Production of data to verify results that can be used for public relations and promoting services

Hanwright, J., & Makinson, S. (2008). Promoting Evaluation Culture: The development and implementation of an evaluation strategy in the Queensland Department of Education, Training and the Arts. Evaluation Journal of Australia, 8(1), 20-25.

Why are we measuring stuff?1. To KNOW the effectiveness of our programs

and ensure our clients are achieving their

OUTCOMES

• It’s client-centred and inclusive

2. To identify what works!

3. To identify what could work better

4. Evidencing our value

• To ourselves, our community, our clients, existing and

future funders

With thanks to Mark FriedmanThe Fiscal Policy Studies Institute

Santa Fe, New Mexico

Results Based Accountability

is made up of two parts:POPULATION ACCOUNTABILITY

Is about the well-being ofWHOLE POPULATIONS

For Communities – Cities – States - Nations

PERFORMANCE ACCOUNTABILITYIs about the well-being of

CUSTOMER POPULATIONSFor Programs – Agencies – Services

RBA Tool 1: The Quadrant Exercise

Three Questions

1. How much did we do?

• Outputs

2. How well did we do it?

• Quality measures

3. Is anyone any better off?

• Outcomes

RBA on RBA QuadrantRBA Self-Evaluation

How Much Did We Do? How Well Did We Do It?

# of active services by phase of implementation

# of staff involved in active services by phase of implementation

% of staff satisfied with training and support

% of staff trained as RBA trainers Time to self-sustainability

o By Service Time programs spend in each phase

of implementation % of measures reviewed for validity % of programs implementing RBA  

Is Anyone Better Off?

 

% of staff who understands what RBA is % of staff who support RBA within Anglicare WA % of staff who support RBA within their own service area% of staff reporting RBA improves their programs % of programs with the capacity to use RBA unassisted % of programs using RBA for program improvement  

RBA Tool 2: Turn the Curve Report (One-page headline measure

report)

Displaying evaluation results in a meaningful, user-friendly

wayOne page, headline measure reportsIt has:

•Graphs for top measures that have data

•Story behind curves

•Action plan

•Partners

•Data development agenda

It is good for:

•Staff

•Managers

•Boards

•Funders

Headline Measure Report: Components

The top 3-5 measures that services have data for make up the Headline Measure Report. These are the things

believed to be most important by the service staff

Yellow graphs show client OUTCOMES while white graphs show outputs (numbers of things) or quality measures (things that add quality to the service but do not necessarily mean clients are ‘better off’

Story behind the Baseline

Story behind the baselines describes why the graphs look the way they do. They explain what went right (in that what the reasons were that clients achieved the desired outcome) as well as diagnose potential reasons some clients did not achieve the outcome

Action PlanThe Action Plan to Turn the Curves

explains what more or different (new) the

service will do in the coming 6 months to try

to ensure all clients receive the desired

outcomes of the service. They should

also focus on low-cost/no-cost ideas as well as out of the

box ideas to encourage innovation

Partners list internal and external partners who could have a role to play in helping turn the curves. This

recognises that no one person or agency can be expected to do this work alone and opens up the possibilities for collaborations – Collective Impact

Data Development

Agenda highlights the additional data

the service currently does

NOT collect but thinks would be

useful to have in the future.

2014 Results Based Accountability Headline Measures Report

Service Description: Anglicare WA has chosen to implement Results Based Accountability (RBA) framework across all its services. This choice is centred on a strong desire to know and have evidence the impact our services have.

Clients

All Anglicare WA staff

Data Development Agenda

1. Determine population outcomes our services contribute to and begin using these across service’s headline measure reports

2. Determine whether or not there are questions we can ask across services that use the same wording and same scale

Story Behind the Baselines

Three of our four headline measures increased in March 2014 from their previous data period in March 2013.Graph 1 shows that more staff in March 2014 understand what RBA is compared to the March 2013 data. This is also true for Graph 3 which shows an increase in the number of programs in phase 3 of RBA implementation. These increases are due to the fact that the RBA team expanded to include a new research officer to help reach more services; do more introductory RBA sessions with staff; and increase visits to regional offices to help staff with their headline reports.Graph 2 shows a 9% decrease in staff who think that RBA will help improve their programs in March 2014 compared to 2013. This could be because of the high turn-over of staff in programs and new staff not having enough exposure to RBA to see its benefits. This could also be due to the ever-changing data collection tools that staff were being asked to use. Because of this, there was no data consistency for some of these programs and could have left staff feeling that RBA was not meaningful for their programs.Interestingly, graph 4 shows an increase (2%) in staff who think that RBA is a good thing for AnglicareWA. It is possible, that staff think that RBA is good for the organisation as a whole but is not something they should be doing themselves . RBA was seen as ‘extra work’ on top of other admin and reporting duties in the open-ended questions where staff also noted that RBA took “time away from their clients”.

Partners Who Can Help Us Do Better

All Anglicare WA Programs and Staff

The Community Sector

Other Not-for-Profit Organisations

National and Federal Government

Trained RBA Trainers

Action Plan to Continue Turning the Curves

In an effort to continue improving performance on our four headline measures, Anglicare WA will:•Work in-depth with one service in each of our sub-brand areas (Relationships, Community, Youth, Housing) and use them as case studies

• This will help us determine if more intensive support correlates to better understanding and, more importantly, high use of the RBA framework for service improvement

•Increase information to all staff on RBA success stories through the intranet and staff meetings•Offer more RBA introduction sessions for staff who are new or need a refresher on the basics of the framework•Encourage teams to present RBA at their team meetings and across teams: integrate RBA in daily business•Share more examples with staff of RBA and where it works•Make RBA a part of induction to help deal with staff turnover issues•Implement an award for “Curve of the Month” to highlight progress being made by service

RBA HEADLINE PERFORMANCE MEASURES

Number of Clients

Anglicare WA has 433 staff

123 staff completed the survey, a response rate of 28%

IMPLEMENTATION

Phase 1 Awareness raising

• Emails• Presentations to groups

Phase 2

Service Planning

• Scoping and readiness assessment• Workshop • Define clients• Agree outcomes• Agree measures• Data strategy• Plot baselines

Phase 3

Service Implementation

• Data collection• Plot curves• Monitor and use

Phase 4

Normalise

• Refine• Use unassisted

Four Phases of Implementation

IMPLEMENTATION CASE STUDY:RBA on RBA

2011: Start of the RBA Journey

• First all staff survey to

get a baseline

• 174 staff out of

~400 completed

survey

• Two page highlight

report sent to staff

Year Two: 2011-2012Adoption of the one-page headline measure report!•Second all staff survey•Prioritised outcomes•Report to all staff and Board

Year three: 2012-2013

• Yellow graphs to highlight outcome measures

• Improved formatting• Strategic focus on

getting services to USE results

• RBA results in annual report for first time!

Year four: 2013-2014

• Development and agreement of Population outcome measures

• Annual report case studies MUST have RBA data to be considered

RBA on RBA

How much did we do? How well did we do it?

Is anyone better off?

Quantity Quality

Effe

ct

E

ffort

• # of services • # of staff

• staff satisfaction with training and support• # RBA facilitators trained• time programs spend in each stage• % services using RBA for program

improvement• validity of measures

•% staff that understand what RBA is•% staff that support RBA within Anglicare WA•% services implementing RBA •% services with the capacity to use RBA unassisted•% of curves turned

(61 out of 68 services) = 89.7%

68%

86%

9

75%

(1 service)

= 2%

RBA on RBAUpdated: April 2014

68

430

Do staff understand what RBA is?

Percentage who agree or strongly agree2011= 53.6% 2012 = 70.6% 2013 = 80.9%2014 = 85.7%

32%

Do staff think RBA is good for Anglicare?

Percentage who agree or strongly agree2011= 55.2% 2012 = 63.1% 2013 = 66.1%2014 = 68.4%

13%

OPPORTUNITIES, CHALLENGES, & LESSONS LEARNT

Staff Comments: challenges“One of the

challenges is how to change

the curve when it is

falling behind”.

“The challenges I feel are to do with how to set up and interpret outcomes”.

“What I perceive to be meaningful data and what a funding body perceives are

two different things”.

“It is simply another reporting tool that takes staff away from doing what we do best, working with people”.

“I think RBA enables AnglicareWA to prove beyond a doubt that its programmes and services are high quality and successful!”

“Reflecting on our achievements provides a sense of value to our work and helps us to focus on what is important”.

Staff Comments: benefits

“I have truly enjoyed the reflective journey of what is a

meaningful outcome/how is the person better off

compared to the old way of conforming data to fit your

funding providers’ KPI’s”.

“I find our teams like

it after they

realise they own

it”.

• Outcomes thinking make sense to staff

• It takes time…. lots of time

• It needs technical support

• High level support is vital

• Data is foreign to most

• Data needs systems

• Use or rigor debate and balance

Lessons Learnt

• Its not a fad• Desire for long-term outcomes but

short-term funding• Enthusiastic services wanting to

measure EVERYTHING need balance

• Don’t make funders the only audience• It is worth it…. really worth it

Lessons Learnt

Evaluation Culture Must Do’s• Support from the top and the middle

• Use a change management model• Be organised – infrastructure and technical

support• Be aspirational• Use the results• Sharing is caring share• Make it fun!

WHERE HAS RBA WORKED?

Where has RBA worked?Young Parents•1 year funding to 3 year contract

YES! Housing

Foyer Oxford

$

$Improved service

design and delivery

FUTURE DIRECTIONS

Now What• Continuing the RBA journey with services

• Composite RBA analysis

• Adding rigour

• Other evaluation and research tools

• Exploring population outcomes

• RBA collaboration with Collective Impact

www.anglicarewa.org.auDesiree.nangle@anglicarewa.org.au

Results Based Accountability March 2013 Headline Measures ReportService Description: Anglicare WA has chosen to implement Results Based Accountability (RBA) framework across all its services. This choice is centred on a strong desire to know and have evidence the impact our services have.

Defined Service Users

All Anglicare WA staff

Data Development Agenda

1. Determine population outcomes our services contribute to and begin using these across service’s headline measure reports

2. Determine whether or not there are questions we can ask across services that use the same wording and same scale

Story Behind the Baselines

Three of our four headline measures increased in March 2013 from their previous baselines in August 2012 while one remained the same at 66% (staff reporting RBA in Anglicare WA as a good thing).

The increase in percentage of services in Phase 3 has been driven by availability of July-Dec data (footwork for data collection plans were set-up Jan-June 2012 with data only now being available for a full 6 month period) and regular RBA monitoring meetings with General and Regional Service Managers. One of the obstacles to moving services in to Phase 3 is likely due to the time it takes to get there. Some services are unable to get together often to go through RBA and move forward with implementation while teams simply have not had enough time to see the usefulness of the RBA process yet.

Staff reporting RBA is good for Anglicare WA has remained constant at 66% in both August of 2012 and March of 2013. This flat lining is a reflection of the large amount of staff who are still undecided (~26%); with time and continued opportunities to use RBA, these undecided staff will most likely be able to definitively answer this in the near future. This result could also be a reflection of teams viewing RBA as something they have to do to “tick the box” and not as something that could be useful to both them and their clients. Teams need to see the value of reflective practice as exciting to be sure they are implementing RBA in a timely manner before losing momentum. A final reason for this result could be that coordinators are not well versed enough in RBA to make it a routine part of reflection during team meetings. To be effective, RBA needs to be an ongoing discussing and incorporated into day-to-day operations at the service level.

Partners Who Can Help Us Do Better

All Anglicare WA Programs and Staff

The Community Sector

Other Not-for-Profit Organisations

National and Federal Government

Trained RBA Trainers

Action Plan to Continue Turning the Curves

In an effort to continue improving performance on our four headline measures, Anglicare WA will:•Work in-depth with one service in each of our sub-brand areas (Relationships, Community, Youth, Housing) and use them as case studies

• This will help us determine if more intensive support correlates to better understanding and, more importantly, high use of the RBA framework for service improvement

•Increase information to all staff on RBA success stories through the intranet and staff meetings•Offer more RBA introduction sessions for staff who are new or need a refresher on the basics of the framework•Encourage teams to present RBA at their team meetings and across teams•Share more examples with staff of RBA and where it works•Make RBA a part of induction to help deal with staff turnover issues•Implement an award for “Curve of the Month” to highlight progress being made by service

RBA HEADLINE PERFORMANCE MEASURES