results based accountability
DESCRIPTION
Results Based Accountability. Charlotte Drury WLGA. Results Based Accountability. Purpose of Workshop: Provide an overview of the RBA framework Show how RBA can help to plan and deliver outcome based services Give some examples of practice in Wales. Results Based Accountability. - PowerPoint PPT PresentationTRANSCRIPT
Results Based Accountability
Charlotte Drury
WLGA
Results Based Accountability
Purpose of Workshop:
• Provide an overview of the RBA framework
• Show how RBA can help to plan and deliver outcome based services
• Give some examples of practice in Wales
Results Based Accountability
RBA increasingly popular in UK and elsewhere . . .
• Embeds performance management into planning and delivery
• Helps turn talk quickly into actions • Explains both collaborative and service
accountability and how they fit back together
A disciplined way of embedding outcome based decision making into planning, delivery and accountability for partnerships and projects.
RBA included in CYP Plan and HSCWB Guidance
www.cartoonstock.com
All these new words are very elitist, and I don’t think there’s a place for it. [By...] using language that a fair proportion of the community won’t understand you’ve already set yourselves apart before you’ve even started.’
Interview transcript - Royal Town Planning Institute research into the language of regeneration, 2001
The Importance of Language
The Importance of LanguageToo many terms. Too few definitions. Too little discipline
Benchmark
Target
Indicator Goal
Result
Objective
Outcome
Measure
Modifiers Measurable Core Urgent Qualitative Priority Programmatic Targeted Performance Incremental Strategic Systemic
Milestone
Aim
Impact
RBA Common Language Tools
IDEAS POSSIBLE LABELS CHOICEWORDS MODIFIERS
4 legged mammal – often kept as pet – barks
• Dog• Hound• Canine
• Large• Brown• Toy• Dangerous
“Toy dog”
What we use to measure the benefits of our service
• Measure• Result• Indicator
• Impact• Customer
“Customer Result”
Two Types of Accountability
Performance Accountability – the responsibility of a project, programme or service system for specific results for its “customers” (e.g. drug cessation programme accountable for “customer results”)
Population Accountability – collective or collaborative responsibility for the well being of populations (e.g. Local Service Board accountable for all people in L.A. Area)
Why This Matters: Responsibility
English National Indicator Set:
Outcome 4: Healthy & Successful Adults
(Population Accountability)
NI131: Timeliness of social care assessment
(Performance Accountability)
Why This Matters: Responsibility
CYP Plan Outcome Measures Framework:
Priority 3.2: Every child and young person is safe and protected from abuse, victimisation and exploitation
(Population Accountability)
Outcome Measure: % of initial Social Services assessments carried out within 7 working days
(Performance Accountability)
Keeping Accountability Straight
Population Outcome A condition of well-being for a population (or sub-pop) stated in plain language
Population Indicator Measurable information which helps quantify achievement of population outcome
Performance Measure Measurable information which helps quantify if a service etc works
• How much did we do?• How well did we do it?• Is anyone better off?
Pop
ulat
ion
Per
form
ance
Children in Wales are born healthy
People in Wrexham live in a safe community
Rate of low birth weight babies in Wales
Crime rate in Wrexham
Population and Performance Accountability
Children in CardiffWHOLE POPULATION
CONDITION OF WELL-BEING
A Flying Start for Children in Cardiff
POPULATION INDICATORS% low birth weight
% breast fed at 6 months% fully immunised
% assessed ready for school
Parenting Project
PERFORMANCE MEASURES% parents attending gaining
better coping skills% parents who use techniques
learnt after 6 months
Contribution
Means
End
Courtesy of David Burnby & Associates
Outcome, Indicator or Performance Measure?
1. People in the community are safe2. Crime Rate3. Average Police Response Time4. People living in safe, stable affordable housing5. First time entrants to Criminal Justice System6. % of low birthweight babies 7. % of LAC reviews carried out on time8. Achievement of 5 A*-C grades at GCSE
Outcome
Outcome
Indicator
IndicatorP.Measure
Indicator
P.Measure?
If the question is “How are we doing as a community on education attainment?”If the question is “How good is XYZ school at raising educational attainment?”
IndicatorP.Measure
From Ends to Means in 7 QuestionsPopulation Accountability
1. What is the quality of life condition we want for the selected population in our community?
Population Outcomes
A condition of well being for a population or sub population stated in plain language.
A Population (or sub)
+
A Place
+
A Condition of Well Being
From Ends to Means in 7 QuestionsPopulation Accountability
1. What is the quality of life condition we want for the selected population in our community?
2. What would this look like if we could see it?
3. How can we measure these conditions? 4. How are we doing on the most important
of these measures? (Baselines and the story behind them)
Today
3 years if nothing changes – OK?
Baselines
Time
% Teen pregnancy
Target for 3 years from now
WHY?
From Ends to Means in 7 QuestionsPopulation Accountability
5. Who are the partners that have a role to play in doing better?
6. What works to do better, including no cost/low cost ideas and off the wall ideas?
7. What do we propose to do? Action – what, how, who, when, how much!
1. Condition of Well Being
2. What does this look like?
3. How can we measure these conditions?
4. How are we doing?
5. Which partners?
6. What works to do better?
7. What are we going to do?
Baselines & stories
DD1
DD2
Children in Powys are safe
Safe at home, from crime & bullying etc
% young people victims of crime
Police, YOTs, schools, families etc
Best ideas, new ideas etc
Action plan and budget – by when and how?
The RBA Commissioning Journey – From Talk to Action
DESIREDOUTCOME
Choosing the
INDICATORS
Drawing the BASELINE (and the
CURVE TO TURN)
The STORY BEHIND
THE BASELINE
Choosing the
PARTNERS
Deciding WHAT
WORKS
The ACTION PLAN
REVIEW
COMMISSIONING
PERFORMANCE MEASURESHow Much?How Well?Better Off?
(CLIENT OUTCOMES)
Contributory Relationship
ENDS (POPULATION ACCOUNTABILITY)
MEANS (PERFORMANCE ACCOUNTABILITY)
Risks
• Less money – fewer grants – more competitive
• Everyone likely to be hit - some more than others
• Less stability = more difficult to retain and develop good people
• Ultimately the price falls on people who need these services
Opportunities
• Agencies which speak a common language best placed improve services & save money
• Greater clarity of expectations • Being able to demonstrate impact of well-
founded programmes• Able establish clear link between service
results and population well being• Opportunity to use flexibility of 3rd Sector
Performance Accountability - for projects/services with a defined customer)
Two Types of Accountability
Performance Accountability – the responsibility of a project, programme or service system for specific results for its “customers” (e.g. drug cessation programme accountable for “customer results”)
Population Accountability – collective or collaborative responsibility for the well being of populations (e.g. Local Service Board accountable for all people in L.A. Area)
Keeping Accountability Straight
Population Outcome A condition of well-being for a population (or sub-pop) stated in plain language
Population Indicator Measurable information which helps quantify achievement of population outcome
Performance Measure Measurable information which helps quantify if a service etc works
• How much did we do?• How well did we do it?• Is anyone better off?
Pop
ulat
ion
Per
form
ance
Older people in Gwent live independent lives
Older people in Flintshire live in a safe community
Number of hospital admissions due to falls
Crime rate in Flintshire
Performance Measures
Quantity Quality
Effort
Effect
or
or
How much did we do?
#
How well did we do it?
%
Is anyone better off as a result?
# %
Performance Measures
How much did we do? How well did we do it?
Is anyone better off as a result?
(#) Customers served
(#) Activities
(%) Common measures (e.g. % staff fully trained)
(%) Activity specific (e.g. % on time, % fully completed
(%) Skills or Knowledge (e.g. qualification)
(%) Attitude or Opinion (e.g. towards school)
(%) Behaviour (e.g. attendance)
(%) Circumstance (e.g. in work)
Types of Performance Measure
How much did we do? How well did we do it?
Is anyone better off as a result?
# young people seen
# 1-to-1 sessions
% staff with higher qualification
% 1st appointments within 2 weeks of referral
% people using service who are off drugs/reduced use at end of 12 week intervention
% people using the service who are off drugs/reduced use 6 weeks after end of intervention
Substance Misuse Service
Performance Measures
How much did we do? How well did we do it?
Is anyone better off as a result?
# of additional learning courses
# young people on courses
% sessions delivered to schedule
% young people completing course
% young people on additional learning courses gaining 1 or more GCSEs
% young people on additional learning course with improved general attendance
Education
Performance Measures
How much did we do? How well did we do it?
Is anyone better off as a result?
# outreach bus sessions
# contacts
% sessions delivered to schedule
% young people writing action plan
% young people accessing appropriate training as a result of contact with youth outreach
% young people reporting increased healthy behaviour
Youth Outreach Service
Performance Measures
Turning Case Management Data into Performance Measures
NAME ISSUE RESULT
John C Anxiety about attending school Significant change
Laura D Feels “lonely & friendless” Partial change
Carl F Involved in nuisance activity No change
Anne P Using alcohol Partial change
Mary H Unauthorised absences No change
Possible service performance measure of “Is Anyone Better Off”? (Lower Right)
• % service users showing “significant” or “partial” positive change (60%)
An aggregate of the difference the service has made to individuals!
1. Customer Population
2. Is anyone better off? LR
3. Are we delivering services well? UR
4. How are we doing?
5. Which partners?
6. What works to do better?
7. What are we going to do?
Baselines & stories
DD1
DD2
Young people on ed. support programme
% Key Stage passes
% sessions delivered on scheduled date
Schools, families, peers etc
Mentoring, alternative curriculum
Action plan and budget
Population Outcome Contributed to – “High Achievement for Young People”
Fitting Population Accountability and Performance Accountability Together
Contributory Relationship
Alignmentof measures
Appropriateresponsibility
POPULATION ACCOUNTABILITYOutcome = Healthy childrenIndicator = % low birth weight
How much?
# sessions
# service users
How well?
% sessions to schedule
% satisfied
# service users stopping while
pregnant
PERFORMANCE ACCOUNTABILITYAnte-Natal Smoking Project
Is anyone better off?% service users stopping while
pregnant
Fitting it all Together
Customer Result
Presenting Your ContributionOutcome(s) to which you most directly contribute
Indicators
Story behind the curves to turn
Partners with a role to play
Actions – what its going to take to turn the curve
Your role – as part of the larger strategy
Pop
ulat
ion
Acc
ount
abili
ty
Story behind the curves to turn
Partners with a role to play
Actions – what its going to take to turn the curve
Your programme & what it delivers
Performance measures
Per
form
ance
A
ccou
ntab
ility
How it all fits together
Community Strategy Outcome = Prosperous City
Indicator = % unemployed
CYP Plan Outcome = Children Live Free From Poverty
Indicator = % children living in workless households
Project = Maintained Childcare
% parents helped to access work, education or training
Economic Strategy
Flying Start
Outcomes & contributions align – indicators/measures make sense – appropriate accountability – what matters is outcomes not plans and process
THE WELSH EPILEPSY UNIT
DEFINED SERVICE USERSPatients with a first suspected seizure or unexplained blackout
HEADLINE PERFORMANCE MEASURES1. % seen by a specialist within 2 weeks2. % DNA first seizure clinic3. % have diagnostic tests within 4 weeks4. % follow the correct pathway
Service Description: The Welsh Epilepsy Unit is a tertiary referral centre for specialist epilepsy services in South Wales. The immediate catchment population covered is 700,000, but many referrals are also taken from elsewhere in Wales. The Unit offers a multidisciplinary approach to epilepsy care and offers a very broad range of services to people with epilepsy, their families and carers.
DATA DEVELOPMENT AGENDA
1. % on inappropriate treatment2. % have clinic letters sent within one week of clinic3. Why patients DNA first seizure clinic
HOW ARE WE DOING?
STORY BEHIND THE BASELINES
Clinic capacity – 1 clinic per week with 5 patient slotsUnpredictable demandSmall MDT – unable to cover absence to prevent clinic cancellationLow frequency of clinics causes delay if appointment not suitableClinic booked by Epilepsy Unit admin staff – if admin staff on leave clinic slots not filledConsultant triage’s fax referrals – delay if unavailablePatient anxietyStigma attached to EpilepsyPatients put off by unit name – diagnosis seems pre-determinedConcerns re implications e.g. diving
WHAT WE PROPOSE TO DO TO IMPROVE PERFORMANCE
Develop nurse led Emergency Unit assessment service Develop nurse led first seizure clinics Enable specialist nurse referral for EEG Change the name of the Epilepsy unit
PARTNERS WHO CAN HELP US DO BETTER
Emergency Unit/MEAU, Radiology, Neurophysiology, Medical records, A&C staff, Consultants, Ambulance Trust, Cardiology, Psychology, Care of the Elderly, Neurosurgery, Prison, Voluntary Sector, CELT, Practice Nurses, Family members/witnesses, Drug and Alcohol Services, Occupational Health, Referral Management Centre, Obstetrics.
% Seen by a Specialist within 2 Weeks
0%5%
10%
15%20%25%
2007 2008 2009 2010 2011 2012
BaselinePredictionCurve to turn
% DNA First Seizure Clinic
0%5%
10%15%20%25%30%
2007 2008 2009 2010 2011 2012
BaselinePredictionCurve to turn
% have diagnostic tests within 4 weeks
0%5%
10%15%20%25%30%35%
2007 2008 2009 2010 2011 2012
BaselinePredictionCurve to turn
% follow correct pathway
0%20%40%
60%80%
100%
2007 2008 2009 2010 2011 2012
BaselinePredictionCurve to turn
HOW ARE WE DOING?
Population Scorecard
POPULATIONChildren aged 0-3yrs 11 months with an eligible post code, resident in one of the eight Cardiff Flying Start catchment areas.
OUTCOMEFlying Start Children are Healthy & Thriving
Headline Indicators (Bellwethers)• % FS Children assessed, scoring >2 deviations below their
chronological age on the ‘Schedule of Growing Skills Assessment at 2yrs (21-29 mths) & at 3yrs (33-41 mths)
• % FS Children fully immunised to schedule at 47 mths• % FS Families who respond to questionnaire, reporting that
FS has helped THEIR CHILD ‘a lot or more’
DATA DEVELOPMENT AGENDA• % FS Children completing primary immunisations• % FS Children with a BMI below the 91st percentile
and above the 2nd percentile at 36 months• % FS Children scoring >2 deviations below their
chronological age on the ‘hearing & language skills SOGS Assessment ‘ when aged (a) between 21-29 mths & (b) 33-41 mths
% Children assessed at 2 & 3yrs, scoring at/above or below their chronological age (1st April 2009 - 31st March 2010)
0%
10%
20%
30%
40%
50%
60%
70%
80%
% At Above Chron. Age % Below Chron. Age
%
2 Years
3 Years0
20
40
60
80
100
%
Catchment
% Children scoring at/above or below chronological age
% Below Ch. Age 66 8 52 26 36 20 35 35 17
% At/Above Ch. Age 33 91 48 73 63 80 64 65 82
Adamsdow
Glan-Yr-
Greenway
Herbert
Mount
Shirenewto
St Mary
Trelai Catch
Windsor
HOW ARE WE DOING (Con’t)? % Parents responding to questionnaire, reporting that Flying Start has helped their child with:
0%10%20%30%40%50%60%70%80%90%
Speec
h
Behav
iour
Social S
kills
Confiden
ce
Learn
ing
Toileti
ng
Eating h
abits
Genera
l Healt
h
Physic
al Deve
lopmen
t
% a
nsw
ered
"a
lot"
or "
a hu
ge a
mou
nt"
DECEMBER 2009 - 38 out of140 questionnaires returned(27%)
MARCH 2010 - 21 out of 98questionnaires returned (22%)
JULY 2010 - 31 out of 182questionnaires returned (17%)
STORY BEHIND THE BASELINE• Children who had their 1st SOGS II assessment at 3 yrs were born in 2006/7, prior to the initiation
of the FS Programme. These are children that would have had very limited exposure to FS interventions. In comparison the children who had their 1st SOGS at 2yrs were born in 2007/8 & would have had greater exposure to FS interventions. The trend line indicates that children having greater exposure to FS are more likely to be developing normally. Initial analysis needs to be treated with caution, as we need to be certain that SOGS assessment are being applied consistently SOGS scores shows marked developmental variations across catchments, with some children in particular catchments scoring lower than average on particular skills sets.
• Schools are telling us that FS children are entering nursery classes with good language & communication skills, are settling quickly and are more independent that their non FS peers.
• The introduction of mandatory training for all FS practitioners on supporting and developing childrens language & communication is possibly having an impact on practice. Childcare settings are also being encouraged to take up and apply the ‘Learning Language & Loving It’ training and we think that this is starting to improve practice.
• Parents are reporting that FS is helping their children to be more confident and sociable and that they are learning more, but more could be done to support parents in developing their childrens language skills; relationships and managing behavioural issues.
• Immunisation data suggests that parents are good at taking their children for Primary Course & Booster immunisations but that it drops below 50% for the final MMR & HIB catch up immunisations.
PARTNERS WHO CAN HELP USFS Service Managers & Teams; FS Catchment Schools; Early Years Forum; Childcare Settings; Childcare Workforce Development Team; Parents; GP Practices
WHAT ARE WE DOING & WHAT IS OUR RESOURCE• Develop a referral pathway for children
with additional needs• Explore development of ANCo role for FS
childcare settings• Interrogate SOGS & Parental
questionnaire data to establish trend lines & what might help parents more
• Develop a standardised induction process for all FS staff, which includes compulsory training on language development and behaviour management strategies.
0%10%20%30%40%50%60%70%80%90%
100%
% 4 yr old FS Population
% Uptake ofCompleted
PrimaryCourses
% UptakeMMR, HIB,
MENCBooster &PNE Imms
% UptakeMMR & HIB
Catch-UpImms
Immunisations
% FS Children Fully Immunised at 47 months
Children Born 1/10/06-31/10/06
Cardiff Flying Start Parents Plus Service : Performance Evaluation Report 2009-10
Service Description: The Parents Plus service is part of the Parenting core strand of the National Flying Start Programme. The Parents Plus Service consists of one Senior Educational Psychologist; three Senior Home Liaison Officers; four Home Liaison Officers; and two p/t Administration Officers. In addition, there is one f/t Specialist Home Liaison Officer (Homeless) and two Home Liaison Officers (former Sure Start) It has a budget of £312,972 and aims to offer time-limited, targeted, home-based parenting interventions in each of the 8 Cardiff Flying Start catchments reaching approximately 110 families annually.Purpose of Service
• Assessment of child’s functioning within family and community context• Agreement with parents on appropriate changes to implement within the home• Evaluate improvements in early play skills; in family interactions and relationships; and in the child’s behaviour
Headline Performance Measures1. Perceived improvements by parents in play/devt; relationships; and/or behaviour2. Perceived improvements by HLO in play/devt; relationships; and/or behaviour3. Use of behaviour techniques by staff with EY children 6 months after PP+ Behaviour Training course
Data Development AgendaDD1 - % Families referred (needing the intervention) who take it up DD2 - % Families receive Initial Assessment Visit within five weeks of referral
How Are We Doing?
Story Behind (last 3 years) Performance
Parents Plus has been through a period of transition: from earlier Sure Start processes to greater Flying Start standards and requirements. It has been an exciting and challenging period. Forming a coherent induction period and regular supervision has ensured new HLOs have become established within the team. The additional supervision and monitoring by Senior HLO have contributed to the successful and relatively smooth move to Flying Start functioning. Unfortunately, significant personnel changes and alterations in working practices by other partners, has had knock-on effects to our referrals and integrated working. Over time, this has been improving. Staff illness and absenteeism has also interfered with our capacity to engage with some families. The developments within the EYFT(Homeless) has ensured an innovative multiagency team approach to working across city hostels, including within Flying Start areas.
Research by Cardiff University followed up families which had received Parents Plus interventions up to six years ago. Parental interviews confirmed the effectiveness of our behavioural techniques; and their long lasting influence. This was in part due to empowering the parents to take ownership of the changes. There was evidence that the families referred to Parents Plus have had more entrenched difficulties and dysfunctional patterns of family life. This has meant they tend to be more difficult to engage and to intervene with to implement change. Nevertheless, evaluations of changes following Parents Plus interventions [ Before v After ] continue to show better play/developmental progress, benefits for family relationships and improvements in behaviour management. Parents Plus contributes to improving the wellbeing and progress of Flying Start children through high quality training to EY staff working with young children.What We Propose To Do To Improve Performance In The Next Year
Low Cost/No Cost Idea:-Provide “telephone clinic” weekly when Senior HLO will be available at base-Print a leaflet for parents attending all settings on “Building Better Behaviour Together”-Publish article on Parents Plus approach-Re-establish termly visits to all Nursery settings [school-links]-Posters for childcare settings about 1. referral to Parents Plus and 2. self referral by families to Parents PlusOff the Wall Idea:-Develop a training manual for childcare staff with introductory, intermediate and -advanced modules on Understanding and Managing Behaviour in Early Years Settings -& identify funding source to publish and disseminate-put together video examples of “before and after”-Team approach with gypsy-traveller families through childcare session
Partners Who Can Help Us Do BetterReferral criteria explicit and awareness raising:-•Childcare staff•Health Visitors•Children’s services•Nursery staff•Play team•Home Start•CAMHS/PSS
Distribution of Parents Plus service activities
Home Interventions
PNP facilitating
FS Training
Behaviour advice intosessions Psychology into FS
EYFT Homeless
Sandfields Family Support Project REPORT CARD : Performance AccountabilityCompliance monitoring and outcome data reporting
The Service will contribute to the Core Aims and the Local Authority’s Vision for Children and Young People .
How much did we do90 referrals received for Summer Playschemes18 parenting courses delivered- focus: resolving conflict and difficulties 12 Child Care:- Creche96 Child Care:- Playgroups39 Summer Playschemes91 Drop-in
How well did we do it% referrals responded to in (x) days% of parents and carers completing parenting courses % parents and carers who report satisfaction with programmes% C/YP engaged in safe activities within their community% C/YP who have re-engaged in education% parents / carers actively participating in C/YP educational reviews/plans % C/YP who report satisfaction with the service
Did we make a difference% of C/YP who report they are more able to deal with conflict and difficulties % of C/YP who develop positive coping strategies % of C/ YP who report increased confidence and self esteem % of C/YP who report they now more able deal with bullying % of C/YP who remain safely in their own homes% increase in school attendance % reduction in offending behaviour% reduction in risk behaviours % parents and carers who report they are more able to deal with conflict and difficulties % parents and carers who report improved relationship with their C/YP% parents and carers who report more confidence when dealing with their C/YP educational needs
Action for Children will work with our Partners to maximise customer outcomes as measured by our “Did we make a difference”.
Action for Children has identified and are working with 43 indicators to evidence our contribution to achieving outcomes defined in the Wales Core Aims
Early Intervention Service :
Reduction in risk factors related to offending Increase in protective factors related to offending Increased presence of positive role models and positive
peer influences in children’s lives Increase in positive coping strategies within children and
young people Increased levels of self esteem within children and young
people Increased levels of emotional well -being Increased engagement between parents and other relevant
services Improved parenting skills Increased numbers of children engaged with education Increased parental participation in the education of their
children
P
leas
e no
te: d
emo
exam
ple
only
Jacky Davies , Diane Hardy - September 2009 ©
Improved outcomes for children and young people – Qtr 1 2009/10 – cohort 52
Enjoy the best possible health and are free from abuse , victimisation and exploitation – The C/YP recognises their own health and developmental needs
Enjoy the best possible health and are free from abuse , victimisation and exploitation – There is an improvement in the C/YP’s emotional wellbeing
Enjoy the best possible health and are free from abuse , victimisation and exploitation – The risks to the C /YP are known and protective factors are in place
Are not disadvantaged by poverty – The young person has suitable accommodation
A comprehensive range of education and learning opportunities – The C /YP achieves satisfactory attendance at school , further education , training or in employment
Are not disadvantaged by poverty – There is an improvement in the child or young person’s practical life skills
Profile of Service Users on Discharge from Action for Children’s Service
How Can RBA Help Us?
1. Planning & performance management framework to deliver on collaborative duties
2. Planning & performance management framework for funded projects (e.g. Cymorth)
3. Organise and explain how population and performance levels fit together e.g. CYP Plan and Cymorth projects
4. Make sense of multitude of partnerships and plans – what matters is results not processes!
5. The future – less money, less small specific grants, more emphasis on outcomes and collaboration
Implementing Results Based Accountability
• Leadership at a senior level• Championship by a senior project
manager and from champions at different organisational levels
• Ownership of a commitment to recognising success as better outcomes, not outputs
• Partnership and multi-agency working. RBA is hard to implement where partnerships are not invested in
Resources
• Mark Friedman Trying Hard Is Not Good Enough (Trafford Publishing, 2005)
• www.raguide.org – website including implementation guide and examples
• www.resultsaccountability.com – website including papers on Results Accountability and links to other resources
Results Based Accountability
Charlotte Drury
Questions?