Measuring success through outcomes:

A way forward

Les Hems July 2014

2

Why Outcomes Measurement is

important

3

‘For purpose’ is a better concept than ‘not-for-profit’

Limitations of existing terminology:| Not-for-profit – contrast with for-profit business - but all

organisations need to make a surplus to be sustainable | Charity – dependence on grants and donations - but all

organisations need a reliable source of revenue to be sustainable

Increasing preference for alternative terminology:| For-purpose – mission driven - creating social change and

impact| Social enterprise – co-creation of commercial and social valueIncreasing interest in alternative structures:| Co-operatives and mutuals – owned and controlled by

members| Civil Society – organisations embedded in communities| Co-operative Collective Impact – real community collaboration

Measuring outcomes rather than Value For Money

For-purpose: The problems with VFM measurement

1. Financial measurement:limited measure of value

2. We allocate resourcesto the things we can measure

3: Stakeholders are often left out of decision making

What are outcomes and impact?

| Outcomes are the significant positive or negative changes for clients (customers / participants), their families, the broader community and other stakeholders.

| Outcomes:Can be short, medium and long termCan be direct and indirect (spill over)Can be for target beneficiaries and other

stakeholdersCan be economic, social, human and environmental

| Impact is the net effect of these outcomes

6

Inputs to impact via outputs and outcomes

Input: The resources you use to undertake tasks and activities – not just financial – human capital etc.Activities: The programs and tasks you undertake to generate outputsOutput: Tells you an activity has taken place and is usually quantitative Intermediate outcomes: The short term change that occurs as a result of an activityOutcome: The long term change that occurs as a result of an activity – the net benefit to recipients – the intended resultsImpact: The long term net benefit to society

7

Why measure outcomes? External perspective

Clients / consumers| Help consumers make good choices and to achieve their

goalsFunders – Government, foundations, donors and corporates| Recognised limitations of funding activities and outputs| “Investment” in prevention, early intervention and

breaking the cycle| Client directed care| Payment by Results (PbR) / for Outcomes, / for Success –

linked to Social Impact Bonds| Collective impact initiatives focused on outcomesEvidence| Focus on what works – the desired outcomes being

achieved Measurement and reporting| Increasing use of approaches such as Social Return on

Investment, Results Based Accountability

Radical / disruptive changes in complex systems

Measuring outcomes is at the heart of social change – interest in social impact has been growing and will become the norm:| Higher expectations| Ineffective responses to

wicked social problems| Increasing demand| Reducing tax revenues

Why measure outcomes? Internal perspective

Mission and strategy| Organisations’ mission statements should focus on

outcomes | Outcomes are the fundamental purpose of an organisation

and therefore the focus for organisational strategyPerformance measurement| Measuring outcomes is the most important measurement

of an organisation / program performanceBenchmarking| Emulating best practice – outcomes frameworks, outcomes

reporting and social impact statementsHigh performing organisations are learning organisations| Organisational learning – the development of a framework

and measurement of outcomes generate learning

13

14

Key concepts for measuring outcomes

Key concepts: Stakeholders

| Stakeholder: Any actor or agency that has an effect on, or is affected by the intervention:They contribute to or experience the changeCan be individuals, families, organisations (e.g.

different tiers of government), collectives, the environment

| Key elements for outcomes measurement:Stakeholder mappingNetwork analysis – interdependence between

stakeholdersStakeholder engagement

15

16

Network analysis – partnerships and collective impact

Key concepts: Client segmentation

Clients / customers / participants are not all the same:| Demographics| Vulnerabilities| Capabilities| Expectations and aspirations Map client segments and their related outcomes| Many to many relationships between client

segments and outcomesHow will the program / intervention respond to these different client segments and outcomes? | High, medium and low – clients & complexity of

service

Key concepts: Materiality

18

| Focus on the stakeholders experiencing material outcomes If I left this outcome out of the measurement, would

somebody make a different decision about the program under analysis:

| Consider relevance and significance to identify materiality

| Iterative process: Make an initial determination to decide which stakeholders to

include in the stakeholder engagement. Then, consult with stakeholders to identify their perspective.

| You can’t measure everything.| Stakeholders with material outcomes will be a subset of the

stakeholders and outcomes you identify

Key concepts: Theory of change

19

| A depiction of how a program or intervention: Is meant to work What it is supposed to achieve Identifies any cause and effect relationships between

activities and outcomes

| Explores the how and the why| Combines evidence and direct experience| A framework for evaluating programs, and

designing and predicting the outcomes of a program

| Can also be used to develop an ‘outcomes hierarchy’ for an organisation as a whole

Key concepts: Impact

20

| Impact refers to the net benefit to society - the ADDED VALUE of your programs

Key concepts to consider:| The counterfactual – what would have

happened otherwise – “Deadweight”| Transferring the problem to or outcome from

elsewhere – “Displacement”| The proportion of the change which relates to

your organisation’s activities – “Attribution” or “Contribution”

| The proportion of the change which relates to working collaboratively – “Interdependence”

Principles for outcomes measurementPrinciple 1: Involve your stakeholders to measure social value| Measure what matters from the perspective of your

stakeholders – the people or organisations that have experienced change as a result of your activities.

Principle 2: Measure and understand the theory of change | Apply appropriate methods and resources to understand

and them measure the change (positive and negative, intended and unintended)

Principle 3: Be accurate and credible| Added value and impact needs to take into account:

What would have happened anyway? Deadweight How much did other organisations contribute to the

change? Attribution / Interdependence Have the issues moved somewhere else? Displacement

21

22

Theory of change – the foundation of

outcomes measurement

Theory of change

23

| A theory of change depicts how a program is intended to achieve meaningful, positive changes for clients — not simply a description of what the intended change is

Demonstrating outcomes leads to more inputs

24

| Feedback loop - demonstrating the outcomes – both in words and numbers – can be instrumental in generating more inputs - the evidence of what works wins tenders, attracts investors, and provides collateral for marketing and fundraising.

25

Example: Community housing for disadvantaged youth

Stakeholder Inputs Activities Outputs Outcomes

Residents

•Accommodation

•Maintenance costs

•Management costs

•Staff expenses

•Access to housing

•Formation of groups

•Number of groups formed

•Number of new friends

• Increased community inclusion

• Increased sense of security

• Improved personal health

Family members

•Not applicable

•Reduced number of hours family member is engaged

• Increased income for family

•Less time spent on care for family member

Gover-nment

•Not applicable

•Health and welfare service usage

•Less usage of intensive child support services

•Reduced costs to Department of Health and Human Services

•Reduced cost to State medical system

26

Improved personal health

Increased sense of security

Provision of secure & affordable housing

Referral to and provision of health, employment and other social services

Building of inclusive communities

More regular sleeping patterns

Appropriate treatment accessed

Reduced substance misuse

Participation in group activities

Use of employment services

Increased sense of personal safetyDecreased financial stress

Greater ability to apply to jobs

Increased employment readiness

Increased community inclusion

Greater social engagement

Decreased mental stress

Activities OutputsIntermediate outcomes Long-term

outcomes

Access to housing

Example: Community housing for disadvantaged youth

Combines evidence and direct experience

| Preparatory desk review of evidence from: Academic research and evaluations – national &

international Population level indicators and validated measurement

tools Existing organisational programs

| Direct experience of those: Delivering the service Receiving the service Other stakeholders

| Iterative process Develop prototype from evidence and those delivering the

service Refine ToC through beneficiaries and other stakeholders

27

The total value created by community services

Program Logic Social Return on Investment

Commercial Return on Social Investment

Outputs & outcomes for stakeholders

Inputs from stakeholders

Recognising all inputs and value created

| Financial– not just revenue but balance sheet| Manufactured– the physical assets, internal systems

you use| Intellectual– the research and development,

innovation and piloting you undertake| Human– the knowledge, capabilities & experience,

wellbeing & health of all those who contribute– paid/unpaid

| Social and Relationship– networks, trust and societal norms – relationship value – within communities (place and interest) and families – between organisations (Interdependence)

| Natural– the environment

29

30

Measuring Change

Quantitative research – evidencing outcomes

| Once we have identified and defined outcomes through the qualitative stakeholder engagement in the Theory of Change, we can quantify them.

Key questions: | How do we know how many people have experienced

this outcome?| How do we know the extent to which this change has

taken place?|By definition - need to measure change over time:

Retrospective construction – “How are you today?” and “Has this got better or worse over the last two years?”

Baseline and follow-up – embedding measurement in to an organisation

31

Some basic principles for quant work

| Maximise the utility of all relevant previous work – external and internal

| Build a counterfactual: Ideally, establish a control group to assess deadweight If not, use self reporting, ask “What would have

happened if you had not received help?”| Appropriate sample size and diversity:

Be pragmatic as can be resource intensive Use of client records

| Pilot your measurement tool / instrument!| Do you need incentives?| Do you need Ethics Approval?

Indicators

| Indicators are key component of measuring outcomes i.e. the changes that have taken place

| Indicators can be Objective e.g. no. of health clinic visits or Subjective e.g. self-reporting on physical health

| Intermediate outcomes achieved can be used as indicators to measure the final outcome

| Lead indicators can also be used to assess likelihood of final outcomes being achieved:

| Some indicators will work for multiple programs| Be pragmatic but don’t cut corners: You may not

have access to data required to measure ideal indicators.

33

Evidencing outcomes – distance travelled

Evidencing outcomes – distance travelled

35

OutcomeAchievement Level Indicators

Improved money management

Very Low- I am always short of money for basic expenses- I am unable to pay any of my regularly scheduled bills- I have no idea of what my outstanding debts are

Low- I often find myself short of money for basic expenses- I often default on my regularly scheduled bills- I know that I owe money but don’t know how much or how I can pay it back

Average- I have tried using a budget but found it difficult to follow - I occasionally fall behind on my regularly scheduled bills- I am aware of the extent of my debts and want some help to clear them up

High

- I am using a budget regularly but sometimes tend to lose track of my expenses- I am making some changes in my habits to be able to afford my regularly

scheduled bills- I am seeking help to begin to payback my previous debts

Very High

- I am able to purchase my basic requirements and stay within a budget- I never default on my regularly scheduled bills- I have been able to eliminate all my outstanding debts OR - I am following a payment plan to wipe out my outstanding debts

36

Evidencing outcomes - Example

Improved Money Management

Very Low Low Medium High Very HighVery Low 5

LowMedium 6

High 7 4Very High 6 2

Sample Size 30

Total 60

Client's Initial Position

Clie

nts E

nd

Posi

tion

37

Approaches for measuring outcomes

What’s the best approach for measuring outcomes?

| There is not a ‘best’ approach | Choose the most suitable and appropriate

approach based on your needs| One program, one organisation or a collective

(shared measurement)| There may be a role for using multiple tools

depending on your needs| Remember the principles of good outcomes

measurement Involve stakeholders Understand what changes Understand impact or added value

38

Social Return On Investment

| A way of understanding how effectively money is spent| A way of comparing the value of change created to the

investment required to create the change

SROI = [Value of change] [Investment]

| It is an enhanced form of cost-benefit analysis which considers triple-bottom-line benefits and investments (economic, social, environmental)

| It measures outcomes, not outputs| It measures change that matters to stakeholders| It can be evaluative or predictive

e.g. $1: $3.60

SROI – a principle-based approach

40

5. Do not over claim

6. Be transparent

7. Verify the result

1. Involve Stakeholders

2. Understand what changes

3. Value the things that matter

4. Only include what is material

Suitable applications for an SROI

| Organisational learning – finding out what works and why| In circumstances where you require a financial valuation:

Where value for money - in its broadest sense - needs to be demonstrated

Social impact bonds Requirement of a funder

| Where a common unit of measurement ($) would be useful: Compare across projects / programs within an organisation Different types of outcomes i.e. social, economic and

environmental| Where there are multiple interventions and contributors –

collective impact: SROI provides a framework for understanding your unique

contribution

41

SROI – strengths and weaknesses

Strengths Weaknesses

• Stakeholder focused process provides great insight into stakeholders and programs

• Powerful as a design and evaluation tool

• Provides a framework for understanding ‘added value’

• Is a basket of best practice tools – can be applied in its full form or using component parts

• Speaks the language of some funders/decision makers

• Has wide applicability across sectors and subject matter

• Overemphasis on the ratio• Heavily reliant on subjectivity and

assumptions -will evolve and strengthen over time

• The valuation process is often a source of discomfort/contention

• It is complex and resource intensive• Often used in a snapshot or

retrospective context which constrains data quality

• Comparison of ratio is inevitable but not always appropriate

42

Measuring outcomes – a proposed approach

| Outcomes Hierarchy for an Organisation – derived from a high level organisational Theory of Change

| Select program(s) for outcomes measurement and organisational learning

| SROI including program level Theory of Change Outcomes Measurement Framework – based on Total Value

Framework Embed measurement processes in to organisational

systems| Formative process evaluation – feedback learning in to

programs in real time| Stories as well as numbers| Reporting – dashboards, infographics, spreadsheet

models

43

44

Reporting outcomes

Case study: MLC Foundation & Lifeline Research Foundation

The MLC Community Foundation and the Lifeline Research Foundation undertook an impact study to measure the social value created by Lifeline Crisis Support Chat Service in suicide prevention in Australia.Social OutcomesThree main social outcomes were identified for the service users:| Reduced suicidality/self harming potential | Improved resourcefulness| Enhanced belongingIn addition, two areas of cost re-allocation were identified for public services:| Reduced use of hospital services| Reduced call outs for police and emergency services

| See flyer, report and presentation

The Social Value Created

Total Social Value Present Value of Social Value

Crisis Intervention (High Risk) $2,367,059 $2,233,075Crisis Aversion (Medium Low Risk)$3,166,579 $2,987,339Emergency Service $469,884 $443,286Medical Service $1,639,367 $1,546,573Total $7,642,889 $7,210,273

Total Value of Inputs $860,517

Social return on investment ratio = ($1:$x) $8.4

47

REDUCED ANXIETY IMPROVED QUALITY OF LIFE

INCREASE IN SELF ESTEEM

Easier access to future creditAbility to improve

credit profile

Discovered stability in their life

Enthusiastic about new opportunities

Explore new opportunities

Take advantage of new opportunitiesAbility to spend

within means

Ability to meet future bill payments

Prepared for any unexpected future payments

Made changes to household expenditure

Awareness of outstanding bills and terms

Ability to budget for future payments

Develop trust In loan officers

Confident of improving life

Trust and support of family and friends

Ability to get things done for self and

family

Set new life goals

Knowledge of how to prepare a

personal budgetReferral to financial counsellor

Access to favourable

payment terms

Access to immediate

credit

Opportunity to make a new start

Ability to pay off debts

Undertake previously

impossible tasks

Liberated from past constraints

Regain independence

Decreased bouts of depressionLeave behind

negative influces

Develop a positive future outlook

“I got into alcohol, got drunk everyday, I don’t want to go back there”

Appropriate

“It has allowed her to have a whole new start”“Has allowed me to start life afresh“The loan saved my life”“God heard my prayers”

“There is the human element here“...they’re like your friends”“...these people really care about you and are not like Cash Converters who are just out to get you”

“I was ecstatic when I was approved, I had found some freedom in my life”

“I just wanted to move things along and not have to wait”

“FFA does good things for people and allows them to do positive things”

1

Read more from case studies

2

3

“Feels like a weight has been lifted off my shoulders” “Without loan, there would be no transport, would not be able to get around would still be trying to figure out how to pay Cash Converters“The payments are flexible, they said that if I wasn’t able to afford

the repayment in 12 months, it could be changed to 18”

financing

“...no one else provides a service like this” “I like the idea of it being a small loan”

48

Reporting outcomes

Be prepared

| Client directed care will become the norm: Increasing competition between providers Greater emphasis on marketing and communication Risk of reliance on brand instead of market intelligence

| Evidence of impact - and value for money - will assist decision-making and allocation of resources

| Increasing interest in social finance which requires: A Payment by results / outcomes mechanism to trigger

reward payments A rigorous method for measuring changes in outcomes Monetising the value of achieving outcomes – cost

savings to Government and/or value of improved outcomes

Challenges: External

Complexity| Complex problems with multiple outcomes – limitations of

linear cause and effect Soft systems & Collective Impact Maximise use of existing evidence and knowledge

Funding systems| Most existing government administrative & funding data is

based on activities and outputs Develop own data systems in preparation for outcomes based funding

Standards| No reporting requirements for outcomes nor repository for

outcomes and frameworks Encourage self reporting and celebrate excellence

Challenges: Internal

Human resource| Limited expertise / competency and capacity

Capacity Building and co-production

Organisation systems| Lack of outcomes related data and and reporting

Implement case management systems

Organisational change| Inertia – organisation culture - resistance to change

Case studies of success

Costs | Dollars and time

This is an investment that will yield long term value

Measuring outcomes to achieve success: The Way Forward

Collective Impact

Biggest challenge for For- Purpose organisations

Experience

Source: Leth & Hems (2013

For further information

Les HemsDirector of Research and DevelopmentNet Balance Research Institute+61 448 588 450les@netbalance.com Twitter: @LesHems

Erica OlessonService Line Lead Net Balance Research Institute & New Zealand co-ordinator erica@netbalance.com

53

Stakeholder engagement: Resources

| Stakeholder engagement: Prove and Improve

http://www.proveandimprove.org/ AccountAbility

http://www.accountability.org/images/content/5/4/542/AA1000SES%202010%20PRINT.pdf

54

55

Theory of Change: Resources

| ActKnowledge’s Theory of Change community:http://www.theoryofchange.org/

| New Philanthropy Capital’s guide to Theory of Change:

http://www.thinknpc.org/publications/theory-of-change/

Indicators and data collection: Resources

| Guide to Measuring Well-being:http://www.neweconomics.org/publications/measuring-well-being | Outcomes Star: http://www.outcomesstar.org.uk/ | National Accounts of Well-being:http://www.nationalaccountsofwellbeing.org/| The Global Value Exchangehttp://www.globalvaluexchange.org | Sample size calculator http://www.nss.gov.au/nss/home.nsf/pages/Sample+size+calculator 56

References

| SROI Network http://www.thesroinetwork.org/| SROI Guide – on your USB| Results-Based Accountability

http://raguide.org/| Logical Framework Approach, Uni of

Wolverhampton Introduction – on your USB| Code of Good Impact Practice – on your USB

57