vMEASURING SOCIAL IMPACTCecilie Hestbaek, NPCThe Big Connect, Tuesday 7th June 2015

NPC: TRANSFORMING THE CHARITY SECTOR

2

NPC works at the nexus between charities and

funders

Charity

SectorFunder

Increasing the impact of charitieseg, impact-focused theories of change

Strengthening the partnership

Eg, collaboration towards shared

goals

Increasing the impact of funders

eg, effective commissioning

Consultancy & Think tank

3

SOCIAL IMPACT

‘The effect of an activity on the social fabric of the community and well-being of the individuals and families.’

http://www.businessdictionary.com

a) articulate how you think a service or intervention will have an impact;

b) and test your thinking against the evidence you have available.

4

THE ESSENCE OF IMPACT MEASUREMENT:

Exercise:Take 10 minutes to discuss with your neighbour why it’s important to

measure social impact and what the challenges are. Write key words on different colour post-its.

5

WHY IS MEASURING IMPACT IMPORTANT?

Saves staff time

Influences the debate on

“what works”Improved services

Raises profileMotivates

staff / volunteers

Taken from: Rickey, B, Lumley, T and Ni Ogain, E . (2011) A Journey to Greater Impact. New Philanthropy Capital.

Helps secure funding

6

CHALLENGES IN MEASUREMENT

Causality, consistency and predictability.

• What worked before might not work again because the external environment has changed

Time frame • Change can happen quickly; or it might take two decades to come about

Contribution and attribution

• A variety of external factors are beyond your organisation’s control and you will often have no

counterfactuals

Data collection• If you are targeting top-level decision makers, it can be

difficult to get an answer from them about why they did (or did not) change their mind on a particular issue.

…and ensuring that you measure outcomes, not outputs.

7

THE FOUR PILLARS APPROACH TO MEASURING IMPACT

Map your theory of change

Prioritise what you measure

Choose your level of

evidence

Select your sources and

tools

Effective measurement framework developed

Strategic visionLeadership

Case for impact measurement

DEVELOPING AND USING A THEORY OF CHANGE

9

WHAT IS A THEORY OF CHANGE?

Links activities è intermediate outcomes è final outcomes

A description of how activities lead to outcomes

- Clarifies what the activities aim to achieve and how

- Provides the case for why achieving intermediate outcomes is important

- Provides a structure for identifying what can be measured

10

HOW TO REPRESENT A THEORY OF CHANGE

Planning Triangle Logic Model Outcomes Chain

However you represent your theory of change, it should be supported by a written description.

11

Children’s emotional resourcefulness

improves

Counselling

Clients’ ability to support their children's healthy development improves

Clients’ emotional or psychological

difficulties decrease

Mother / Child interaction improves

Clients’ capacity for self care increases

EXAMPLE (SIMPLIFIED) THEORY OF CHANGEMOTHERS’ COUNSELLING SERVICE

Activities Intermediate Outcomes Final outcome

12

EXERCISE

• Take 2 minutes to think about 1-2 social outcomes that are central to what you do.

• Write them down on a piece of paper.

• As we go through the rest of the presentation, try to reflect on the measurement of that/those outcomes. How would you apply the theory and measure those?

Categories of outcomes include: ‘soft’ outcomes, such as attitudes, knowledge, skills, behaviour; and ‘hard’ outcomes, such as employment,

educational attainment, reduced reoffending etc.

PRIORITISING OUTCOMES

14

PRIORITISE OUTCOMES THAT:

• are directly influenced (rather than indirectly supported)

• are important / material to the mission

• are not too costly to measure

• will produce credible data

CHOOSING THE RIGHT LEVEL OF EVIDENCE

16

COUNTERFACTUAL

• Comparing the world with your organisation in it with what the world would be like without it.

• Control group

• Attribution

17

LEVELS OF EVIDENCE

Randomised control trial

Anecdotes / quotes

Before and after survey

Self-reported change

Case studies

Control groups

Credibility

Basic Advanced

SELECTING DATA SOURCES AND TOOLS

• Quantitative data (numbers)• Statistical estimates• Prevalence of views,

attitudes and experiences• Admin data or questionnaires

(paper, web, etc.)

• Qualitative data (words)• Detailed understanding • In-depth interviewing

(telephone or face-to-face)• Observation• Stakeholders’ views

Proportion of beneficiaries whose outcomes have improved, and by how

much.

What did beneficiaries think, did it make a

difference to them? How? How could it have been

better?

DIFFERENT TYPES OF EVIDENCE

20

EVALUATION TRAPS

Collect data that matters, and work together

Don’t force squares into circles & don’t

collect arbitrary data

21

INSPIRING IMPACT: MEASURING UP! AND THE IMPACT HUBMeasuring Up!

• online, step-by-step self-assessment tool

• looks at the way you plan, evidence, communicate and learn from the difference your work makes

The Inspiring Impact Hub

• pulls together resources relevant to improving impact practice

• enables users to search and filter results according to their needs

inspiringimpact.org

22

QUESTIONS AND COMMENTS