X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

v NPC’S APPROACH TO MEASUREMENT Iona Joy, Head of charity team, NPC

CSV conference March 2015

#impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

NPC: TRANSFORMING THE CHARITY SECTOR

2

NPC works at the nexus between charities and

funders

Charity

Sector Funder

Increasing the impact of charities

eg, impact-focused theories of change

Strengthening the partnership

Eg, collaboration towards shared

goals

Increasing the impact of funders

eg, effective commissioning

Consultancy

Think tank

#impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

OUR STRATEGY PROCESS

3

Strategy development sits within a wider process

Development Implementation Preparation

Leadership

Stakeholders

Culture/ values

Plan

Deliver

Assess

Review Purpose

Purpose

Strategy

External Resources

Leadership, governance and culture

focused on continuous learning

Development Implementation Preparation

#impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

IMPACT CYCLE

4

Plan for impact

Deliver with impact

Assess impact

Review for future impact

• Short term service design:

rapid response to data

• Long term strategy design:

adaptation of strategy

#impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

WHY USE IMPACT DATA IN YOUR STRATEGY? INCENTIVES

5

Internal (improving services)

External (pressure

from funders)

• Internal pressure to

improve service delivery

25% use measurement to

improve services* NPC hopes

more will take up cause!

*Pritchard, D., Ni Ogain, E., Lumley, T. 2013. Making an Impact. New Philanthropy Capital.

• External pressure from

funders

Half of charities* driven by

funders to measure

#impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

THE FOUR PILLARS APPROACH TO

MEASURING IMPACT

6

Map your

theory of

change

Prioritise

what you

measure

Choose

your level

of evidence

Select your

sources and

tools

Effective measurement

framework developed

Strategic vision

Leadership

Case for impact measurement

#impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

DEVELOPING AND USING

A THEORY OF CHANGE

#impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

WHAT IS A THEORY OF CHANGE?

Links activities intermediate outcomes final outcomes

A description of how activities lead to outcomes

8

- Clarifies what the activities aim to achieve and how

- Provides the case for why achieving intermediate outcomes is important

- Provides a structure for identifying what can be measured

#impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

HOW TO REPRESENT A THEORY OF CHANGE

9

Planning Triangle Logic Model Outcomes Chain

However you represent your theory of change, it should be supported by a written description.

#impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

THEORY OF CHANGE FOR A PARENTING

CHARITY: OUTCOMES CHAIN EXAMPLE

Improved life

chances for

children from

disadvantaged

backgrounds

Practical 1-

1 support

to parents Increased

parental

commitment to

school Improved

attainment of

children from

disadvantaged

backgrounds

Improvement in

children’s

abilities and

temperament

Improved

attendance and

behaviour in

school

Address specific

problems

stopping children

from attending

school

10

Parents better

support

children’s

learning

Improved

parenting skills

Parenting

classes

School-based

activities to help

children from

disadvantaged

backgrounds

Activity Enabling

factor

Intermediate

outcome

Long-term

goal

#impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

PRIORITISING OUTCOMES

#impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

PRIORITISE OUTCOMES THAT:

• Are directly influenced (rather than indirectly supported)

• are important / material to the mission

• are not too costly to measure

• will produce credible data

12 #impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

CHOOSING THE RIGHT

LEVEL OF EVIDENCE

#impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

THREE AUDIENCES FOR EVALUATION

14

1) Funders/commissioners:

- Does the intervention make a difference?

- What’s your track record?

- What are the costs/benefits?

3) The evidence-base:

- What have you learned?

- How can we all achieve more?

2) Your own organisation

- Do we really make a difference?

- How can we improve?

#impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

COUNTERFACTUAL

• Comparing the world with the

service in it with what the world

would be like without it.

• Control group

• Attribution

15 #impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

LEVELS OF EVIDENCE

16

Randomised

control trial

Anecdotes /

quotes

Before and

after survey

Self-reported

change

Case

studies

Control

groups

Credibility

Basic Advanced

#impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

ALTERNATIVES

• Waiting lists

• Dosage: users who’ve received a smaller ‘dose’ of a

service

• Type of service: comparing users who’ve received one

type of service compared to another

• Use existing data sources to create control groups (quasi-

experimental approaches)

• Ask users whether it’s made a difference

17

Comparison groups

No comparison group

#impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

SELECTING DATA

SOURCES AND TOOLS

#impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

QUESTIONS TO ANSWER (ITERATIVELY)

19

• Who?

• When?

• How?

Which

beneficiaries?

Stakeholders?

Staff?

Everyone or

sample?

Before and

after?

How

frequently?

Quantitative

and/or

qualitative?

How credible

does the data

need to be?

#impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

• Quantitative data (numbers)

• Statistical estimates

• Prevalence of views, attitudes and experiences

• Admin data or questionnaires (paper, web, etc.)

• Qualitative data (words)

• Detailed understanding

• In-depth interviewing (telephone or face-to-face)

• Observation

• Stakeholders’ views

Proportion of

beneficiaries whose

outcomes have

improved, and by how

much.

What did beneficiaries

think, did it make a

difference to them?

How? How could it have

been better?

DIFFERENT TYPES OF EVIDENCE

#impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

A LOT OF IMPACT EVIDENCE ALREADY EXISTS

21

Improved life

chances for

children from

disadvantaged

backgrounds

Practical 1-1

support to

parents

Increased

parental

commitment to

school Improved

attainment of

children from

disadvantaged

backgrounds

Improvement in

children’s

abilities/

temperament

Improved

attendance and

behaviour in

school

Address specific

problems

stopping children

from attending

school

Parents better

support

children’s

learning

Improved

parenting skills

Parenting

classes

School-based

activities to help

children from

disadvantaged

backgrounds

✔

✔

✔ ✔ ✔

✔ ?

?

?

?

https://www.gov.uk/government/publications/national-evaluation-of-the-troubled-families-programme

http://www.local.gov.uk/c/document_library/get_file?uuid=1f145177-2552-47dc-978f-e46f6c65f4cd&groupId=10180

#impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

EVALUATION TRAPS

22

Collect data that matters, and work together

Don’t force

squares into

circles & don’t

collect arbitrary

data

#impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

INSPIRING IMPACT: MEASURING UP! AND

THE IMPACT HUB

Measuring Up!

• online, step-by-step self-assessment tool

• enables you to review and improve your

organisation’s impact practice

• looks at the way you plan, evidence, communicate

and learn from the difference your work makes

The Inspiring Impact Hub

• pulls together resources relevant to improving

impact practice

• enables users to search and filter results according

to their needs

23 #impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

RESOURCES

24

http://www.thinknpc.org/publications/theory-of-change/

http://www.ces-vol.org.uk/Publications-Research/publications-

free-downloads/making-connections

NPC – Brief Introduction to Theory of Change

http://inspiringimpact.org/

http://www.thinknpc.org/publications/npcs-four-pillar-approach/

http://www.thinknpc.org/publications/creating-your-

theory-of-change/

#impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

NPC’S STRATEGY TRIANGLE

25

Focus on what achieves impact

Core purpose

Resources and capabilities

Strategy

External environment

Mission, vision, values

Theory of change

Impact data into ToC

Needs mapping

PESTLE/STEEPL

5 forces

Policy

Organisational/

governance review

eg Little blue book

Options analysis

#impactmatters

X AXIS

LOWER LIMIT

UPPER LIMIT

CHART TOP

Y AXIS Y AXIS LIMIT

v

THANK YOU Iona.Joy@thinknpc.org

London, March 2015