Creating Value, Evaluating Impact

Kerry Traynor November 2012

What is Evaluation and why do it?About the ProjectMeasuring ImpactTheories and ModelsNext Steps

‘Self-evaluation is a vital part of the discipline imposed in undertaking creative work’

(Moriarty, 2002)

What is evaluation?Evaluation involves gathering evidence

before, during and after a project and using it to make judgements about what happened.

The evidence should prove what happened and why, and what effect it had.

The evidence can also help you to improve what you are doing during the project and what you do next time.

(Woolf, 2004 cited in Arts Council, 2011)

Why evaluate?‘Self-evaluation is hard work and time-consuming. The

reward is that it can give us the ability to do things beyond the best of our present available knowledge’

(Moriarty, 2002)

evaluation helps with planning, as it makes you think about what you’re aiming to do, how you will do it and how you will know if you’ve succeeded

on-going feedback keeps you on track and helps to avoid disasters evaluation helps you to adapt/change as you go along evaluation is a good way of dealing with ‘quality assurance’ – you’re

keeping an eye on things to make sure quality is maintained evaluation helps prove the value of what you are doing evaluation records your contribution to the field you are working in your evaluation can help others working in the same field information you collect can also be used for reporting back to those with

an interest in the project (eg participants, funders) and telling others about what you’ve done

the evidence you collect can support future funding applications (Arts Council, 2011)

Evaluation helps the arts… Rigorous evaluation of our work enables us to : accumulate a collective body of evidence contributes to the ‘collective practice wisdom of the sector’ builds a record of our ‘history and achievement’

(Arts Victoria, 2002)

Cannot evaluate all projects in the same way and in the same depth Can be more or less formal, and more or less detailed Artistic judgements about process, materials, form and content Judgements about the results of what you did and what you have

produced Process as well as product - quality and impact of both

(Arts Council, 2011)

Planning your evaluation: What kinds of information or evidence you are going to include in

your evaluation e.g. what people say, what they have done (process and finished

work), what you have done, how participants/audiences responded? What questions you are going to ask? How you plan to answer those questions –what sort of information

you need to answer the questions and how you will collect it. When you should collect the information. How you will collect the information? e.g. keeping a register, asking people in a questionnaire, asking them

to video their thoughts about a project, keeping a diary, taking photographs, etc.

How you are going to make sense of the information you have collected?

How you are going to present the results of the evaluation? Who you are going to share it with and how?

(Arts Council, 2011)

About the Project

Awards for All funding / Nov 12 – Aug 13

Aims Understand how impact is currently measured and evaluated in small

enterprises in the creative industries Understand the strengths/weaknesses/opportunities/risks of evaluating

impact Review models of impact evaluation and identify appropriate model(s)

for use in creative communities network Identify needs and deliver support to evaluate impact more effectively

Outcomes◦ Impact Evaluation Strategy for Safe Productions and the Creative

Communities Network – finding a shared approach that can measure the impact of the whole network

◦ One-to-one support for members in implementing impact evaluation◦ Interactive blog / web archive of useful info

http://evaluatingimpact.wordpress.com/

Background Charity Manager in Toxteth for 6 yrsDirector / Chair of Safe Productions for 10 yrsGrants Assessor for BBC CIN for 10 yrsResearcher & lecturer at LJMU for 8 yrs Manager of Community Media EnterpriseMSc in Governance (Creative Industries)PhD in Impact of Local Media

What do we mean by impact?Inputs + Outputs = Outcomes Outcomes – What would have happened anyway = ImpactDifference achieved by a project/serviceSocial, environmental, financial

ExerciseIntroduce yourself and your organisationWhat do you believe are the main

impact(s) of your organisation?What is your experience of impact

evaluation?What has worked well for you? What hasn’t worked so well? What would you like to do better?What do you hope this could achieve?What are your fears / concerns?

Models of Impact Evaluation Social Return on

Investment (SROI) European Foundation for

Quality Management (EFQM) Excellence Model

Prove It? Quality First Social Enterprise

Balanced Scorecard SIMPLE 3rd Sector Performance

Dashboard AA1000 Assurance

Standard

The Big PictureVolunteering Impact

Assessment Toolkit ISO9001:2008Global Reporting

Index (GRI) Investors in People

(IIP)Eco-MappingEU Eco-Management

and Audit Scheme (EMAS)

PQASSO

Why SROI?Robust methodology developed

through academic peer review process

Flexible, universally applicable(Relatively) straightforwardEndorsed by Cabinet OfficeBecoming common practise

SROI: A simplified example Inputs : £20,000 Outputs : 20 unemployed people complete a 6 month

training programme and get a qualification Outcomes : 5 people get a job But : 2 people would have got a job anyway Impact : 3 people move into employment Attach monetary values (returns) to that impact e.g.

◦ Reduced costs to state of paying unemployment benefits ◦ Increased income to the state from employment taxes

Calculate returns over 5 years Divide total returns by investment e.g.

◦ £100,000 / £20,000 = 5:1 Or, for every £1 invested, £5 is created in benefit for

society

A collective approach – find your common impacts

Moving people towards employment / rehousing / financial independence

Improving the physical environment Improving people’s health & wellbeing

Agree common methodology

Measure and evaluate over time

Convert into monetary values

Find your individual and collective Social Return On Investment Rates

H&W: A simplified exampleWEMWBS 7 or 14 point

questionnaire5 response codesFor participants aged 13+Each participant completes at

start/endPoints awarded for each response

H&W: A simplified exampleOver 1 year, 500 people

participate across all member organisations

Results in an average increase of 5 points per person, or

A total increase of 2,500 points across the area

Can be analysed by postcode/age/gender etc.

Visualising our collective impact

Visualising our collective impact

Visualising our collective impactThe Health and Wealth of Nations

The Decline: The Geography of a Recession

But does that really capture impact… ?

Can we really put a price on social impacts? Will SROI let us tell the whole story? Or will we need more? Eg. case study, images, quotes, narrative etc. Creativity and the Creative Industries – arts & health Community development – what is valuable and sustainable? How

do we build strong and resilient communities? Social Capital theory – our social, economic and cultural stock

(Bourdieu, 1984) The ties that bind us - bonding & bridging (Putnam, 2000) Decline in social capital = decline in moral and social standards Increasing social, democratic and civic engagement Pride, shared identity, shared vision - challenging the negative,

promoting the positive Alternative socio-economic models – 21 hours (nef) Can we use these theories and models to find a better way to

demonstrate our impact?