The f Word

Naomi JonesAngela PaineAmanda ProsserAmanda ProsserNatalie LowSt h M KStephen McKay

What difference do we make? Evaluating v and v projects Evaluating v and v projects

OverviewOverview

• Who are we? • Why bother with evaluation?

Wh t diff d • What difference do you make?

• How do we all go about proving it? proving it?

Who we areWho we are

• A consortium of four organisationsU d t ki i d d t • Undertaking an independent evaluation for v

• Working with v over the next two • Working with v over the next two years

• Using a variety of evaluation Using a variety of evaluation methods

• Constantly feeding back findings y g g• Shaping the future development of

v

Turn to the person on your right and introduce yourself and say and introduce yourself and say two words to sum up your view

f l ti of evaluation

Why bother with evaluation?

For accountability and learning

Why bother with evaluation?

• For accountability and learning• Evidence funders - what difference their

money makesy • Evidence for trustee - strategic decision

making • Evidence for volunteers - what

difference their time makes• Attracting new volunteers – what they • Attracting new volunteers what they

will get out of it • Organisational learning - what works;

h t b i d what can be improved

Less about Less about…

More about More about…

What does it all mean? • Outputs: The countable direct products of a

What does it all mean? • Outputs: The countable, direct products of a

project or organisation’s activities (e.g. number of training courses delivered)

• Outcomes: Changes, benefits, learning or other g , , g effects that happen as a result of your project (e.g. increased skills and confidence among service users)

• Impact: Effect of a project at a higher or broader • Impact: Effect of a project at a higher or broader level in the longer term after a range of outcomes has been achieved (e.g. reduce poverty)

• Monitoring: Collecting information on a project or Monitoring: Collecting information on a project or an activity to help report on a project. Needs to be planned and managed.

• Evaluation: Using monitoring and other i f ti t k j d t b t ff ti

g ginformation to make judgments about effectiveness of a project and to make changes and improvements

(adapted from Charities Evaluation Service, 2007, Your Project and its Outcomes, CES: London)

Overall aim

What you ultimately hope to achieve.

Impact

Specific aimsOutcomes

Should be brief and focused.

The changes you hope to achieve as a result of your workUse words like ‘improve’, ‘increase’, ‘reduce’, ‘develop’

O t tActivities

The activities you undertake and the services you offer to bring these changes about

Outputs

Inputs Inputs Inputs Inputs

Adapted from Charities Evaluation Planning TriangleAdapted from Charities Evaluation Planning Triangle

What difference does your What difference does your project make?

• Your project will make a difference to lots of different people:– Volunteers

– Service users

Your organisation as a whole– Your organisation as a whole

– The community in which you work

What difference does What difference does your project make?

• It will make a difference in a number of ways:

y p j

• Things [aka ‘physical capital’]:– The things produced by volunteering 

• Number of training courses delivered or peopleNumber of training courses delivered or people mentored

– Need to consider not just quantity but also quality

[ ]• People [aka ‘human capital’]– Personal development 

• Changes in levels of confidence, self‐esteem, wellbeingwellbeing

– Gaining new skills or enhancing existing ones• Soft skills – e.g. team work, communication, inter‐personal

• Hard skills – e.g. IT skills, construction, horticulture

• Money [aka ‘economic capital’]: Money [aka economic capital ]: – The monetary value of volunteering and its outcomes on all stakeholders, e.g. value of training received changes in employabilitytraining received, changes in employability, value of work done by volunteers

[ ]• Relationships [aka ‘social capital’]:– Relationships, networks, bonds of trust between people and reciprocitybetween people and reciprocity 

• Attitudes [aka ‘cultural capital’]: [ p ]– Shared sense of cultural and religious identity– Awareness and understanding of other peoples 

culturescultures

How do we go about How do we go about proving it?

• You can collect evidence to evaluate your project in a number of ways:

– Monitoring information

– Surveys of volunteers or staff members

– Focus groups of volunteers or service users

– Interviews with service users and key stakeholders

– Photograph projects before, during and after

– Video diaries by volunteers or service users

How do we go about How do we go about proving it? proving it?

• You don’t have to do it all yourself: – The evaluation website will include guidance 

– You can get others involved: volunteers or students from a local university might be able to helpa local university might be able to help

– You could make use of the questions within IVR’s 

Volunteering Impact Assessment Toolkit

Case study: Children’s Hospice Assoc.

• Volunteers trained to undertake interviews and surveys y

• Volunteers reported positive experience:– 84% had built friendships and networks – But, 18% felt their skills weren’t being utilisedBut, 18% felt their skills weren t being utilised

• Staff valued the role of volunteers:– 83% felt volunteers helped create open and diverse culture– But 18% felt they were over reliant on volunteersBut, 18% felt they were over reliant on volunteers

• Families reported a significant impact of volunteers:– 73% said volunteers led to new friendships and networks– But some concerns about vetting and role appropriatenessBut, some concerns about vetting and role appropriateness

• Recommendations were built into a new strategy:– Reviewing how volunteer skills are utilised to full effect– Raising awareness of the role and volunteer status of the board– Raising awareness of the role and volunteer status of the board– Exploring the possibility of extending volunteer roles – Developing a leaflet for families about volunteers

On your table work to complete the planning complete the planning and evaluation triangle g

How will we go about How will we go about evaluating v? g

• Analysing national data from the Citizenship SurveySurvey

• Analysing monitoring data collected by vSpeaking to key national stakeholders• Speaking to key national stakeholders

• Undertaking a survey of young peopleD i t l h i t i ith t • Doing telephone interviews with grant recipients

• Undertaking a series of case studies• Undertaking a series of case studies

How can you get How can you get involved?

• Answer our phone calls and surveys!• Become a case study• Become a case study• Participate in pilot initiatives• Provide good quality monitoring information• Provide good quality monitoring information• Visit our website

– Launching in JulyLaunching in July– Housed on vinspired.com/evaluation– Purpose is to:

• Share any insights we’re picking up• Share any insights we re picking up• Enable you to share experiences and best practice• Enable you to feed into evaluation

d d lf l• Provide guidance on self evaluation