1

Inspiring Impact Scotland Champions Network Event: “Embedding evaluation for learning” About Inspiring Impact in Scotland Inspiring Impact is a UK-wide programme that aims to change the way the UK voluntary sector thinks about impact. By putting impact at the heart of the non-profit sector, we aim to ensure every pound spent makes the biggest possible difference to beneficiaries. Evaluation Support Scotland (ESS) leads Inspiring Impact in Scotland.

Inspiring Impact materials Inspiring Impact has developed materials in consultation with the UK non-profit sector to help the non-profit sector to:

• Make sense of what we mean by ‘impact measurement’

• Understand why impact measurement is important, and the benefits that it has for organisations and beneficiaries

• Follow best impact practice and agreed guidelines

• Review and improve their impact practice using a step-by-step self-assessment tool

• Access readily the best tools, systems, guidance, services and resources to help them improve their impact practice

Inspiring Impact materials can be used by anyone who wants to start or improve impact measurement practice as they serve as useful guides with helpful tips and examples of how to take things forward.

Inspiring Impact Scotland Champions Network ESS facilitates an Inspiring Impact Champions Network in Scotland to:

1. Share learning from Scotland (and in Scotland from the rest of the UK)

2. Provoke and encourage discussions around specific issues

3. Provide practical advice and examples of good impact practice

4. Provide opportunities for organisations (i.e. champions!) to get together to learn from and support peers

This report provides highlights from the event that ESS ran on 18 June 2015 in Edinburgh. It sets out information about presentations and discussions, as well as champions’ feedback on what the Inspiring Impact programme, ESS and/ or the Inspiring Impact Scotland Champions Network could do in the future to support organisations to measure their impact.

http://inspiringimpact.org/http://www.evaluationsupportscotland.org.uk/http://www.evaluationsupportscotland.org.uk/our-work-partners/inspiring-impact/http://www.evaluationsupportscotland.org.uk/our-work-partners/inspiring-impact/http://inspiringimpact.org/where-to-start-a-guide-to-inspiring-impacts-resources/

2

The Inspiring Impact Champions Network event Evaluation Support Scotland (ESS) ran an Inspiring Impact Champions Network event on 18 June 2015 in Edinburgh to promote Measuring Up! (Inspiring Impact’s online step-by-step self-assessment tool) and provide an opportunity and space for participating organisations to:

Have a better understanding of how the Inspiring Impact online self-assessment tool, Measuring Up!, can help with impact measurement

Learn about what helps/ gets in the way of embedding evaluation and impact practice in their organisation

Network with peers

The event was based on what participants of our previous Inspiring Impact events told us they wanted more of, including: having time to reflect on impact practice; sharing experiences/ views; hearing stories about impact measurement; meeting new people, and learning more about Inspiring Impact and plans in Scotland.

What was the format of the event?

The event was designed to maximize peer interaction and included a mixture of presentations and thematic discussions on: the latest from Inspiring Impact; case studies of organisations using Measuring Up! with support from ESS, and embedding evaluation for learning. The programme of the event can be found in Annex A.

The event was chaired by Steven Marwick, Director of Evaluation Support Scotland. The event was well attended, with 38 participants from third sector, public sector and academia and 5 ESS staff (see Annex B for details).

What has Inspiring Impact being up to? Patty Lozano-Casal, Evidence into Policy and Practice Manager at ESS, provided a short presentation about Inspiring Impact and its online self-assessment tool, Measuring Up!.

http://inspiringimpact.org/http://inspiringimpact.org/measuringup/

3

Tom Scott, Training Officer at ESS, shared ESS’s experience of using Measuring Up! Tom explained that, although ESS is an expert in the field of evaluation and impact practice, the tool identified a few areas in which the organisation could do to improve its practice.

Jenny Wood, Fundraiser at Impact Arts, spoke to a Pecka Kucha presentation of 20 photos that illustrated her journey of using Measuring Up! and other II materials and workshops. Jenny highlighted key challenges that organisations often go through, such as staff moving on/ joining the team, which can get in the way of impact practice. Jenny explained that for her organisation the three Inspiring Impact workshops that ESS delivered in 2014/15 came in at the right time for her and her organisation. In fact, Jenny took attending the workshops as an opportunity to embed the use of Measuring Up! in her organisation. For more information, see Jenny’s case study in Annex C.

Fraser Thompson, Performance and Research Officer at Children’s Hearings Scotland, provided an overview of some of the practical issues that organisations go through when it comes to measuring their impact. This included issues around collecting and analysing data at local level, especially when the organisation relies on volunteers to deliver its work, and minimising failure demand, in other words, 'demand caused by a failure to do something or do something right for the customer'1.

1 http://www.amazon.co.uk/dp/095197310X

http://www.amazon.co.uk/dp/095197310X

4

Assessing impact practice with Measuring Up! Participants were asked to use a grid with indicators extracted from the Measuring Up! tool to self-assess where their organisations/ projects are in terms of the ‘plan’, ‘do’, ‘assess’ and ‘review’ stages of the impact cycle. Participants used sticky dots to record their answers.

Our champions told us they fully/ partially met issues around:

• Clarity around information needed • Plans for impact measurement • Involving beneficiaries in data collection • Shared understanding of why and to

what purpose they are collecting information

• IT systems to analyse data quickly and easily

• Other factors that might have influenced the outcomes that beneficiaries experience

• Using findings to improve the way their deliver their work • Communicating findings about failures as well as successes.

The table below shows the number of sticky dots used by participants to record their answers for each indicator.

5

Part and parcel: Embedding evaluation

After a comfort break, Jane Marryat, Research and Communications Officer at ESS, presented findings from the research project “Embedding evaluation: ‘Are we nearly there yet? Just a bit further!’ The long term impact of ESS evaluation support”.

Jane explained that this research is the result of a survey and ten semi-structured interviews of key stakeholders.

Key findings about third sector organisations:

• All organisations felt they had embedded evaluation either completely or partially. To some extent this is related to the quality of ESS’ support, rather than the quantity of support given.

• Organisational changes, short term funding and capacity hinder organisational ability to evaluate their work.

• Organisations embed evaluation by having systems to collect feedback which is used to improve practice/services, reporting and for future planning. They have skilled and knowledgeable staff; some of which had responsibility for evaluation.

• Having a learning culture and ownership of evaluation across the whole of an organisation is helpful to embed evaluation practice. This includes having supportive senior management teams that work well together and who are committed to learning and using that learning to improve practice.

• Evaluation issues are sometimes technological and not always about lack of knowledge and skills.

• Funders although they support funded organisations to evaluate, have their own challenges evaluating their organisation as a whole or evaluating a grant making programme. They are aware of the need to revise and improve their own evaluation especially in the light of organisational or strategic change.

Key findings about Evaluation Support Scotland:

• ESS training, website, and having access to information and guidance helps organisations embed evaluation.

• ESS approach and staff are particularly valued.

6

• People attending ESS courses particularly, but not exclusively, in smaller organisations or isolated geographical locations lack support post training.

• Funded training and support enables some organisations to take advantage of ESS’s services.

• Organisations may not know the breadth of ESS’s work and how to engage with it.

For more information about this research download this poster and read this blog (both can be found at www.evaluationsupportscotland.org.uk).

Learning from Realising Ambition programme Jay Crangle, Communications and Influencing Manager, Realising Ambition, at Catch22, set out the context around Realising Ambition. Jay explained that Realising Ambition is a UK-wide £25m Big Lottery Fund programme replicating 25 services aimed at preventing children and young people from entering the criminal justice system. Launched in 2012, the five years programme is providing grant funding and specialist support to 22 organisations to refine and build the evidence base of their services.

“[Funders] Be explicit with organisations about the data you

need from them. If you get buy-in in relation to the data collection tool you

use that is very valuable.” Jay Crangle

Realising Ambition Programme Insights can be found at http://www.catch-22.org.uk/realising-ambition/programme-insights.

Heather Boyce, Head of Education Development at Anne Frank Trust, shared her learning from being involved in Realising Ambition. Heather talked about the challenges of measuring shifts in attitudes and how triangulation of evidence was helpful to overcome this issue. She also touched on difficulties around mainstreaming one database for funded projects to use – Heather explained that this did not work for them as well as it was expected.

“Having evaluation embedded in our work is absolutely

essential for the third sector to demonstrate the difference

we make.” Heather Boyce

Heather informed participant that Anne Frank Trust’s first external impact report is now published and can be accessed at http://www.annefrank.org.uk/uploads/images/Impact%20report%20final%20version.pdf

The last presentation of the day was provided by Colin MacFarlane, Association Manager at YMCA Scotland. After introducing participants to the

http://www.evaluationsupportscotland.org.uk/media/uploads/embedding_evaluation_poster1.pdfhttp://www.evaluationsupportscotland.org.uk/news/2015/Jun/11/blog-part-and-parcel-all-we-do/http://www.evaluationsupportscotland.org.uk/news/2015/Jun/11/blog-part-and-parcel-all-we-do/http://www.evaluationsupportscotland.org.uk/http://www.catch-22.org.uk/http://www.catch-22.org.uk/programmes-services/realising-ambition/http://www.catch-22.org.uk/realising-ambition/programme-insightshttp://www.catch-22.org.uk/realising-ambition/programme-insightshttp://www.annefrank.org.uk/uploads/images/Impact%20report%20final%20version.pdfhttp://www.annefrank.org.uk/uploads/images/Impact%20report%20final%20version.pdf

7

PlusOne pilot, Colin shared his experience of logic modelling and common approaches to gathering and record data. He finally touched on the challenges his organisation went through to set up a Randomised Control Trial (RCT) that never took off the ground.

“We have learned what to collect and how to move

forward. We also learned how to best use the skills of

partnership working.” Colin MacFarlane

All Realising Ambition presentation slides were e-mailed to participants alongside this report).

Thematic discussion: embedding evaluation into an organisation’s practice Participants were asked to take a few minutes to individually self-assess a series of elements of embedding evaluation into an organisation using a scale of 1=poor and 5=excellent.

Feedback from tables indicated that different organisations are at different stages of the embedding process. Some people raised ‘contribution analysis’ as a concern; in other words, not being able to demonstrate their contribution to the difference made. Others were still struggling with basic understanding of evaluation, such as setting outcomes and indicators, or using appropriate methods to collect information.

8

Participants were also asked to consider what Inspiring Impact and/ or ESS could do to help with some of the issues around embedding evaluation and impact practice for learning. Suggestions made include:

Guide/ tips for facilitating a whole team approach [to using MU!]

Promoting Inspiring Impact tool [MU!] more (e.g. through Third Sector Interfaces, TSIs)

More opportunities to share resources/ experiences

More resources online

Regular forum/ meeting for organisations to share their evaluation journey

Explore what is the organisation culture that allow you to create time to reflect and evaluate

Advice on how to create time to reflect and evaluate – a tool to allow this to happen (e.g. a game)

Tag functionality to organise evaluation and reflection based on reoccurring words (both in MU! and in other tools)

Tips to engage at board/ trustee level [make the case for MU!]

Information/ discussion on how we support people to find the right thing for them in a complex and challenging evaluation landscape

Your wishes, our commands – feedback from Champions At the end of the session participants were asked to complete a feedback form.

Please tell us what you thought about this event. Did you learn what you wanted to learn?

“Glad I came – interesting morning. Gave opportunity to think – though not

all the answers! More examples of evaluating embedded would have been

helpful.” Susan Buckingham, Asthma UK Centre for Applied Research

“Thank you! Stood in for a colleague and found all very interesting. Sharing

experiences was a highlight.” Heather Boyce, Anne Frank Trust

“Thought-provoking.” Emma Goodlad, Health and Social Care Alliance

“Good event. Expected more discussion on impact.” Darren Rocks, NHS Health Scotland

“Yes! Great case studies and roundtable discussion.” Lorraine Simpson, The Lines Between

“Great event and I ‘think’ I got more out of the event than I thought I would.” Colin MacFarlane, YMCA Scotland

9

“Very useful for networking, learning from other new models!” Isobel Meek, West Lothian Council

“Fab – informative, great change to network and great resources.” Catherine-Rose Stocks-Rankin, Institute for Research and Innovation in the Social Sciences

“Yes – I was interested in what the ‘measuring up’ tool is and how it should be

used.” Tamsyn Wilson, Scottish Government

“I very much enjoyed the event. I wasn’t sure what I wanted or needed to

learn but this has been helpful in starting to clarify this for me.” Sally McElroy, Archdiocese of Edinburgh and St Andrews

“Yes and more! Great to see so many organisations involved in impact

measurement and discuss different stages of the journey” Jenny Wood, Impact Arts

“Good basic intro, so yes. Great speakers too.” Ilse McKinnon, Scottish Council for Voluntary Organisations

“Brilliant to hear more about Measuring Up! and organisations that have used

it.” Jay Crangle, Catch 22

“I wasn’t sure what the day would be like – it was very stimulating. I would

have encouraged more folk to come if I was clearer about what it was.” Kate Tregaskis, Bethany Christian Trust

“Well put together. Very interesting.” Jennifer McDonald, Fife Rape and Sexual Assault Centre

What are you taking away?

Participants’ responses can be summarised as:

• Useful information and discussion (e.g. there are pockets of great practice; don’t have to reinvent the wheel)

• Shared learning

• Useful tools (MU!, logic modelling

• Good contacts/ links with other organisations

• Evaluation advice and tips (e.g. “how to build evaluation into induction; evaluation takes time and requires buy-in from senior management”; “needs to be self-led and home-grown”)

• Desire and enthusiasm to embed impact practice more thoroughly

What will you do next?

• Check out Inspiring Impact website

10

• Explore/ use resources (e.g.MU! tool, ESS wheel)

• Complete MU! assessment

• Share learning with team/ organisation

• Digest information and relate it to own organisation

• Review evaluation plan

• Learn about and/ or develop logic models

• Promote Inspiring Impact to others

• Etc.

What should Inspiring Impact and/ or ESS do next?

• Offer more events with good practice examples

• Discuss what success looks like in impact practice

• Produce guidance on whole system/ team approaches

• Make events more relevant to mixture of organisations/ size

• Get more information out to third sector about resources and tools available through Inspiring Impact

• A network or contact (physical or virtual) between organisations who are trying it out so we can communicate with each other

• Continue to speak to the third sector to analyse their needs

What ESS will do next Evaluation Support Scotland (ESS) will consider all the feedback and points made during the discussions and feedback to shape the development of the network in a way that is helpful and useful for stakeholders. We will continue to consult with participants whenever possible and to find further opportunities to provide networking and engagement in the Inspiring Impact programme.

The Inspiring Impact Programme Board has put a bid for funding to Big Lottery Fund. A decision is expected sometime in mid July and Inspiring Impact stakeholders will be kept informed through newsletters. Sign up here for Inspiring Impact newsletter.

ESS will continue to engage with the Inspiring Impact Champions Network through its newsletter. Sign up here for ESS newsletter.

For enquiries about Inspiring Impact Scotland and/ or the Inspiring Impact Champions Network, please contact:

Patty Lozano-Casal

Evidence into Policy and Practice Manager

Evaluation Support Scotland

E: patty@evaluationsupportscotland.org.uk

T: 0131 243 2777

http://ctt-news.org/InspiringImpact/UL9-H6X/Network.aspxhttp://ctt-news.org/InspiringImpact/UL9-H6X/Network.aspxhttp://www.evaluationsupportscotland.org.uk/news/newsletters-updates/mailto:patty@evaluationsupportscotland.org.uk

Annex A

11

Inspiring Impact Champions Network Event:

Embedding evaluation for learning 18 June 2015 – 10 am until 1 pm

Venue: COSLA, Verity House, 19 Haymarket Yards, Edinburgh,

EH12 5BH By the end of the session you should:

• Have a better understanding of how the Inspiring Impact online self-

assessment tool, Measuring Up!, can help with impact measurement

• Have learned about what helps/ gets in the way of embedding evaluation

and impact practice in your organisation

• Have had opportunities to network with peers

Programme

9:30 am Registration

10 am Welcome and introductions

10:15 am What has Inspiring Impact being up to? Patty Lozano-Casal, Evaluation Support Scotland

Presentation

10:30 am Assessing impact practice with Measuring Up! Presentation/ discussion

11:15 am Coffee Break

11:30 am Part and parcel: Embedding evaluation

Jane Marryat, Evaluation Support Scotland

Presentation

11:40 am Learning from Realising Ambition programme Presentation

12:05 am Thematic discussions Discussion

12:50 pm Your wishes, our commands Discussion

Next steps, closing remarks and evaluation

1 pm Close and networking lunch

https://www.google.co.uk/maps/place/Edinburgh+EH12+5BH,+UK/data=!4m2!3m1!1s0x4887c7aed772192b:0x8f15561bc58eb7a5?sa=X&ei=Y8NRVZyuAYnXUey0geAE&ved=0CCgQ8gEwAQhttp://inspiringimpact.org/

Annex B

12

Full list of attendees

Inspiring Impact Champions Network Event

18 June 2015

Name Job title Organisation

Alison Clancy Project Worker Parenting Across Scotland

Amrita Ahluwalia Fundraising Officer BE United

Andrew Murray Group Head of Community Support Bethany Christian Trust

Beth Cross Lecturer University of the West of Scotland

Bill Clements Community Development Officer CVS Inverclyde

Catherine-Rose Stocks-Rankin

Impact and Engagement Manager

Institute for Research and Innovation in the Social Sciences (IRISS)

Christine Riddoch House Manager Garvald West Linton Ltd

Colin MacFarlane Association Manager YMCA Scotland

Colleen Souness Researcher Individual

Darren Rocks Senior Health Improvement Officer NHS Health Scotland

Eilidh Watson Communications Executive Inspiring Scotland

Emma Goodlad Grants and Impact Officer The Health and Social Care Alliance

Fiona Sword Key Worker Scottish Spina Bifida Association

Fraser Thompson Performance and Research Officer Children's Hearings Scotland

Gary McGrow Social Researcher Scottish Health Council

Gillian Urquhart Manager Moira Anderson Foundation

Heather Boyce Head of Education Development Anne Frank Trust

Ilse MacKinnon Research Officer SCVO

Isobel Meek Learning & Quality Assurance Manager West Lothian Council

Jane Marryat Research and Communications Officer

Evaluation Support Scotland

Jay Crangle Communication and Influencing Manager, Catch22

Annex B

13

Realising Ambition

Jean Smith Office Manager Falkirk and District Womens Aid

Jennifer McDonald Sexual Violence Prevention Worker Fife Rape and Sexual Assault Centre (FRASAC)

Jenny Wood Impact and Innovation Coordinator Impact Arts

Julie Wild National Business & Partnership Advisor Care and Learning Alliance

Karen Carrick Evaluation Officer SCVO

Karen Wawrzyniak Key Worker Scottish Spina Bifida Association

Kate Tregaskis Trust and Grants Fundraiser Bethany Christian Trust

Laura Plumb Project Manager Edinburgh Voluntary Organisations Council (EVOC)

Leigh-Anne Fyfe Supervising Social Worker West Lothian Council

Linda Rodgers Chief Executive Officer Edinburgh Women's Aid

Lorna Mackay Performance Advisor Inspiring Scotland

Lorraine Simpson Research Director The Lines Between

Lydia Morrow Finance and Administration Manager

Evaluation Support Scotland

Noreen Kelly Centre Development Rape Crisis Scotland

Patty Lozano-Casal Evidence into Policy and Practice Manager Evaluation Support Scotland

Rachel Lawson Project Coordinator Scottish Spina Bifida Association

Richa Chaudry Evaluation Support Co-ordinator CrossReach

Sally McElroy Family Life Coordinator Archdiocese of Edinburgh and St Andrews

Sharon Bissett Organisational Development Advisor Voluntary Action North Lanarkshire

Steven Marwick Director Evaluation Support Scotland

Susan Buckingham Dissemination Officer Asthma UK Centre for Applied Research

Tamsyn Wilson Senior Social Researcher Scottish Government

Tanya Anderson Administrator CrossReach

Tom Scott Training Officer Evaluation Support Scotland

Annex C

14

Inspiring Impact Scotland case study: “Impact measurement in practice” workshops Jenny Wood, Impact Arts

About Impact Arts Impact Arts is a forward-thinking community arts organisation which uses the arts and creativity to enable and empower social change. We consider the arts as a positive tool for change –helping people to build confidence, attain new skills, get back into work or education, source work in creative industries, enjoy better health or simply improve quality of life. #artchangeslives

Drivers to measure our impact Our organisation has been engaging Scotland’s communities through the arts for 20 years- 20 years! We’ve made amazing progress since Susan Aktemel started Impact Arts in her wee Partick flat in 1994. We now have full charitable status, 32 full-time staff, offices in Glasgow, Edinburgh and Irvine and an annual turnover of around £1.5million per year.

As a fast-paced arts organisation relying on multiple funding streams to enable us to work with thousands of people each year, and employ creatives and artists as well as a core team, it’s not always an easy ride and can be down-right exhausting. Like many other 3rd sector organisations in Scotland, we have been experiencing further demands from funders on evidencing our impact, justifying our methodology and even our reason for existing.

Team this up with a 2 year service redesign programme with Big Lottery Fund’s ‘Better by Design’ and some recent major staffing changes: our Business Development Director, who was instrumental in raising the profile of Impact Arts, moved on after 10 years; our Chief Executive retired at the end of March 2015; our Head of Delivery moved on at the end of May; and we have a new Director; it’s been all go!

Running parallel to this, my role in the Business Development Team has slowly been evolving over 3 years, from Assistant to Coordinator to Fundraiser. I had been noticing gaps and trends in how we research and develop projects, evaluate our work, and use our data for impact. I had been chipping away at this, but I knew I needed to bring clarity and focus to my findings in order to present them, along with a plan, to the Senior Management Team: the Measuring Up training came along at exactly the right time.

Annex C

15

Taking part in the workshops I attended the first of the three Measuring Impact in Practice sessions run by Evaluation Support Scotland (ESS) in November 2014. I went along with an open mind. I wasn’t too sure what to expect as I had heard rumblings about the tool via newsletters and emails, but couldn’t quite see how it would work for us.

The three training and support sessions proved invaluable. I learned a great deal from Patty and Tom [ESS trainers], their delivery approach was informative, well planned and full of great resources. It was also flexible which allowed the group to learn from each other. A few of the organisations had had similar experiences to Impact Arts, which was actually quite comforting!

My first piece of homework was to complete the MU tool. I had attempted to do this in the EES session, but felt that to really cover all bases; I had to share it with a variety of staff members. So, I downloaded the questions and circulated them to key members in our SMT, Delivery, Development and Finance teams to complete individually.

There was a little bit of hesitation at first, especially as the SMT had been going through the ‘Better by Design’ process, which in a way seemed similar. While gathering everyone’s feedback, I noticed at times that there were very varied opinions. For some areas, this was to be expected depending on role responsibilities. However, there were a couple of areas which we needed to address and so, I circulated the collated feedback and scheduled a 2-part meeting for the team to sit round the table to discuss.

At this point, our new Director was preparing to revisit our Strategic/Business Plan and Fundraising Strategy so the timing was pertinent. The meetings proved vital in filling in gaps and blanks amongst old and new staff. We found that we were actually doing well in certain areas, but just needed the conversation to happen to provide clarity across the team. It also flagged up areas which we hadn’t considered or needed to focus more on.

Why this experience was worth it The MU tool provided a clear platform to enable these discussions to happen. It has been vital in shaping our work going forward and unifying the team. Through systematically discussing each point, considering each other’s knowledge and experience and working through solutions together, we now have clear action points and timeframes.

Annex C

16

For example, our issue around evaluation and data collection has been addressed. Prior to MU, we were in the process of implementing a new database (which is consequently provided by Synovations, who create bespoke systems using Salesforce for 3rd sector organisations). Both EES and Child Brain Injury Trust have Synovations databases, so it was very useful to hear about their learning experiences and how to identify and tackle any issues at an early stage. To accompany the new database, I have now created an Evaluation Plan for the organisation, which features resources from the sessions, and have mapped out a training plan to embed this with the wider team. This will help us become more effective in measuring the impact of our work, and ultimately continue to develop and tailor our programmes for our beneficiaries.

In addition, our SMT use the MU Action Plan at their monthly meetings as a prompt to mark progress, and will review assessment criterion at 6/12 month stages.

I would highly recommend using MU to any organisation, especially those going through change, regardless of service, structure or size. I also think it is important to revisit it on an ongoing basis. As Hugh Prather said, “Just when I think I have learned the way to live, life changes.”

Contact details: Jenny Wood, Fundraiser, Impact Arts (jenny@impactarts.co.uk)

impactarts.co.uk - Join us on: facebook • twitter • Youtube • linkedin

mailto:jenny@impactarts.co.ukhttp://www.impactarts.co.uk/https://www.facebook.com/impactartsofficialhttps://twitter.com/impact_artshttp://www.youtube.com/user/ImpactArtsOfficialhttp://uk.linkedin.com/company/impact-arts