advice on data used to measure outcomes

Advice on Data Used to Measure Outcomes

Friday 20th March 2009

Today’s Workshop• Introduce myself & background• 2008/09 SOA – SBC experience• 2009/10 SOA – Context• How we developed the 2009/10 SOA

– Consultation • Linking activities to SOA – the 4 areas PED leads• How to measure success

– The outcome– What SBC needs to measure – a worked example

• Issues in what to measure– Primary

• Some conclusions • Discussion

Introduction• Planning & Economic Development

• Business Information Unit has led development of Outcome Agreements

• Own experience of developing evaluation frameworks contributing to PSA in England Local Authorities

• Process of Logical Frameworks, Logic Chains etc

Area of activity• Regeneration – town centre, retail, events &

tourism expertise• Property – economic development, infrastructure

and property management expertise• Economic Development – tourism, inward

investment, business support expertise• Rural Development – economic development,

key sectors, funding, regeneration expertise• Business Gateway – key sectors, business

support expertise

Our Experience

• Scottish Government’s National Outcomes• Development for 2008/09• Scottish Borders Council - Observations

– New way of working– Community Planning Partnership = 20 partners– c 48 Outcomes– c 15+ under ‘economic’– Mix of outcome, indicators, output, activity– Difficulties in performance management

• Different times

2009/10 SOA – Context • NO - We realise our full economic potential with more

and better employment opportunities for our people

• NO - We are better educated, more skilled and more successful, renowned for our research and innovation

• NO - Our young people are successful learners, confident individuals, effective contributors and responsible citizens

• NO - We live in well-designed, sustainable places where we are able to access the amenities and services we need

9 Outcomes that relate to the ‘economy’

• 2.1 – supporting & growing business activity in key sectors

• 2.2 – promoting self-employment & creating sustainable businesses

• 2.3 – maximising employment opportunities

• 10.3 – improving the socio-economic performance of our town centres & growing stronger communities

• 3.1 – maximising participation in education, training & employment amongst economically inactive

• 3.2 – existing workforce is highly skilled and responsive to the needs of employers

• 3.3 – the Scottish Borders Campus is a World Class Centre of Learning

• 4.2 – all young people will be in education, training or employment

• 4.3 – young people demonstrate achievement & attainment

How did we decide?• Examined where SBC PED is responsible

& accountable?– Probably 4 key areas– Looked at the nature of our contribution

• Where SBC contributes with other Stakeholders?– Through Competitive Borders

• Where SBC’s other departments are responsible & accountable– Skills, education, learning

Consultation

• Internal consultation – SBC– Logic Chain Approach

• External consultation– Partners eg SEn, VisitScotland, SDS,

Sectoral Groups– Education sector

• Business Planning

Objective of exercise

• To understand the activities, projects we do

• How these link to strategy, policy, objectives

• And how these need to link to Single and National Outcomes

• What is needed to deliver activities=> And this is what we have come up with

Linking activities to SOA

National OutcomeWe realise our full economic potential with more and better employment opportunities for our people

Local or Single Outcome (2.1)

Sustaining and growing business activity in key sectors of the Borders economy

Likely Activity

- Management of amended Business Gateway- Food & Drink Activity- Tourism activity, including Homecoming 2009- VisitScotland SLA- Procurement Support Project- Events support & planning- Creative Clusters Project

Inputs (£)

Inputs (Human) Business Gateway, Events




Promoting self-employment and creating sustainable businesses

Likely Activity- Management of Business Gateway contract - SBCC, Export Club support

Inputs (£)

Inputs (Human) Business Gateway




Maximising employment opportunities

Likely Activity

- Employment Land development - capital bids- Management of building/ premises, tenants- Social Enterprise - Inward Investment, OSB, Film Focus- Employability Coordination- External Funding

Inputs (£)Inputs (Human) Property, Events, Rural Development


National OutcomeWe live in well-designed, sustainable places where we are able to access the amenities and services we need


Improving the socio-economic performance of our town centres and growing stronger communities

Likely Activity

- Regeneration - capital bids- Regeneration - town centre plans & 3 year project databases- Shop Local - year-around - Employability Coordination- LEADER (Projects & Grants)- SSMEI II- Events - support & planning

Inputs (£)Inputs (Human) Regeneration, Rural & Economic Development

But how do we measure success?

• Outcome = result from outputs – why we should be spending ££ (cumulative = impact)

• Output = The services delivered as a result of activities – measurable

• Activities = The way(s) in which we deliver/ implement

• Inputs = The resources used to deliver or produce activities (& outputs) eg human and financial resources

How to Measure the OutcomeRef Description Indicators Verification

2.1Sustaining and growing business activity in key sectors of the Borders economy

- GVA/ employment in key sectors- SME survival rates in key sectors- Rural productivity

Annual Business Inquiry, BERR, Earnings per head of working age population resident in Borders (Annual Survey of Hours & Earnings)

2.2Promoting self-employment and creating sustainable businesses

- Self Employment rates vs. Scotland- Start Up rates vs. Scotland- Business Stock per WA population

Nomis, BERR

2.3 Maximising employment opportunities

- Employment rate- Claimant Count unemployment rates vs. Scotland- ILO unemployment rates vs. Scotland- ILO inactivity rates vs. Scotland

Nomis, Annual Population Survey

3.1Maximise participation in education, training and employment amongst economically inactive

- Employment rate - Working age population qualifications- Average earnings of employees- Proportion in employment, education or training

- Nomis APS- Nomis APS; Annual Business Inquiry- ASHE- Scottish Government

3.2Existing workforce is highly skilled and responsive to the needs of employers

- Working Age Population qualifications- Earnings- Standard Occupation Classifications- People falling out of work and on to incapacity benefits

- Nomis, APS; ABI- ASHE- Nomis, APS; Scottish Employers Skill Survey- Scottish Government/ DWP

3.3The Scottish Borders Campus Becomes a World Class Centre of Learning

- Learners achieving Level 2, 3, 4+- FE/ HE participation rates- RAE accreditation

- Nomis APS- Scottish Government- HWU/ Scottish Funding Council

4.2All young people will be in education, training or employment

- School leaver destination- FE/ HE participation rates- Vulnerable people in employment, education and training- School exam attainment- Learners achieving entry level qualifications in literacy/ numeracy

- Scottish Government- Scottish Government, Nomis APS- Scottish Government- Scottish Government Destination of School Leavers- Scottish Government - SQA Exam results in Scottish Schools

4.3Young people demonstrate achievement and attainment

- Achievement at English & Maths - Standard Grades- Achievement of Level 1/ 2 qualifications by age 17 or 19- School leaver destination- FE/ HE participation rates by 17 year olds

- Scottish Government SQA results in Scottish Schools- Scottish Government SQA results in Scottish Schools- Scottish Government School Leaver data- Scottish Government

10.3Improving the socio-economic performance of our town centres and growing stronger communities

- TTWA Employment Rates- Population trends- Quality Assessment (vitality & vibrancy)

Nomis, Annual Population Survey, GRO, SBC (Plans & Research)

And what is SBC measuring? • Outcome? Outputs? • Effectiveness in meeting Objectives?• Inputs - Project Managers must consider

the causal link between activities & ultimate performance

• So we use ‘if/ then’ methodology - at each level we can develop targets and indicators

=> For example, the data we need to collect

Example: Business Start Up SupportColumn 1

Objectives Indicators Verification Assumptions

OutcomePromoting self-employment & creating sustainable businesses

ObjectiveTo address barriers to enterpriseTo create sustainable business

Outputs

Marketing & PR of the Business Gateway200 Business Starts pa75% 3-year survival rate

ActivitiesHigh quality management & delivery of the BG service

Critical Condition





If barriers to business support are lowered, it will result in greater self employment levels & sustaining economic growth

Outputs


If these start ups occur, then the overall objective will have been met


Delivery of BG activity to spec will lead to outputs. There is a risk that the recession will have a greater influence

Critical Condition SE transfer of BG activities




Self Employment rates vs ScotlandStart Up rates vs ScotlandBusiness Stock per WA population


Business stock per 10,000 WA people vs ScotlandSurvival rates vs Scotland

If barriers to business support are lowered, it will result in more economic activity and economic growth

Outputs


# of start up enquiries (tied to promos); # of assists and % converted# of VAT Registrations# of new business bank accounts% of business survival - 12- 36-month

If these start ups occur, then the overall objective will have been met


Project Inputs: Economic Development Staff and Revenue Budget:Business Gateway - £k;Project Plans;Client satisfaction levels

Delivery of BG activity to spec will lead to outputs. There is a risk that the recession will have a greater influence

Critical Condition SE transfer of BG activities

Example: Business Start Up Support

Regardless of ‘who measures’, there may be issues

• What to monitor – hard & soft outcomes? – Baseline

• Tracking - a data capture strategy; – Primary – Business or Beneficiary – survey?– Secondary – spatial availability? Sampling issues? Delay

• Frequency – over what time period– Longitudinal

• Attribution– Reference Case? Control Groups

• Collection & Ownership– Who and at what cost? – Single point of capture? Shared assessment? Benefits?

Pros & Cons - PrimaryMethodology Benefit Drawback

In project/ programme – data collection

Vital in assessing baseline position against which to measure any change

Requires coordination & resource

Control Informed view of benefits that are attributable to an intervention vs deadweight ie additionality

Complicated, may be costly to recruit and challenge to ensure it is representative

Longitudinal Important to assess longer term effects of intervention; for labour market, allows tracking of changes

Challenge to maintain reliability as non-response rates increase over time; cost

Advice on data used will depend on• Why it needs to be collected; at which level of

the ‘Logic Tree’; and by who• The audience – is it

– Internal (eg Economic Development, budget holders, project managers)

– External (eg partners, funders, public, policy makers)• Their different levels of interest

– Achievements against objectives– Monitoring– Outcomes & Impacts– Value & Lessons– Strategic Added Value

Close

Richard Sweetnam

01835 825 069

[email protected]

advice on data used to measure outcomes

Documents

soalinking activities

withlinking activities

young people

sbc ped

key sectors2

key areaslooked

economic potential

training employment