pro bono economics

2© Pro Bono Economics

PBE aims to match professional economists with charities;

providing pro bono help to measure performance and understand charity impact, and

fostering a culture of volunteering within the economics profession


History2009 - incorporated and registered with Charity Commission

11 projects – no management – none completed

2010 – 2 members of staff from SeptemberManaging 24 projects – 1 completed

2012 – + part-time economistManaging ± 40 projects – 41 completed to date


PeopleSupported by over 260 volunteers from the public sector (Government Economic Service, Bank of England and others), private consultancy firms (Charles Rivers Associates, Europe Economics, Frontier Economics, FTI Consulting, and Oxera), academics and individual economistsTrustees Martin Brookes (Chair), Andy Haldane (Treasurer), Kate Barker,Lynne Berry, Lucy Heady,Nicola Pollock, Dave RamsdenPatrons include Gus O’Donnell and Adair Turner


Projects160 charities have registered 168 projects

Where: 127 London & South East, 25 North & East, 5 in Scotland,1 Northern Ireland and 2 WalesWho for: 13 with homeless people,13 with ex-offenders, 9 with disable people, 4 with the elderly, 8 with other charitiesWhat: 12 employment support, 15 using arts or sport, 20 looking for analysis to support campaigning88 proposals have been withdrawn or rejected 79 projects have had 162 economists allocated to them41 reports completed to date16 projects in pipeline, 14 being scoped, 9 being worked on

© Pro Bono Economics

Vfm of interventions UK analysis

Data collection

advice

Other economic

advice

Some charities PBE has helped

http://www.stgilestrust.org.uk/

http://www.google.co.uk/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=3Qt08Vh9cUSBwM&tbnid=LqJbmRsruBvXCM:&ved=0CAUQjRw&url=http://dartington.org.uk/&ei=hCNAUvTCMuGw0QXyy4HQCw&bvm=bv.52434380,d.d2k&psig=AFQjCNFhv0UGQEAY7kTiThe4y-KAI333cA&ust=1380021505323720

http://www.google.co.uk/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=uE9BhaB0QRMZrM&tbnid=1HhsJwOLBHK-pM:&ved=0CAUQjRw&url=http://www.guess2give.com/national-childrens-bureau-on-guess2give/&ei=sCVAUrq5AsqI0AW8woGYCw&bvm=bv.52434380,d.d2k&psig=AFQjCNG0FoS-yk4DpfWLQ9WyUM4SZK3LiQ&ust=1380022056935241

http://www.google.co.uk/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=g_SkiQI9KOkMqM&tbnid=VTV5aAzuPuUfVM:&ved=0CAUQjRw&url=http://uk.virginmoneygiving.com/fundraiser-web/fundraiser/showFundraiserProfilePage.action?userUrl=SalandNick&isTeam=true&ei=7MHTUsOIIKj70gWhjoDoAw&bvm=bv.59026428,d.ZGU&psig=AFQjCNGtrGMg-1ouw71NiEyhFI_jYl6HXQ&ust=1389695851322220

http://www.google.co.uk/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=g_SkiQI9KOkMqM&tbnid=VTV5aAzuPuUfVM:&ved=0CAUQjRw&url=http://uk.virginmoneygiving.com/fundraiser-web/fundraiser/showFundraiserProfilePage.action?userUrl=SalandNick&isTeam=true&ei=7MHTUsOIIKj70gWhjoDoAw&bvm=bv.59026428,d.ZGU&psig=AFQjCNGtrGMg-1ouw71NiEyhFI_jYl6HXQ&ust=1389695851322220

http://www.google.co.uk/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=woSb4oXG4BE9sM&tbnid=kOPvWJaerpdGsM:&ved=0CAUQjRw&url=http://www.david-tennant-news-dev.com/charity/circle-scotland&ei=2RgCU6ywBeam0AXphICYBA&bvm=bv.61535280,d.d2k&psig=AFQjCNGQ_P4MHzOr6UlWeZBupU3_9aLVUQ&ust=1392732736384729

http://www.google.co.uk/url?sa=t&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&uact=8&ved=0CAQQjRw&url=http://www.nationalnumeracy.org.uk/&ei=2LQxU86VO8rLhAfK_IDICw&usg=AFQjCNF839_mGKVcu6jBGiPB0Dl15oxBWA&bvm=bv.63587204,d.ZG4

The counterfactual


Types of evaluation

Process: how and how well is the project working?

Impact: what change occurred as a result of the project?

Economic: were the benefits worth the costs? requires an impact evaluation


Measuring outcomesMeasure a characteristic before the interventione.g. number of people with a coldDeliver the interventione.g. cold and flu medicineMeasure the same characteristic after the interventionDifference = impact?

Most people get better from a cold with or without medicine

10


Measuring outcomesMeasure a characteristic before the intervention

e.g. school attendanceDeliver the interventione.g. mentoring scheme to improve attendanceMeasure the same characteristic after the interventionDifference = impact?What other things might be going on at the same time? New breakfast club New attendance officer

11


What would have happened anyway?i.e. without the interventionN.B. the intervention did take place, so we can never know for certainCan we measure changes in outcomes for a credible comparison group, who didn’t get the intervention?

credible = no reason to think that there may be differences (bias) which affect the outcomese.g. employment programme for women cf. national employment outcomes – unbiased?

12


Control GroupsNational average

PBE: Foundation Training CompanyBeforePBE: MEAM pilots – adults with multiple needsMatched records in admin datasetPeterborough Social Impact Bond/MoJ datalabNatural experiment in charity dataPBE: BarnardosRCTEducation Endowment Foundation/Social Research Unit

13


Estimating impact

Measure outcomesBest estimate of what would have happened anyway (the counterfactual)Outcomes minus counterfactual = impactA measurement minus an estimate = an estimate

14


Discuss

What are you using/might you use as a control group?

Data: methods & issues

16


Data 1: project admin

Can provide data on activities and outputs

(monitoring)

Rarely useful for outcomes (evaluation)

Tends to be less reliable over time at an individual

level

Often essential for identifiers to link to large data sets need to ensure this is gathered from the start

17


Data 2: surveyProduce your own survey Pros:

can tailor to your outcomesCons: time consuming (design, trialing, collection)usually need specialist inputbiases data collection from control groups

Sometimes possible to link to other large surveys

18


Data 3: large data setsEducational attainment

National Pupil Database (NPD) Re-offending PNC, prison, probation, MoJ Data labPros: good measures of specific outcomes can ‘match’ / control – ‘synthetic’ counterfactualCons: access (security)use….needs v high level of expertise bias (unrecorded variable bias)

19


Data 4: RCTs

Randomised control trials

Randomly allocating participants between ‘treatment / project’ and ‘control’ groups & compare outcomes.

Produces the best counterfactual with least bias → best estimation of impact

20


Issues 1: biasResponse bias

especially to surveysUnrecorded variable biasself selection attitude personality

RCTs are the best way to overcome bias

21


Issues 2: sample size

Only relevant once bias is eliminated or known

Needs to be large enough to be representative, but too large is wasteful

What is large enough?Ask a statistician

22


Issues 3: timing

Outcomes happen as a result of activity Therefore can only be measured after the activity not during it!

Can be difficult for funders who want reports at a project’s close

23


DiscussSample evaluations:

What level of evidence?Strengths and weaknesses?

24


Levels of Evidence

Valuation

26


DefinitionsSocial value

monetary worth of something to society orSocial value act“how what is proposed to be procured might improve the economic, social and environmental well-being of the relevant area”Social cost benefit analysisCBA including wider benefitsSocial return on investmentparticular methodology for social CBA emphasising role and input of stakeholders

27


Whose perspective?Value to society

won’t include transfer payments e.g. tax and benefitsValue to individualschange in income including benefits (+)Value to ExchequerIncrease in tax receipts (+) or increase in benefit payments (-)Potential costs avoidedCashable savings

28


Which benefits?Hard outcomes

Qualifications → employment → lifetime earningsCosts avoided by public servicesSoft outcomesEarly stages of work in this areaRelated to income, so only for those who are earningBut still important so

No numbers without stories, no stories without numbers

29


Which costs and values?

Input costsProgramme costsVolunteer inputOther activities

Benefit valuesAverage costsMarginal cost

30


DiscussWhat type of valuation might be most relevant to you?

pro bono economics

Economy & Finance