‘What Works’The role of evidence based policy and

research in Britain’s welfare to work policies

Professor Dan FinnProfessor Dan Finn

University of PortsmouthUniversity of Portsmouth

“.. Social science should be at the heart of policy making. We need a revolution in the relationship between government and the social research community – we need social scientists to help determine what works and why, and what type of policy initiatives are likely to be most effective”(David Blunkett, Secretary of State for Education and Employment, 2000)

Context

Relationship between applied research and British social Relationship between applied research and British social policy develops from late 1890spolicy develops from late 1890s

Took considerable time to establish that research was of Took considerable time to establish that research was of use in both formulating social policies and administering use in both formulating social policies and administering themthem

Research into, and evaluation, of government policies now Research into, and evaluation, of government policies now systematic & routine – interest in impacts, efficiency and systematic & routine – interest in impacts, efficiency and value for money (post New Public Management)value for money (post New Public Management)

Increase in organisations producing ‘evidence’ for Increase in organisations producing ‘evidence’ for Government (growth in size and diversity of research Government (growth in size and diversity of research community)community)

New Labour and Evidence Based Policy Making (EBPM)

New Labour regard EBPM as central to the New Labour regard EBPM as central to the modernisation of government modernisation of government

Multiple influences on policy but in context Multiple influences on policy but in context of increase in public expenditure and major of increase in public expenditure and major social and economic development social and economic development programmes, ambition that evidence should programmes, ambition that evidence should form a key part of the reform processform a key part of the reform process

Review of evidence use by Government

Little of the research commissioned by Government Little of the research commissioned by Government Departments or other academic research appears to be Departments or other academic research appears to be used by policy makersused by policy makers

Prime Minister’s Cabinet Office develops strategy:Prime Minister’s Cabinet Office develops strategy: Policy making must be soundly based on Policy making must be soundly based on

evidence of what worksevidence of what works Improve the accessibility of the evidence Improve the accessibility of the evidence

available to policy makersavailable to policy makers Improve departments’ capacity to make best use Improve departments’ capacity to make best use

of evidence (training officials in how to of evidence (training officials in how to interpret, use and apply evidence)interpret, use and apply evidence)

EBP – Government Strategy

Cabinet Office – ‘Policy Hub’Cabinet Office – ‘Policy Hub’ Evidence Based Policy Fund – strengthening links between Evidence Based Policy Fund – strengthening links between

universities, research institutes and government (e.g., EPPI – universities, research institutes and government (e.g., EPPI – ‘‘Evidence for Policy and Practice Information and Coordination Evidence for Policy and Practice Information and Coordination Centre’Centre’))

Economic and Social Research Council programme (see ‘Evidence Economic and Social Research Council programme (see ‘Evidence Network’)Network’)

‘‘Campbell Collaboration’ – new collaboration for systematic Campbell Collaboration’ – new collaboration for systematic research synthesis in the social science and policy fields, modelled research synthesis in the social science and policy fields, modelled on the ‘Cochrane Collaboration’.on the ‘Cochrane Collaboration’.

Government Social Research Office oversees use of social research Government Social Research Office oversees use of social research – purpose “to provide government with objective, reliable, relevant – purpose “to provide government with objective, reliable, relevant and timely research to inform policy making and delivery” and timely research to inform policy making and delivery”

Evidence of ‘what works’

Rules of evidence differ greatly between research Rules of evidence differ greatly between research cultures (healthcare, education, criminal justice, cultures (healthcare, education, criminal justice, social care, welfare, housing, transport and urban social care, welfare, housing, transport and urban renewal)renewal)

Balance between quantitative and qualitative Balance between quantitative and qualitative research in assessing what works and what use to research in assessing what works and what use to make of secondary researchmake of secondary research

Systematic evidence reviews replacing traditional Systematic evidence reviews replacing traditional literature reviewsliterature reviews

Primacy given to ‘hard’ quantitative data, esp. Primacy given to ‘hard’ quantitative data, esp. randomised control trials, meta analysis.randomised control trials, meta analysis.

UK policy now more informed by evidence (‘Better Policy Making’, Cab. Office, 2001)

In UK, evidence based government/policy making well In UK, evidence based government/policy making well establishedestablished

Reviews of existing & proposed policies (international policy Reviews of existing & proposed policies (international policy learning)learning)

Commissioning new research/evaluating policy design and Commissioning new research/evaluating policy design and impacts by commissioning/undertaking research. Typically:impacts by commissioning/undertaking research. Typically: Expert evidence and systematic evidence reviews (internet)Expert evidence and systematic evidence reviews (internet) Pilots and case studiesPilots and case studies Rapid/Interim evidence assessmentsRapid/Interim evidence assessments Some Random Control Trials but more typically Some Random Control Trials but more typically

statistical/econometric ‘net impact’ studiesstatistical/econometric ‘net impact’ studies Emphasis also on learning from practice – gathering evidence Emphasis also on learning from practice – gathering evidence

from delivery agencies and frontline workersfrom delivery agencies and frontline workers

Evidence Based Policy Making and Welfare to Work Reforms

Department for Work and Pensions (about 80 social researchers Department for Work and Pensions (about 80 social researchers work with other analysts - economists, statisticians):work with other analysts - economists, statisticians):

internal research capacity contracted out research programmes (about £18m, 2007)

DWP commissions research using a variety of methods – large DWP commissions research using a variety of methods – large scale surveys, in-depth interviews, experiments, ethnographic scale surveys, in-depth interviews, experiments, ethnographic methods and analysis of administrative datamethods and analysis of administrative data

Not directly engaged with social theory - the approach tends to Not directly engaged with social theory - the approach tends to be inductive rather deductive – work on building up ideas, be inductive rather deductive – work on building up ideas, theories, concepts from the data rather than using data to verify theories, concepts from the data rather than using data to verify theories.theories.

Development of Welfare to Work policies has been informed by Development of Welfare to Work policies has been informed by continuous process of research and evaluation – major continuous process of research and evaluation – major influence influence of US welfare experiments and evaluation cultureof US welfare experiments and evaluation culture

Pathways to Work

New way of engaging with people on disability benefits:New way of engaging with people on disability benefits: Compulsory Work Focused Interviews with Case ManagersCompulsory Work Focused Interviews with Case Managers ‘‘Choices’ - innovative package of employment and ‘condition Choices’ - innovative package of employment and ‘condition

management’ measuresmanagement’ measures MethodologyMethodology

Review of Existing EvidenceReview of Existing Evidence IntegratedIntegrated Evaluation with Policy Development and Evaluation with Policy Development and

ImplementationImplementation Qualitative Interviews (clients, officials, case managers and Qualitative Interviews (clients, officials, case managers and

employers)employers) Administrative Data and Surveys – measuring impacts of pilot Administrative Data and Surveys – measuring impacts of pilot

areas with matched statistical control groupsareas with matched statistical control groups Local Context Analysis (Qualitative and Quantitative)Local Context Analysis (Qualitative and Quantitative) Economic Appraisal (Cost Benefit Analysis)Economic Appraisal (Cost Benefit Analysis)

UK Employment, Retention & Advancement Demonstration Project

What is the most effective and efficient way of:What is the most effective and efficient way of: Retaining people in workRetaining people in work Advancing low paid people in the labour marketAdvancing low paid people in the labour market

MethodologyMethodology Review of Existing EvidenceReview of Existing Evidence IntegratedIntegrated Evaluation with Policy Development and Evaluation with Policy Development and

ImplementationImplementation Qualitative Interviews (clients, officials, case managers and Qualitative Interviews (clients, officials, case managers and

employers)employers) Experimental Method (Randomised Control Trial)Experimental Method (Randomised Control Trial) Administrative Data and SurveysAdministrative Data and Surveys Local Context Analysis (Qualitative and Quantitative)Local Context Analysis (Qualitative and Quantitative) Economic Appraisal (Cost Benefit Analysis)Economic Appraisal (Cost Benefit Analysis)

Realities

Limitations of ‘evidence’ produced from a Limitations of ‘evidence’ produced from a Government managed research process (Government managed research process (not all not all research is of sufficient quality)research is of sufficient quality)

Time scale of evaluations (esp. randomised trials) Time scale of evaluations (esp. randomised trials) – often overtaken by events– often overtaken by events

Absence of impact evidence is not necessarily Absence of impact evidence is not necessarily evidence of absence of effectevidence of absence of effect

Evidence requires analysis and interpretation!Evidence requires analysis and interpretation!

Conclusion

Reality of policy making – contingent and Reality of policy making – contingent and unpredictable, evidence just one factor that policy unpredictable, evidence just one factor that policy makers take into account.makers take into account.

Other (maybe more important) influences in Other (maybe more important) influences in policy making – rpolicy making – rresources; lobbyists and pressure resources; lobbyists and pressure groups; pragmatism; expediency and groups; pragmatism; expediency and contingencies. contingencies.

Should ask also why politicians are showing an Should ask also why politicians are showing an increased interest in evaluation and EBP - danger increased interest in evaluation and EBP - danger that legitimising functions are being used in a that legitimising functions are being used in a purely symbolic (political) fashion.purely symbolic (political) fashion.

Conclusion

Evidence Based Policy Making is a contested Evidence Based Policy Making is a contested concept – closely tied to the politics of a strong concept – closely tied to the politics of a strong central state. central state.

Best assessment is there is more ‘evidence Best assessment is there is more ‘evidence informed policy making’informed policy making’

UK political realities – ‘speaking truth to power’, UK political realities – ‘speaking truth to power’, academic freedom, and importance of critical academic freedom, and importance of critical social research social research

Role of independent foundations – Joseph Role of independent foundations – Joseph Rowntree Trust, Leverhulme Trust, Cadbory Rowntree Trust, Leverhulme Trust, Cadbory Trusts, etc.Trusts, etc.

Some useful sources Britain Britain

Policy Hub and ‘Magenta Book: Guidance Notes on Policy Evaluation’ at Policy Hub and ‘Magenta Book: Guidance Notes on Policy Evaluation’ at http://http://policyhub.cmps.gov.ukpolicyhub.cmps.gov.uk))

Evidence Based Policy Network (www.evidencenetwork.org) Evidence Based Policy Network (www.evidencenetwork.org) Evidence for Policy and Practice Information and Coordination Centre Evidence for Policy and Practice Information and Coordination Centre

(http//eppi.ioe.ac.uk) (http//eppi.ioe.ac.uk) Social Care Institute for Excellence (Social Care Institute for Excellence (www.scie.org.ukwww.scie.org.uk)) Evidence & Policy: A journal of research, debate and practice (Evidence & Policy: A journal of research, debate and practice (

https://www.policypress.org.uk/journals/evidence_policyhttps://www.policypress.org.uk/journals/evidence_policy)) DWP Research (www.dwp.gov.uk/asd/asd5/branch.asp)DWP Research (www.dwp.gov.uk/asd/asd5/branch.asp) Office for National Statistics (www.ons.gov.uk)Office for National Statistics (www.ons.gov.uk)

International:International: Cochrane Collaboration (medical: Cochrane Collaboration (medical: www.cochrane.orgwww.cochrane.org) ) Campbell Collaboration (social science and public policy: Campbell Collaboration (social science and public policy:

www.campbellcollaboration.orgwww.campbellcollaboration.org) )