Outline

1. Programmes and standards of evidence2. Where do targeted interventions fit?3. A key challenge for targeted provision4. A brave new world?

Evidence-based programmes (EBPs)

• A programme is a discrete, organised package of practices, spelled out in guidance (sometimes called a manual) that explains what should be delivered to whom, when, where and how.

• A programme is ‘evidence-based’ when it is ‘tested and effective’:

- Tested’ means that the programme has been put through its paces by a high-quality impact evaluation

- ‘Effective’ means that there is strong evidence from that evaluation that the programme makes life better for children or families

Standards of evidence

1. Intervention specificity – what is it?2. Impact – does it work?3. Evaluation quality – can we be confident in the results?4. System readiness – can it be replicated?

www.investinginchildren.eu/standards-evidence

Continuum of prevention

1. Promotion – universal (promoting good)2. Universal prevention – universal (preventing bad)3. Selective prevention – elevated risk (group) 4. Indicated prevention – elevated risk (individual) / early problems 5. Treatment – established disorder6. Maintenance – preventing relapse

O’Connell et al. (2009)

Spend on children in Northern Ireland

Executive Departments Levels 1-2 Levels 3-4 Levels 5-6 Not disaggregated

£m £m £m £m

Agriculture & Rural Development 1.34

Culture, Arts & Leisure 17.39 0.65 0.05

Education 1,193.99 212.41 0.97 225.24

Employment & Learning 48.95 54.41 0.03

Enterprise, Trade & Investment

Finance & Personnel

Health, Social Services, Safety 29.35 17.00 424.08 13.57

Environment 3.06 0.05

Justice 0.80 14.90 0.03

Regional Development 7.67

Social Development 7.71 0.77

First & deputy First Minister 0.25 1.57

Total 1,301.75 293.28 440.75 240.46

Percentage split 57% 13% 19% 11%

4 circles

Children in need

Children in contact with specialist services

Children with informal support

Provide more services

4 circles

Re-focus services

4 circles

Reduce need

4 circles

KiVa – bullying prevention

• “Bullying”: verbal/psychological/physical behaviour, designed to cause harm/distress, direct/indirect, repeated over time, power differential

• Universal element:- Classroom lessons (10 over year)- Posters, playground monitors- Parent engagement• Targeted element:- Structured process to deal with bullying incidents

Hard-to-reach families, or hard-to-access services?

What we learnt (and maybe should have known before…)

1. If we don’t engage the right parents, the programme won’t work2. Engage providers first if you want them to engage parents3. Have a clear recruitment process and train everyone involved 4. Invest in and incentivise recruitment and retention5. Get out there! Go to parents; don’t expect them to come to you6. Build relationships: visit, call, then visit and call again7. Be practical: make it attractive and easy to come along8. Be creative!9. Recruitment is nothing without retention10. Aim high but be realistic: life gets in the way of the best intentions

What’s up with evidence-based programmes?

1. Limited impact: no “breakthrough outcomes”, poor transportability2. “Pseudo-science”: cherry-picking, developer bias3. RCTs are not the gold standard4. Fiddling trumps fidelity5. Poor fit with systems (therefore not “scalable”)6. Automatons replacing autonomy7. Limit innovation8. “Privatised” solutions to “private” problems9. Technical not relational10. ‘Real’ children and ‘My children’

Poor tansportability?

Programme Name Effects in the US Number of studies in the US Effects in Europe Number of studies in Europe

Big Brothers Big Sister Positive effect 4 No effect 1

Functional Family Therapy Positive effect 8 Positive effect 3

Good Behaviour Game Positive effect 5 Positive Effect 4

Incredible Years Parent Training Programme Positive effect 26 Positive effect 18

Incredible Years Child Training Programme Positive effect 5 No effect 3

Multisystemic Therapy Positive effect 12 Mixed effects 5

Nurse Family Partnership Positive effect 3 Mixed effects 4

Promoting Alternative Thinking Strategies (PATHS) Positive effect 8 Mixed effects 6

Multidimensional Treatment Foster Care (Oregon) Positive effect 8 Mixed effects 4

Strengthening Families 10-14 Positive effect 2 No effect 3

Programme Effect Size in the US Effect Size in Europe Countries Included

Functional Family Therapy -0.09 to -0.59 -0.96 Sweden

Good Behaviour Game -0.37 -0.35 The Netherlands

Incredible Years Parent Training Programme -0.02 to -1.18 -0.10 to -0.72 UK, Norway

Multisystemic Therapy -0.13 to -1.74 --0.40 UK

Multidimensional Treatment Foster Care (Oregon) -0.49 to -1.6 -0.92UK (In Sweden, the effect on internalizing (ES = -0.39) and

externalizing (ES = -0.58) behaviour was measured.)

*Effect sizes based on effects on primary outcomes.Some effect sizes have been calculated by the Washington State Institute of Public Policy

Developer bias?

ProgrammeFindings in developer-led studies

Findings in independent evaluations

Reconnecting Youth (Drug prevention programme)

Increased GPA; increased self-esteem; increased school bonding; decreased hard drug use; and decreased drug control problems (Eggert et al. 1994)

Negative effects on most outcome measures, no positive effects. Negative effects the stronger the better implementation fidelity (Sanchez et al. 2007)

Triple P Positive Parenting Programme

Positive mean effect on child problem behavior of d=0.35 in 33 trials (Nowak and Heinrichs 2008)

No positive effects on any aspect of problem behavior evaluated by teachers, parents, or child self-reports (Eisner et al. 2007)

Olweus Bullying Prevention Programme

Reductions of up to 50% in bullying in the original study (Olweus 1994)

No overall effects on either attitudinal measures or victimization (Bauer et al. 2007)

ALERT (Drug prevention programme)

Reduction in cigarette, marijuana and alcohol use by 19–39% (Ellickson et al. 2003)

No effects on mediators or substance abuse itself (St Pierre et al. 2006)

Source: Eisner, M. (2009). No effects in independent prevention trials: can we reject the cynical view?. Journal of Experimental Criminology, 5(2), 163-183.

Towards EBP 2.0?

1. Do the same but better (e.g. progressive standards, implementation) 2. Re-invent EBPs (e.g. build on neuroscience, build in flex)3. Develop and apply evidence-based “kernels” and “bundles”4. Make “services as usual” more evidence-based (e.g. Lipsey, PDSA, rct)5. Engage multiple systems (e.g. HPS, “collective impact”)6. Explore common logic models / meta-theory7. Segment, and then target better8. Improve prevention training for practitioners9. Empower civil society (e.g. community engagement, kernels)10. Reform systems (e.g. de-commission, earmarking)

Evidence-based kernels

A “fundamental unit of behavioural influence” that underlie effective prevention and treatment and a small and simple entity that holds the potential to transform into something much larger.

They must: • be inexpensive;• show an immediate effect;• be easily useable; and• be flexible to simultaneously solving additional

problems that might arise in the course of prevention or treatment

• be empirically found to be effective• be indivisible (i.e. it would not work if altered).

Selected sources

Axford, N., Lehtonen, M., Tobin, K., Kaoukji, D. & Berry, V. (2012) ‘Engaging parents in parenting programs: lessons from research and practice’, Children and Youth Services Review, 34 (10), 2061-2071.

Axford, N. and Morpeth, L. (2013) ‘Evidence-based programs in children’s services: a critical appraisal’, Children and Youth Services Review 35(1), 268-277.

Davies, F. A., McDonald, L. & Axford, N. (2012) Technique is Not Enough: Making Evidence-based Programmes Socially Inclusive. Discussion Paper for the British Psychological Society Professional Practice Board’s Social Inclusion Group. Leicester, BPS.

Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327-350.

Eisner, M. (2009). No effects in independent prevention trials: can we reject the cynical view? Journal of Experimental Criminology, 5(2), 163-183.

Embry, D. D., & Biglan, A. (2008). Evidence-based kernels: fundamental units of behavioral influence. Clinical Child and Family PsychologyReview, 11(3), 75-113.

Gottfredson, D. C., Cook, T. D., Gardner, F. E. M., Gorman-Smith, D., Howe, G. W., Sndler, I. W. & Zafft, K. M. (2015) Standards of evidence for efficacy, effectiveness, and scale-up research in prevention science: next generation. Prevention Science 16 (7) 893-926.

Selected sources

Hanleybrown, F., Kania, J. and Kramer, M. (2012) Channelling Change: Making Collective Impact Work. Stanford Social Innovation Review. http://ssir.org/articles/entry/channeling_change_making_collective_impact_work [Accessed 8th February 2016]

Kemp, F., Ohlson, C., Raja, A., Morpeth, L. & Axford, N. (2015) Fund-mapping: The Investment of Public Resources in the Wellbeing of Children and Young People in Northern Ireland. Belfast: NICCY

Langford, R., Bonell, C.P., Jones, H.E., Pouliou, T., Murphy, S.M., Waters, E., Komro, K.A., Gibbs, L.F., Magnus, D., and Campbell, R. (2014), The WHO Health Promoting School framework for improving the health and well-being of students and their academic achievement, Cochrane Database of Systematic Reviews 2014, Issue 4.

Lipsey, M., Howell, J. C., Kelly, M. R., Chapman, G., & Carver, D. (2010). Improving the effective- ness of juvenile justice programs: A new perspective on evidence-based practice. Center for Juvenile Justice Reform: Georgetown Public Policy Institute, Georgetown University.

O’Connell, M. E., Boat, T. and Warner, K. E. (Eds.) (2009) Preventing Mental, Emotional, and Behavioural Disorders Among Young People: Progress and Possibilities. Washington DC: The National Academies Press.

Santucci, L. C., Thomassin, K., Petrovic, L. & Weisz, J. R. (2015) Building evidence-based interventions for the youth, providers, and contexts of real-world mental-health care. Child Development Perspectives, 9 (2), 67-73.

Shonkoff, J. P. and Fisher, P. A. (2013) Rethinking evidence-based practice and two-generation programs to create the future of early childhood policy. Development and Psychopathology, 25, 1635-1653.

Contact details

nick.axford@dartington.org.ukwww.dartington.org.uk@nick_axford