ESSENTIALS

Evaluating the impact of healthcare interventions usingroutine data

OPEN ACCESS

Geraldine M Clarke senior data analyst 1 Stefano Conti senior statistician 2 Arne T Wolters senioranalytics manager 1 Adam Steventon director of data analytics 1

1The Health Foundation London UK 2NHS England and NHS Improvement London UK

What you need to knowAssessing the impact of healthcare interventions is critical to inform futuredecisionsCompare observed outcomes with what you would have expected if theintervention had not been implementedA wide range of routinely collected data is available for the evaluation ofhealthcare interventions

Interventions to transform the delivery of health and social careare being implemented widely such as those linked toAccountable Care Organizations in the United States1 or tointegrated care systems in the UK2 Assessing the impact ofthese health interventions enables healthcare teams to learn andto improve services and can inform future policy3 Howeversome healthcare interventions are implemented without highquality evaluation in ways that require onerous data collectionor may not be evaluated at all4

A range of routinely collected administrative and clinicallygenerated healthcare data could be used to evaluate the impactof interventions to improve care However there is a lack ofguidance as to where relevant routine data can be found oraccessed and how they can be linked to other data A diversearray of methodological literature can also make it hard tounderstand which methods to apply to analyse the data Thisarticle provides an introduction to help clinicianscommissioners and other healthcare professionals wishing tocommission interpret or perform an impact evaluation of ahealth intervention We highlight what to consider and discusskey concepts relating to design analysis implementation andinterpretation

What are interventions impacts andimpact evaluationsA health intervention is a combination of activities or strategiesdesigned to assess improve maintain promote or modify healthamong individuals or an entire population Interventions caninclude educational or care programmes policy changesenvironmental improvements or health promotion campaignsInterventions that include multiple independent or interactingcomponents are referred to as complex5 The impact of anyintervention is likely to be shaped as much by the context (egcommunities work places homes schools or hospitals) inwhich it is delivered as the details of the intervention itself6-9

An impact is a positive or negative direct or indirect intendedor unintended change produced by an intervention An impactevaluation is a systematic and empirical investigation of theeffects of an intervention it assesses to what extent the outcomesexperienced by affected individuals were caused by theintervention in question and what can be attributed to otherfactors such as other interventions socioeconomic trends andpolitical or environmental conditions Evaluations can becategorised as formative or summative (table 1)Approaches such as the Plan Do Study Act cycle11 which ispart of the Model for Improvement a commonly used tool totest and understand small changes in quality improvement work12

may be used to undertake formative evaluationWith either type of evaluation it is important to be realisticabout how long it will take to see the intended effectsAssessment that takes place too soon risks incorrectlyconcluding that there was no impact This might leadstakeholders to question the value of the intervention whenlater assessment might have shown a different picture Forexample in a small case study of cost savings from proactivelymanaging high risk patients the costs of healthcare for the

Correspondence to G Clarke Geraldineclarkehealthorguk

Open Access Reuse allowed Subscribe httpwwwbmjcomsubscribe

BMJ 2019365l2239 doi 101136bmjl2239 (Published 20 June 2019) Page 1 of 7

Practice

PRACTICE

on 5 June 2020 by guest Protected by copyright

httpww

wbm

jcom

BM

J first published as 101136bmjl2239 on 20 June 2019 D

ownloaded from

eligible intervention population initially increased comparedwith the comparison population but after six months wereconsistently lower14

This article focuses on impact evaluation but this can only everaddress a fraction of questions15 Much more can beaccomplished if it is supplemented with other qualitative andquantitative methods including process evaluation Thisprovides context assesses how the intervention wasimplemented identifies any emerging unintended pathwaysand is important for understanding what happened in practiceand for identifying areas for improvement16 The economicevaluation of healthcare interventions is also important forhealthcare decision making especially with ongoing financialpressures on health services17

What are the right evaluation questionsAn effective impact evaluation begins with the formulation ofone or more clear questions driven by the purpose of theevaluation and what you and your stakeholders want to learnFor example ldquoWhat is the impact of case management onpatientsrsquo experience of carerdquoFormulate your evaluation questions using your understandingof the idea behind your intervention the implementationchallenges and your knowledge of what data are available tomeasure outcomes Review your theory of change or logicmodel21 22 to understand what inputs and activities were plannedand what outcomes were expected and when Once you haveunderstood the intended causal pathway consider the practicalaspects of implementation which include the barriers to changeunexpected changes by recipients or providers and otherinfluences not previously accounted for Patient and publicinvolvement (PPI) in setting the right question is stronglyrecommended for additional insights and meaningful resultsFor example if evaluating the impact of case management youcould engage patients to understand what outcomes matter mostto them Healthcare leaders may emphasise metrics such asemergency admissions but other aspects such as the experienceof care might matter more to patients5 23

What methods can be used to perform animpact evaluationRandomised control designs where individuals are randomlyselected to receive either an intervention or a control treatmentare often referred to as the ldquogold standardrdquo of causal impactevaluation24 In large enough samples the process ofrandomisation ensures a balance in observed and unobservedcharacteristics between treatment and control groups Howeverwhile often suitable for assessing for example the safety andefficacy of medicines these designs may be impracticalunethical or irrelevant when assessing the impact of complexchanges to health service deliveryObservational studies are an alternative approach to estimatecausal effects They use the natural or unplanned variation ina population in relation to the exposure to an intervention orthe factors that affect its outcomes to remove the consequencesof a non-randomised selection process25 The idea is to mimica randomised control design by ensuring treated and controlgroups are equivalentmdashat least in terms of observedcharacteristics This can be achieved using a variety of welldocumented methods including regression control andmatching26 eg propensity scoring27 or genetic matching28 Ifthe matching is successful at producing such groups and thereare also no differences in unobserved characteristics then it can

be assumed that the control group outcomes are representativeof those that the treated group would have experienced if nothinghad changed ie the counterfactual For example an evaluationof alternative elective surgical interventions for primary totalhip replacement on osteoarthritis patients in England and Walesused genetic matching to compare patients across three differentprosthesis groups and reported that the most prevalent type ofhip replacement was the least cost effective29

Assessing similarity is only possible in relation to observedcharacteristics and matching can result in biased estimates ifthe groups differ in relation to unobserved variables that arepredictive of the outcome (confounders) It is rarely possible toeliminate this possibility of bias when conducting observationalstudies meaning that the interpretation of the findings mustalways be sensitive to the possibility that the differences inoutcomes were caused by a factor other than the interventionMethods that can help when selection is on unobservedcharacteristics include difference-in-difference30 regressiondiscontinuity31 instrumental variables18 or syntheticcontrols32Table 2 gives a summary of selected observationalstudy designsObservational studies are often referred to as natural (for naturalor unplanned interventions) or quasi (for planned or intentionalinterventions) experiments Natural experiments are discussedto evaluate population health interventions41

Whatrsquos wrong with a simplebefore-and-after studyBefore-and-after studies compare changes in outcomes for thesame group of patients at a single time point before and afterreceiving an intervention without reference to a control groupThese differ from interrupted time series studies which comparechanges in outcomes for successive groups of patients beforeand after receiving an intervention (the interruption)Before-and-after studies are useful when it is not possible toinclude an unexposed control group or for hypothesisgeneration However they are inherently susceptible to biassince changes observed may simply reflect regression to themean (any changes in outcomes that might occur naturally inthe absence of the intervention) or influences or secular trendsunrelated to the intervention eg changes in the economic orpolitical environment or a heightened public awareness ofissuesFor example a before-and-after study of the impact of a carecoordination service for older people tracked the hospitalutilisation of the same patients before and after they wereaccepted into the service They found that the service resultedin savings in hospital bed days and attendances at the emergencydepartment42 Reduced hospital utilisation could have reflectedregression to the mean here rather than the effects of theintervention for example a patient could have had a specifichealth crisis before being invited to join the service and thenreverted back to their previous state of health and hospitalutilisation for reasons unconnected with the care coordinationserviceVarious tools are available to evaluate the risk of bias innon-randomised designs due to confounding and other potentialbiases43 44

Where can I find suitable routine dataHealthcare systems generate vast amounts of data as part oftheir routine operation These datasets are often designed to

Open Access Reuse allowed Subscribe httpwwwbmjcomsubscribe

BMJ 2019365l2239 doi 101136bmjl2239 (Published 20 June 2019) Page 2 of 7

PRACTICE

on 5 June 2020 by guest Protected by copyright

httpww

wbm

jcom

BM

J first published as 101136bmjl2239 on 20 June 2019 D

ownloaded from

support direct care and for administrative purposes rather thanfor research and use of routinely collected data for evaluatingchanges in health service delivery is not without pitfalls Forexample any variation observed between geographical regionsproviders and sometimes individual clinicians may reflect realand important variations in the actual healthcare qualityprovided but can also result from differences in measurement45

However routine data can be a rich source of information on alarge group of patients with different conditions across differentgeographical regions Often data have been collected for manyyears enabling construction of individual patient historiesdescribing healthcare utilisation diagnoses comorbiditiesprescription of medication and other treatmentsSome of these data are collected centrally across a wider systemand routinely shared for research and evaluation purposes egsecondary care data in England (Hospital Episode Statistics)or Medicare Claims data in the United States Other sourcessuch as primary care data are often collected at a more locallevel but can be accessed through or on behalf of healthcarecommissioners provided the right information governancearrangements are in place Pseudonymised records where anyidentifying information is removed or replaced by an artificialidentifier are often used to support evaluation while maintainingpatient confidentiality See table 3 for commonly used routinedatasets available in EnglandHealthcare records can often be linked across different sourcesas a single patient identifier is commonly used across ahealthcare system eg the use of an NHS number in the UKUsing a common pseudonym across different data sources cansupport linkage of pseudonymised records Linking into publiclyavailable sources of administrative data and surveys can furtherenrich healthcare records Commonly used administrative dataavailable for UK populations include measures of GP practicequality and outcomes from the Quality and OutcomesFramework (QOF)52 deprivation rurality and demographicsfrom the 2011 Census53 and patient experience from the GPPatient Survey54

Are there any additional considerationsIt is essential to consider threats to validity when designing andevaluating an impact evaluation validity relates to whether anevaluation is measuring what it is claiming to measure SeeRothman et al55 for further discussionInternal validity refers to whether the effects observed are dueto the intervention and not some other confounding factorSelection bias which results from the way in which subjectsare recruited or from differing rates of participation due forexample to age gender cultural or socioeconomic factors isoften a problem in non-randomised designs Care must be takento account for such biases when interpreting the results of animpact evaluation Sensitivity analyses should be performed toprovide reassurance regarding the plausibility of causalinferencesExternal validity refers to the extent to which the results of astudy can be generalised to other settings Understanding thesocietal economic health system and environmental contextin which an intervention is delivered and which makes itsimpact unique is critical when interpreting the results ofevaluations and considering whether they apply to your setting56

Descriptions of context should be as rich as possibleOften the impact of an intervention is likely to vary dependingon the characteristics of patients These can be usefully exploredin subgroup analyses57

Clear and transparent reporting using established guidelines(eg STROBE58 or TREND59)to describe the intervention studypopulation assignment of treatment and control groups andmethods used to estimate impact should be followed Limitationsarising as a result of inherent biases or validity should beclearly acknowledgedAround the world many interventions designed to improvehealth and healthcare are under way An evaluation is anessential part of understanding what impact these changes arehaving for whom and in what circumstances and help informfuture decisions about improvement and further roll out Thereis no standard lsquolsquoone size fits allrsquorsquo recipe for a good evaluationit must be tailored to the project at hand Understanding theoverarching principles and standards is the first step towards agood evaluation

Further ResourcesSee The Health Foundation Evaluation what to consider 201560 for a list ofwebsites articles webinars and other guidance on various aspects of impactevaluation which may help locate further information for the planninginterpretation and development of a successful impact evaluation5 23 55

Education into practicebull What interventions have you designed or experienced aimed at

transforming your service Have they been evaluatedbull What types of routine data are collected about the care you deliver

Do you know how to access them and use them to evaluate caredelivery

bull What resources are available to you to support impact evaluations forinterventions

Contributors GMC SC ATW and AS designed the structure of the report GMCwrote the first draft of the manuscript SC wrote table 2 ATW wrote table 3 ASand GMC critically revised the manuscript for important intellectual content Allauthors approved the final version of the manuscript

Competing interests We have read and understood BMJ policy on declarationof interests All authors work in the Improvements Analytics Unit a joint projectbetween NHS England and the Health Foundation which provided support forwork reported in references of this report13 37 60

Provenance and peer review This article is part of a series commissioned by TheBMJ based on ideas generated by a joint editorial group with members from theHealth Foundation and The BMJ including a patientcarer The BMJ retained fulleditorial control over external peer review editing and publication Open accessfees and The BMJrsquos quality improvement editor post are funded by the HealthFoundation

Patient andor members of the public were not involved in the creation of this article

1 Davis K Guterman S Collins S Stremikis G Rustgi S Nuzum R Starting on the path toa high performance health system Analysis of the payment and system reform provisionsin the Patient Protection and Affordable Care Act of The Commonwealth Fund 2010httpswwwcommonwealthfundorgpublicationsfund-reports2010sepstarting-path-high-performance-health-system-analysis-payment

2 NHS NHS Long Term Plan 2019 httpswwwenglandnhsuklong-term-plan3 Djulbegovic B A framework to bridge the gaps between evidence-based medicine health

outcomes and improvement and implementation science J Oncol Pract 201410200-2101200JOP2013001364 24839282

4 Bickerdike L Booth A Wilson PM etal Social prescribing less rhetoric and more realityA systematic review of the evidence BMJ Open 20177e013384

5 Campbell M Fitzpatrick R Haines A etal Framework for design and evaluation ofcomplex interventions to improve health BMJ 2000321694-6101136bmj3217262694 10987780

6 Rickles D Causality in complex interventions Med Health Care Philos 20091277-90101007s11019-008-9140-4 18465202

7 Hawe P Lessons from complex interventions to improve health Annu Rev Public Health201536307-23 101146annurev-publhealth-031912-114421 25581153

8 Greenhalgh T Papoutsi C Studying complexity in health services research desperatelyseeking an overdue paradigm shift BMC Med 20181695101186s12916-018-1089-4 29921272

9 Pawson R Tilley N Realistic evaluation Sage 1997

Open Access Reuse allowed Subscribe httpwwwbmjcomsubscribe

BMJ 2019365l2239 doi 101136bmjl2239 (Published 20 June 2019) Page 3 of 7

PRACTICE

on 5 June 2020 by guest Protected by copyright

httpww

wbm

jcom

BM

J first published as 101136bmjl2239 on 20 June 2019 D

ownloaded from

10 Smith J Wistow G (Nuffield Trust comment) Learning from an intrepid pioneer integratedcare in North West London httpswwwnuffieldtrustorguknews-itemlearning-from-an-intrepid-pioneer-integrated-care-in-north-west-london

11 Improvement NHS Plan Do Study Act (PDSA) cycles and the model for improvementHandb Qual Serv Improv Tools 2010

12 Academy of Medical Royal Colleges Quality Improvementmdashtraining for better outcomes2016 httpswwwaomrcorgukwp-contentuploads201606Quality_improvement_key_findings_140316-2pdf

13 Lloyd T Wolters A Steventon A The impact of providing enhanced support for care homeresidents in Rushcliffe 2017 httpwwwhealthorguksiteshealthfilesIAURushcliffepdf

14 Ferris TG Weil E Meyer GS Neagle M Heffernan JL Torchiana DF Cost savings frommanaging high-risk patients In Yong PL Saunders RS Olsen LA editors The healthcareimperative lowering costs and improving outcomes workshop series summaryNat AcadPress (US) 2010301 httpswwwncbinlmnihgovbooksNBK53910

15 Greenhalgh T Papoutsi C Studying complexity in health services research desperatelyseeking an overdue paradigm shift BMC Med 2018164-9

16 Moore GF Audrey S Barker M etal Process evaluation of complex interventions MedicalResearch Council guidance BMJ 2015350h1258 101136bmjh1258 25791983

17 Drummond M Weatherly H Ferguson B Economic evaluation of health interventionsBMJ 2008337a1204 101136bmja1204 18824485

18 Baiocchi M Cheng J Small DS Instrumental variable methods for causal inference StatMed 2014332297-340 101002sim6128 24599889

19 Lorch SA Baiocchi M Ahlberg CE Small DS The differential impact of delivery hospitalon the outcomes of premature infants Pediatrics 2012130270-8101542peds2011-2820 22778301

20 Martens EP Pestman WR de Boer A Belitser SV Klungel OH Instrumental variablesapplication and limitations Epidemiology 200617260-710109701ede000021516088317cb 16617274

21 Center for Theory of Change httpwwwtheoryofchangeorg22 Davidoff F Dixon-Woods M Leviton L Michie S Demystifying theory and its use in

improvement BMJ Qual Saf 201524228-38 101136bmjqs-2014-003627 2561627923 Gertler PJ Martinez S Premand P Rawlings LB Vermeersch CMJ Impact evaluation

in practice The World Bank Publications 2017 httpssiteresourcesworldbankorgEXTHDOFFICEResources5485726-1295455628620Impact_Evaluation_in_Practicepdf

24 Cochrane A Effectiveness and efficiency random reflections on health services London1972 httpswwwnuffieldtrustorgukresearcheffectiveness-and-efficiency-random-reflections-on-health-services

25 Portela MC Pronovost PJ Woodcock T Carter P Dixon-Woods M How to studyimprovement interventions a brief overview of possible study types Postgrad Med J201591343-54 101136postgradmedj-2014-003620rep 26045562

26 Stuart EA Matching methods for causal inference A review and a look forward Stat Sci2010251-21 10121409-STS313 20871802

27 Austin PC An introduction to propensity score methods for reducing the effects ofconfounding in observational studies Multivariate Behav Res 201146399-424101080002731712011568786 21818162

28 Diamond A Sekhon JS Genetic matching for estimating causal effects A generalmultivariate matching method for achieving balance in observational studies Rev EconStat 201395932-45101162REST_a_00318

29 Pennington M Grieve R Sekhon JS Gregg P Black N van der Meulen JH Cementedcementless and hybrid prostheses for total hip replacement cost effectiveness analysisBMJ 2013346f1026

30 Wing C Simon K Bello-Gomez RA Designing difference in difference studies bestpractices for public health policy research Annu Rev Public Health 201839453-69101146annurev-publhealth-040617-013507 29328877

31 Venkataramani AS Bor J Jena AB Regression discontinuity designs in healthcareresearch BMJ 2016352i1216 101136bmji1216 26977086

32 Abadie A Gardeazabal J The economic costs of conflict a case study of the Basquecountry Am Econ Rev 200393113-32101257000282803321455188

33 Stuart EA Matching methods for causal inference A review and a look forward Stat Sci2010251-21 10121409-STS313 20871802

34 McNamee R Regression modelling and other methods to control confounding OccupEnviron Med 200562500-6 472 101136oem2002001115 15961628

35 Ho DE Imai K King G Stuart EA Matching as nonparametric preprocessing for reducingmodel dependence in parametric causal inference Polit Anal200715199-236101093panmpl013

36 Sutton M Nikolova S Boaden R Lester H McDonald R Roland M Reduced mortalitywith hospital pay for performance in England N Engl J Med 20123671821-8101056NEJMsa1114951 23134382

37 Stephen O Wolters A Steventon A Briefing The impact of redesigning urgent andemergency care in Northumberland 2017 httpswwwhealthorguksiteshealthfilesIAUNorthumberlandpdf

38 Geneletti S OrsquoKeeffe AG Sharples LD Richardson S Baio G Bayesian regressiondiscontinuity designs incorporating clinical knowledge in the causal analysis of primarycare data Stat Med 2015342334-52 101002sim6486 25809691

39 Bernal JL Cummins S Gasparrini A Interrupted time series regression for the evaluationof public health interventions a tutorialInt J Epidemiol 2016 46348-55

40 Donegan K Fox N Black N Livingston G Banerjee S Burns A Trends in diagnosis andtreatment for people with dementia in the UK from 2005 to 2015 a longitudinal retrospectivecohort study Lancet Public Health 201726671-8

41 Craig P Cooper C Gunnell D etal Using natural experiments to evaluate populationhealth interventions new Medical Research Council guidance J Epidemiol CommunityHealth 2012661182-6 101136jech-2011-200375 22577181

42 Mayhew L On the effectiveness of care co-ordination services aimed at preventing hospitaladmissions and emergency attendances Health Care Manag Sci 200912269-84101007s10729-008-9092-5 19739360

43 Sterne JA Hernaacuten MA Reeves BC etal ROBINS-I a tool for assessing risk of bias innon-randomised studies of interventions BMJ 2016355i4919101136bmji4919 27733354

44 Chen YF Hemming K Stevens AJ Lilford RJ Secular trends and evaluation of complexinterventions the rising tide phenomenon BMJ Qual Saf 201625303-10101136bmjqs-2015-004372 26442789

45 Powell AE Davies HT Thomson RG Using routine comparative data to assess the qualityof health care understanding and avoiding common pitfalls Qual Saf Health Care200312122-8 101136qhc122122 12679509

46 NHS Digital Hospital Episode Statistics httpsdigitalnhsukdata-and-informationdata-tools-and-servicesdata-serviceshospital-episode-statistics

47 NHS Digital Data Access Request Service (DARS) httpsdigitalnhsukservicesdata-access-request-service-dars

48 NHS Digital Secondary Uses Service (SUS) httpsdigitalnhsukservicessecondary-uses-service-sus

49 Medicines and Healthcare Regulatory Agency and National Institute for Health Research(NIHR) Clinical Practice Research Datalink(CPRD) httpswwwcprdcom

50 Office for National Statistics Deaths httpswwwonsgovukpeoplepopulationandcommunitybirthsdeathsandmarriagesdeaths

51 NHS Digital Mental Health Services Data Set httpsdigitalnhsukdata-and-informationdata-collections-and-data-setsdata-setsmental-health-services-data-set

52 NHS Digital Quality Outcomes Framework (QOF) httpsdigitalnhsukdata-and-informationdata-tools-and-servicesdata-servicesgeneral-practice-data-hubquality-outcomes-framework-qof

53 Office for National Statistics 2011 Census httpswwwonsgovukcensus2011census54 NHS England GP Patient Survey (GPPS) httpswwwgp-patientcouk55 Rothman KJ Greenland S Lash T Modern Epidemiology Lippincott Williams amp Williams

200556 Minary L Alla F Cambon L Kivits J Potvin L Addressing complexity in population health

intervention research the contextintervention interface J Epidemiol Community Health201872319-2329321174

57 Sun X Briel M Walter SD Guyatt GH Is a subgroup effect believable Updating criteriato evaluate the credibility of subgroup analyses BMJ 2010340c117

58 von Elm E Altman DG Egger M Pocock SJ Gotzsche PC Vandenbroucke JP Thestrengthening the reporting of observational studies in epidemiology (STROBE) statementguidelines for reporting observational studies Ann Intern Med 2007147573-7

59 Des Jarlais DC Lyles C Crepaz N the TREND Improving the reporting quality ofnonrandomized evaluations the TREND statement Am J Public Health 200494361-6

60 The Health Foundation Evaluation what to consider 2015 httpswwwhealthorgukpublicationsevaluation-what-to-consider

Published by the BMJ Publishing Group Limited For permission to use (where not alreadygranted under a licence) please go to httpgroupbmjcomgrouprights-licensingpermissionsThis is an Open Access article distributed in accordance with the terms ofthe Creative Commons Attribution (CC BY 40) license which permits others to distributeremix adapt and build upon this work for commercial use provided the original work isproperly cited See httpcreativecommonsorglicensesby40

Open Access Reuse allowed Subscribe httpwwwbmjcomsubscribe

BMJ 2019365l2239 doi 101136bmjl2239 (Published 20 June 2019) Page 4 of 7

PRACTICE

on 5 June 2020 by guest Protected by copyright

httpww

wbm

jcom

BM

J first published as 101136bmjl2239 on 20 June 2019 D

ownloaded from

Tables

Table 1| Impact evaluations

ExamplesSummativeFormative

A formative evaluation of the Whole Systems Integrated Care (WSIC) programmeaimed at integrating health and social care in London found that difficulties inestablishing data sharing and information governance and differences inprofessional culture were hampering efforts to implement change10

Conducted after the interventionrsquoscompletion or at the end of a programmecycle

Conducted during the developmentor implementation of an intervention

A summative impact evaluation of an NHS new care model vanguard initiative foundthat care home residents in Nottinghamshire who received enhanced support hadsubstantially fewer attendances at emergency departments and fewer emergencyadmissions than a matched control group13 This evidence supported the decisionby the NHS to roll out the Enhanced Health in Care Homes Model across thecountry2

Aims to render judgment or makedecisions about the future of theintervention

Aims to fine tune or reorient theintervention

Open Access Reuse allowed Subscribe httpwwwbmjcomsubscribe

BMJ 2019365l2239 doi 101136bmjl2239 (Published 20 June 2019) Page 5 of 7

PRACTICE

on 5 June 2020 by guest Protected by copyright

httpww

wbm

jcom

BM

J first published as 101136bmjl2239 on 20 June 2019 D

ownloaded from

Table 2| Observational study designs for quantitative impact evaluation

Strengths and limitationsMethod

Can be combined with other methods eg difference-in-differences andregression Enables straightforward comparison between intervention andcontrol groups Methods include propensity score matching and genetic

matching

Matching33 Aims to find a subset of control group units (eg individuals or hospitals)with similar characteristics to the intervention group units in the pre-intervention periodFor example impact of enhanced support in care homes in RushcliffeNottinghamshire13

Can be beneficial to pre-process the data using matching in addition toregression control This reduces the dependence of the estimated treatment

effect on how the regression models are specified35

Regression control34 Refers to use of regression techniques to estimate associationbetween an intervention and an outcome while holding the value of the other variablesconstant thus adjusting for these variables

Simple to implement and intuitive to interpret Depends on the assumptionthat there are no unobserved differences between the intervention and controlgroups that vary over time also referred to as the ldquoparallel trendsrdquo assumption

Difference-in-differences (DiD)30 Compares outcomes before and after an interventionin intervention and control group units Controls for the effects of unobservedconfounders that do not vary over time eg impact of hospital pay for performance onmortality in England36

Allows for unobserved differences between the intervention and controlgroups to vary over time The uncertainty of effect estimates is hard toquantify Produces biased estimates over short pre-intervention periods

Synthetic controls32 Typically used when an intervention affects a whole population(eg region or hospital) for whom a well matched control group comprising wholecontrol units is not available Builds a ldquosyntheticrdquo control from a weighted average ofthe control group units eg impact of redesigning urgent and emergency care inNorthumberland37

There is usually a strong basis for assuming that patients close to either sideof the threshold are similar Because the method only uses data for patients

near the threshold the results might not be generalisable

Regression discontinuity design31 Uses quasi-random variations in interventionexposure eg when patients are assigned to comparator groups depending on athreshold Outcomes of patients just below the threshold are compared with thosejust above eg impact of statins on cholesterol by exploiting differences in statinprescribing38

Ensures limited impact of selection bias and confounding as a result ofpopulation differences but does not generally control for confounding as aresult of other interventions or events occurring at the same time as theintervention

Interrupted time-series39 Compares outcomes at multiple time points before andafter an intervention (interruption) is implemented to determine whether the interventionhas an effect that is statistically significantly greater than the underlying trend eg toexamine the trends in diagnosis for people with dementia in the UK40

Explicitly addresses unmeasured confounding but conceptually difficult andeasily misused Identification of instrumental variables is not straightforwardEstimates are imprecise (large standard error) biased when sample size issmall and can be biased in large samples if assumptions are even slightly

violated20

Instrumental variables18 An instrumental variable is a variable that affects the outcomesolely through the effect on whether the patient receives the treatment An instrumentalvariable can be used to counteract issues of measurement error and unobservedconfounders eg used to assess delivery of premature babies in dedicated v hospitalintensive care units19

Open Access Reuse allowed Subscribe httpwwwbmjcomsubscribe

BMJ 2019365l2239 doi 101136bmjl2239 (Published 20 June 2019) Page 6 of 7

PRACTICE

on 5 June 2020 by guest Protected by copyright

httpww

wbm

jcom

BM

J first published as 101136bmjl2239 on 20 June 2019 D

ownloaded from

Table 3| Commonly used routine datasets available in the NHS in England

Dissemination and alternativesDataset

HES is available through the Data Access Request Service (DARS)47 a serviceprovided by NHS Digital Commissioners providers in the NHS and analyticsteams working on their behalf can also access hospital data directly via the

Secondary Use Service (SUS)48 These data are very similar to HES processedby NHS Digital and are available for non-clinical uses including research and

planning health services

Hospital episode statistics (HES)46 HES is a database containing details of alladmissions accident and emergency attendances and outpatient appointmentsat NHS England hospitals and NHS England funded treatment centres Informationcaptured includes clinical information about diagnoses and operations patientdemographics geographical information and administrative information such asthe data and method of admissions and discharge

Commissioners and analytics teams working on their behalf can work with anintermediary service called Data Service for Commissioning Regional Office torequest access to anonymised patient level general practice data (possibly linkedto SUS described above) for the purpose of risk stratification invoice validation

and to support commissioning Anonymised UK primary care records for arepresentative sample of the population are available for public health research

through for instance the Clinical Practice Research Datalink49

Primary care data is collected by general practices Although there is no nationalstandard on how primary care data should be collected andor reported there area limited number of commonly used software providers to record these dataInformation captured includes clinical information about diagnoses treatmentand prescriptions patient demographics geographical information andadministrative information on booking and attendance of appointments andwhether appointments relate to a telephone consultation an in-practiceappointment or a home visit

ONS mortality data are routinely processed by NHS Digital and can be linkedto HES data These data can be requested through the DARS service

When deaths occur in hospital this is typically recorded as part of dischargeinformation

Mortality data50 The Office for National Statistics (ONS) maintains a dataset ofall registered deaths in England These data can be linked to routine health datato record deaths that occur outside of hospital

Like HES MHSDS is available through the DARS service Mental health datafrom before April 2016 have been recorded in the Mental Health Minimum Dataset

also disseminated through NHS Digital

The Mental Health Services Data Set (MHSDS)51 contains record level dataabout the care of children young people and adults who are in contact with mentalhealth learning disabilities or autism spectrum disorder services These datacover data from April 2016

Open Access Reuse allowed Subscribe httpwwwbmjcomsubscribe

BMJ 2019365l2239 doi 101136bmjl2239 (Published 20 June 2019) Page 7 of 7

PRACTICE

on 5 June 2020 by guest Protected by copyright

httpww

wbm

jcom

BM

J first published as 101136bmjl2239 on 20 June 2019 D

ownloaded from