leite, a; andrews, nj; thomas, sl (2016) near real...
TRANSCRIPT
Leite, A; Andrews,NJ; Thomas,SL (2016) Near real-time vaccinesafety surveillance using electronic health records-a systematic re-view of the application ofstatisticalmethods. Pharmacoepidemi-ology and drug safety,25 (3). pp. 225-37. ISSN 1053-8569 DOI:https://doi.org/10.1002/pds.3966
Downloaded from:http://researchonline.lshtm.ac.uk/2532230/
DOI: 10.1002/pds.3966
Usage Guidelines
Please refer to usage guidelinesat http://researchonline.lshtm.ac.uk/policies.htmlor alterna-tively contact [email protected].
Available under license:http://creativecommons.org/licenses/by/2.5/
REVIEW
Near real-time vaccine safety surveillance using electronic healthrecords—a systematic review of the application of statistical methods†
Andreia Leite1*, Nick J. Andrews2 and Sara L. Thomas1
1Faculty of Epidemiology and Population Health, London School of Hygiene & Tropical Medicine, London, UK2Statistics, Modelling and Economics Department, Public Health England, London, UK
ABSTRACTPurpose Pre-licensure studies have limited ability to detect rare adverse events (AEs) to vaccines, requiring timely post-licensure studies.With the increasing availability of electronic health records (EHR) near real-time vaccine safety surveillance using these data has emerged asan option. We reviewed methods currently used to inform development of similar systems for countries considering their introduction.Methods Medline, EMBASE and Web of Science were searched, with additional searches of conference abstract books. Questionnaireswere sent to organizations worldwide to ascertain unpublished studies. Eligible studies used EHR and regularly assessed pre-specified AEto vaccine(s). Key features of studies were compared descriptively.Results From 2779 studies,31 were included from the USA (23),UK (6), and Taiwan and New Zealand (1 each).These werepublished/conducted between May 2005 and April2015.Thirty-eightdifferentvaccines were studied,focusing mainly on influenza(47.4%), especially 2009 H1N1 vaccines. Forty-six analytic approaches were used, reflecting frequency of EHR updates and the AE studied.Poisson-based maximized sequential probability ratio test was the most common (43.5%), followed by its binomial (23.9%) and conditionalversions (10.9%). Thirty-seven of 49 analyses (75.5%) mentioned control for confounding, using an adjusted expected rate (51.4% of thoseadjusting), stratification (16.2%) or a combination of a self-controlled design and stratification (13.5%). Guillain-Barré syndrome (11.9%),meningitis/encephalitis/myelitis (11.9%) and seizures (10.8%) were studied most often.ConclusionsNear real-time vaccine safety surveillance using EHR has developed over the past decade but is not yet widely used. As morecountries have access to EHR, it will be important that appropriate methods are selected, considering the data available and AE of interest.© 2016 The Authors. Pharmacoepidemiology and Drug Safety Published by John Wiley & Sons Ltd.
key words—electronic health records; safety; sequential tests; statistical process control; vaccines; pharmacoepidemiology
Received 19 June 2015; Revised 16 December 2015; Accepted 17 December 2015
INTRODUCTIONVaccines are considered to be one of the mostcost-effective interventionsin public health.1,2 As withother drugs,vaccines are nottotally safe,3 butsafetyrequirementsareparticularly high asvaccinesaregiven to healthy individuals,mostoften children.4
All vaccines go through extensive safety assessmentbefore licensure;however,pre-licensure studies havelimited ability to detectrare adverse events (AEs) tovaccines (with frequency <1/10 000-1/100 000)5, AE
occurring among specific sub-populations who werenot included in clinical trials,and long-term AE.6 Toovercome these limitations, timely post-licensure stud-ies are required.These can be broadly divided intopassive (spontaneous reports) and active studies andshould be followed by confirmatory epidemiologicstudies. While spontaneous reporting of AE is widelyimplemented worldwideas a simpleand low-costmethod,usefulto detectnew,unanticipated AE,ithas limitations.2 These include difficulties in denomi-nator calculation, potential reporting biases (e.g. over-reporting ofpotentialAE receiving extensive mediacoverage) and incomplete reporting. In contrast, activesurveillance triesto identify allthose experiencing(or atleastseeking medicalattention for) a potentialAE to vaccines.This approach includesanalysesof large population datasets (using electronic healthrecords (EHR)),targeted hospital-based surveillance
*Correspondence to: A. Leite, Department of Infectious Disease Epidemiology,London School of Hygiene & Tropical Medicine,Keppel Street,WC1E 7HT,London, UK. E-mail: [email protected]†Prior postings and presentations statement: This work has not been submittedor accepted elsewhere.Preliminary results have been presented atthe NIHRHealth Protection Research Uniton Immunisation annualmeeting in March2015 and have been presented as a poster presentation to the 31st InternationalConference on Pharmacoepidemiology & Therapeutic Risk Management.
© 2016 The Authors. Pharmacoepidemiology and Drug Safety Published by John Wiley & Sons Ltd.
pharmacoepidemiology and drug safety 2016; 25: 225–237Published online 28 January 2016 in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/pds.3966
This is an open access article under the terms of the Creative Commons Attribution-NonCommercial License, which permits use, distributionand reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes.
(where trained health workersdaily seek potentialcases of conditions of interest) and recruitment of vac-cinated cohorts for detection of AE (using face-to-faceinterviews,phone interviews,short-message servicesor web-based tools).7,8With the increased availabilityof large population datasets,nearreal-time vaccinesafety surveillance(NRTVSS) hasemerged asanoption.9
Near real-timevaccinesafetysurveillance,alsoknown as rapid cycle analysis,involves regular inter-rogation ofEHR to investigate pre-specified AE tovaccines.2 By testing these AE on a regular basis afterintroduction of a new vaccine, these methods ensure atimely detection of possible safety problems.10When asignal is detected by this approach, it needs to be fur-ther analysed, including a signal refinement stage andeventualconfirmatory analyses.These stepsshouldbe predetermined and willlead to the decision ofwhether to validate or invalidate the signal. NRTVSSis thus partof a systematic approach to signaldetec-tion, with a dual role of signalling possible AE to vac-cines and reassuring authorities and populations thatevents are being monitored.11 For a given vaccine,NRTVSS only considers a small number of suspectedAE (e.g. 5 to 10); complementary information is pro-vided by existingmethodssuch as spontaneousreports.12
The growing use of NRTVSS methods,along withthe increasing availability of EHR, highlights the needto review studies using this approach.Such a reviewcan provide crucialinformation on the developmentof systems for vaccine safety surveillance for countriesconsidering their introduction.
OBJECTIVEThe aim of this study was to carry out a systematic re-view of publishedand unpublisheddataon themethods used for NRTVSS using EHR.
METHODSStudies were included in the review if they (i) usedroutinely collected health data (atleastfor the ex-pected numberof events);(ii) studied pre-specifiedoutcome(s) to assess the safety of one or more vac-cines; and (iii) regularlytestedthe outcomes.Studies(i) includingonly informationbasedonspontaneous reporting systems,(ii) aimed attestinghypothesis/confirming previously generated/suspectedsignals or (iii) aimed atdeveloping new methods forNRTVSS (unlessa specific application ofthe new
method was given) were excluded. No limits were im-posed in terms of language or year.
Medline and EMBASE were searched forstudiespublished until6 January 2015,using a combinationof thesaurus and free-text terms (search strategy is pro-vided in Supporting Information Appendix A).Titlesand abstracts were reviewed to determine eligibilitystatus,followed by the fulltextfor those consideredpotentially eligible.References from the papers col-lected were also reviewed. Reviews of the topic wereselected forreference mining.A. L. was responsiblefor evaluating eligibility of the identified studies.Toensure quality, eligibility of a random sample of 10%of the results was evaluated by S. T.and N. A.Wheneligibility was unclear, the study was discussed amongthe authors until a consensus was reached.
To complementthe database searches,a citationsearch was conducted. To the best of our knowledge,the methods under study were first applied to the fieldof vaccinesafety by theVaccineSafety Datalink(VSD). Two key VSD papers that describe the testingand implementation of rapid cycle analysis using rou-tinely collected health data were selected to perform acitation search.9,13
The same search strategy was used in the Web ofScience Core Collection to cover meetings and confer-ences,restricting the search to meeting abstractsorproceedings papers.Also,the AnnualConference onVaccine Research and the Vaccine and ISV Congressabstract book and programme, respectively, were analysed(Supporting Information Appendix B).The BrightonCollaboration newsletter was also searched as a potentialsource of relevant new studies or contacts.14
A second stage ofthe review included contactingexperts in vaccine safety, as follows:
• Specialists in vaccine safety (from the Global Advi-sory Committee on Vaccine Safety (GACVS),15
Brighton Collaboration16 and Accelerated Devel-opmentof Vaccine benefit–risk collaboration inEurope (ADVANCE)17) were asked if they wereaware ofwork being conducted in the area andfulfilling our inclusion criteria.
• Authors with known work using routinely collecteddata and the potential to have implemented/conductedeligiblestudieswere contacted(MedicinesandHealthcare products Regulatory Agency (MHRA),18
VSD19 and Statens Serum Institute20). Further con-tacts were also asked for at this stage.
• Finally, authors with a previous published work butincompleteinformation,and thosesuggested byotherexperts,were contacted to ask forfurtherinformation to characterize the methods.
a. leite et al.226
© 2016 The Authors. Pharmacoepidemiology and Drug SafetyPublished by John Wiley & Sons Ltd.
Pharmacoepidemiology and Drug Safety, 2016; 25: 225–237DOI: 10.1002/pds
An online questionnaire was used to capture infor-mation on studies conducted (Supporting InformationAppendix C).When othersourcesof information(e.g.reports)were available and shared by the con-tactsthesewereused.Expertcontactstook placefrom February to March 2015.
The information identified wasextracted using astandardized extraction form.Data extracted includedtimeline, country/institutions where the study was con-ducted,vaccines studied,study population,outcomesassessed and their method of ascertainment,methodsused to perform the analyses, frequency of assessment,confounding,data-accruallag (i.e.delays in the dataavailable to perform surveillance,which may affectthe results), assessment of the validity of the outcomesof interest (e.g.chart review) and main results.A de-scriptivesummary ofcountry/institution,vaccines,outcomes studied,confounding and data-accruallaghandling was drawn up.
RESULTSA totalof 29 reports were included for data extrac-tion (includinginformationprovidedby expertcontacts),9,13,21–45representing31 studies/systems(Figure 1). A brief description of the studies/systems in-cluded by country,methods used and adjustmentforconfounding strategies is given in Table 1.A detailedcharacterization of the studies is provided in SupportingInformation Appendix D.
Nearreal-timevaccinesafety surveillanceusingEHRs was first reported by Davis et al. in 2005, whena retrospectivestudyassessingthe feasibilityofimplementing such methods was published. Since thistime,we identified a further13 studies conductedby the VSD and 17 other studies in three countries(Figure 2).The firststudy conducted outside theVSD was conducted in New Zealand and publishedin 2007.The reportfrom the laststudy includedwas published online in 2015.Fourstudies (allinthe USA) were conducted completely orpartiallyin a retrospective manner,to testthe feasibility ofimplementing this kind ofsystem (Table 1).Twoof these studies attempted to replicate known sig-nals (rotavirusvaccineand intussusceptionandacellulardiphtheria-tetanus-pertussis (DTaP)/wholecelldiphtheria-tetanus-pertussis vaccine and febrileseizures).Of the prospectivestudies,mostwereconducted in the USA (n = 20),with studiesalsoconductedin the UK (n = 6), and Taiwan andNew Zealand(n = 1for each).The prospectivestudies looked mainly at influenza vaccines (n = 16), es-pecially the 2009 H1N1 pandemic influenza vaccine
(n = 7). Rotavirus (n = 5), DTaP-based (n = 3) and humanpapillomavirus vaccines (n = 3) also received attention.
The outcomes studied were most often neurological(58.5%). Looking at specific outcomes, Guillain-Barrésyndrome (GBS) (11.9% of studied known outcomes),meningitis/encephalitis/myelitis (11.9%) and seizures(10.8%) were the most often included. Outcome ascer-tainmentfor the near real-time analysis was,in mostcases, based on automated data (with no a priori con-firmation of the diagnosis). In these cases, chart reviewand confirmation were used whenever a potential AEwas signalled.Only two studies performed this kindof confirmation forthe nearreal-time analysis,21,35
and one compared the analysis considering the chart-reviewed and non-reviewed outcome for GBS.33Fromthe outcomes studied,11 signals were identified,butonly threeconfirmed (measles-mumps-rubella-vari-cella combinationvaccineand febrileseizures,27
2010–2011trivalentinactivatedinfluenzavaccineand febrile seizures,37 and monovalentrotavirus vac-cine and intussusception41).
Table 2 summarizes the methods used by the stud-ies included in this review. These can be broadly di-vided into continuoussequentialtesting,whichallows examination ofthe data as often as desired(n = 25),9,13,22–34,37,38,40–43,45group sequential testing(n = 4)35,36,38,39and statisticalprocess control(SPC;n = 3).21,44The choice ofthe group of methods hasbeen determined by the frequency of updates to theEHR data used (Table 2).
When considering specificversionsof the testsavailable, the choice has been guided by the increasingavailability of new methods and knowledge of thesemethods over time,as shown in Figure 2,as wellasthe frequency of AE studied.In VSD, the sequentialprobability ratio test(SPRT) was firstapplied9 beingsubsequentlyreplacedby its maximizedversion(MaxSPRT) with the advantage of not having to spec-ify a singlealternativehypothesis.13 The use ofMaxSPRT and its variations also evolved over time.While in the beginning the Poisson and binomial ver-sionswere simultaneously used forthe same out-come,13 from 2010,a targeted selection ofthe testversion and its extensions,based on the strengths ofeach method (Table 2) and the characteristics of theoutcomeunderstudy,was preferred.24,33,34,42,43Inparticular,Poisson-basedMaxSPRT (PMaxSPRT)has been used when less than 50 events were antici-pated and the conditional version when the ratio of ob-served historicaleventsto upperlimit was ≤2.5.Outside VSD, a pattern in the use of continuous sequen-tial methods was less clear. Overall, these tests were themost often employed—PMaxSPRT(45.7%),10,50
near real-time vaccine safety surveillance 227
© 2016 The Authors. Pharmacoepidemiology and Drug SafetyPublished by John Wiley & Sons Ltd.
Pharmacoepidemiology and Drug Safety, 2016; 25: 225–237DOI: 10.1002/pds
followed by the binomial(BMaxSPRT—23.9%)10,50
and conditional (10.9%) versions.51
More recently,fourstudies used group sequentialtesting.Two of theseused an alpha-spending ap-proach,38,39(a function controlling how much of thealpha willbe ‘spent’every time a new analysis isrun52), one the Updating Sequential Probability RatioTest53 and otherthe Abt’s modification ofSPRT.54
An alpha-spending approach was thus preferred overthe two othertests employed in a group sequentialway. Both the Pocock-type and O’Brien–Fleming-typefunctions have been used.12,55The remaining methodsdid notfollow a clearevolution and include use ofSPC56at different times by two non-USA institutions
(New Zealand Ministry of Health,Health ProtectionScotland).21,44
Thirty-seven of 49 analyses (75.5%) mentioned con-trol for confounding.Strategieschosen were oftendesign-based and included (alone or in combination)the following: (i) using a self-controlled design, whichautomatically addressestime-invariantconfounders;(ii) matching baseline confounders, through a concur-rentcomparatordesign;(iii) adjusting the expectedrateobtained from ahistoricalcomparison groupbased on the confounders’distribution in the studycohort (iv) stratifying the results according to relevantconfounder categories.Analyses adjusting for poten-tial confounders used mainly an expected rate adjusted
Figure 1. Flowchart of included studies. Studies were excluded for (i) not considering vaccines (nonvaccine), (ii) not analysing the safety of a vaccine (notsafety), (iii) considering safety issues but not applying the methods of interest (other safety), (iv) only developing new methods (methods only) and (v) havingno abstract available (not available)
a. leite et al.228
© 2016 The Authors. Pharmacoepidemiology and Drug SafetyPublished by John Wiley & Sons Ltd.
Pharmacoepidemiology and Drug Safety, 2016; 25: 225–237DOI: 10.1002/pds
Tabl
e 1.
Inclu
ded
stud
ies a
ccor
ding
to th
e co
untry
, met
hods
use
d an
d co
ntro
l for
con
foun
ding
stra
tegi
es (s
ee S
uppo
rting
Info
rmat
ion
Appe
ndix
D fo
r fur
ther
det
ails)
Stud
yCo
untry
,or
gani
zatio
nM
etho
dCo
nfou
ndin
gDa
ta‐a
ccru
al la
g or
unde
rrepo
rting
adj
ustm
ent
Retro
spec
tive
Davi
s9US
A, V
SDSP
RTRi
sk a
djus
tmen
t* (s
ite, a
ge, t
ime,
seas
on, s
ex)
Retro
spec
tive
Lieu13
†US
A, V
SDPM
axSP
RTUn
clear
Retro
spec
tive
Brow
n22US
A, i3
Dru
gSa
fety
PMax
SPRT
Expe
cted
cou
nts
(sex
, age
, reg
ion,
mon
th,
conc
omita
nt v
accin
atio
n)Re
trosp
ectiv
e; d
ata
lags
ass
esse
d du
ring
the
stud
yGr
eene24
†US
A, V
SDPM
axSP
RTEx
pect
ed ra
tes (
age
and
site)
Retro
spec
tive—
data
ass
umed
to a
ccru
ewi
thou
t del
ayBM
axSP
RT‡SC
; stra
tifica
tion
(age
, sea
son)
Pros
pect
ive
Lieu13
USA,
VSD
PMax
SPRT
No a
djus
tmen
tAn
alys
es w
aite
d at
leas
t 6 w
eeks
from
the
vacc
inat
ion
or p
reve
ntiv
e vi
sitBM
axSP
RTM
atch
ing
(age
, wee
k, si
te)
McN
ichol
as21Ne
wZe
alan
d,M
oH
SPC
Stra
tifica
tion
(age
)Da
ily re
view
of d
atab
ases
, med
ical
char
ts, d
ischa
rge
lette
rs a
nd la
bora
tory
reco
rds
Yih23
USA,
VSD
PMax
SPRT
Expe
cted
cou
nts
(GBS
/sei
zure
s—ag
e;ot
her A
E—ag
e, se
x)An
alys
is st
arte
d at
leas
t 8 w
eeks
from
the
date
of v
accin
atio
n46
and
redo
neat
the
end
of th
e st
udy
Belo
ngia25
†US
A, V
SDPM
axSP
RTEx
pect
ed ra
tes (
intu
ssus
cept
ion—
trend
,ag
e, si
te b
y Po
isson
regr
essio
n; o
ther
AE—
site)
Anal
ysis
star
ted
at le
ast 8
wee
ks fr
omth
e da
te o
f vac
cinat
ion
46
Brya
n28UK
, MHR
APM
axSP
RTEx
pect
ed ra
tes (
age
and
gend
er)
Adju
sted
for u
nder
repo
rting
(yel
low-
card
dat
a)Hu
ang30
†Ta
iwan
, CDC
PMax
SPRT
Stra
tifica
tion
(age
)Da
taba
se u
pdat
ed d
aily
BMax
SPRT
‡SC
Enge
r29US
A, i3
Dru
gSa
fety
Uncle
arUn
clear
Uncle
ar
DMSS
26,3
2,47
,48
USA,
DoD
PMax
SPRT
Uncle
arUn
clear
VA26
,32,
48US
A, V
APM
axSP
RTUn
clear
Uncle
arIH
S26,3
2,48
.49
USA,
IHS/
FDA
PMax
SPRT
Uncle
arUn
clear
PRIS
M26,3
2,48
USA,
FDA
/NV
POPM
axSP
RTUn
clear
Uncle
arBM
axSP
RT‡
Klei
n27†
USA,
VSD
BMax
SPRT
Mat
chin
g (a
ge g
roup
, site
, cal
enda
rye
ar a
nd re
spira
tory
viru
s sea
son)
Anal
ysis
dela
yed
at le
ast 8
wee
ksfro
m d
ate
of v
accin
atio
n46
Gee34
USA,
VSD
PMax
SPRT
Expe
cted
rate
s (ag
e, si
te)
Uncle
arBM
axSP
RT§
Mat
chin
g (a
ge, s
ite, v
accin
atio
n da
te)
Lee33
USA,
VSD
PMax
SPRT¶
Expe
cted
rate
s (ag
e an
d sit
e)Ad
just
ed fo
r par
tially
ela
psed
risk
inte
rval
and
del
ay in
the
arriv
al o
fin
patie
nt d
ata
BMax
SPRT‡
SCBo
thSt
ratifi
catio
n** (
age)
Brya
n31†
MHR
A, U
KPM
axSP
RTEx
pect
ed ra
tes (
age)
Adju
sted
for u
nder
repo
rting
(yel
low-
card
dat
a)Bu
rwen36
USA,
FDA
USPR
TNo
Criti
cal l
imits
adj
uste
d fo
rde
lays
in th
e cl
aim
s (ba
sed
onpr
evio
us s
easo
ns)
Loug
hlin35
†US
A,Op
tum
Insig
htAb
t’sm
odifi
catio
n of
SPRT
NoNo
(Con
tinue
s)near real-time vaccine safety surveillance 229
© 2016 The Authors. Pharmacoepidemiology and Drug SafetyPublished by John Wiley & Sons Ltd.
Pharmacoepidemiology and Drug Safety, 2016; 25: 225–237DOI: 10.1002/pds
Tabl
e 1.
(Con
tinue
d)
Stud
yCo
untry
,or
gani
zatio
nM
etho
dCo
nfou
ndin
gDa
ta‐a
ccru
al la
g or
unde
rrepo
rting
adj
ustm
ent
Tse37
USA,
VSD
PMax
SPRT¶
Stra
tifica
tion
(age
, site
)Ad
just
ed fo
r par
tially
ela
psed
risk
inte
rval
and
dela
y in
the
arriv
al o
f inp
atie
nt d
ata
BMax
SPRT‡
SCDo
nega
n40†
UK, M
HRA
PMax
SPRT
Stra
tifica
tion
(age
)—fir
st y
ear o
fsu
rvei
llanc
eSe
nsiti
vity
ana
lyse
s ass
umin
g va
rious
deg
rees
of u
nder
repo
rting
(yel
low-
card
dat
a)Ne
lson38
USA,
VSD
GS PMax
SPRT
Expe
cted
cou
nts
(site
, gen
der,
age
grou
p,sit
e ×
age—
Poiss
on re
gres
sion)
No††
Tsen
g39US
A, V
SDGS
Stra
tifica
tion
(age
, dos
e nu
mbe
r—on
ly fo
r feb
rile
seizu
res,
urtic
aria
/ang
ione
urot
ic o
edem
a, a
sthm
a)No
††
Dale
y42†
USA,
VSD
PMax
SPRT¶
Expe
cted
rate
s (sit
e—ex
cept
for G
BS a
nd S
JS—
weig
hted
ave
rage
use
d)Ex
clusio
n of
the
mos
t rec
ent 1
4 we
eks o
fda
ta11
Kawa
i43US
A, V
SDPM
axSP
RT¶Ex
pect
ed ra
tes a
djus
ted
(age
, site
)De
laye
d an
alys
is un
til e
stim
ated
dat
a la
gac
crua
l and
follo
w-up
tim
e wa
s com
plet
edBM
axSP
RT‡SC
, stra
tifica
tion
(age
)W
eint
raub41
†US
A, V
SDPM
axSP
RTEx
pect
ed ra
tes (
age,
site
)An
alys
is de
laye
d 2
week
sM
urdo
ch44†
UK, H
PSSP
CSt
ratifi
catio
n (a
ge, s
ite)
NoYi
h45US
A, F
DAPM
axSP
RT¶Ex
pect
ed ra
tes (
age
for a
naph
ylax
is an
dse
izure
s and
dat
a pa
rtner
for s
eizu
res)
Adju
sted
for p
artia
lly e
laps
ed ri
sk in
terv
al a
ndde
lay
in th
e ar
rival
of i
npat
ient
dat
aBM
axSP
RT‡SC
, stra
tifica
tion
(sei
zure
s—ag
e,co
ncom
itant
PCV
13 6
–23
mon
ths)
HPS†
(unp
ublis
hed)
UK, H
PSSP
CSt
ratifi
catio
n (a
ge, s
ex fo
r her
pes
zost
er, s
ite)
No
MHR
A†(u
npub
lishe
d)MHR
A, U
KPM
axSP
RTEx
pect
ed ra
tes (
age)
Adju
sted
for u
nder
repo
rting
(yel
low-
card
dat
a)
Stud
ies
in it
alic
are
the
ones
iden
tified
from
exp
ert c
onta
cts.
AE‐A
dver
se e
vent
; BM
axSP
RT, b
inom
ial‐b
ased
max
imize
d se
quen
tial p
roba
bilit
y ra
tio te
st; C
DC, C
ente
rs fo
r Dise
ase
Cont
rol a
nd P
reve
ntio
n; D
MSS
, Def
ense
Med
ical S
urve
illan
ce S
yste
m; D
oD,
Depa
rtmen
t of D
efen
se; F
DA, F
ood
and
Drug
Adm
inist
ratio
n; H
PS, H
ealth
Pro
tect
ion
Scot
land
; IHS
, Ind
ian
Heal
th S
ervi
ce; M
HRA,
Med
icine
s and
Hea
lthca
re p
rodu
cts R
egul
ator
y Ag
ency
; MoH
,M
inist
ry o
f Hea
lth; N
VPO,Na
tiona
l Vac
cine
Prog
ram
Offi
ce; P
CV13
,13
‐val
ent p
neum
ococ
cal c
onju
gate
vac
cine;
PM
axSP
RT,
Poiss
on‐b
ased
max
imize
d se
quen
tial p
roba
bilit
y ra
tio te
st; P
RISM
,Po
st‐L
icens
ure
Rapi
d Im
mun
izatio
n Sa
fety
Mon
itorin
g; S
C, se
lf‐co
ntro
lled
desig
n; S
JS, S
teve
ns–Jo
hnso
n sy
ndro
me;
SPC
, sta
tistic
al p
roce
ss c
ontro
l; SP
RT, s
eque
ntia
l pro
babi
lity
ratio
test
; USP
RT,
upda
ting
sequ
entia
l pro
babi
lity
ratio
test
; VA,
Vet
eran
s Affa
irs; V
SD, V
accin
e Sa
fety
Dat
alin
k.*E
ach
uniq
ue c
ombi
natio
n of
pot
entia
l con
foun
ders
is id
entifi
ed, f
orm
ing
a st
ratu
m, a
nd a
bas
elin
e ris
k is
calcu
late
d. F
or e
ach
stra
tum
, a te
st st
atist
ic is
calc
ulat
ed, a
nd th
e te
st s
tatis
tics a
re c
ombi
ned.
† Add
ition
al in
form
atio
n ob
tain
ed fr
om th
e au
thor
s.‡ U
ses a
self‐
cont
rolle
d de
sign.
§ Use
s an
exac
t ver
sion
of th
e te
st, w
ith fl
exib
le m
atch
ing.
¶ Use
s the
con
ditio
nal v
ersio
n of
the
test
.**
Only
for i
nact
ivat
ed v
accin
es a
nd s
peci
fic o
utco
mes
(dem
yelin
atin
g di
seas
e of
the
cent
ral n
ervo
us sy
stem
, diso
rder
s of t
he p
erip
hera
l ner
vous
syst
em a
nd n
euro
path
y, se
izure
s, B
ell’s
pal
sy a
ndot
her c
rani
al n
erve
diso
rder
s).
††An
alys
is ba
sed
on th
e nu
mbe
r of d
oses
mig
ht m
inim
ize d
elay
s for
initi
al p
erio
ds o
f sur
veill
ance
.
a. leite et al.230
© 2016 The Authors. Pharmacoepidemiology and Drug SafetyPublished by John Wiley & Sons Ltd.
Pharmacoepidemiology and Drug Safety, 2016; 25: 225–237DOI: 10.1002/pds
for potentialconfounders (51.4% of those adjusting),stratification (16.2%)or a combination ofa self-controlleddesignand stratification(13.5%).Thechoice of approaches also depended on the analyticalmethod selected.For group sequentialmethods andSPC, strategies to dealwith confounders were evenmore limited.When employinggroupsequentialmethods,only expected rate calculationsbased onthe confounders’distribution and stratification wereconsidered. For SPC, only stratification was used. Po-tentialconfounders considered include age,sex,geo-graphicsite, concomitantvaccineadministration,season and trend (Table 1).
Some ofthe prospective studies considered data-accrual lags in their analysis. Most often, the analysiswas delayed by some weeks (n = 7).Others adjustedfor partially elapsed risk intervals and delays in the ar-rival of inpatient data (n = 3).46For studies using spon-taneous report for the observed number of events (andEHR for the expected number of events),sensitivity
analyses with several degrees of underreporting wereconducted(n = 4).28,31,40Updatesto the previousdatasets already analysed were notconsidered a spe-cific strategy to adjustfor data-accruallags as theywould notreduce the time to signal.The majority ofstudies did notmention ways ordid notadjustfordata-accrual lags (n = 11).
DISCUSSIONOurcomprehensive systematic review has identifiedan increasingnumberof studiesand systemsimplementing NRTVSS.All the studiesidentifiedwere performed in high-income countries/regions withmost in the USA. This might reflect limited capacity inmany settings to provide registry data in a timely fash-ion and theinfra-structurerequired to setup thesystem.
A clear effortwas putinto using these methods toassess pandemic influenza vaccine safety. This vaccine
Figure 2.Studies included in the review, ordered by the year of publication. Continuous sequential test are underlined with single line, group sequential withbold line,and statisticalprocess controlwith dashed line.Grey background indicates non-published studies.*Results with previous published results.maxSPRT – Maximized Sequential Probability Ratio Test, P – Poisson version (†use of the conditional version), B – binomial version (‡use of self-controlledcase-series or extensions of the test). DMSS – Defense Medical Surveillance System, DTaP – acellular diphtheria-tetanus-pertussis vaccine, DTwP – wholecell diphtheria-tetanus-pertussis vaccine, GBMV – Group-B Meningococcal Vaccine, HPS – Health Protection Scotland, HPV2 – bivalent human papillomavirusvaccine, HPV4 – quadrivalent human papillomavirus vaccine, IHS – (US) Indian Health Service, IPV – inactivated poliovirus vaccine, MCV – meningococcalconjugate vaccine,MHRA – Medicines and Healthcare products Regulatory Agency,MMRV – Measles-Mumps-Rubella-Varicella combination vaccine,PCV13 – 13-valent pneumococcal conjugate vaccine, PRISM – Post-Licensure Rapid Immunization Safety Monitoring, RRV – Rhesus-Rotavirus vaccine,RV5 – pentavalent rotavirus vaccine, VA – Veteran’s Affaires, VSD – Vaccine Safety Datalink
near real-time vaccine safety surveillance 231
© 2016 The Authors. Pharmacoepidemiology and Drug SafetyPublished by John Wiley & Sons Ltd.
Pharmacoepidemiology and Drug Safety, 2016; 25: 225–237DOI: 10.1002/pds
Tabl
e 2.
Met
hods
and
resp
ectiv
e ex
tens
ions
use
d by
the
elig
ible
stud
ies.
Mai
n ad
vant
ages
and
cha
lleng
es o
f eac
h m
etho
d ar
e pr
ovid
edGe
neric
met
hod
Vers
ion
Gene
ral d
escr
iptio
nCo
mpa
rato
rAd
vant
ages
and
disa
dvan
tage
sCo
nfou
ndin
g
Cont
inuo
us s
eque
ntia
l—al
low
exam
inat
ion
of th
e da
ta a
s ofte
n as
des
ired,
the
vario
us v
ersio
ns a
re d
escr
ibed
late
r (SP
RT a
nd M
axSP
RT)
Wal
d’s S
PRT
Gene
ral
desc
riptio
nTh
is is
the
gene
ricm
etho
d pr
opos
edby
Wal
d in
the
1940
s.
For v
accin
e sa
fety
, a P
oiss
on m
odel
wou
ldty
pica
lly b
e us
ed w
ith th
e ob
serv
ed c
ount
com
pare
d wi
th a
fixe
d ex
pect
ed c
ount
.9,
50
Adva
ntag
e—Ea
sy im
plem
enta
tion
of th
e Po
isson
mod
el.
Cova
riate
adj
uste
d ex
pect
ed le
vels
can
be o
btai
ned
to a
llow
for
poss
ible
con
foun
ding
.9
Disa
dvan
tage
(com
pare
d wi
thM
axSP
RT)—
Fixe
d sin
gle
alte
rnat
ive
hypo
thes
is (e
.g. R
R =
3)wh
ose
choi
ce w
ill u
sual
ly b
e ar
bitra
ry.
50
Max
SPRT
Gene
ral
desc
riptio
nTh
is ge
neric
ally
desc
ribes
all
SPRT
met
hods
that
hav
e a
com
posit
eal
tern
ativ
ehy
poth
esis
(RR
> 1
).
Depe
nds o
n th
e ve
rsio
n of
the
test
(refe
r to
succ
eedi
ng d
ata)
.No
nee
d to
spe
cify
a sin
gle
alte
rnat
ive.
50De
pend
s on
the
vers
ion
of th
e te
st (r
efer
tosu
ccee
ding
dat
a).
Poiss
on—
This
impl
emen
tatio
n as
sum
es a
Poiss
on d
istrib
utio
n fo
r obs
erve
dco
unts
and
com
pare
s to
a fix
edex
pect
ed m
ean.
50
Adva
ntag
e—Si
mpl
e to
impl
emen
t.Th
e us
e of
a fi
xed
expe
cted
leve
lin
crea
ses p
ower
.10
Cova
riate
adj
uste
d ex
pect
edle
vels
can
be o
btai
ned
toal
low
for p
ossib
le c
onfo
undi
ng.
Pote
ntia
l for
con
foun
ding
due
tose
ason
al o
r tem
pora
l cha
nges
indi
seas
e in
cide
nce
or c
odin
g.10
Disa
dvan
tage
—Re
lies o
n ac
cura
teda
ta fo
r the
exp
ecte
d le
vel,
whic
hm
ay n
ot b
e th
e ca
se if
dat
a ar
elim
ited
or o
nly
hist
oric
al.
51
Bino
mia
l—
Base
d on
a b
inom
ial d
istrib
utio
nev
ents
occ
urrin
g am
ong
vacc
ine
expo
sed
indi
vidu
als/
perio
dsve
rsus
com
paris
on (u
nexp
osed
indi
vidu
als/
perio
ds).
50
Adva
ntag
e—Do
es n
ot re
ly o
n a
fixed
exp
ecte
d va
lue
and
can
mat
ch o
n co
nfou
nder
s or
com
pare
to o
ther
per
iods
with
in in
divi
dual
s.50
Can
be u
sed
in d
iffer
ent v
ersio
ns—
mat
chin
g co
ntro
ls (fi
xed
or fl
exib
lem
atch
ing
ratio
—ex
act s
eque
ntia
lan
alys
is57) o
r sel
f-con
trolle
d de
sign
(SCC
S or
SCR
I) or
con
sider
ing
prev
ious
sea
sons
, avo
idin
g th
ehe
alth
y va
ccin
ee e
ffect
(DID
24).
Disa
dvan
tage
—Le
ss p
ower
ful
than
Poi
sson
unl
ess m
ultip
leun
vacc
inat
ed a
vaila
ble
per v
accin
ated
.50
The
use
of a
self-
cont
rolle
dde
sign
with
pos
t-exp
osur
eco
mpa
rison
inte
rval
s mig
htre
sult
in d
elay
s.46
Pote
ntia
l for
con
foun
ding
dep
ends
on th
e ve
rsio
n of
the
test
use
d.
Cond
ition
al—
Assu
mes
a P
oiss
on p
roce
ss fo
rth
e cu
mul
ativ
e pe
rson
-tim
e to
obse
rve
a nu
mbe
r of a
dver
seev
ents
.51
Adva
ntag
e—Do
es n
ot a
ssum
e th
eex
pect
ed n
umbe
r of c
ases
is k
nown
(as
the
Poiss
on-b
ased
Max
SPRT
).
Sam
e as
Poi
sson
Acco
unts
for u
ncer
tain
ty in
hist
orica
l dat
a.51
Disa
dvan
tage
—As
sum
es c
onst
ant
even
t rat
es a
re in
hist
orica
l and
surv
eilla
nce
data
.51
Grou
pse
quen
tial
test
ing
Gene
ral
desc
riptio
nDa
ta a
re e
xam
ined
at d
iscre
tepo
ints
in ti
me.10
Seve
ral a
ppro
ache
s use
d a
grou
p se
quen
tial w
ay (P
Max
SPRT
,Ab
t’s m
odifi
catio
n of
SPR
T,US
PRT)
ofte
n im
plem
entin
g an
alph
a-sp
endi
ng a
ppro
ach
(usin
g a
func
tion
to d
eter
min
e ho
w to
‘spe
nd’
the
alph
a in
the
diffe
rent
test
s).
12
Adva
ntag
e—Re
quire
sle
ss fr
eque
nt u
pdat
es.
Depe
nds o
n th
e sp
ecifi
cve
rsio
n us
ed.
Disa
dvan
tage
—Co
ntin
uous
test
s are
mor
e po
werfu
l.58
Less
exp
lore
d(c
ompa
red
with
con
tinuo
us te
sts)
in th
e ob
serv
atio
nal s
ettin
g,in
cludi
ng a
djus
tmen
t for
con
foun
ders
.M
ore
com
plex
des
igns
.12
(Con
tinue
s)a. leite et al.232
© 2016 The Authors. Pharmacoepidemiology and Drug SafetyPublished by John Wiley & Sons Ltd.
Pharmacoepidemiology and Drug Safety, 2016; 25: 225–237DOI: 10.1002/pds
is a good example of the importance of post-licensuresurveillance due to potential safety concerns.32Menin-gococcalgroup B vaccine in New Zealand21 repre-sents a similar situation,where NRTVSS,along withenhancedpassivesurveillanceand other activemethods,was implemented after the vaccine was ap-proved without phase III trials. Other situations wherethese methods have been particularly usefulincludevaccines/AE ofconcerndue to experienceswithpreviousversionsof the vaccine—forexample,rotavirus/intussusception25 and influenza/GBS.32 Forpreviously suspected AE,the setof methodsherereviewed has the advantage of informing in a timelymanner the existence of a safety concern or reassuringregulatory authoritiesand the public aboutvaccinesafety.
In this review, we have identified different methodsto perform NRTVSS using EHR and the way thesehave been applied, both by VSD and by other institu-tions.All the methodsidentified are derived fromWald’s sequential test.50,59,60When choosing a partic-ular method, it is important to be aware of its proper-ties. Propertiesof the continuousand groupsequentialmethods have been studied in the contextof drug safety.12 Group sequentialmethodsweredeemed to be more appropriate when data updatesare less frequent,12 butmore recentwork comparingthese methods has found that for any group sequentialdesign, there is a better continuous method and recom-mended thatthe data are looked atas frequently aspossible.58 After selecting themethodologicalap-proach, it is necessary to choose the specific test to em-ploy. For example,using the PMaxSPRT andBMaxSPRT simultaneously mightbe a more robustapproach owing to complementary strengths.How-ever,as previously suggested,BMaxSPRT might failto identify a signalwhen investigating very rareevents.Hence,an alternative is to use PMaxSPRTwhen less than 50 events are anticipated and the condi-tionalversion when the ratio ofobserved historicalevents to upperlimitis ≤2.5.The use ofa targetedapproach has been considered in VSD’s more recentwork.24,33,34,42,43
On the otherhand,the propertiesof SPC-basedmethods applied to vaccine safety have notbeen ex-tensively studied. Both Kulldorff et al.50and Musondaet al.61have argued that SPC-based methods such ascumulative sum are not appropriate to perform surveil-lance for newly introduced products as the aim is todetecta safety problem thatis already presentandnot a sudden change. These authors defend the use ofsuch methodsin the contextof surveillanceforbatch-related problems (problems arising atthe timeTa
ble
2.(C
ontin
ued)
Gene
ric m
etho
dVe
rsio
nGe
nera
l des
crip
tion
Com
para
tor
Adva
ntag
es a
nd d
isadv
anta
ges
Conf
ound
ing
Stat
istica
lpr
oces
s con
trolGe
nera
lde
scrip
tion
Grap
hica
l app
roac
hwh
ere
the
num
ber o
f eve
nts i
sco
mpa
red
with
an
uppe
r lim
it(th
e th
resh
old
is ty
pica
lly—
mea
n+
a ce
rtain
num
ber o
f SD)
.56
Expe
cted
cou
nt.
Adva
ntag
e—Ea
sy to
impl
emen
t.St
ratifi
catio
n ca
n be
use
dto
han
dle
conf
ound
ing.
Disa
dvan
tage
—Le
ss m
etho
dolo
gica
lwo
rk o
n ap
plic
atio
ns to
vac
cine
safe
ty.
No fo
rmal
way
to c
ontro
l for
mul
tiple
test
.
AE, a
dver
se e
vent
; DID
, diff
eren
ce-in
-diff
eren
ce; (
P)M
axSP
RT, (
Poiss
on-b
ased
) max
imize
d pr
obab
ility
ratio
test
; RR,
rela
tive
risk;
SCC
S, se
lf-co
ntro
lled
case
serie
s; S
CRI,
self-
cont
rolle
d ris
k in
terv
al;
SD, s
tand
ard
devi
atio
n; S
PRT,
sequ
entia
l pro
babi
lity
ratio
test
; USP
RT, u
pdat
ing
sequ
entia
l pro
babi
lity
ratio
test
; UL,
upp
er li
mit.
near real-time vaccine safety surveillance 233
© 2016 The Authors. Pharmacoepidemiology and Drug SafetyPublished by John Wiley & Sons Ltd.
Pharmacoepidemiology and Drug Safety, 2016; 25: 225–237DOI: 10.1002/pds
of manufactureratherthan related to theproductitself).However,we should consider thatatthe timeof introduction,if there is a safety problem with thatspecific vaccine and an appropriate comparison groupis used, a sudden change would be observable as well.Given its ease of implantation,SPC is attractive,butrecommendations on the use of SPC are deferred untilfurther research on their properties is available.
Controlfor potentialconfounders has been limitedin both the strategies employed and factors adjustedfor. This observation isin agreementwith Nelsonet al.,12who have argued for better methods for con-founder adjustment, in particular at the analysis stage.Recent work has been performed in this area, adaptinggroup sequentialmethods with regression adjustmentand comparing thisto existing approaches.62,63Tothe best of our knowledge, these promising approachesare stillat the developmentstage and have notyetbeen applied to new studies.As pointed outbyYih,11it might not be possible to adjust for all possi-ble confounders in this setting,which can lead tospurious signals.However,it should be noted that,as a near real-timeanalysis,aimedat quicklyidentifying/strengthening signals,priority isgivento rapid results.As such,confounding adjustment isnot deemedas critical—morecompleteanalysescan be performed atconfirmatory stages.11 Thesemightinclude adjusting foradditionalconfoundersor a more detailed adjustment (e.g.using finer cate-gorization of a variable) to avoid residual confound-ing. The specific confounders to adjust for should bedecided on the basis of the vaccine, outcome and agegroups studied.In addition to those factors consid-ered by studies,adjustmentsfor day-of-the-weekeffects or co-morbidities might be required.11Never-theless,12 studies13,24–27,29,30,35,36did notrefertopotential confounding in at least one of the analysesreported in their published texts.
Best practice using EHR apply equally to NRTVSSas to any study using these kind of data. For example,Lanes et al. provide an approach to identify outcomesin healthcare databases.64One of the aspects to considerwhile doing so is misclassification. In some occasions,manualreview ofindividualmedicalrecords can beused,particularly if a signalis found.In this review,only two studies21,35performed this confirmation beforerunning the NRTVSS analysis, as doing so might delaythe surveillance process.Alternatively,multiple algo-rithms mightbe developed,providing a trade-off be-tween sensitivity and positive predictive values (PPV).In the NRTVSS,an algorithm with higher sensitivityand moderate PPV is generally considered to be timelierthan algorithms with moderate sensitivity algorithm and
high PPV. This should be considered for the specificoutcome understudy,its seriousnessand the dataavailable.65 Misclassification ofthe exposure mightalso be problematic. A possible approach is to restrictthe analysis to vaccinated individuals, avoiding poten-tial biases.11
A key aspect to consider while using these methods isthe availability of timely data. ‘Real-time’ analyses aredifficult to achieve, and thus, the expression ‘near real-time’ is preferred.In fact,delays can occur at variousstages, including delays in diagnosis (e.g. for conditionswith more insidious onset), recording (e.g. retrospectiverecording of vaccination administration or diagnosis),receiving the data for analysis (due to either incompletedata accrualor partially accrued risk windows)andreporting.The timeliness of data should thus be con-sidered.Some studies have delayed the analysis forsome weeks.13,23,25,27,41–43While this approach givestime for data to accrue,it will notreduce the time tosignal.The use of group sequential methods with lessfrequenttesting portraysa similarsituation wheremore time has been given fordata to accrue.35,38,39
Nevertheless,for events occurring closerto the timeof testing,data-accruallags may stillbe problematic.Finally,adjustments for partially elapsed risk intervaland delays in the arrivalof inpatientdata have beenproposed (through the expected numberof events)46
or integrated in the criticallimits calculation36. Thesecan decrease the time to signal,based on previouslyobserved data-accrualpatterns.They have been ap-plied in a few,influenza vaccine,studies.Influenzavaccinespose particularchallengeswhen using de-layed data asfailure to detecta signalbefore theseason ends willimpede adequate action.Strategiesproposed so fardo not specifically addressdelaysbetween illness onset and diagnosis.
Only three of the 11 outcomes identified in the pro-spective studies were confirmed as true signals. In ad-dition to issues already raised (confounding factorsthathave notbeen considered,misclassification ofthe outcome),unconfirmed signalswere due to (i)changes in the true incidence or coding practices; (ii)inappropriate comparison groups;(iii) uncertainty inbackground rates; and (iv) type I errors.11,33For typeI errors,additionalstrategies to reduce the false dis-covery rate are available atthe planning stage:theseinclude delaying the first test,66requiring a minimumnumber of events to occur before rejecting the null hy-pothesis67 or, in the case ofgroup sequentialtests,selecting an O’Brien–Fleming threshold.The latterspends lessalpha in earliertests and wasused byNelson et al.38During the surveillance period, it is im-portant to update the critical limits as data arrive, as the
a. leite et al.234
© 2016 The Authors. Pharmacoepidemiology and Drug SafetyPublished by John Wiley & Sons Ltd.
Pharmacoepidemiology and Drug Safety, 2016; 25: 225–237DOI: 10.1002/pds
observed data might differ from those planned.66As inthe case ofoutcome identification,these consider-ations should be balanced againstthe importance ofdetecting signals in a timely manner. Even after care-ful consideration of all these aspects before and duringsurveillance,possible spurious signals may still arise.This emphasizes the need for a predetermined plan ofaction forsignalrefinementif a signalis found.11
The plan should include a careful decision on the datasource to use to test the hypothesis in subsequent anal-yses if needed, owing to potential biases with the useof the same data to identify and testthe signals.NRTVSS is thus nota stand-alone method butpartof the signal detection and evaluation process.
This review aimed at capturing studies and systemsworldwide using EHR to perform NRTVSS.Our rig-orous search strategy and further contacts with manyexpertson vaccinesafety from differentcountriesand institutions(with a satisfactory responserate,70.6%) should have minimized the risk of missing sys-tems currently in use. However, we cannot exclude theexistence of similar systems elsewhere.Furthermore,some information was missing from the studies in-cluded,which we have tried to reduce by contactingthe authors.The missing information mostoften re-lated to confounding controlstrategies and the data-accruallag adjustmentemployed.This mightreflectthe limited options to address these issues,especiallyfor the earlier studies.
Countries considering introduction of these methodsshould benefitfrom the work developed so farandfrom strategies under development.There should bea cautious reflection on the availability of timely dataand theircharacteristics(including discussion withthe data providers),the vaccine(s) and outcome(s) tobe studied and the infra-structure needed in case a sig-nal is detected.Future directions forresearch mightinclude further development and application of strate-gies for adjustmentfor confounding and data-accruallag, as well as consideration of other methods not yetapplied to observational settings but in use in clinicaltrials,for example,Bayesian approaches to group se-quentialtests.68 Bayesian methodscan incorporateprevious information (such as the data generated bypre-licensure studies) and potentially provide a moreflexible approach.
In conclusion,NRTVSS using EHR to assess thesafety of newly introduced vaccines is being increas-ingly used in the USA,with limited introduction in afew other countries. These methods ensure timely de-tection of safety signals. New methods have been inte-grated over time, but strategies to account for potentialconfounders and data-accruallags have received less
attention.As new vaccines are expected to be intro-duced and the public questionsvaccine safety,thedemand for strong post-licensure surveillance systemswill increase.
CONFLICT OF INTERESTThe authors declare no conflict of interest.
KEY POINTS• Near real-time vaccine safety surveillance using
electronic health records (EHR) is one of the op-tions available to identify vaccine safety signals.
• Use of near real-time vaccine safety surveillanceusing EHR has been increasing in the USA but todate has only been considered in a few othercountries.
• Methods available have developed over time andhave been integrated into systems using this kindof surveillance.Continuoussequentialtestinghas been the preferred approach.
• Strategiesto addresspotentialconfoundingfactors are currently limited, but further develop-ments may address this in the near future.
• Timelinessand allowing fordata-accruallagare importantfactorsfor consideration whenimplementing nearreal-time surveillance usingEHR. Lags haveonly beenaddressedin afew studies.
ETHICS STATEMENTThe authors state that no ethical approval was needed.
ACKNOWLEDGEMENTSThe research was funded by the National Institute forHealthResearchHealthProtectionResearchUnit(NIHR HPRU) in Immunisation at the London Schoolof Hygiene & TropicalMedicine in partnership withPublic Health England (PHE).The views expressedare those of the author(s) and not necessarily those ofthe NHS,the NIHR, the Departmentof Health orPublic Health England. The funders had no role in thestudy design, data collection, analysis or interpretation.The authors thank Dr Abdoulreza Esteghamati (TehranUniversity of MedicalSciences/GACVS),Dr AnandaAmarasinghe (Ministry of Health, Sri Lanka/GACVS),ProfBrigitte Keller-Stanislawski(PaulEhrlich-Institut/GACVS),Bruce Fireman (Kaiser Permanente),ClaireCameron (Health Protection Scotland), Dr Daniel Salmon
near real-time vaccine safety surveillance 235
© 2016 The Authors. Pharmacoepidemiology and Drug SafetyPublished by John Wiley & Sons Ltd.
Pharmacoepidemiology and Drug Safety, 2016; 25: 225–237DOI: 10.1002/pds
(JohnsHopkinsUniversity Schoolof Public Health),Dr David Martin (Food and Drug Administration),Dr Edward Belongia(Marshfield ClinicResearchFoundation), Dr Gagandeep Kang (Christian MedicalCollege),Dr Hanna Nohynek (NationalInstitute forHealth and Welfare),Heather Murdoch (Health Pro-tection Scotland),Jeanne Loughlin (OptumInsight),Jorgen Bauwens (Brighton Collaboration), Dr KatherineDonegan (MHRA), Dr Katherine Yih (Harvard PilgrimHealth Care Institute), Kevin Pollock (Health ProtectionScotland),Lorenz Von Seidlein,Dr Matthew Daley(KaiserPermanente Colorado),Dr Melinda Wharton(Centers for Disease Control and Prevention/GACVS),Dr Nicola Klein (Kaiser Permanente), Ned Lewis (KaiserPermanente),Dr Patrick Garman (US Army),Dr PhilBryan (MHRA),Dr Punam Mangtani (London Schoolof Hygiene & TropicalMedicine),Dr RogerBaxter(Kaiser Permanente), Dr Silvia Perez-Vilar (Foundationfor the Promotion of Health and BiomedicalResearchof Valencia Region), Dr Sharon K. Greene (New YorkCity Departmentof Health and MentalHygiene),ProfStephen Evans (London School of Hygiene & TropicalMedicine),Dr SteveBlack (CincinnatiChildren’sHospital Medical Center), Dr Suzie Seabroke (MHRA),Dr Wan-Ting Huang (Taiwan Centers for Disease Control),Dr Xavier Kurz (European Medicines Agency/GACVS)and allother researchers who generously gave theirtime to answer queries arising from this research.
REFERENCES1. Bonhoeffer J, Black S, Izurieta H, Zuber P, Sturkenboom M. Current status and
future directions of post-marketing vaccine safety monitoring with focus on USAand Europe. Biologicals 2012 Sep; 40: 393–397.
2. Chen RT,Glanz JM, VellozziC. Pharmacoepidemiologic studies ofvaccinesafety.In Pharmacoepidemiology.Strom BL,KimmelSE and Hennessy S(eds). Wiley-Backwell: Chicester, 2011; 423–468.
3. O’Hagan DT, Rappuoli R. The safety of vaccines. Drug Discov Today 2004; 9:846–854.
4. McPhillips H, Marcuse EK. Vaccine safety. Curr Probl Pediatr 2001; 31: 95–121.5. Lopalco PL, Johansen K, Ciancio B, Gomes HC, Kramarz P, Giesecke J. Mon-
itoring and assessing vaccine safety:a European perspective.ExpertRev Vac-cines 2010; 9: 371–380.
6. Dal Pan GJ, Lindquist M, Gelperin K. Postmarketingspontaneouspharmacovigilance reporting systems.In Pharmacoepidemiology.Strom BL,Kimmel SE and HennessyS (eds). Wiley-Backwell:Chicester,2011;137–157.
7. Crawford NW,ClothierH, Hodgson K,SelvarajG, Easton ML,Buttery JP.Active surveillance foradverse eventsfollowing immunization.ExpertRevVaccines 2014; 13: 265–76.
8. Waldman EA, Luhm KR, Monteiro SA, Freitas FR. Surveillance of adverse ef-fectfollowing vaccination and safety ofimmunization programs.Rev SaudePublica 2011; 45: 173–84.
9. Davis RL, Kolczak M, Lewis E, Nordin J, Goodman M, Shay DK, et al. Activesurveillance of vaccine safety: a system to detect early signs of adverse events.Epidemiology 2005; 16: 336–41.
10. Kulldorff M. Sequential statistical methods for prospective postmarketing safetysurveillance. In Pharmacoepidemiology. Strom BL, Kimmel SE and Hennessy S(eds). Wiley-Backwell: Chicester, 2011; 852–867.
11. Yih WK, Kulldorff M, Fireman BH, Shui IM, Lewis EM, Klein NP, et al. Activesurveillance for adverse events:the experience of the Vaccine Safety Datalinkproject. Pediatrics 2011; 127(S1): S54–64.
12. Nelson JC, Cook AJ, Yu O, Zhao S, Jackson LA, Psaty BM. Methods for obser-vationalpost-licensure medicalproductsafety surveillance.StatMethods MedRes 2015; 24: 177–193.
13. Lieu TA, Kulldorff M, Davis RL, Lewis EM,Weintraub E, Yih K, et al. Real-time vaccine safety surveillance for the early detection of adverse events.MedCare 2007; 45(S2): S89–95.
14. BrightonCollaboration.Newsletter.2015;.https://brightoncollaboration.org/public/newsletter.html (accessed 19 June 2015).
15. World Health Organization. The Global Advisory Committee on Vaccine Safety.2015; http://www.who.int/vaccine_safety/committee/en/(accessed19 May2015).
16. Brighton Collaboration.Who we are.2015;https://brightoncollaboration.org/public/who-we-are.html (accessed 19 May 2015).
17. ADVANCE. About ADVANCE. 2015; http://www.advance-vaccines.eu/(accessed 2015-05-19).
18. GovernmentDigitalService.Medicines& HealthcareproductsRegulatoryAgency.2015;https://www.gov.uk/government/organisations/medicines-and-healthcare-products-regulatory-agency (accessed 19 May 2015).
19. Centers forDisease Controland Prevention.Vaccine Safety Datalink (VSD).2015; http://www.cdc.gov/vaccinesafety/Activities/VSD.html (accessed 19 May2015).
20. Statens Serum Institut.Statens Serum Institut.2015; http://www.ssi.dk/english.aspx (accessed 19 May 2015).
21. McNicholas A,Galloway Y,Stehr-Green P,Reid S,Radke S,Sexton K,et al.Post-marketing safety monitoring of a new group B meningococcalvaccine inNew Zealand, 2004–2006. Hum Vacc 2007; 3: 196–204.
22. Brown JS, Moore KM, Braun MM, Ziyadeh N, Chan KA, Lee GM, et al. Activeinfluenza vaccine safety surveillance: potential within a healthcare claims envi-ronment. Med Care 2009; 47: 1251–7.
23. Yih WK, Nordin JD, Kulldorff M, Lewis E, Lieu TA, Shi P, et al. An assessmentof the safety of adolescent and adult tetanus-diphtheria-acellular pertussis (Tdap)vaccine,using active surveillance foradverse eventsin the Vaccine SafetyDatalink. Vaccine 2009; 27: 4257–62.
24. Greene SK, Kulldorff M, Lewis EM, Li R, Yin R, Weintraub ES, et al. Near real-time surveillance for influenza vaccine safety:proof-of-conceptin the VaccineSafety Datalink Project. Am J Epidemiol 2010; 171: 177–88.
25. Belongia EA, Irving SA, Shui IM, Kulldorff M, Lewis E, Yin R, et al. Real-timesurveillance to assess risk of intussusception and other adverse events after pen-tavalent, bovine-derived rotavirus vaccine. Pediatr Infect Dis J 2010; 29: 1–5.
26. National Vaccine Advisory Committee (NVAC). Report on 2009 H1N1 vaccinesafety risk assessment.NVAC. 2010;http://www.hhs.gov/nvpo/nvac/reports/vsrawg_report_apr2010.html (accessed 19 June 2015).
27. Klein NP,Fireman B,Yih WK, Lewis E,Kulldorff M,Ray P,etal. Measles-mumps-rubella-varicella combination vaccine and the risk offebrile seizures.Pediatrics 2010; 126: e1–8.
28. Bryan P, Seabroke S, Davies C. H1N1 vaccine safety: real-time surveillance inthe UK. Lancet 2010; 376: 417–8.
29. Enger C, Turnbull B, Gately R, Loughlin J, Chan KA. H1N1 influenza vaccinesurveillance project. Pharmacoepidemiol Drug Saf 2010; 19: S14–S5.
30. Huang WT, Chen WW, Yang HW, Chen WC, Chao YN, Huang YW,et al. Design ofa robustinfrastructureto monitorthe safety ofthe pan-demicA(H1N1)2009 vaccination program in Taiwan.Vaccine2010;28:7161–6.
31. Bryan P, Seabroke S. No increased risk of febrile convulsions after seasonal in-fluenza immunisation in UK. Lancet 2011; 377: 904.
32. Salmon DA, Akhtar A, Mergler MJ, Vannice KS, Izurieta H, Ball R, et al. Immu-nization-safety monitoring systems forthe 2009 H1N1 monovalentinfluenzavaccination program. Pediatrics 2011; 127(S1): S78–86.
33. Lee GM,Greene SK, Weintraub ES, Baggs J, Kulldorff M, Fireman BH, et al.H1N1 and seasonalinfluenza vaccine safety in the Vaccine Safety Datalinkproject. Am J Prev Med 2011; 41: 121–8.
34. Gee J, Naleway A, Shui I, Baggs J, Yin R, Li R, et al. Monitoring the safety ofquadrivalenthuman papillomavirus vaccine:findings from the Vaccine SafetyDatalink. Vaccine 2011; 29: 8279–84.
35. Loughlin J, Mast TC, DohertyMC, Wang FT, Wong J, SeegerJD.Postmarketing evaluation of the short-term safety of the pentavalentrotavirusvaccine. Pediatr Infect Dis J 2012; 31: 292–6.
36. Burwen DR,Sandhu SK,MaCurdy TE,Kelman JA,GibbsJM, Garcia B,et al. Surveillancefor Guillain-Barresyndromeafterinfluenzavaccinationamong the Medicare population,2009–2010.Am J Public Health 2012;102:1921–7.
37. Tse A,Tseng HF,Greene SK,VellozziC, Lee GM.Signalidentification andevaluation for risk of febrile seizures in children following trivalent inactivatedinfluenza vaccine in the Vaccine Safety Datalink Project,2010–2011.Vaccine2012; 30: 2024–31.
38. Nelson JC, Yu O, Dominguez-Islas CP, Cook AJ, Peterson D, Greene SK, et al.Adapting group sequential methods to observational postlicensure vaccine safetysurveillance: results of a pentavalent combination DTaP-IPV-Hib vaccine safetystudy. Am J Epidemiol 2013; 177: 131–41.
a. leite et al.236
© 2016 The Authors. Pharmacoepidemiology and Drug SafetyPublished by John Wiley & Sons Ltd.
Pharmacoepidemiology and Drug Safety, 2016; 25: 225–237DOI: 10.1002/pds
39. Tseng HF, Sy LS, Liu IL, Qian L, Marcy SM, Weintraub E, et al. Postlicensuresurveillance for pre-specified adverse events following the 13-valent pneumococ-cal conjugate vaccine in children. Vaccine 2013; 31: 2578–83.
40. Donegan K,Beau-Lejdstrom R,King B, Seabroke S,Thomson A,Bryan P.Bivalent human papillomavirus vaccine and the risk of fatigue syndromes in girlsin the UK. Vaccine 2013; 31: 4961–7.
41. Weintraub ES, Baggs J, Duffy J, Vellozzi C, Belongia EA, Irving S, et al. Risk ofintussusception after monovalent rotavirus vaccination. N Engl J Med 2014; 370:513–9.
42. Daley MF, Yih WK, Glanz JM, Hambidge SJ, Narwaney KJ, Yin R, et al. Safetyof diphtheria, tetanus, acellular pertussis and inactivated poliovirus (DTaP-IPV)vaccine. Vaccine 2014; 32: 3019–24.
43. Kawai AT, Li L, Kulldorff M, Vellozzi C, Weintraub E, Baxter R, et al. Absenceof associationsbetween influenza vaccinesand increased risksof seizures,Guillain-Barre syndrome, encephalitis, or anaphylaxis in the 2012–2013 season.Pharmacoepidemiol Drug Saf 2014; 23: 548–53.
44. Murdoch H, McFadden M, Smith-Palmer A, von Wissmann B, Cameron C. Ac-tive monitoring of potentialadverse immunisation events with hospitaladmis-sion data and linked analysis in Scotland. Lancet 2014; 384(S2): S10.
45. Yih K, Zichitella L, Sandhu S, Nguyen M, Kuldorff M, Cole D, et al. Accessingthe freshestfeasibledatafor conductingactiveinfluenzavaccinesafetysurveillance.Mini-Sentinel.2015; http://www.mini-sentinel.org/work_products/PRISM/Mini-Sentinel_PRISM_Active-Influenza-Vaccine-Safety-Surveillance-Report.pdf (accessed 19 June 2015).
46. Greene SK, Kulldorff M, Yin R, Yih WK, Lieu TA, Weintraub ES, et al. Near real-time vaccine safety surveillance with partially accrued data.PharmacoepidemiolDrug Saf 2011; 20: 583–90.
47. Garman P. Enhanced surveillance of novel H1N1 vaccines among military person-nel.Defense MedicalSurveillance System.2009;http://www.fda.gov/downloads/AdvisoryCommittees/CommitteesMeetingMaterials/BloodVaccines-BloodVaccinesandO/VaccinesandRelatedBiologicalProductsAdvisoryCommittee/UCM198078.ppt(accessed 19 June 2015), ND.
48. National Vaccine Advisory Committee (NVAC). H1N1 Vaccine Safety Risk As-sessment Working Group Report. NVAC. 2012; http://www.hhs.gov/nvpo/nvac/re-ports/vsrawg_report_january_2012.pdf (accessed 19 June 2015).
49. SandhuSK. Updateon Surveillancefor Guillain-BarréSyndromeafterVaccination with PandemicInfluenza A/H1N1 2009-containing Vaccines,2009–2011. Vaccines and Related Biological Products Advisory Committee. 2011;http://fda.yorkcast.com/webcast/Viewer/?peid=75dcd91903204870aff160cb9d5528151d (accessed 19 June 2015).
50. Kulldorff M,Davis RL,Kolczak M,Lewis E,Lieu T, PlattR. A maximizedsequentialprobability ratio testfor drug and vaccine safety surveillance.SeqAnal 2011; 30: 58–78.
51. Li L, Kulldorff M. A conditional maximized sequential probability ratio test forpharmacovigilance. Stat Med 2010; 29: 284–95.
52. Ellenberg SS,Fleming TR,DeMets DL.Statistical,philosophicaland ethicalissues in data monitoring. In Data Monitoring Committees in Clinical Trials: APractical Perspective. Wiley: Chicester, 2003; 119–152.
53. Franks R,Sandhu S,Avagyan A,Lu Y, Hong H,Garcia B,etal. Robustnesspropertiesof a sequentialtest for vaccinesafetyin the presenceofmisspecification. Stat Anal Data Min 2014; 7: 368–75.
54. Abt K. Poisson sequential sampling modified towards maximal safety in adverseevent monitoring. Biometrical J 1998; 40: 21–41.
55. Cook AJ, Tiwari RC, Wellman RD, Heckbert SR, Li LL, Heagerty P, et al. Sta-tistical approaches to group sequential monitoring of postmarket safety surveil-lancedata:currentstateof the art for use in the Mini-Sentinelpilot.Pharmacoepidemiol Drug Saf 2012; 21: 72–81.
56. Carey RG,Stake LV.Improving Healthcare with ControlCharts:Basic andAdvanced SPC Methods and Case Studies.ASQ Quality Press:Milwakukee,Wisconsin, 2003.
57. Lewis E, Fireman B,Klein NP, BaxterR. Exactsequentialanalysisforvaccine safety surveillance.12th AnnualConference on Vaccine Research,Bethesda; 2009. http://www.nfid.org/professional-education/archives/acvr/acvr09.pdf(accessed 19 June 2015).
58. Silva IR,KulldorffM. Continuous versus group sequentialanalysis forpost-marketdrug and vaccine safety surveillance.Biometrics2015.doi:10.1111/biom.12324.
59. Jennison C,TurnbullBW. Introduction.In Group SequentialMethodswithApplications to Clinical Trials. CRC Press: Florida, 1999; 1–19.
60. Grigg O, FarewellV, SpiegelhalterD. Use of risk-adjusted CUSUM andRSPRTcharts for monitoring in medical contexts.Stat Methods Med Res 2003;12: 147–70.
61. Musonda P, Hocine MN, Andrews NJ, Tubert-Bitter P, Farrington CP. Monitoringvaccine safety using case series cumulative sum charts. Vaccine 2008; 26: 5358–67.
62. Cook AJ,Wellman RD,Nelson JC,Jackson LA,Tiwari RC.Group sequentialmethod for observational data by using generalized estimating equations: appli-cation to Vaccine Safety Datalink.J Roy Stat Soc C–App 2015; 64: 319–38.
63. Stratton KG, Cook AJ, Jackson LA, Nelson JC. Simulation study comparing ex-posure matching with regression adjustmentin an observationalsafety settingwith group sequential monitoring. Stat Med 2014; 34: 1117–1133.
64. Lanes S, Brown JS, Haynes K, Pollack MF, Walker AM. Identifying health out-comes in healthcare databases. Pharmacoepidemiol Drug Saf 2015; 24: 1009–16.
65. Maro JC, Brown JS, Dal Pan GJ, Kulldorff M. Minimizing signal detection timein postmarket sequential analysis: balancing positive predictive value and sensi-tivity. Pharmacoepidemiol Drug Saf 2014; 23: 839–48.
66. Nelson JC,Cook AJ, Yu O, et al. Challenges in the design and analysis ofsequentially monitored postmarket safety surveillance evaluations using electronicobservational health care data. Pharmacoepidemiol Drug Saf 2012; 21: 62–71.
67. KuldorffM, Silva IR. Continuous post-marketsequentialsafety surveillancewith minimum events to signal. Rev Stat 2015; http://arxiv.org/abs/1503.01978.
68. GsponerT, GerberF, Bornkamp B,Ohlssen D,VandemeulebroeckeM,Schmidli H. A practical guide to Bayesian group sequential designs. Pharm Stat2014; 13: 71–80.
SUPPORTING INFORMATIONAdditional supporting information may be found in theonline version of this article at the publisher’s web site.
near real-time vaccine safety surveillance 237
© 2016 The Authors. Pharmacoepidemiology and Drug SafetyPublished by John Wiley & Sons Ltd.
Pharmacoepidemiology and Drug Safety, 2016; 25: 225–237DOI: 10.1002/pds