model selection in dose-response meta-analysis of summarized … · department of public health...
TRANSCRIPT
Model selection in dose-response meta-analysis ofsummarized data
Nicola Orsini, PhD
Biostatistics TeamDepartment of Public Health Sciences
Karolinska Institutet
2019 Nordic and Baltic Stata Users Group meeting, Stockholm
August 30, 2019
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 1 / 34
Outline
• Background
• Aim
• Simulation study
• Results
• Summary
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 2 / 34
Background
• A dose-response analysis describes the changes of a response acrosslevels of a quantitative factor. The quantitative factor could be anadministered drug or an exposure.
• A meta-analysis of dose-response (exposure-disease) relations aims atidentifying the trend underlying multiple studies trying to answer thesame research question.
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 3 / 34
Increasing number of dose-response meta-analyses
0
20
40
60
80
100
120
140
160
180P
ub
lish
ed
do
se
−re
sp
on
se
me
ta−
an
aly
sis
1992
1994
1996
1998
2000
2002
2004
2006
2008
2010
2012
2014
2016
Publication year
Data source: Web of Science
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 4 / 34
Current applications
• Potassium intake in relation to blood pressure levels in adultpopulation
• Antipsychotic drugs in relation to symptoms in acute schizophreniapatients
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 5 / 34
Example of summarized data from 5 studies
+--------------------------------------+| id md semd dose n sd ||--------------------------------------|| 1 0.0 0.0 2.7 500 30.3 || 1 0.9 1.9 7.6 500 29.7 ||--------------------------------------|| 2 0.0 0.0 2.1 334 27.9 || 2 -2.9 2.3 4.4 333 29.3 || 2 4.9 2.3 8.8 333 30.0 ||--------------------------------------|| 3 0.0 0.0 2.6 500 30.5 || 3 4.1 1.9 7.5 500 30.9 ||--------------------------------------|| 4 0.0 0.0 2.7 500 30.1 || 4 1.5 2.0 7.6 500 31.8 ||--------------------------------------|| 5 0.0 0.0 2.0 334 31.9 || 5 2.6 2.4 4.3 333 30.5 || 5 2.9 2.4 8.4 333 29.4 ||--------------------------------------|
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 6 / 34
Linear Mixed Model
A one-stage approach for meta-analysis of summarized dose-response datahas been proposed in the general framework of linear mixed model (StatMeth Med Res, 2019).
γ̂ i = X iβ + Zibi + εi
γ̂ i is the vector of empirical constrasts (mean differences) estimated in thei-th study
X i is the design matrix for the fixed-effects β
It is implemented in the drmeta command (Type ssc install drmeta).
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 7 / 34
Random effects and residual error term
bi ∼ N (0,Ψ)
The random-effects bi represent study-specific deviations from thepopulation average dose-response coefficients β.
Z i is the analogous design matrix for the random-effects.
The residual error term εi ∼ N (0,Si ), whose variance matrix Si isassumed known.
Si can be either given or approximated using available summarized data(BMC Med Res Meth, 2016).
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 8 / 34
Splines according to the research question Am J Epi, 2012
0.8
1.0
1.2
1.5
2.0
Rate
ratio
of c
olor
ecta
l can
cer
0 10 20 30 40 50 60 70
a) Restricted cubic splines
0.8
1.0
1.2
1.5
2.0
0 10 20 30 40 50 60 70
b) Piecewise linear
0.8
1.0
1.2
1.5
2.0
Rate
ratio
of c
olor
ecta
l can
cer
0 10 20 30 40 50 60 70Alcohol intake, grams/day
c) Piecewise constant
0.8
1.0
1.2
1.5
2.0
0 10 20 30 40 50 60 70Alcohol intake, grams/day
d) Mix of splines
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 9 / 34
Aim
• Explore the ability of the Akaike Information Criterion (AIC) tosuggest the correct functional relationship using linear mixed modelsfor meta-analysis of summarized dose-response data.
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 10 / 34
Sketch of the Monte-Carlo simulation
• Generate multiple individual data according to a certain dose-responserelationship
• Create a table of summarized data upon categorization of the dose
• Fit a linear mixed-effects model on the summarized data usingalternative dose-response functions
• Tag the dose-response functions associated with lowest AIC
• Repeat the steps above a large number of times
• Examine the frequency of correctly identified dose-responserelationships
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 11 / 34
Design matrix
Since the γ̂ i is a set of response contrasts relative to the baseline dose xi0,X i needs to be constructed in a similar way by centering the ptransformations of the dose levels to the corresponding values in xi0.
Let consider, for example, a transformation g ; the generic j-th row of X i
would be defined as g(xij)− g(xi0).
As a consequence X i does not contain the intercept term (γ̂ i = 0 forx = xi0).
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 12 / 34
Random effects and residual error term
bi ∼ N (0,Ψ)
The random-effects bi represent study-specific deviations from thepopulation average dose-response coefficients β.
Z i is the analogous design matrix for the random-effects.
The residual error term εi ∼ N (0,Si ), whose variance matrix Si isassumed known.
Si can be either given or approximated using available summarized data(BMC Med Res Meth, 2016).
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 13 / 34
Regression splines (cubic) are very popular (AJE, 2012)
0.8
1.0
1.2
1.5
2.0
Rate
ratio
of c
olor
ecta
l can
cer
0 10 20 30 40 50 60 70
a) Restricted cubic splines
0.8
1.0
1.2
1.5
2.0
0 10 20 30 40 50 60 70
b) Piecewise linear
0.8
1.0
1.2
1.5
2.0
Rate
ratio
of c
olor
ecta
l can
cer
0 10 20 30 40 50 60 70Alcohol intake, grams/day
c) Piecewise constant
0.8
1.0
1.2
1.5
2.0
0 10 20 30 40 50 60 70Alcohol intake, grams/day
d) Mix of splines
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 14 / 34
The point is
• dose-response meta-analysis are likely to be published in top journalsand highly influential
• given the limited number of data points, can you really trust theresults of selected models?
• what are the chances of misleading conclusions/artefacts?
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 15 / 34
Aim
• Explore the ability of the Akaike Information Criterion (AIC) tosuggest the correct functional relationship using linear mixed modelsfor meta-analysis of summarized dose-response data.
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 16 / 34
Sketch of the Monte-Carlo simulations
• Generate multiple individual data according to a certain dose-responserelationship
• Create a table of summarized data upon categorization of the dose
• Fit a linear mixed-effects model on the summarized data usingalternative dose-response functions
• Tag the dose-response functions associated with lowest AIC
• Repeat the steps above a large number of times
• Examine the frequency of correctly identified dose-responserelationships
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 17 / 34
Simulating individual data for a single study
Random values X drawn from a χ2 distribution with 5 degrees of freedom
Random values Y drawn according the the following functions
Linear function Sl
Y = β0 + β1x + ε
Quadratic function Sq
Y = β0 + β1x + β2x2 + ε
with ε ∼ N(0, 30).
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 18 / 34
Mechanism generating data
Common-effect. Regression coefficients are fixed constant across studies
E (Y |x) = 10 + 0.5x
E (Y |x) = 10 + 0.5x − 0.5x2
Random-effects. Regression coefficients (β1, β2)T across studies arevectors randomly drawn from a multivariate normal with specified meansand var/covariance structures
β1 ∼ N(0.5, .1)(β1β2
)∼ MVN
((0.5−0.5
),
(0.1 0.05
0.05 0.1
))
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 19 / 34
Create a table of summarized data
Quantiles. Dose is categorized into quantiles (2, 3). Mean dose withineach quantile is assigned to each dose interval.
Measure of effect. Differences in mean responses (std errors) comparingeach dose interval relative to the baseline dose using a linear regressionmodel.
Additional basic information. Sample size and sample standarddeviation of the response for each dose interval.
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 20 / 34
A single simulated study from E (Y |x) = 10 + 0.5x−
10
0−
50
05
01
00
Re
sp
on
se
0 5 10 15 20 25Dose
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 21 / 34
A single simulated study from E (Y |x) = 10 + 0.5x − 0.5x2
−6
00
−4
00
−2
00
02
00
Re
sp
on
se
0 5 10 15 20 25Dose
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 22 / 34
Estimation
We consider estimation methods based on maximum likelihood (ML). Thelog-likelihood for the linear mixed model is defined as
` (β, ξ) = −1
2n log(2π)− 1
2
k∑i=1
log |Σi (ξ) |+
− 1
2
k∑i=1
[(γ̂ i − Xiβ)>Σi (ξ)−1 (γ̂ i − Xiβ)
]where n =
∑ki=1 ni and ξ is the vector of the variance components in Ψ to
be estimated.
Number of studies included in the simulated dose-response meta-analysisis k = 10.
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 23 / 34
Candidate Models
Linear function Ml
γ̂ij = (β1 + b1i )(xij − xi0) + εij
Restricted cubic spline function Ms
γ̂ij = (β1 + b1i )[g1(xij)− g1(xi0)] + (β2 + b2i )[g2(xij)− g2(xi0)] + εij
with three knots (k1, k2, k3) at fixed percentiles (10th, 50th, 90th) of thedose is defined only in terms of p = 2 regression coefficients (AJE, 2012).The two splines are
g1(xij) = xij
g2(xij) =(xij − k1)3+ −
k3−k1k3−k2 (xij − k2)3+ + k2−k1
k3−k2 (xij − k3)3+(k3 − k1)2
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 24 / 34
Definition of Akaike Information Criteria
AIC = −2`(β̂, ξ̂) + 2(p + q)
`(β̂, ξ̂) maximized log-likelihood using ML method
p number of fixed effects (Ml = 1; Ms = 2)
q number of variance/covariance components (Ml = 1; Ms = 3)
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 25 / 34
Performance measures
Proportion of simulated dose-response meta-analysis for which theminimum AIC corresponds to the true data-generating mechanism.
If data are generated under Sl (linear)
Pl =
∑[min{AICl ,AICs} = AICl ]
nsim
If data are generated under Sq (quadratic)
Ps =
∑[min{AICl ,AICs} = AICs ]
nsim
nsim = 1, 000
AICl and AICs correspond to the candidate models Ml and Ms ,respectively.
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 26 / 34
Results I: True dose-response shape is linear (Sl)
Table: Proportion (Pl) of correctly identified linear (Sl) dose-responserelationships according to different categorizations of the dose and datagenerating mechanism.
Common-effect Random-effects
2 Doses 0.99 0.98Mix 2/3 Doses 0.98 0.97
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 27 / 34
Results II: True dose-response shape is quadratic (Sq)
Table: Proportion (Ps) of correctly identified non-linear (Sq) dose-responserelationships according to different categorizations of the dose and datagenerating mechanism.
Common-effect Random-effects
2 Doses 0.03 0.26Mix 2/3 Doses 0.99 0.97
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 28 / 34
How many studies with just two doses?
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Pe
rfo
rma
mce
of
the
AIC
0
50
100
150
200
250
Fre
qu
en
cy
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Proportion of studies with 2 categories in the dose−response meta−analysis
Data generated under a random and non−linear mechanism
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 29 / 34
What about increasing from k = 10 to k = 30 the numberof studies included in each dose-response meta-analysis?
Table: Proportion (Ps) of correctly identified non-linear (Sq) dose-responserelationships according to different categorizations of the dose and datagenerating mechanism.
Common-effect Random-effects
2 Doses 0.12 0.13Mix 2/3 Doses 1.00 1.00
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 30 / 34
Are 1,000 predicted dose-response models of type Ml
estimating the right shape under Sl?
E (Y |X = x)− E (Y |X = 2) = 0.5(X − 2)
Settings: Random-effects mechanism, truly linear, mix of 2/3 doses.Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 31 / 34
Are 1,000 predicted dose-response models of type Ms
estimating the right shape under Sq?
E (Y |X = x)− E (Y |X = 2) = 0.5(X − 2)− 0.5(X − 22)
Settings: Random-effects mechanism, truly quadratic, mix of 2/3 doses.Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 32 / 34
Summary
• We evaluated the performance of the AIC based on linear mixedmodels (ML method) suitable for summarized data in realisticMonte-Carlo simulations.
• If the dose-response relationship underlying multiple studies is linear,the AIC is very good in suggesting linearity even when all studiescategorize the dose into two quantiles.
• If the dose-response relationship underlying multiple studies isnon-linear (quadratic), the AIC is very bad in suggesting non-linearitywhen all studies categorize the dose into two quantiles.
• In such a case, a mix of studies categorizing the dose into either 2 or3 quantiles increased substantially the performance of the AIC.
• Model selection was not sensitive to the data-generating mechanism(common-effect, random-effects) of the individual studies.
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 33 / 34
References
• Crippa A, Discacciati A, Bottai M, Spiegelman D, Orsini N.One-stage dose-response meta-analysis for aggregated data. StatMethods Med Res. 2019 May;28(5):1579-1596.
• Crippa A, Thomas I, Orsini N. A pointwise approach todose-response meta-analysis of aggregated data. International Journalof Statistics in Medical Research. 2018 May 8;7(2):25-32.
• Discacciati A, Crippa A, Orsini N. Goodness of fit tools fordose-response meta-analysis of binary outcomes. Research SynthesisMethods. 2017 Jun;8(2):149-160.
• Crippa A, Orsini N. Multivariate Dose-Response Meta-Analysis: thedosresmeta R Package. 2016. J Stat Softw. Vol. 72.
• Crippa A, Orsini N. Dose-response meta-analysis of differences inmeans. BMC Med Res Methodol. 2016 Aug 2;16(1):91.
Orsini N (PHS, KI) Dose-response meta-analysis August 30, 2019 34 / 34