Download - Auc silver spring
Discrimination measures for survival outcomes:Connection between the AUC and the predictiveness
curve
V. Viallon, A. Latouche
University Lyon 1University Versailles Saint Quentin
October 13, 2011
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 1 / 22
1 The binary outcome settingThe AUCThe predictiveness curveConnecting the AUC and the predictiveness curve
2 The survival outcome settingTime-dependent outcomes definitionsConnecting time-dependent AUC and time-dependent predictivenesscurveEstimates of the time-dependent AUCSome synthetic examples
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 2 / 22
AUC for binary outcomes
Let D be a (0, 1)-variable representing the status regarding a givendisease:
D = 1 for diseased individuals;D = 0 for non-diseased individuals.
Further let X be a continuous marker. For any c ∈ IR
X > c: the test is positive;X ≤ c: the test is negative.TPR(c) = IP(X > c|D = 1) and FPR(c) = IP(X > c|D = 0).
ROC curve:{(
FPR(c),TPR(c)), c ∈ IR
}AUC:
∫TPR(c)dFRP(c).
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 3 / 22
Binary outcomes: a toy example
FPR(c) = IP(X > c|D = 0)
TPR(c) = IP(X > c|D = 1)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
FPR
TP
R
0 1
−4
−2
02
46
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 4 / 22
Binary outcomes: a toy example
FPR(c) = IP(X > c|D = 0)
TPR(c) = IP(X > c|D = 1)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
FPR
TP
R
0 1
−4
−2
02
46
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 5 / 22
Binary outcomes: a toy example
FPR(c) = IP(X > c|D = 0)
TPR(c) = IP(X > c|D = 1)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
FPR
TP
R
0 1
−4
−2
02
46
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 6 / 22
Binary outcomes: a toy example
FPR(c) = IP(X > c|D = 0)
TPR(c) = IP(X > c|D = 1)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
FPR
TP
R
0 1
−4
−2
02
46
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 7 / 22
Binary outcomes: a toy example
FPR(c) = IP(X > c|D = 0)
TPR(c) = IP(X > c|D = 1)
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
FPR
TP
R
0 1
−4
−2
02
46
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 8 / 22
Predictiveness curve for binary outcomes
Many alternative criteria have been proposed for evaluatingdiscrimination
proportion of explained variation,standardized total gainrisk reclassification measures (Pencina et al., SiM, 2006)
Most of them express as simple functions of the predictiveness curve(Gu and Pepe, Int. J. Biostatistics, 2009).Denote by G−1 the quantile function of X . For any q ∈ [0, 1], let
R(q) = P[D = 1|X = G−1(q)
]be the risk associated to the qth quantile of X .
The predictiveness curve plots R(q) versus q.
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 9 / 22
Predictiveness curves and their corresponding AUC values
With p = IP(D = 1) =∫ 10 R(q)dq=0.5
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Quantiles
Pre
dic
tive
ness C
urv
e
R1 (AUC=0.500)
R2 (AUC=0.700)
R3 (AUC=0.833)
R4 (AUC=0.928)
R5 (AUC=1.000)
A flat predictiveness curve, R(q) = p, where p is the proportion of events,is associated to an AUC of 0.5
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 10 / 22
The relation in the binary outcome setting
Still denote by R the predictiveness curve of marker X ,R(q) = P
[D = 1|X = G−1(q)
].
Then, denoting by p = IP(D = 1) =∫ 10 R(q)dq the disease
prevalence, the AUC of marker X is given by
AUC =
∫ 10 qR(q)dq − p2/2
p(1− p)
We can check that
AUC = 0.5 when R(q) = p;AUC = 1 when R(q) = 1I[1−p,1](q).
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 11 / 22
Extensions to survival outcomes
In prospective cohort study, the outcome (e.g., the disease status) canchange over time⇒ we consider time-dependent outcomes, TPR, FPR, ROC curves,AUC and predictiveness curve.
Notations:
Ti and Ci : survival and censoring times for subject i(Zi , δi )1≤i≤n with Zi = min(Ti ,Ci ) and δi = 1I(Ti ≤ Ci )Di (t): time-dependent outcome status for subject i at time t.
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 12 / 22
Heagerty and Zheng’s Taxonomy
Today’s talk focus on Cumulative cases & Dynamic controls:
cumulative cases: Di (t) = 1 if Ti ≤ t;
dynamic controls Di (t) = 0 if Ti > t;
so that Di (t) = 1I{Ti ≤ t}.
⇒discrimination between subjects who had the event prior to time tand those who were still event-free at time t.
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 13 / 22
Cumulative cases and Dynamic controls
For a given evaluation time t0
Cumulative true positive rates are
TPRC(c, t0) = IP(X > c|D(t0) = 1)
= IP(X > c|T ≤ t0);
Dynamic false positive rates are
FPRD(c, t0) = IP(X > c|D(t0) = 0)
= IP(X > c|T > t0);
AUCC,D(t0) =∫∞−∞ TPRC(c, t0)d
[FPRD(c, t0)
].
But 1I(Ti ≤ t0) is not observed for all i due to censoring!!
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 14 / 22
Cumulative cases and Dynamic controls
For a given evaluation time t0
Cumulative true positive rates are
TPRC(c, t0) = IP(X > c|D(t0) = 1)
= IP(X > c|T ≤ t0);
Dynamic false positive rates are
FPRD(c, t0) = IP(X > c|D(t0) = 0)
= IP(X > c|T > t0);
AUCC,D(t0) =∫∞−∞ TPRC(c, t0)d
[FPRD(c, t0)
].
But 1I(Ti ≤ t0) is not observed for all i due to censoring!!
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 14 / 22
Workaround for AUCC,D
Using Bayes’s theorem (see, e.g., Chambless & Diao)
AUCC,D(t0) =
∫ ∞−∞
∫ ∞c
F (t0;X = x)[1− F (t0;X = c)]
[1− F (t0)]F (t0)g(x)g(c)dxdc
with
F (t) = IP(T ≤ t) be the risk function at time t;
F (t;X = x) = IP(T ≤ t|X = x) be the conditional risk function attime t;
g the density function of marker X .
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 15 / 22
Predictiveness curve and AUCC,D
Introduce
R(t; q) := IP(D(t) = 1|X = G−1(q))
= IP(T ≤ t|X = G−1(q))
the time-dependent predictiveness curve
We established that
AUCC,D(t0) =
∫ 10 qR(t0; q)dq − F (t0)2
2
F (t0)[1− F (t0)]
Proper estimation of R(t0; q) (especially for q ' 1) should yieldproper estimation of AUCC,D(t0)
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 16 / 22
Deriving estimates for AUCC,D(t)
G and g : cdf and pdf of X .
X(i): i-th order statistic of (X1, . . . ,Xn).
F̂n(t0; x): estimator of the conditional risk F (t0;X = x).
Using the change of variable x = G−1(q), we have∫ 10 qR(t0; q)dq =
∫∞−∞ G (x)F (t0;X = x)g(x)dx , so that
1
n
n∑i=1
i
nF̂n(t0;X(i)),
is the empirical counterpart of the∫ 10 qR(t0; q)dq.
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 17 / 22
Deriving estimates for AUCC,D(t)
To estimate the marginal risk function F
the KM estimator F̂n,(1)(t0).
Since F (t0) =∫F (t0; x)g(x)dx , we can also use
F̂n,(2)(t0) =1
n
n∑i=1
F̂n(t0;Xi ).
This yields two estimators for AUCC,D(t0), namely, for k = 1, 2,
AUCC,Dn,(k)(t0) =
1n
∑ni=1
in F̂n(t0;X(i))− F̂ 2
n,(k)(t0)/2
F̂n,(k)(t0)[1− F̂n,(k)(t0)
] .
Experimental results (not shown) suggested better performances resultsobtained with k = 2.
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 18 / 22
Simulation study
X : continuous marker
T : survival time generated under
a Cox model λ(t) = λ0(t) exp(αX )a TV coeff. Cox model λ(t) = λ0(t) exp(α(t)X )
Various censoring schemes were considered.
Estimation of the cond. risk F (t;X = x)
a Cox modelan Aalen additive modelconditional KM estimator
Goal: assess the effect of model misspecification – when estimatingthe conditional risk function– on the AUCC,D(t) estimation.
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 19 / 22
Simulation under a Cox model
0 1 2 3 4 5
0.5
0.6
0.7
0.8
0.9
1.0
Time
AU
C C
/DTrue
HLP
KM cond.
Add. Aalen
Cox
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 20 / 22
Simulation under a Time-varying coefficient Cox model
0.2 0.4 0.6 0.8 1.0 1.2
0.5
0.6
0.7
0.8
0.9
1.0
Time
AU
C C
/DTrue
HLP
KM cond.
Add. Aalen
Cox
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 21 / 22
Conclusion
Relation between the predictiveness curve and the AUCC,D(t).
Enables to easily derive estimates of the AUCC,D(t) given estimates ofthe cond. risk function.
Correctly specifying the model (when estimating the cond. riskfunction) is crucial to get proper estimation of AUCC,D(t);
A similar relation can be obtained to connect the partial AUC (orpartial AUCC,D(t)) to the (time-dependent) predictiveness curve;
The conditional risk function, through the predictiveness curve, is thekey when assessing discrimination of prognostic tools
Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 22 / 22
Sketch of the proof
∫ ∞−∞
∫ ∞c
F (t; X = x)[1− F (t; X = c)]g(x)g(c)dxdc
=
∫ 1
0
∫ 1
vF (t; X = G−1(u))[1− F (t; X = G−1(v))]dudv
=
∫ 1
0
∫ 1
v[1− S(t; X = G−1(u))]S(t; X = G−1(v))dudv
=
∫ 1
0(1− v)S(t; X = G−1(v))dv −
∫ 1
0
∫ 1
vS(t; X = G−1(u))S(t; X = G−1(v))dudv
=
∫ 1
0(1− v)S(t; X = G−1(v))dv −
∫ 1
0
∫ 1
0S(t; X = G−1(u))S(t; X = G−1(v))1I(u ≥ v)dudv.
Setting
L(u, v) = S(t; X = G−1(u))S(t; X = G−1(v)),
we have L(u, v) = L(v, u) so that
∫ ∞−∞
∫ ∞c
F (t; X = x)[1− F (t; X = c)]g(x)g(c)dxdc
=
∫ 1
0(1− v)S(t; X = G−1(v))dv −
1
2
∫ 1
0
∫ 1
0S(t; X = G−1(u))S(t; X = G−1(v))dudv
=
∫ 1
0(1− v)S(t; X = G−1(v))dv −
1
2
(∫ 1
0S(t; X = G−1(v))dv
)2
.
Simulation under a Time-varying coefficient Cox model
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Quantile of marker
Pre
dic
tive
ne
ss c
urv
e
True
KM cond.
Add. Aalen
Cox
0.0 0.2 0.4 0.6 0.8 1.00
.00
.20
.40
.60
.81
.0
Quantile of marker
Pre
dic
tive
ne
ss c
urv
e
True
KM cond.
Add. Aalen
Cox