auc silver spring

Discrimination measures for survival outcomes:Connection between the AUC and the predictiveness

curve

V. Viallon, A. Latouche

University Lyon 1University Versailles Saint Quentin

October 13, 2011

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 1 / 22

1 The binary outcome settingThe AUCThe predictiveness curveConnecting the AUC and the predictiveness curve

2 The survival outcome settingTime-dependent outcomes definitionsConnecting time-dependent AUC and time-dependent predictivenesscurveEstimates of the time-dependent AUCSome synthetic examples


AUC for binary outcomes

Let D be a (0, 1)-variable representing the status regarding a givendisease:

D = 1 for diseased individuals;D = 0 for non-diseased individuals.

Further let X be a continuous marker. For any c ∈ IR

X > c: the test is positive;X ≤ c: the test is negative.TPR(c) = IP(X > c|D = 1) and FPR(c) = IP(X > c|D = 0).

ROC curve:{(

FPR(c),TPR(c)), c ∈ IR

}AUC:

∫TPR(c)dFRP(c).


Binary outcomes: a toy example

FPR(c) = IP(X > c|D = 0)

TPR(c) = IP(X > c|D = 1)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

FPR

TP

R

0 1

−4

−2

02

46



FPR(c) = IP(X > c|D = 0)

TPR(c) = IP(X > c|D = 1)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

FPR

TP

R

0 1

−4

−2

02

46


Predictiveness curve for binary outcomes

Many alternative criteria have been proposed for evaluatingdiscrimination

proportion of explained variation,standardized total gainrisk reclassification measures (Pencina et al., SiM, 2006)

Most of them express as simple functions of the predictiveness curve(Gu and Pepe, Int. J. Biostatistics, 2009).Denote by G−1 the quantile function of X . For any q ∈ [0, 1], let

R(q) = P[D = 1|X = G−1(q)

]be the risk associated to the qth quantile of X .

The predictiveness curve plots R(q) versus q.


Predictiveness curves and their corresponding AUC values

With p = IP(D = 1) =∫ 10 R(q)dq=0.5

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Quantiles

Pre

dic

tive

ness C

urv

e

R1 (AUC=0.500)

R2 (AUC=0.700)

R3 (AUC=0.833)

R4 (AUC=0.928)

R5 (AUC=1.000)

A flat predictiveness curve, R(q) = p, where p is the proportion of events,is associated to an AUC of 0.5


The relation in the binary outcome setting

Still denote by R the predictiveness curve of marker X ,R(q) = P

[D = 1|X = G−1(q)

].

Then, denoting by p = IP(D = 1) =∫ 10 R(q)dq the disease

prevalence, the AUC of marker X is given by

AUC =

∫ 10 qR(q)dq − p2/2

p(1− p)

We can check that

AUC = 0.5 when R(q) = p;AUC = 1 when R(q) = 1I[1−p,1](q).


Extensions to survival outcomes

In prospective cohort study, the outcome (e.g., the disease status) canchange over time⇒ we consider time-dependent outcomes, TPR, FPR, ROC curves,AUC and predictiveness curve.

Notations:

Ti and Ci : survival and censoring times for subject i(Zi , δi )1≤i≤n with Zi = min(Ti ,Ci ) and δi = 1I(Ti ≤ Ci )Di (t): time-dependent outcome status for subject i at time t.


Heagerty and Zheng’s Taxonomy

Today’s talk focus on Cumulative cases & Dynamic controls:

cumulative cases: Di (t) = 1 if Ti ≤ t;

dynamic controls Di (t) = 0 if Ti > t;

so that Di (t) = 1I{Ti ≤ t}.

⇒discrimination between subjects who had the event prior to time tand those who were still event-free at time t.


Cumulative cases and Dynamic controls

For a given evaluation time t0

Cumulative true positive rates are

TPRC(c, t0) = IP(X > c|D(t0) = 1)

= IP(X > c|T ≤ t0);

Dynamic false positive rates are

FPRD(c, t0) = IP(X > c|D(t0) = 0)

= IP(X > c|T > t0);

AUCC,D(t0) =∫∞−∞ TPRC(c, t0)d

[FPRD(c, t0)

].

But 1I(Ti ≤ t0) is not observed for all i due to censoring!!


Workaround for AUCC,D

Using Bayes’s theorem (see, e.g., Chambless & Diao)

AUCC,D(t0) =

∫ ∞−∞

∫ ∞c

F (t0;X = x)[1− F (t0;X = c)]

[1− F (t0)]F (t0)g(x)g(c)dxdc

with

F (t) = IP(T ≤ t) be the risk function at time t;

F (t;X = x) = IP(T ≤ t|X = x) be the conditional risk function attime t;

g the density function of marker X .


Predictiveness curve and AUCC,D

Introduce

R(t; q) := IP(D(t) = 1|X = G−1(q))

= IP(T ≤ t|X = G−1(q))

the time-dependent predictiveness curve

We established that

AUCC,D(t0) =

∫ 10 qR(t0; q)dq − F (t0)2

2

F (t0)[1− F (t0)]

Proper estimation of R(t0; q) (especially for q ' 1) should yieldproper estimation of AUCC,D(t0)


Deriving estimates for AUCC,D(t)

G and g : cdf and pdf of X .

X(i): i-th order statistic of (X1, . . . ,Xn).

F̂n(t0; x): estimator of the conditional risk F (t0;X = x).

Using the change of variable x = G−1(q), we have∫ 10 qR(t0; q)dq =

∫∞−∞ G (x)F (t0;X = x)g(x)dx , so that

1

n

n∑i=1

i

nF̂n(t0;X(i)),

is the empirical counterpart of the∫ 10 qR(t0; q)dq.


Deriving estimates for AUCC,D(t)

To estimate the marginal risk function F

the KM estimator F̂n,(1)(t0).

Since F (t0) =∫F (t0; x)g(x)dx , we can also use

F̂n,(2)(t0) =1

n

n∑i=1

F̂n(t0;Xi ).

This yields two estimators for AUCC,D(t0), namely, for k = 1, 2,

AUCC,Dn,(k)(t0) =

1n

∑ni=1

in F̂n(t0;X(i))− F̂ 2

n,(k)(t0)/2

F̂n,(k)(t0)[1− F̂n,(k)(t0)

] .

Experimental results (not shown) suggested better performances resultsobtained with k = 2.


Simulation study

X : continuous marker

T : survival time generated under

a Cox model λ(t) = λ0(t) exp(αX )a TV coeff. Cox model λ(t) = λ0(t) exp(α(t)X )

Various censoring schemes were considered.

Estimation of the cond. risk F (t;X = x)

a Cox modelan Aalen additive modelconditional KM estimator

Goal: assess the effect of model misspecification – when estimatingthe conditional risk function– on the AUCC,D(t) estimation.


Simulation under a Cox model

0 1 2 3 4 5

0.5

0.6

0.7

0.8

0.9

1.0

Time

AU

C C

/DTrue

HLP

KM cond.

Add. Aalen

Cox


Simulation under a Time-varying coefficient Cox model

0.2 0.4 0.6 0.8 1.0 1.2

0.5

0.6

0.7

0.8

0.9

1.0

Time

AU

C C

/DTrue

HLP

KM cond.

Add. Aalen

Cox


Conclusion

Relation between the predictiveness curve and the AUCC,D(t).

Enables to easily derive estimates of the AUCC,D(t) given estimates ofthe cond. risk function.

Correctly specifying the model (when estimating the cond. riskfunction) is crucial to get proper estimation of AUCC,D(t);

A similar relation can be obtained to connect the partial AUC (orpartial AUCC,D(t)) to the (time-dependent) predictiveness curve;

The conditional risk function, through the predictiveness curve, is thekey when assessing discrimination of prognostic tools


Sketch of the proof

∫ ∞−∞

∫ ∞c

F (t; X = x)[1− F (t; X = c)]g(x)g(c)dxdc

=

∫ 1

0

∫ 1

vF (t; X = G−1(u))[1− F (t; X = G−1(v))]dudv

=

∫ 1

0

∫ 1

v[1− S(t; X = G−1(u))]S(t; X = G−1(v))dudv

=

∫ 1

0(1− v)S(t; X = G−1(v))dv −

∫ 1

0

∫ 1

vS(t; X = G−1(u))S(t; X = G−1(v))dudv

=

∫ 1

0(1− v)S(t; X = G−1(v))dv −

∫ 1

0

∫ 1

0S(t; X = G−1(u))S(t; X = G−1(v))1I(u ≥ v)dudv.

Setting

L(u, v) = S(t; X = G−1(u))S(t; X = G−1(v)),

we have L(u, v) = L(v, u) so that

∫ ∞−∞

∫ ∞c

F (t; X = x)[1− F (t; X = c)]g(x)g(c)dxdc

=

∫ 1

0(1− v)S(t; X = G−1(v))dv −

1

2

∫ 1

0

∫ 1

0S(t; X = G−1(u))S(t; X = G−1(v))dudv

=

∫ 1

0(1− v)S(t; X = G−1(v))dv −

1

2

(∫ 1

0S(t; X = G−1(v))dv

)2

.

Simulation under a Time-varying coefficient Cox model

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Quantile of marker

Pre

dic

tive

ne

ss c

urv

e

True

KM cond.

Add. Aalen

Cox

0.0 0.2 0.4 0.6 0.8 1.00

.00

.20

.40

.60

.81

.0

Quantile of marker

Pre

dic

tive

ne

ss c

urv

e

True

KM cond.

Add. Aalen

Cox

auc silver spring

Education

survival outcome settingtime

evaluation time t0

deriving estimates

dynamic controls

cumulative cases

ti t0

risk function

survival outcomes