auc silver spring

Discrimination measures for survival outcomes:Connection between the AUC and the predictiveness

curve

V. Viallon, A. Latouche

University Lyon 1University Versailles Saint Quentin

October 13, 2011

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 1 / 22

1 The binary outcome settingThe AUCThe predictiveness curveConnecting the AUC and the predictiveness curve

2 The survival outcome settingTime-dependent outcomes definitionsConnecting time-dependent AUC and time-dependent predictivenesscurveEstimates of the time-dependent AUCSome synthetic examples

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 2 / 22

AUC for binary outcomes

Let D be a (0, 1)-variable representing the status regarding a givendisease:

D = 1 for diseased individuals;D = 0 for non-diseased individuals.

Further let X be a continuous marker. For any c ∈ IR

X > c: the test is positive;X ≤ c: the test is negative.TPR(c) = IP(X > c|D = 1) and FPR(c) = IP(X > c|D = 0).

ROC curve:{(

FPR(c),TPR(c)), c ∈ IR

}AUC:

∫TPR(c)dFRP(c).

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 3 / 22

Binary outcomes: a toy example

FPR(c) = IP(X > c|D = 0)

TPR(c) = IP(X > c|D = 1)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

FPR

TP

R

0 1

−4

−2

02

46

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 4 / 22

Binary outcomes: a toy example

FPR(c) = IP(X > c|D = 0)

TPR(c) = IP(X > c|D = 1)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

FPR

TP

R

0 1

−4

−2

02

46

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 5 / 22

Binary outcomes: a toy example

FPR(c) = IP(X > c|D = 0)

TPR(c) = IP(X > c|D = 1)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

FPR

TP

R

0 1

−4

−2

02

46

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 6 / 22

Binary outcomes: a toy example

FPR(c) = IP(X > c|D = 0)

TPR(c) = IP(X > c|D = 1)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

FPR

TP

R

0 1

−4

−2

02

46

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 7 / 22

Binary outcomes: a toy example

FPR(c) = IP(X > c|D = 0)

TPR(c) = IP(X > c|D = 1)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

FPR

TP

R

0 1

−4

−2

02

46

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 8 / 22

Predictiveness curve for binary outcomes

Many alternative criteria have been proposed for evaluatingdiscrimination

proportion of explained variation,standardized total gainrisk reclassification measures (Pencina et al., SiM, 2006)

Most of them express as simple functions of the predictiveness curve(Gu and Pepe, Int. J. Biostatistics, 2009).Denote by G−1 the quantile function of X . For any q ∈ [0, 1], let

R(q) = P[D = 1|X = G−1(q)

]be the risk associated to the qth quantile of X .

The predictiveness curve plots R(q) versus q.

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 9 / 22

Predictiveness curves and their corresponding AUC values

With p = IP(D = 1) =∫ 10 R(q)dq=0.5

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Quantiles

Pre

dic

tive

ness C

urv

e

R1 (AUC=0.500)

R2 (AUC=0.700)

R3 (AUC=0.833)

R4 (AUC=0.928)

R5 (AUC=1.000)

A flat predictiveness curve, R(q) = p, where p is the proportion of events,is associated to an AUC of 0.5

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 10 / 22

The relation in the binary outcome setting

Still denote by R the predictiveness curve of marker X ,R(q) = P

[D = 1|X = G−1(q)

].

Then, denoting by p = IP(D = 1) =∫ 10 R(q)dq the disease

prevalence, the AUC of marker X is given by

AUC =

∫ 10 qR(q)dq − p2/2

p(1− p)

We can check that

AUC = 0.5 when R(q) = p;AUC = 1 when R(q) = 1I[1−p,1](q).

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 11 / 22

Extensions to survival outcomes

In prospective cohort study, the outcome (e.g., the disease status) canchange over time⇒ we consider time-dependent outcomes, TPR, FPR, ROC curves,AUC and predictiveness curve.

Notations:

Ti and Ci : survival and censoring times for subject i(Zi , δi )1≤i≤n with Zi = min(Ti ,Ci ) and δi = 1I(Ti ≤ Ci )Di (t): time-dependent outcome status for subject i at time t.

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 12 / 22

Heagerty and Zheng’s Taxonomy

Today’s talk focus on Cumulative cases & Dynamic controls:

cumulative cases: Di (t) = 1 if Ti ≤ t;

dynamic controls Di (t) = 0 if Ti > t;

so that Di (t) = 1I{Ti ≤ t}.

⇒discrimination between subjects who had the event prior to time tand those who were still event-free at time t.

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 13 / 22

Cumulative cases and Dynamic controls

For a given evaluation time t0

Cumulative true positive rates are

TPRC(c, t0) = IP(X > c|D(t0) = 1)

= IP(X > c|T ≤ t0);

Dynamic false positive rates are

FPRD(c, t0) = IP(X > c|D(t0) = 0)

= IP(X > c|T > t0);

AUCC,D(t0) =∫∞−∞ TPRC(c, t0)d

[FPRD(c, t0)

].

But 1I(Ti ≤ t0) is not observed for all i due to censoring!!

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 14 / 22

Cumulative cases and Dynamic controls

For a given evaluation time t0

Cumulative true positive rates are

TPRC(c, t0) = IP(X > c|D(t0) = 1)

= IP(X > c|T ≤ t0);

Dynamic false positive rates are

FPRD(c, t0) = IP(X > c|D(t0) = 0)

= IP(X > c|T > t0);

AUCC,D(t0) =∫∞−∞ TPRC(c, t0)d

[FPRD(c, t0)

].

But 1I(Ti ≤ t0) is not observed for all i due to censoring!!

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 14 / 22

Workaround for AUCC,D

Using Bayes’s theorem (see, e.g., Chambless & Diao)

AUCC,D(t0) =

∫ ∞−∞

∫ ∞c

F (t0;X = x)[1− F (t0;X = c)]

[1− F (t0)]F (t0)g(x)g(c)dxdc

with

F (t) = IP(T ≤ t) be the risk function at time t;

F (t;X = x) = IP(T ≤ t|X = x) be the conditional risk function attime t;

g the density function of marker X .

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 15 / 22

Predictiveness curve and AUCC,D

Introduce

R(t; q) := IP(D(t) = 1|X = G−1(q))

= IP(T ≤ t|X = G−1(q))

the time-dependent predictiveness curve

We established that

AUCC,D(t0) =

∫ 10 qR(t0; q)dq − F (t0)2

2

F (t0)[1− F (t0)]

Proper estimation of R(t0; q) (especially for q ' 1) should yieldproper estimation of AUCC,D(t0)

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 16 / 22

Deriving estimates for AUCC,D(t)

G and g : cdf and pdf of X .

X(i): i-th order statistic of (X1, . . . ,Xn).

F̂n(t0; x): estimator of the conditional risk F (t0;X = x).

Using the change of variable x = G−1(q), we have∫ 10 qR(t0; q)dq =

∫∞−∞ G (x)F (t0;X = x)g(x)dx , so that

1

n

n∑i=1

i

nF̂n(t0;X(i)),

is the empirical counterpart of the∫ 10 qR(t0; q)dq.

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 17 / 22

Deriving estimates for AUCC,D(t)

To estimate the marginal risk function F

the KM estimator F̂n,(1)(t0).

Since F (t0) =∫F (t0; x)g(x)dx , we can also use

F̂n,(2)(t0) =1

n

n∑i=1

F̂n(t0;Xi ).

This yields two estimators for AUCC,D(t0), namely, for k = 1, 2,

AUCC,Dn,(k)(t0) =

1n

∑ni=1

in F̂n(t0;X(i))− F̂ 2

n,(k)(t0)/2

F̂n,(k)(t0)[1− F̂n,(k)(t0)

] .

Experimental results (not shown) suggested better performances resultsobtained with k = 2.

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 18 / 22

Simulation study

X : continuous marker

T : survival time generated under

a Cox model λ(t) = λ0(t) exp(αX )a TV coeff. Cox model λ(t) = λ0(t) exp(α(t)X )

Various censoring schemes were considered.

Estimation of the cond. risk F (t;X = x)

a Cox modelan Aalen additive modelconditional KM estimator

Goal: assess the effect of model misspecification – when estimatingthe conditional risk function– on the AUCC,D(t) estimation.

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 19 / 22

Simulation under a Cox model

0 1 2 3 4 5

0.5

0.6

0.7

0.8

0.9

1.0

Time

AU

C C

/DTrue

HLP

KM cond.

Add. Aalen

Cox

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 20 / 22

Simulation under a Time-varying coefficient Cox model

0.2 0.4 0.6 0.8 1.0 1.2

0.5

0.6

0.7

0.8

0.9

1.0

Time

AU

C C

/DTrue

HLP

KM cond.

Add. Aalen

Cox

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 21 / 22

Conclusion

Relation between the predictiveness curve and the AUCC,D(t).

Enables to easily derive estimates of the AUCC,D(t) given estimates ofthe cond. risk function.

Correctly specifying the model (when estimating the cond. riskfunction) is crucial to get proper estimation of AUCC,D(t);

A similar relation can be obtained to connect the partial AUC (orpartial AUCC,D(t)) to the (time-dependent) predictiveness curve;

The conditional risk function, through the predictiveness curve, is thekey when assessing discrimination of prognostic tools

Viallon (Univ. Lyon 1) Connecting the pred. curve and the AUC 22 / 22

Sketch of the proof

∫ ∞−∞

∫ ∞c

F (t; X = x)[1− F (t; X = c)]g(x)g(c)dxdc

=

∫ 1

0

∫ 1

vF (t; X = G−1(u))[1− F (t; X = G−1(v))]dudv

=

∫ 1

0

∫ 1

v[1− S(t; X = G−1(u))]S(t; X = G−1(v))dudv

=

∫ 1

0(1− v)S(t; X = G−1(v))dv −

∫ 1

0

∫ 1

vS(t; X = G−1(u))S(t; X = G−1(v))dudv

=

∫ 1

0(1− v)S(t; X = G−1(v))dv −

∫ 1

0

∫ 1

0S(t; X = G−1(u))S(t; X = G−1(v))1I(u ≥ v)dudv.

Setting

L(u, v) = S(t; X = G−1(u))S(t; X = G−1(v)),

we have L(u, v) = L(v, u) so that

∫ ∞−∞

∫ ∞c

F (t; X = x)[1− F (t; X = c)]g(x)g(c)dxdc

=

∫ 1

0(1− v)S(t; X = G−1(v))dv −

1

2

∫ 1

0

∫ 1

0S(t; X = G−1(u))S(t; X = G−1(v))dudv

=

∫ 1

0(1− v)S(t; X = G−1(v))dv −

1

2

(∫ 1

0S(t; X = G−1(v))dv

)2

.

Simulation under a Time-varying coefficient Cox model

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Quantile of marker

Pre

dic

tive

ne

ss c

urv

e

True

KM cond.

Add. Aalen

Cox

0.0 0.2 0.4 0.6 0.8 1.00

.00

.20

.40

.60

.81

.0

Quantile of marker

Pre

dic

tive

ne

ss c

urv

e

True

KM cond.

Add. Aalen

Cox