Adjusting the Generalized ROC Curve for Covariates

By

Enrique F. Schisterman1, David Faraggi2 and Benjamin Reiser2

1. Division of Epidemiology, Statistics and Prevention, NICHD, NIH.

2. Department of Statistics, University of Haifa, Mount Carmel, Haifa, Israel

SUMMARY

Receiver Operating Characteristic (ROC) curves and in particular the area under the

curve (AUC), are widely used to examine the effectiveness of diagnostic markers.

Diagnostic markers and their corresponding ROC curves can be strongly influenced by

covariate variables. When several diagnostic markers are available, they can be combined

by a best linear combination such that the area under the ROC curve of the combination

is maximized among all possible linear combinations. In this paper we discuss covariate

effects on this linear combination assuming that the multiple markers, possibly

transformed, follow a multivariate normal distribution. The ROC curve of this linear

combination when markers are adjusted for covariates is estimated and approximate

confidence intervals for the corresponding AUC are derived. An example of two

biomarkers of coronary heart disease for which covariate information on age and gender

is available is used to illustrate this methodology.

Key words: diagnostic markers, Box-Cox transformations, best linear combination,

sensitivity, specificity.

2

1. Introduction

This paper deals with obtaining linear combinations of multiple continuous markers

adjusted for covariate information in order to better distinguish between healthy and

diseased populations.

The effectiveness of continuous markers in distinguishing between healthy and

diseased subjects is generally assessed through the use of the Receiver Operating

Characteristic (ROC) curve [1]. A subject is assessed as diseased (positive) or healthy

(negative) according to whether the subject's marker value is greater than or less than or

equal to a specified threshold value. Associated with any threshold value is the

probability of a true positive (sensitivity) and the probability of a true negative

(specificity). The resulting theoretical ROC curve is the plot of sensitivity versus

1-specificity for all possible threshold values.

The ROC curve can be estimated from sample data taken on both diseased (Y) and

healthy (X) subjects. This estimation can be carried out under parametric or non-

parametric assumptions [1]-[3].

A commonly used global summary measure of marker accuracy is the area under the

ROC curve (AUC). Bamber [4] proved that AUC=P(Y>X) with larger values of AUC

indicating higher diagnostic accuracy. The functional P(Y>X) appears in many statistical

problems not connected with marker evaluation [5]. Both parametric and nonparametric

procedures have been suggested for statistical inference on AUC [1], [5]. Faraggi and

Reiser [6] compare a number of procedures for estimating the AUC.

The effectiveness of a continuous marker can be influenced by covariates/factors such

as age, gender, general health status etc. The ROC curve itself and the summary index

AUC can be adjusted for covariate effects by regression modeling of the relationship

between the marker and the covariates [7], [8]. An alternative approach can be based on

modeling the ROC curve [9]. Faraggi [8] discusses some of the advantages in directly

modeling the markers.

When multiple markers are available a comparison of the areas under the different

ROC curve is often used to decide on which marker is best. Su and Liu [10]

recommended that instead of trying to decide on single marker one should use a linear

3

combination of all the markers. They discuss choosing a best linear combination for

which the area under the corresponding ROC curve is maximized. Reiser and Faraggi

[11] derived a confidence interval for this maximal area. This maximal area, which they

called the generalized ROC criterion, provides a measure of how well the vector of

markers distinguishes between the healthy and diseased groups. The ROC curve

corresponding to this linear combination can be termed the generalized ROC curve.

Further discussion of these linear combinations and some examples of their use can be

found in [12]-[14]. Alternative approaches to Su and Liu [10] for obtaining optimal

combinations of diagnostic markers are discussed in Baker [15] and McIntosh and Pepe

[16].

In this paper we discuss how the Su and Liu methodology can be adjusted to account

for covariate effects. In Section 2 we discuss a motivating example that deals with

oxidative stress and antioxidant biomarkers for cardiovascular disease. In Section 3 we

derive the covariate adjustments by extending the Su and Liu procedure [10]. We further

show how confidence intervals can be obtained for the generalized ROC criterion

conditional on given covariate values. In Section 4 we apply this methodology to the

example and in Section 5 provide concluding remarks.

The methodology developed in this paper is based on the assumption that the marker

values are normally distributed. When data analysis indicates that this assumption is

untenable a power transformation of the Box-Cox [17] type can be used to improve the

normal fit. This approach has been found effective in estimating the AUC and ROC

curves in a wide variety of cases [6], [12], [18]-[21].

2. Example: Oxidative Stress and Antioxidant Biomarkers

Biomarkers of individual oxidative stress and antioxidant status have been suggested

for discriminating between individuals with and without coronary heart disease (CHD)

[22].

Schisterman et al. [14] discuss data from a population-based sample of randomly

selected residents of New York State’s Erie and Niagara counties that provides

information on a number of biomarkers. We consider, for illustrative purposes, only the

4

markers TBARS (thiobarbuturic acid reacting substances) and TEAC (trolox equivalent

antioxidant capacity). Data are available for 45 diseased and 891 healthy individuals.

Examination of the data shows that both the TBARS and TEAC distributions are strongly

non-symmetric. In order to improve the normality of the data the TBARS and TEAC

values were taken to the power of –1 and 4 respectively. These powers were obtained by

applying the Box-Cox method of estimating transformations. Figure 1 provides the

estimated ROC curves for each transformed marker separately. It also presents the

generalized ROC curve for their best linear combination obtained following [10]. The

corresponding AUCs for TBARS, TEAC and their linear combination are 0.695, 0.622

and 0.751 respectively. The difference between the AUCs of TBARS and TEAC was

tested using the methodology of Liu and Schisterman [23] and found to be marginally

significant (p-value=0.05). Both the ROC curves and their AUCs show that TBARS is

better than TEAC while the linear combination resulted in an improvement in separating

the healthy and diseased populations.

However, covariate information is available for the subjects in this study. More

specifically information on the gender and age of each subject were obtained. Gender and

age may influence the discriminatory accuracy of the markers themselves and their linear

combination. In the following section we present the theory for adjusting the best linear

combination for covariates and then in Section 4 apply this theory to the data under

consideration.

3. Covariate Adjustment of the Linear Combination of Markers

3.1 Notation and Assumptions

Let X and Y be p dimensional column vectors denoting p different markers on the

healthy and diseased groups respectively on which samples of size m ( )miX i ,...,1, = and

n ( )njYj ,...,1, = are available. Further let ixZ represent a vector of size xr whose first

element is 1 and whose remaining 1−xr elements are observations on 1−xr explanatory

variables for the thi healthy subject. We assume for the healthy subjects that the marker

vector is related to the covariate vector by the usual multivariate regression model

5

ii xxXi ZBX ε+= (1)

where XB is a p by xr matrix composed of unknown coefficients and ixε is the residual

vector of size p such that ixε , mi ,...,1= are independently distributed as ( )xN Σ,0 with

0 being a vector of p zeros and xΣ a positive definite p by p residual covariance

matrix. We denote the design matrix corresponding to the model (1) by the m by xr

matrix )',...,,( '''21 mxxxx ZZZZ = , the first column of which is composed of ones.

Similarly to (1) we assume that the jY (diseased subjects) follow

jj yyYj ZBY ε+= (2).

Letting 1−yr represent the number of explanatory variables for the diseased subjects,

YB is a p by yr matrix where jyZ has yr elements. The n independent

jyε vectors are

( )yN Σ,0 variates and are further assumed to be independent of the ixε . The

corresponding n by yr design matrix is denoted by )',...,,( '''21 nyxyy ZZZZ = . Note that the

explanatory variables associated with the healthy subjects do not need to be identical to

those of the diseased subjects.

Note that the explanatory variables may include interaction terms or other functions of

the covariates which may be of interest. Standard model selection techniques for

multivariate regression can be employed as described for example in Rencher [24, pp

382-391]. Residual analysis should be used to examine the model assumptions [25].

3.2 The Covariate Adjusted Optimal Linear Combination

Let 0xZ and

0yZ be column vectors of size xr and yr of given covariate values

corresponding to X and Y respectively. Both have ones as the first element. Conditional

on these given covariate values

6

( )000

| xXxx ZBZXE == µ

and (3)

( )000

| yYyy ZBZYE == µ

while the conditional variances remain as xΣ and yΣ respectively. We want to find the

best linear combination XaU '= and YaV '= in the sense of having the largest AUC

over all linear combinations, conditional on 0xZ and

0yZ .

The results of Su and Liu [10] can be applied to show that

( ) µ1−Σ+Σ= yxa (4)

where 00 xy µµµ −= with corresponding

( )( ) 2/11' µµ −Σ+ΣΦ= yxAUC (5)

where Φ denotes the standard normal cumulative distribution function.

The AUC, µ and a are functions of the given covariate vectors 0xZ and

0yZ but for

notational convenience we do not write them as such explicitly.

The corresponding sensitivity (TP) and specificity (TN) are for given threshold value

C and given covariates 0xZ and

0yZ

⎟⎟

⎠

⎞

⎜⎜

⎝

⎛

Σ

−Φ=

aa

CaTP

y

y

'

'0

µ (6)

and

7

⎟⎟⎠

⎞⎜⎜⎝

⎛

Σ

−Φ=

aa

aCTN

x

x

'

'0

µ (7).

The conditional ROC curve for covariates 0xZ and

0yZ is the plot of TP versus 1-TN

for all possible values of C . This can alternatively be written as

( ) ( )

⎟⎟

⎠

⎞

⎜⎜

⎝

⎛

Σ

−ΦΣ+Φ=

−

aaTNaaa

TPy

x

'1'' 1µ

(8).

Thus (8) represents the covariate adjusted generalized ROC curve.

For estimation of these ROC curves and their corresponding AUCs we consider two

cases, (I) Σ=Σ=Σ yx and (II) yx Σ≠Σ .

Let XB and YB denote the standard least squares estimators of XB and YB while xS

and yS denote the multivariate regression residual sum of squares matrices for models

(1) and (2) respectively. Set 00

ˆˆ xXx ZB=µ , 00

ˆˆ yYy ZB=µ and 00

ˆˆˆ xy µµµ −= .

3.2.1 Case I ( Σ=Σ=Σ yx )

Set 2

' 1µµδ−Σ

= , then ( )2/1δΦ=AUC . Since AUC is monotonically related to δ , a

confidence interval for AUC readily follows from that for δ . yx

yxP rrnm

SSS

−−+

+=

provides an unbiased estimate of Σ=Σ=Σ yx . The linear combination coefficients given

by (4), the generalized ROC criteria (5) and the corresponding generalized ROC curve (8)

are estimated by substituting PS for xΣ and yΣ and µ for µ . All these estimators are

for given covariates 0xZ and

0yZ .

8

Confidence intervals for AUC can now be obtained similarly to the argument used in

[11] but in our case conditional on the given 0xZ and

0yZ . Letting W denote the Wishart

distribution, it follows from standard multivariate regression theory that

( ) ( )Σ−−+−−+ ,,~ prrnmWSrrnm yxPyx (9)

independent of

( )Σ== 200000

,~ˆˆ xxXxxXx aZBNZB µµ and ( )Σ== 200000

,~ˆˆ yyYyyYy aZBNZB µµ where

( )000

1''2xXXxx ZZZZa −

= and ( )000

1''2yYYyy ZZZZa −

= . This results in

( )Σ+−= )(,~ˆˆˆ 220000 yxxy aaN µµµµ (10)

or

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛Σ

++,1~ˆ1

22220000

µµyxyx aa

Naa

independently of PS . Consequently the Hotelling

2T statistic is

δµµ ˆ2'ˆ122

122

2

0000 yxP

yx aaS

aaT

+=

+= −

and using the standard connection between 2T and non central F variates we obtain that

)(~ˆ))(()1(2ˆ

1,22*

00

λδδ +−−−+−−++

+−−−+= prrnmp

yxyx

yxyx

Fprrnmaa

prrnm (11)

where the non centrality parameter 2200

2

yx aa +=

δλ .

Applying Lam [26] to (11) we obtain a confidence interval for λ and hence AUC (via

δ ) by numerically solving

9

21ˆ2Pr *

221,00

αδδ−=

⎟⎟

⎠

⎞

⎜⎜

⎝

⎛≤⎟

⎟⎠

⎞⎜⎜⎝

⎛

++−−−+yx

prrnmp aaFob

yx (12)

2ˆ2Pr *

221,00

αδδ=

⎟⎟

⎠

⎞

⎜⎜

⎝

⎛≤⎟

⎟⎠

⎞⎜⎜⎝

⎛

++−−−+yx

prrnmp aaFob

yx (13)

for δ and δ respectively. The resulting interval )](),([2/12/1 δδ ΦΦ provides a α−1

confidence interval for δ . If ( )( )*1,

ˆ0Pr δ<+−−−+ prrnmp yxFob is less than 2/1 α− [ 2/α ]

then there is no solution for (12) [(13)] and δ [δ ] is assigned the value zero.

3.2.1 Case II ( yx Σ≠Σ )

For this general case

)( 2/1δΦ=AUC (14)

for µµδ 1' −Σ= C , yxC Σ+Σ=Σ . x

xx rm

S−

=Σ and y

yy rn

S−

=Σ provide unbiased

estimates of xΣ and yΣ while yxC Σ+Σ=Σ ˆˆˆ estimates CΣ unbiasedly. Letting

µµδ ˆˆ'ˆˆ 1−Σ= C and substituting estimates for the parameters in (4), (8) and (14) provides

estimates for the best linear combination and its corresponding ROC curve and AUC.

The confidence interval for the AUC is more complicated in this case due to the

inequality of the residual covariance matrices and resembles the multivariate Behrens-

Fisher problem. We follow [11] and use approximations developed in the literature for

the Behrens- Fisher problem. For the general case ),(~ˆ 2200 yyxx aaN Σ+Σµµ

),,(~ xxx prmWS Σ− and ),,(~ yyy prnWS Σ− , all independently distributed. Thus

10

( ) ( ))/(,,)/(,,~ˆyyyxxxC rnprnWrmprmW −Σ−+−Σ−Σ .

Following [11] and the references they cite, we consider that approximately

⎟⎠⎞

⎜⎝⎛ Σ

MN C,~ˆ µµ & (15)

independently of

f ( )CC pfW ΣΣ ,,~ˆ & (16)

where

( )

( )yyxx

C

aatrtr

MΣ+Σ

Σ= 22

00

(17)

and the formula for f is given below. The above formulae (15) and (16) are of the same

form as (10) and (9) obtained for case I. In parallel to (11) we obtain that approximately

)(~1ˆˆ1,

* δδδ MFppf

fM

pfp +−⎟⎟⎠

⎞⎜⎜⎝

⎛ +−= & (18).

Based on a simulation study [11] recommended that f be estimated following [27]

whose method gives for our situation the formula

( ) { } { }⎥⎥⎦

⎤

⎢⎢⎣

⎡ΣΣΣ

−+ΣΣΣ

−Σ= −−−−−− 2121212121

00'1'1'1 µµµµµµ WyyW

yWxxW

xW a

rna

rmf (19)

where yyxxW aa Σ+Σ=Σ 2200

.

11

In order to obtain confidence intervals )](),([2/12/1 δδ ΦΦ for the AUC we have from

(18)

( )( )2

1ˆPr *1,

αδδ −=≤+− MFob pfp (20)

( )( )2

ˆPr *1,

αδδ =≤+− MFob pfp (21)

which need to solved numerically for δ and δ . These parallel (12) and (13). M and f in

(20) and (21) are obtained by the obvious substitutions in (17) and (19).

4. The Example Revisited

Multivariate regression was carried out on the TBARS and TEAC data using the

transformations given in Section 2. Age, gender and the age-gender interaction were used

as explanatory variables. For the diseased (Y) subjects both the interaction term (p-

value=0.198) and gender (p-value=0.13) were not found to be significant while for the

healthy (X) subjects all three terms were significant (p-value<0.001). Scatter plots of the

residuals for both the healthy and diseased groups show the “cloud” pattern typical for

normality (Figure 2a). Other residual analyses such as Q-Q plots show no reason to reject

the normality assumption for the transformed markers (Figure 2b). Consequently the

covariate-adjusted linear combinations were calculated as described in Section 3.2.2. A

referee raised the possibility that the residual covariance matrices in the multivariate

regression models (1) and (2) may depend on the covariates and thus violate the linear

model assumptions of variance homogeneity. An examination of the pattern of the

residuals when plotted against age (for males and females separately) found no violation

of the variance homogeneity assumption. For brevity these graphs are not presented.

Table I presents the weights (a’s) for the transformed TEAC and TBARS variables for

both females and males as a function of age. Figures 3a and 3b provide graphs of the

12

AUC for the best linear combination as a function of age along with point wise 95%

confidence intervals, which give an indication of the variability of the estimation process.

Figures 4a and 4b show the AUCs for both markers individually as well as their linear

combination adjusted for age for females and males respectively.

In Section 2 we found, when ignoring the age and gender, that the AUC for the linear

combination of TEAC and TBARS was 0.751. Figures 3a and 3b show that in fact the

AUCs differ greatly with age and that those for males are larger than those for females of

the same age. In addition for both males and females the AUCs first decrease and then

increase with age. For many ages the AUCs are quite higher than the ‘overall’ value of

0.751. The relatively short confidence intervals show the accuracy of the area estimates.

The weights of TBARS and TEAC (transformed) vary with age for both genders.

Their relative influence on the linear combination is made clear in Figures 4a and 4b. For

females younger than about 45 the AUC for TBARS alone is as good as that of the linear

combination and is substantially larger than that of TEAC alone. For females older than

65, TEAC alone does about as well as the combination and better than TBARS alone.

The linear combination does better than each marker alone for woman in the 45-65 age

group. The picture for males is similar although for older males the linear combination

still tends to show improvement over each marker separately.

The estimated adjusted ROC curves for both females and males are plotted in Figures

5a and 5b along with their corresponding AUCs for the linear combination of TBARS

and TEAC at ages 45, 55, 65 and 75. It is clear that for a given age the discrimination

between the healthy and diseased groups is better for males than for females. In addition

we see the strong effect of age. Comparing Figures 5a and 5b with the generalized ROC

curve in Figure 1 shows that ignoring gender and age will lead to misleading conclusions

about the discriminatory effectiveness of combining TBARS and TEAC.

5. Concluding Remarks

In many applications multiple diagnostic markers are available. Since different

markers may be sensitive to different aspects of the disease been studied, creating a

“new” marker as a linear combination of the multiple markers provides a simple readily

13

implementable procedure for improving diagnostic capability. Since diagnostic markers

will frequently be subject to covariate effects it becomes important to adjust the

combination process for covariates. The theory presented in Section 3 shows how this

adjustment can be carried out using standard multivariate regression modeling

techniques.

Schisterman et al. [14] consider a different procedure for handling explanatory

variables such as age and gender. They do not distinguish between the biomarkers and

the explanatory variables but treat them as a vector of markers and compute a linear

combination of all of them using the Su and Liu procedure. This approach has several

difficulties: (i) binary variables such as gender cannot be transformed to have an

approximate normal distribution (ii) the weighting given to each biomarker is fixed and

does not depend on the covariate values. As seen in our example such a dependency can

be very meaningful.

Although the theory in Section 3 is restricted by the normality assumption it can be

extended to many non-normal situations by using the Box-Cox type transformations.

Standard residual analysis methods should be used to examine the fit of the model.

Once a generalized ROC curve adjusted for covariates values of interest is estimated

not only the AUC but also other indices of interest such as the partial area and Youden

Index as well as the critical decision threshold value can be readily obtained.

Acknowledgement

We would like to thank the reviewers for their helpful comments

14

References

1. Zhou, X.H., Obuchowski, N.A. and McClish, D.K. (2002). Statistical Methods in Diagnostic Medicine. Wiley. New York.

2. Shapiro, D.E. (1999). The Interpretation of Diagnostic Tests. Statistical Methods in Medical Research, 8, 113-134.

3. Zou, K.H., Hall, W.J. and Shapiro, D.E. (1997). Smooth Non-Parametric Receiver Operating Characteristic (ROC) Curves for Continuous Diagnostic Tests. Statistics in Medicine, 16, 2143-2156.

4. Bamber, D.C. (1975). The Area Above the Ordinal Dominance Graph and the Area Below the Receiver Operating Characteristic Graph. Journal Math. Psychol. 12, 387-415.

5. Kotz, S., Lumelskii, Y. and Pensky, M. (2003). The Stress-Strength Model and Its Generalizations. World Scientific. Singapore.

6. Faraggi, D. and Reiser, B. (2002). Estimation of the Area Under the ROC Curve. Statistics in Medicine. 21, 3093-3106.

7. Smith, P.J., and Thompson, T.J. (1996). Correcting for Confounding in Analyzing Receiver Operating Characteristic Curves. Biometrical Journal, 38, 857-863.

8. Faraggi, D. (2003). Adjusting ROC Curves and Related Indices for Covariates. Journal of the Royal Statistical Society - Series D - The Statistician, 52, 179-192.

9. Pepe, M.S. (2000). An Interpretation for the ROC curve and inference using GLM procedures. Biometrics, 56, 352-359.

10. Su, J.Q. and Liu, J.S. (1993). Linear Combinations of Multiple Diagnostic Markers. Journal of the American Statistical Association 88, 1350-1355.

11. Reiser, B. and Faraggi, D. (1997). Confidence Intervals for the Generalized ROC Criterion. Biometrics 53, 644-652.

12. Kramar, A., Faraggi, D., Ychou, M., Reiser, B. and Grenier, J. (1999). Criteres ROC Generalises Pour L’evaluation de Plusieurs Marqueurs Tumoraux. Revue d’Epidemiologie et de Sante Publique. 47, 217-226.

13. Kramar, A., Faraggi, D., Fortune, A. and Reiser, B. (2001). mROC: a computer program for combining tumour markers in predicting disease states. Computer Methods and Programs in Biomedicine. l66, 199-207.

14. Schisterman, E., Faraggi, D., Browne, R., Freudenhime, J., Dorn, J., Muti, P., Armstrong, D., Reiser, B. and Trevisan, M. (2002). Minimal and Best Linear Combination of Oxidative Stress and Antioxidants Biomarkers in Discriminate Cardiovascular Disease. Nutrition, Metabolism and Cardiovascular Diseased, 12, 259-266.

15. Baker S. G. (2000). Identifying Combination of Cancer Markers for Further Study as Triggers of Early Intervention. Biometrics, 56, 1082-1087.

16. McIntoch, M. W. and Pepe, M. S. (2002). Combining Several Screening Tests: Optimality of the Risk Score. Biometrics, 58, 657-664.

17. Box, G. E. P. and Cox, D. R. (1964). An Analysis of Transformations. Journal of the Royal Statistical Society, Series B, 26, 211-243.

18. O'Malley, A. and Zou, K.H. (2002). Bayesian hierarchical transformation models for receiver operating characteristic curve analysis. ASA Proceedings of the Joint Statistical Meetings, 2541-2549.

15

19. Zou, K.H., and Hall, W.J. (2002). Semiparametric and parametric transformation models for comparing diagnostic markers with paired design. Journal of Applied Statistics, 29 (6), 803-816.

20. Zou, K.H. and Hall, W.J. (2000). Two transformation models for estimating an ROC curve derived from continuous data'', Journal of Applied Statistics, 27 (5), 621-631.

21. Zou, K.H., Tempany, C.M., Fielding, J.R., and Silverman, S.G. (1998). Original Smooth Receiver Operating Characteristic Curves Estimation from Continuous Data: Statistical Methods for Analyzing the Predictive Value of Spiral CT of Ureteral Stones. Academic Radiology, 5, 680-687.

22. Amstrong, D. (2002). Oxidative Stress Biomarkers and Antioxidant Protocols. Humana Press, New Jersey.

23. Liu, A. and Schisterman, E. F. (2003). Comparison of Diagnostic Accuracy of Biomarkers With Pooled Assessments. 45, 631-644.

24. Rencher, A. C. (1995). Methods of Multivariate analysis. Wiley, New York. 25. Gnanadesikan, R. (1997). Methods for Statistical Data Analysis of Multivariate

Observations. Wiley, New York. 26. Lam, Y.M. (1987). Confidence Limits for Noncentrality Parameters of Noncentral

Chi-Squared and F Distributions. ASA Proceedings of Statistical Computing Section, 441-443.

27. Yao, Y. (1965). An approximate degrees of freedom solution to the multivariate Behrens-Fisher problem. Biometrika, 52, 139-147.

16

Table I. Weights for Best Linear Combination

Age Females Males

TEAC TBARS TEAC TBARS

30 0.1528 6.256 -0.3583 8.832

40 -0.0664 4.725 -0.4639 6.440

50 -0.2856 3.195 -0.5695 4.047

60 -0.5047 1.665 -0.6751 1.655

70 -0.7239 0.134 -0.7806 -0.737

80 -0.9431 -1.396 -0.8862 -3.129

17

Figure 1. Unadjusted ROC Curves

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

1-specificity

Sen

sitiv

ity

TEAC

TBARS

LinearCombination

18

Figure 2. Residual analysis Healthy Diseased

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

-3 -2 -1 0 1 2 3 4

Residuals TEAC

Res

idua

ls T

BAR

S

-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

-1.5 -1 -0.5 0 0.5 1 1.5

Residuals TEAC

Res

idua

ls T

BA

RS

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

-0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8Residuals TBARS

-0.5

-0.3

-0.1

0.1

0.3

0.5

-0.5 -0.3 -0.1 0.1 0.3 0.5Residuals TBARS

-3

-2

-1

0

1

2

3

-3 -2 -1 0 1 2 3Residuals TEAC

-1.5

-0.5

0.5

1.5

-1.5 -0.5 0.5 1.5Residuals TEAC

a) Scatter Plots

b) normal Q-Q plots

19

Figure 3. Generalized AUC as a function of age with pointwise 95% confidence interval (a) Females

0.5

0.6

0.7

0.8

0.9

1

30 40 50 60 70 80

Age

AU

C

upperAUClower

(b) Males

0.5

0.6

0.7

0.8

0.9

1

30 40 50 60 70 80Age

AU

C

upper

AUC

lower

20

Figure 4. AUCs of Individuals Markers and Their Linear Combination. (a) Females

0.5

0.6

0.7

0.8

0.9

1

30 40 50 60 70 80

Age

AU

C

CombinationTBARSTEAC

(b) Males

0.5

0.6

0.7

0.8

0.9

1

30 40 50 60 70 80Age

AU

C

Combination

TBARS

TEAC

21

Figure 5. Adjusted Generalized ROC Curves. (a) Females

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 11-Specificity

Sen

sitiv

ity

Age 45, AUC=0.839

Age 55, AUC=0.735

Age 65, AUC=0.699

Age 75, AUC=0.785

(b) Males

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 11-Specificity

Sen

sitiv

ity

Age 45, AUC=0.904

Age 55, AUC=0.784

Age 65, AUC=0.730

Age 75, AUC=0.839