WorkshopSemiparametric frailty models

coxph and coxme

The data set: Time to first insemination

• Database of regional Dairy Herd Improvement Association (DHIA)– Milk recording service– Artificial insemination– Select sample– Subset of 2567 cows from 49 dairy farms

Fixed covariates data setinsemfix.dat

Time-varying covariates data setinsemtvc.dat

Fitting semiparametric modelswith fixed covariates

setwd("c://docs//onderwijs//survival//Flames//")insemfix<-read.table("insemfix.dat",header=T)coxph(Surv(timeto,stat)~heifer+frailty(herd))

Call:coxph(formula = Surv(timeto, stat) ~ heifer + frailty(herd)) coef se(coef) se2 Chisq DF p heifer -0.24 0.0432 0.0430 30.9 1.0 2.7e-08frailty(herd) 205.3 40.9 0.0e+00

Iterations: 10 outer, 23 Newton-Raphson Variance of random effect= 0.123 I-likelihood = -16953

Degrees of freedom for terms= 1.0 40.9 Likelihood ratio test=281 on 41.9 df, p=0 n= 2579

HR=exp(-0.24)=0.79

Exercise

• Can we maximise the penalised partial likelihood at once?

• Depict the penalised partial likelihood as a function of theta

• The PPL is the second item of the named list loglik, e.g., for our example– insemfix$loglik[2]

Penalised partial log likelihood

0.06 0.08 0.10 0.12 0.14

-16

89

8-1

68

96

-16

89

4-1

68

92

-16

89

0

theta

pe

na

lise

d p

art

ial l

og

like

liho

od

detloglik<-function(x){coxph(Surv(timeto,stat)~heifer+frailty(herd,theta=x),data=insemfix)$loglik[2]}thetlist<-seq(0.05,0.15,0.005)logliklist<-sapply(thetlist,detloglik)plot(thetlist,logliklist,type='l',xlab='theta',ylab='penalised partial loglikelihood')

Exercise

• Do we obtain the maximum profile likelihood?

• Depict the profile likelihood as a function of theta– Theta can be fixed at x by frailty(herd,theta=x)– The marginal likelihood can be obtained as

insemfix$history[[1]]$c.loglik

Profile log likelihood

0.06 0.08 0.10 0.12 0.14

-16

95

4.5

-16

95

4.0

-16

95

3.5

-16

95

3.0

-16

95

2.5

-16

95

2.0

theta

pro

file

log

like

liho

od

detloglik<-function(x){coxph(Surv(timeto,stat)~heifer+frailty(herd,theta=x),data=insemfix)$history[[1]]$c.loglik}thetlist<-seq(0.05,0.15,0.005)logliklist<-sapply(thetlist,detloglik)plot(thetlist,logliklist,type='l',xlab='theta',ylab='profile loglikelihood')

Adjusting outer number of iterations

coxfit<-coxph(Surv(timeto,stat)~heifer+frailty(herd,eps=0.00000001),data=insemfix,outer.max=100)coxfitCall:coxph(formula = Surv(timeto, stat) ~ heifer + frailty(herd, eps = 1e-08), outer.max = 100)

coef se(coef) se2 Chisq DF p heifer -0.241 0.0431 0.043 31.1 1.0 2.4e-08frailty(herd, eps = 1e-08 194.5 38.5 0.0e+00

Iterations: 21 outer, 39 Newton-Raphson Variance of random effect= 0.0878 I-likelihood = -16952.1 Degrees of freedom for terms= 1.0 38.5 Likelihood ratio test=277 on 39.5 df, p=0 n= 2579

Time-varying covariates data

Contribution to the denominator:

tij=10 : 2.29tij=55 : 2.61

Fitting Cox models withtime-varying covariates (1)

#Read datasetwd("c://docs//onderwijs//survival//Flames//")insemtvc<-read.table("insemtvc.dat",header=T)coxph(Surv(begin,end,stat)~ureum+frailty(herd,eps=0.0000001), outer.max=100,data=insemtvc)

Fitting Cox models withtime-varying covariates (2)

> coxph(Surv(begin,end,stat)~ureum+frailty(herd,eps=0.0000001), outer.max=100,data=insemtvc)Call:coxph(formula = Surv(begin, end, stat) ~ ureum + frailty(herd, eps = 1e-07), data = insemtvc, outer.max = 100)

coef se(coef) se2 Chisq DF p ureum -0.0525 0.0187 0.0186 7.85 1 0.0051frailty(herd, eps = 1e-07 1238.47 165 0.0000

Iterations: 35 outer, 82 Newton-Raphson Variance of random effect= 0.334 I-likelihood = -69868.9 Degrees of freedom for terms= 1 165 Likelihood ratio test=2066 on 166 df, p=0 n= 93608

Alternative semiparametric models

• Unadjusted and marginal semiparametric model

• Stratified semiparametric model

• Semiparametric frailty model

Unadjusted semiparametric modelureumtv.unadjust <-coxph(Surv(begin,end,stat)~ureum,data=insemtvc)summary(ureumtv.unadjust)Call:coxph(formula = Surv(begin, end, stat) ~ ureum, data = insemtvc)

n= 93608 coef exp(coef) se(coef) z pureum -0.0273 0.973 0.0162 -1.68 0.093

exp(coef) exp(-coef) lower .95 upper .95ureum 0.973 1.03 0.943 1.00

Rsquare= 0 (max possible= 0.779 )Likelihood ratio test= 2.83 on 1 df, p=0.0926Wald test = 2.82 on 1 df, p=0.0929Score (logrank) test = 2.82 on 1 df, p=0.0929

Marginal semiparametric modelureumtv.marg<-coxph(Surv(begin,end,stat)~ureum+cluster(herd),data=insemtvc)summary(ureumtv.marg)Call:coxph(formula=Surv(begin,end,stat)~ureum+cluster(herd),data=insemtvc) n= 93608 coef exp(coef) se(coef) robust se z pureum -0.0273 0.973 0.0162 0.0285 -0.957 0.34

exp(coef) exp(-coef) lower .95 upper .95ureum 0.973 1.03 0.92 1.03

Rsquare= 0 (max possible= 0.779 )Likelihood ratio test= 2.83 on 1 df, p=0.0926Wald test = 0.92 on 1 df, p=0.338Score (logrank) test = 2.82 on 1 df, p=0.0929, Robust = 0.91 p=0.339

Stratified semiparametric modelureumtv.strat<-coxph(Surv(begin,end,stat)~ureum+strata(herd),data=insemtvc)summary(ureumtv.strat)Call:coxph(formula = Surv(begin, end, stat) ~ ureum + strata(herd), data = insemtvc)

n= 93608 coef exp(coef) se(coef) z pureum -0.0588 0.943 0.0198 -2.97 0.003

exp(coef) exp(-coef) lower .95 upper .95ureum 0.943 1.06 0.907 0.98

Rsquare= 0 (max possible= 0.444 )Likelihood ratio test= 8.86 on 1 df, p=0.00291Wald test = 8.83 on 1 df, p=0.00296Score (logrank) test = 8.83 on 1 df, p=0.00296

Semiparametric frailty modelureumtv.frail<-coxph(Surv(begin,end,stat)~ureum+frailty(herd,eps=0.0000001),outer.max=100,data=insemtvc)summary(ureumtv.frail)Call:coxph(formula = Surv(begin, end, stat) ~ ureum + frailty(herd, eps = 1e-07), data = insemtvc, outer.max = 100)

n= 93608 coef se(coef) se2 Chisq DF p exp(coef) lower.95 upper.95ureum -0.0525 0.0187 0.0186 7.85 1 0.0051 0.949 0.915 0.984frailty(herd, eps = 1e-07 1238.5 165 0.0000

Iterations: 35 outer, 60 Newton-Raphson Variance of random effect= 0.334 I-likelihood = -69868.9 Degrees of freedom for terms= 1 165 Rsquare= 0.022 (max possible= 0.779 )Likelihood ratio test= 2066 on 166 df, p=0Wald test = 7.85 on 166 df, p=1

Conclusions

• With adjustment, either by stratification or frailties, significant ureum effect

• Without adjustment, either with or without variance adjustment, no significant ureum effect

• Why?• Plot herd specific ureum effect as a

function of mean herd ureum concentration

Exercise

• Plot percentage events as a function of mean herd ureum concentration

• Using following scheme– Derive average ureum conc. for each cow

• avureum.cow<-tapply(insemtvc$ureum,list(insemtvc$cowid),mean)

– Derive average ureum conc. for each herd– Derive percentage events– Plot

Mean ureum and %insemination

setwd("c://docs//onderwijs//survival//Flames//")insemtvc<-read.table("insemtvc.dat",header=T)herd<-insemtvc$herdnr;timeto<-(insemtvc$end*12/365.25)stat<-insemtvc$stat;heifer<-insemtvc$par2avureum.cow<-tapply(insemtvc$ureum,list(insemtvc$cowid),mean)herd.cow<-tapply(insemtvc$herd,list(insemtvc$cowid),mean)avureum.herd<-tapply(avureum.cow,list(herd.cow),mean)numinsem.herd<-tapply(insemtvc$stat,insemtvc$herd,sum)totanim.herd<-tapply(insemtvc$stat,insemtvc$herd,length)percinsem<- 100*numinsem.herd/totanim.herdplot(avureum.herd,percinsem,xlab="Ureum",ylab="Insemination (%)")

Exercise

• Plot frailty term as a function of mean herd ureum concentration

• Using following scheme– Save frailties from Cox frailty model– Derive average ureum conc. for each herd– Plot

Mean ureum and frailties

ureumtv.frail<-coxph(Surv(begin,end,stat)~ureum+frailty(herd,eps=0.0000001),outer.max=100,data=insemtvc)avureum.cow<-tapply(insemtvc$ureum,list(insemtvc$cowid),mean)avureum.herd<-tapply(avureum.cow,list(herd.cow),mean)herd.cow<-tapply(insemtvc$herd,list(insemtvc$cowid),mean)avureum.herd<-tapply(avureum.cow,list(herd.cow),mean)stat.herd<-tapply(insemtvc$stat,list(insemtvc$herd),sum)results<-data.frame(frailty=ureumtv.frail$frail,ureum=avureum.herd,stat=stat.herd)results0<-results[results$stat==0,]plot(results$frailty,results$ureum,xlab="Frailty",ylab="Mean ureum concentration")points(results0$frailty,results0$ureum,pch=3)

Mean ureum and frailties plot

-4 -3 -2 -1 0 1

2.0

2.5

3.0

3.5

4.0

4.5

Frailty

Me

an

ure

um

co

nce

ntr

atio

n

Exercise

• Derive herd specific ureum effects– Fit the model for each herd separately

Herd specific treatment effectavureum.cow<-tapply(insemtvc$ureum,list(insemtvc$cowid),mean)herd.cow<-tapply(insemtvc$herd,list(insemtvc$cowid),mean)avureum.herd<-tapply(avureum.cow,list(herd.cow),mean)

fitmod<-function(herdnr){coxph(Surv(begin,end,stat)~ureum,subset=(insemtvc$herd==herdnr),data=insemtvc)$coef}

ureumeffect.herd<-sapply(herd.cow,fitmod)Error in fitter(X, Y, strats, offset, init, control, weights = weights, : Can't fit a Cox model with zero failures

Exercise

• Refit the frailty model and the marginal model leaving out the herds without inseminations

Results deleting farms without inseminations

insemtvcr<-insemtvc[insemtvc$herd!=1302100819 & insemtvc$herd!=1302310377 & insemtvc$herd!=3103311094 & insemtvc$herd!=3402102723 & insemtvc$herd!=3403801839 & insemtvc$herd!=4300110030,]ureumtvcr.unadjust<-coxph(Surv(begin,end,stat)~ureum,data=insemtvcr)summary(ureumtvcr.unadjust)Call:coxph(formula = Surv(begin, end, stat) ~ ureum, data = insemtvcr)

n= 90232 coef exp(coef) se(coef) z p lower .95 upper .95ureum -0.0433 0.958 0.0163 -2.65 0.008 0.927 0.989 Rsquare= 0 (max possible= 0.789 )Likelihood ratio test= 7.04 on 1 df, p=0.00795Wald test = 7.02 on 1 df, p=0.00804Score (logrank) test = 7.02 on 1 df, p=0.00805

Results deleting farms without inseminations - alternative

insemtvcr<-insemtvc[insemtvc$herd!=1302100819 & insemtvc$herd!=1302310377 & insemtvc$herd!=3103311094 & insemtvc$herd!=3402102723 & insemtvc$herd!=3403801839 & insemtvc$herd!=4300110030,]ureumtvcr.unadjust<-coxph(Surv(begin,end,stat)~ureum,data=insemtvcr)summary(ureumtvcr.unadjust)Call:coxph(formula = Surv(begin, end, stat) ~ ureum, data = insemtvcr)

n= 90232 coef exp(coef) se(coef) z p lower .95 upper .95ureum -0.0433 0.958 0.0163 -2.65 0.008 0.927 0.989 Rsquare= 0 (max possible= 0.789 )Likelihood ratio test= 7.04 on 1 df, p=0.00795Wald test = 7.02 on 1 df, p=0.00804Score (logrank) test = 7.02 on 1 df, p=0.00805

Deleting farms without inseminations

#alternativeherdstat<-data.frame(herd=names(stat.herd),numinsem=stat.herd)insemtvcr2<-merge(insemtvc,herdstat,by="herd")

Exercise

• Fit the frailty model assuming the lognormal distribution for the frailties

Fitting the shared normal frailty model

coxph(Surv(begin,end,stat)~ureum+frailty(herd,dist="gaussian",eps=0.00001),outer.max=100,data=insemtvc)Call:coxph(formula = Surv(begin, end, stat) ~ ureum + frailty(herd, dist = "gaussian", eps = 1e-05), data = insemtvc, outer.max = 100)

coef se(coef) se2 Chisq DF p ureum -0.0513 0.0188 0.0186 7.48 1 0.0062frailty(herd, dist = "gau 1581.56 165 0.0000

Iterations: 6 outer, 21 Newton-Raphson Variance of random effect= 0.411 Degrees of freedom for terms= 1 165 Likelihood ratio test=2030 on 166 df, p=0 n= 93608

Lognormal frailties versus normal random effects

• Assume U has lognormal distribution, with U=exp(W) and W~N(m,s2)

• The gamma density with mean 1, variance q is given by

• For q = 0.1, 0.5, 1, 1.5, plot the gamma density function and lognormal density chosing values for m and s2 so that– Var(exp(W))= q and m = 0– Var(exp(W))= q and E(U)=1

2

2

2 2

logexp

2

1)(

i

i

iU

u

uuf

1

exp1

11

iiiU

uuuf

1exp2expVar

5.0expE22

2

U

U

Lognormal frailties versus normal random effects (2)

• With m equal to 0, we have

• With E(U)=1, we have

1expexpVar

5.0expE22

2

U

U

2

411log2

1log

5.02

2

1exp2expVar

5.0expE22

2

U

U

Plotting densitiestheta<-c(0.1,0.5,1,1.5)s1<-log((1+sqrt(1+4*theta))/2)s2<-log(theta+1)m2<- -0.5*s2u<-seq(0,3,0.01)for (i in 1:length(u)){dlnorm.theta0.1.EWeq0<-dlnorm(u,0,sqrt(s1[1]))dlnorm.theta0.5.EWeq0<-dlnorm(u,0,sqrt(s1[2]))dlnorm.theta1.EWeq0<-dlnorm(u,0,sqrt(s1[3]))dlnorm.theta1.5.EWeq0<-dlnorm(u,0,sqrt(s1[4]))dlnorm.theta0.1.EUeq1<-dlnorm(u,m2[1],sqrt(s2[1]))dlnorm.theta0.5.EUeq1<-dlnorm(u, m2[2],sqrt(s2[2]))dlnorm.theta1.EUeq1<-dlnorm(u, m2[3],sqrt(s2[3]))dlnorm.theta1.5.EUeq1<-dlnorm(u, m2[4],sqrt(s2[4]))dgam.theta0.1.EUeq1<-dgamma(u,1/theta[1], 1/theta[1])dgam.theta0.5.EUeq1<- dgamma(u,1/theta[2], 1/theta[2])dgam.theta1.EUeq1<- dgamma(u,1/theta[3], 1/theta[3])dgam.theta1.5.EUeq1<- dgamma(u,1/theta[4], 1/theta[4])}

0.0 0.5 1.0 1.5 2.0 2.5 3.0

0.0

0.5

1.0

1.5

2.0

u

Den

sity

theta=0.1

gammalnormEU=1lnormEW=0

0.0 0.5 1.0 1.5 2.0 2.5 3.0

0.0

0.5

1.0

1.5

2.0

u

Den

sity

theta=0.5

0.0 0.5 1.0 1.5 2.0 2.5 3.0

0.0

0.5

1.0

1.5

2.0

u

Den

sity

theta=1

0.0 0.5 1.0 1.5 2.0 2.5 3.0

0.0

0.5

1.0

1.5

2.0

u

Den

sity

theta=1.5