nonparametric predictive comparison of lifetime data under progressive censoring

Journal of Statistical Planning and Inference 140 (2010) 515 -- 525

Contents lists available at ScienceDirect

Journal of Statistical Planning and Inference

journal homepage: www.e lsev ier .com/ locate / jsp i

Nonparametric predictive comparison of lifetime data under progressivecensoring

Tahani A. Maturi, Pauline Coolen-Schrijner1, Frank P.A. Coolen∗Department of Mathematical Sciences, Durham University, Durham, DH1 3LE, UK

A R T I C L E I N F O A B S T R A C T

Article history:Received 31 October 2008Received in revised form28 July 2009Accepted 28 July 2009Available online 6 August 2009

Keywords:Lifetime testingLower and upper probabilityNonparametric predictive inferencePairwise comparisonProgressive censoring

In reliability and lifetime testing, comparison of two groups of data is a common problem. It isoften attractive, or even necessary, to make a quick and efficient decision in order to save timeand costs. This paper presents a nonparametric predictive inference (NPI) approach to comparetwo groups, say X and Y, when one (or both) is (are) progressively censored. NPI can easily beapplied to different types of progressive censoring schemes. NPI is a statistical approach basedon few assumptions, with inferences strongly based on data and with uncertainty quantifiedvia lower and upper probabilities. These inferences consider the event that the lifetime of afuture unit from Y is greater than the lifetime of a future unit from X.

© 2009 Elsevier B.V. All rights reserved.

1. Introduction

In reliability and lifetime testing, comparison of two groups of data (e.g. related to two treatments or units from twoproductionlines) is a common problem. In some lifetime experiments, making a quick and efficient decision is desirable in order to save timeand costs, and the lifetime experiment may involve expensive units, which, if they have not failed during the experiment mightbe used for other purposes (Ng and Balakrishnan, 2005). An example of an application is presented by Montanari and Cacciari(1988). To this end, a progressive censoring scheme can be useful, with censoring occurring at different stages. There may alsobe specific circumstances which cause some units to fail due to reasons unrelated to the experimentation (Cohen, 1966), and itmay occur that an individual or unit drops out of the study before the end of the experiment (Balakrishnan and Aggarwala, 2000),which also makes progressive censoring schemes useful. Several progressive censoring schemes are considered in the literature,including progressive Type-I censoring, progressive Type-II censoring and Type-II progressively hybrid censoring.

In a progressive Type-I censoring scheme, see Fig. 1(a), n units are placed on a lifetime experiment. Of these n units, r failduring the experiment, we assume (as we do throughout this paper, in order to simplify the presentation of the novel approachpresented) that they fail at r different failure times x1<x2< · · ·<xr . At m times T1<T2< · · ·<Tm, some further units may berandomly withdrawn from the experiment, leading to right-censored observations for their corresponding lifetimes. At such atime Tj (j = 1, . . . ,m) where progressive censoring is taking place, let Rj denote the number of units that are removed from theexperiment without having failed. We assume that the experiment finishes at time Tm, hence Tm >xr and Rm = n − r − ∑m−1

j=1 Rj.For use later in this paper, we define sj to be the number of failures between the consecutive right-censoring times Tj−1 and Tj,so sj = #{Tj−1<xi � Tj : i = 1, . . . , r} (j = 2, . . . ,m), and s1 is the number of failures before T1. Then the data from this experiment,under a progressive Type-I censoring scheme, consist of r = ∑m

j=1 sj observed failure times and n − r right-censoring times.

∗ Corresponding author.E-mail addresses: [email protected] (T.A. Maturi), [email protected] (F.P.A. Coolen).

1 Pauline died in April 2008, when the research in this paper was at an advanced stage. This paper is dedicated to her.

0378-3758/$ - see front matter © 2009 Elsevier B.V. All rights reserved.doi:10.1016/j.jspi.2009.07.027

http://www.sciencedirect.com/science/journal/jspi

http://www.elsevier.com/locate/jspi

mailto:[email protected]

mailto:[email protected]

516 T.A. Maturi et al. / Journal of Statistical Planning and Inference 140 (2010) 515 -- 525

Fig. 1. Three different progressive censoring schemes. (a) Progressive Type-I censoring, (b) Progressive Type-II censoring and (c) Type-II progressively hybridcensoring.

In a progressive Type-II censoring scheme, see Fig. 1(b), the number of units to be observed to fail is fixed, let this number ber. At each observed failure time, which we again assume to be r different times x1<x2< · · ·<xr , some further units which havenot failed are randomly removed from the experiment, and at the last failure time xr all the remaining units are removed from theexperiment. Let Ri denote the number of units that have not failed but are removed from the experiment at the failure time xi, fori= 1, . . . , r, then Rr = n− r − ∑r−1

i=1 Ri. The data consist of the r observed failure times x1<x2< · · ·<xr , together with the numbersof units with right-censored lifetimes at each of these failure times, which information we denote by R = (R1,R2, . . . ,Rr). Somespecial cases of this censoring scheme occur if r = n, then Ri = 0 for all i= 1, . . . , r, which means that there is no censoring actuallyoccurring, and if we have Ri = 0 for all i = 1, . . . , r − 1 and Rr = n − r then we obtain a conventional Type-II censored sample withcensoring only due to the experiment being stopped before all units have failed. Such special cases are not highlighted further inthis paper, but are briefly considered in the example in Section 4.

In a progressive Type-II censored experiment, it might take a very long time to reach the prefixed number r. Therefore, itmay be attractive to consider an experiment which is ended as soon as either r failures have been observed, or at a specific time,say T, whichever occurs first. In the latter case, the lifetimes of all the remaining units in the experiment at time T are right-censored at this time, in addition to the right-censored lifetimes of units that were progressively censored during the experimentat different failure times before T. This scenario is called Type-II progressive hybrid censoring, and is a mixture of progressiveType-II censoring and conventional Type-I censoring, see Fig. 1(c). Let xJ denote the largest observed failure time prior to T, andagain we assume that no failures coincide, so the observed failure times are x1<x2< · · ·<xJ <T. At xi, for i= 1, . . . , J, Ri units arerandomly withdrawn from the experiment. Finally, all the remaining RT units are withdrawn from the experiment at time T, soRT = n − J − ∑J

i=1 Ri.In this paper we consider the progressive Type-II censoring scheme in most detail, as it has received most attention in the

literature. However, we also briefly present results of our method for progressive Type-I censoring and Type-II progressivelyhybrid censoring, the proofs of these follow the same lines of reasoning as for our progressive Type-II censoring results and arenot included in full detail.

As mentioned above, one may be interested in comparing two independent populations or treatments, say X and Y. Forexample, X may refer to a control group and Y to a new treatment group (Balakrishnan et al., 2008a), where statistical inferencewould be aimed at investigating whether or not Y can be considered to provide an improvement compared to X. Most classicalstatistics methods presented in the literature (Balakrishnan et al., 2008a), including several nonparametric methods, approachsuch comparison problems by hypothesis testing. In particular, they tend to assume continuous cumulative distribution functionsfor the random quantities of interest, say F(·) corresponding to X and G(·) corresponding to Y, and test the null hypothesis thatthe two groups X and Y are the same with regard to the random quantity of interest, so they test H0 : F(x)=G(x), for all x, againstthe hypothesis that group Y tends to have greater lifetimes than group X, expressed via the stochastic dominance hypothesisH1 : F(x)�G(x) with strict inequality holding for at least one x. The approach presented in this paper is fundamentally different,with comparisons formulated directly in terms of a future observation for each of the two groups considered, a method whichdoes not involve any hypotheses to be tested.

We focus on the progressive Type-II censoring scheme, for which in the literature two cases are considered, depending onwhether progressive censoring has been applied to only one group or to both groups. In the first case, the progressive Type-IIcensoring scheme applies to only one group, say group Y, and it is assumed that the data from group X result from an experiment

T.A. Maturi et al. / Journal of Statistical Planning and Inference 140 (2010) 515 -- 525 517

without progressive censoring, but which is also ended when the experiment of group Y ends, so at the last failure time yrfrom group Y, so the group X data consist of failure times prior to yr and right-censored lifetimes at yr , resulting from standardType-II censoring at yr . Ng and Balakrishnan have proposed several tests, including the weighted precedence test, the weightedmaximal precedence test and the maximal Wilcoxon rank-sum precedence test as extensions of classical precedence tests thatare suitable for this scenario (Ng and Balakrishnan, 2005; Balakrishnan and Ng, 2006). Bairamov and Eryilmaz (2006) consideredexceedance statistics for the same setting. As the second case, one considers the situation with progressive Type-II censoringapplied independently to both groups X and Y. Recently, Balakrishnan et al. (2008a) introduced a precedence test based onplacement statistics with progressive censoring for both groups. The proposed precedence test statistic, P(s), is basically thenumber of failures from group X that precede the s-th (1� s� ry) failure from group Y, where ry is the number of failures fromgroup Y. Two further precedence tests are proposed by Balakrishnan et al. (2008b) for progressive censoring in both samples.The first is a Wilcoxon type rank-sum precedence test, T(ry), and the second based on the Kaplan–Meier estimator of the survivalfunctions, Q (ry). These methods are compared to the NPI approach in the example in Section 4 (case B). A recent survey onprogressive censoring is presented by Balakrishnan (2007).

This paper presents nonparametric predictive inference (NPI) to compare two groups, say X and Y, when one (or both) is (are)progressively censored. Suppose that nx and ny units from group X and group Y, respectively, are placed on a lifetime experiment.As mentioned before, we focus on progressive Type-II censoring, and in line with the notation introduced above, let the data forgroup X consist of rx different observed failure times and nx − rx progressively censored observations represented by Rx, and forgroup Y let the data consist of ry different observed failure times and ny − ry progressively censored observations representedby Ry.

In NPI, as discussed in Section 2, the comparison of groups X and Y is in terms of lower and upper probabilities for the eventthat a single future observation from group Y is greater than a single future observation from group X, where lower and upperprobabilities are used in order to keep inferential assumptions, added to the data observed, restricted. The NPI method has theattractive feature that it is applicable whether the progressive censoring is adopted on one group or on both groups, and also fordifferent censoring schemes. This latter advantage is related to the similar general advantage of statistical methods that adhereto the likelihood principle, for which stopping rules tend not to affect the inferences, and is a direct consequence of the fact thatno hypotheses are being tested, hence no counterfactual data (i.e. data that could have occurred, under a specific experimentalset-up, but which did not occur) play any role, which is noticeably different to traditional hypothesis tests which are influencedby counterfactual data. Of course, one must be happy to accept the assumptions, related to exchangeability, underlying NPI, asdiscussed in Section 2.

Section 2 provides a short overview of NPI for as far as required in this paper. In Section 3 we present the main results of thispaper, namely the lower and upper probabilities for the event that the lifetime of a future unit from group Y is greater than thelifetime of a future unit from group X, under the three different progressive censoring schemes discussed above. The main focusis on the progressive Type-II censoring scheme, for which the results stated are proved in detail, while for the two other schemesthe results are presented without proofs. NPI for these scenarios, and for a few related situations, is illustrated and discussed viaan example in Section 4, and the paper ends with some concluding remarks in Section 5 and an appendix with the proofs of someresults in Section 2.

2. Nonparametric predictive inference (NPI)

Nonparametric predictive inference (NPI) is a statistical method based on Hill's assumption A(n) (Hill, 1968), which givesdirect probabilities for a future observable random quantity, given the observed values of related random quantities (Augustinand Coolen, 2004; Coolen, 2006). Suppose that X1, . . . ,Xn,Xn+1 are positive, continuous and exchangeable random quantitiesrepresenting lifetimes. Let the ordered observed values of X1, . . . ,Xn be denoted by x1<x2< · · ·<xn <∞, and let x0 = 0 andxn+1 = ∞ for ease of notation. We assume that no ties occur, our results can be generalized to allow ties (Hill, 1993). For positiveXn+1, representing a future observation, based on n observations, A(n) (Hill, 1968) is P(Xn+1 ∈ (xi, xi+1)) = 1/(n + 1), i = 0, 1, . . . ,n.

A(n) does not assume anything else, and can be considered to be a post-data assumption related to exchangeability (De Finetti,1974). Hill (1988) discussed A(n) in detail. Inferences based on A(n) are predictive and nonparametric, and can be consideredsuitable if there is hardly any knowledge about the random quantity of interest, other than the n observations, or if one does notwant to use such information, e.g. to study the effects of additional assumptions underlying other statistical methods. A(n) is notsufficient to derive precise probabilities for many events of interest, but it provides bounds for probabilities via the `fundamentaltheorem of probability' (De Finetti, 1974), which are lower and upper probabilities in interval probability theory (Walley, 1991;Weichselberger, 2001).

Coolen and Yan (2004) presented a generalization of A(n), called rc-A(n), suitable for right-censored data. In comparison to A(n),rc-A(n) uses the extra assumption that, at the moment of censoring, the residual lifetime of a right-censored unit is exchangeablewith the residual lifetimes of all other units that have not yet failed or been censored.

For the problem considered in this paper, namely NPI for comparison of two groups of lifetime data under progressivecensoring schemes, we consider the two groups to be fully independent and apply the suitable rc-A(n) assumption per group,for which observations on n units are available, as the basis of our inferences. For the progressive Type-II censoring scheme, nunits are placed on a lifetime experiment, and for r of these units the actual failure times are observed during the experiment,while at each observed failure time for one of these r units, some of the remaining units may be withdrawn from the experiment,


until the r-th failure time when the experiment is ended, and hence all remaining units are removed from the experiment. Wecan consider the n − r progressively censored units as being grouped in blocks, each consisting of units censored at a specificobserved failure time. Hence, this leads to all censored units in one block to be censored at the same time, which is dealt withby rc-A(n) as described below. For ease of notation, we assume throughout that there are no ties between the observed failuretimes, the tied right-censoring times do not provide complications and actually simplify the approach as discussed below. Toformulate the required form of rc-A(n), we need notation for probability mass assigned to intervals without further restrictions onthe spread within the intervals. Such a partial specification of a probability distribution is called anM-function (Coolen and Yan,2004). A probability mass assigned for a real-valued random quantity X to an interval (a, b) is denoted by MX(a, b), and referredto as M-function value for X on (a, b). This concept is very similar to Shafer's basic probability assignments (Shafer, 1976).

When considering a single group of units, let r denote the number of observed failure times for the n units considered, with(n−r) units progressively censored according to the scheme R=(R1, R2, . . . ,Rr). The following definition provides theM-functionsrequired for NPI applied to the comparison of lifetime data under the progressive Type-II censoring scheme, together with thetotal probabilitymass assigned to the interval (xi, xi+1). These follows quite directly from rc-A(n) (Coolen and Yan, 2004), as provenin the appendix.

Definition 1. For nonparametric predictive inference under a progressive Type-II censoring scheme with R = (R1,R2, . . . ,Rr), theassumption rc-A(n) implies that the probability distribution for a nonnegative random quantity Xn+1 on the basis of data includingr real and (n − r) progressively censored observations is partially specified by the followingM-function values; for i = 0, 1, . . . , r,

MX(xi, xi+1) = MXn+1 (xi, xi+1) = 1n + 1

i−1∏k=1

n − k − ∑k−1l=1 Rl + 1

n − k − ∑kl=1Rl + 1

(1)

MX(x+i , xi+1) = MXn+1 (x

+i , xi+1) =

[Ri

n − i − ∑il=1Rl + 1

]MX(xi, xi+1) (2)

where x+i is used to indicate a value infinitesimally greater than xi, which can be interpreted as representing the lower bound

for the interval that would contain the actual lifetimes for all units censored at xi. In addition, we use x0 = 0 and xr+1 = ∞. Thenthe total probability mass assigned to the interval (xi, xi+1) is the sum of the two M-functions corresponding to (xi, xi+1) and(x+

i , xi+1) (for i = 0, 1, . . . , r), and is given by

PX(xi, xi+1) = P(Xn+1 ∈ (xi, xi+1)) = 1n + 1

i∏k=1

n − k − ∑k−1l=1 Rl + 1

n − k − ∑kl=1Rl + 1

(3)

In this paper we consider NPI for comparison of two independent groups, X and Y, and we mostly focus on the progressiveType-II censoring scheme with Rx = (Rx1, R

x2, . . . ,R

xrx ) and Ry = (Ry1, R

y2, . . . ,R

yry ), respectively. We are interested in the lower and

upper probabilities that a future observation, Xnx+1, of group X is less than a future observation of group Y, Yny+1, based on nx andny observations, the schemes Rx and Ry, and the assumptions rc-A(nx) and rc-A(ny), all for the respective groups. The derivationof these lower and upper probabilities, which is the main contribution of this paper and is presented in Section 3, requires thefollowing lemma from (and proven by) Coolen and Yan (2003).

Lemma 1. For s�2, let Jl = (jl, R), with j1<j2< · · ·<js <R, so we have nested intervals J1 ⊃ J2 ⊃ · · · ⊃ Js with the same rightend point R (which may be infinity). We consider two independent real-valued random quantities, say U and V. Let the probabilitydistribution for U be partially specified via M-function values, with all probability mass P(U ∈ J1) described by the s M-function valuesMU(Jl), l = 1, . . . , s, so

∑sl=1 MU(Jl) = P(U ∈ J1). Then, without additional assumptions

s∑l=1

P(V < jl)MU(Jl)� P(V <U,U ∈ J1)� P(V <R)P(U ∈ J1)

provides the maximum lower and minimum upper bounds for the probability P(V <U,U ∈ J1).

We are now ready to consider NPI for progressive censoring in Section 3. However, let us briefly comment on what couldbe considered a special case of the above described progressive censoring schemes, namely if we just decide to terminate thelifetime experiment at a certain time point, say T0, which could be a specific failure time, and with no other censoring applied.In this case, we have Rx = (0, 0, . . . ,Rxrx ) and Ry = (0, 0, . . . ,Ryry ), where Rxrx = nx − rx and Ryry = ny − ry, and rx and ry are the numberof failures from groups X and Y that precede T0, respectively. Comparison of different groups under this setting is also known as`precedence testing'. The NPI approach for such precedence testing for two groups is presented by Coolen-Schrijner et al. (2009),whereas the generalization of such NPI precedence testing to more than two groups will be presented elsewhere. In the examplein Section 4, this precedence testing scenario is also included.


3. NPI for progressive censoring

This section presents the main contribution of this paper, these results are illustrated in an example in Section 4. Due tosimilarities in arguments used to derive the results, the derivations are presented in detail for the case of progressive Type-II censoring, which is the most frequently used in the literature, and corresponding results are presented with less detailedderivations included for the two other progressive censoring schemes discussed in Section 1.

3.1. NPI for progressive Type-II censoring

Suppose that we have two independent groups, X and Y, consisting of nx and ny units, all placed on a lifetime experiment.Units of both groups are progressively Type-II censored with the schemes Rx = (Rx1,R

x2, . . . ,R

xrx ) and Ry = (Ry1,R

y2, . . . ,R

yry ). In practice,

for example, group X could be a control group, with a new treatment applied to units in group Y, and the aim might be to drawconclusions on whether or not the new treatment group tends to provide improved lifetimes. Given the data, Rx, Ry, and withthe appropriate assumptions rc-A(nx) and rc-A(ny) for the respective groups, Theorem 1 presents the NPI lower and the upperprobabilities that the next future observation from group Y is greater than the next future observation from group X.

Theorem 1. The NPI lower and the upper probabilities for the event that the next future observation from group Y is greater than thenext future observation from group X, under the progressive Type-II censoring scheme for both groups, are given by

P(Yny+1>Xnx+1) =ry∑j=0

⎧⎨⎩

rx∑i=0

1(xi+1<yj)PX(xi, xi+1)

⎫⎬⎭ PY (yj, yj+1) (4)

P(Yny+1>Xnx+1) =ry∑j=0

⎧⎨⎩

rx∑i=0

1(xi <yj+1)PX(xi, xi+1)

⎫⎬⎭ PY (yj, yj+1) (5)

with PX and PY according to Eq. (3).

Proof. The NPI lower probability for the event Xnx+1<Yny+1, given the data and progressive Type-II censoring schemes Rx =(Rx1,R

x2, . . . ,R

xrx ) and Ry = (Ry1,R

y2, . . . ,R

yry ), is derived as follows:

PXY = P(Xnx+1<Yny+1) =ry∑j=0

P(Xnx+1<Yny+1,Yny+1 ∈ (yj, yj+1))

�ry∑j=0

{P(Xnx+1<yj)MY (yj, yj+1) + P(Xnx+1<y+

j )MY (y+

j , yj+1)}

=ry∑j=0

P(Xnx+1<yj){MY (yj, yj+1) + MY (y+j , yj+1)}

=ry∑j=0

P(Xnx+1<yj)PY (yj, yj+1)

�ry∑j=0

rx∑i=0

1(xi+1<yj)PX(xi, xi+1)P

Y (yj, yj+1)

The first inequality follows by putting all mass of Yny+1 corresponding to the intervals (yj, yj+1) and (y+j , yj+1) (j = 1, . . . , ry)

to the left end points of these intervals, and by using Lemma 1 for the nested intervals (yj, yj+1) and (y+j , yj+1). The second

inequality follows by putting all mass of Xnx+1 corresponding to the intervals (xi, xi+1) and (x+i , xi+1) (i=1, . . . , rx) to the right end

points of these intervals. With these configurations of probability masses, it is easily seen that the derived lower bound is themaximum possible general lower bound corresponding to the NPI probability assignments, and hence it can be interpreted asa lower probability (Augustin and Coolen, 2004). We should notice that P(Xnx+1<y+

j ) = P(Xnx+1<yj) since the Ryj units that areright-censored at yj do not cause these probabilities to be different due to the assumption of an infinitesimal difference betweeny+j and yj, and due to the fact that the M-functions in NPI are generally assigned to open intervals between observations.The derivation of the corresponding upper probability for the event Xnx+1<Yny+1 is given below. The first inequality follows

by putting all mass of Yny+1 corresponding to the intervals (yj, yj+1) and (y+j , yj+1) (j = 1, . . . , ry) to the right end points of these

intervals, and by using Lemma 1 for the nested intervals (yj, yj+1) and (y+j , yj+1). The second inequality follows by putting all

mass of Xnx+1 corresponding to the intervals (xi, xi+1) and (x+i , xi+1) (i = 1, . . . , rx) to the left end points of these intervals. We

should notice that 1(x+i < yj+1)=1(xi <yj+1) by arguments similar to those used in the derivation above for the lower probability.


Again, this configuration ensures that the inequalities cannot be improved generally, and hence that this upper bound can beinterpreted as an upper probability

PXY = P(Xnx+1<Yny+1) =ry∑j=0

P(Xnx+1<Yny+1,Yny+1 ∈ (yj, yj+1))

�ry∑j=0

P(Xnx+1<yj+1)PY (yj, yj+1)

�ry∑j=0

PY (yj, yj+1)rx∑i=0

{1(xi <yj+1)MX(xi, xi+1) + 1(x+

i < yj+1)MX(x+

i , xi+1)}

=ry∑j=0

rx∑i=0

PY (yj, yj+1)1(xi <yj+1){MX(xi, xi+1) + MX(x+i , xi+1)}

=ry∑j=0

rx∑i=0

1(xi <yj+1)PX(xi, xi+1)P

Y (yj, yj+1) �

The use of these NPI lower and upper probabilities are illustrated via an example in Section 4. Next we present the NPI lowerand upper probabilities for the two other progressive censoring schemes discussed in Section 1, for all these ingredients requiredfor the complete derivations are provided, but detailed proofs are deleted as these follow the general lines of the proof above.

3.2. NPI for progressive Type-I censoring

In a progressive Type-I censoring scheme for n units on a lifetime experiment, as discussed in Section 1, Rj units arewithdrawnfrom the experiment at Tj (j= 1, . . . ,m), and for a total of r = ∑m

j=1 sj units the actual failure times will be observed, where sj is thenumber of observed failure times between Tj−1 and Tj. Again assuming no ties among the observed failure times, the data can bewritten as

where xjij is the ij-th observed failure time between Tj−1 and Tj (ij = 1, . . . , sj, j = 1, . . . ,m). For this situation, the NPI approach for

the comparison of two groups, X and Y, similarly as presented in the previous subsection, is as follows. Let

Bj = 1n + 1

j∏k=1

n − ∑kl=1sl −

∑k−1l=1 Rl + 1

n − ∑kl=1sl −

∑kl=1Rl + 1

then the M-functions corresponding to a progressive Type-I censoring scheme are (for j = 1, . . . ,m and ij = 1, . . . , sj)

MX(0, x11) = B1, MX(xjij , xjij+1) = Bj−1, MX(Tj, x

j+11 ) =

⎡⎣ Rj

n − ∑jl=1sl −

∑jl=1Rl + 1

⎤⎦Bj−1, PX(xjij , x

jij+1) = Bj

where xj+11 ( xjsj ) is the first (last) failure time observed after (before) we removed Rj units at time Tj, and where xjsj+1 = xj+1

1 and

xm+11 = ∞.

Now we consider two groups X and Y under such a progressive Type-I censoring scheme, with right-censoring times Tx�(� = 1, . . . , p) and Ty� (� = 1, . . . , q), such that Rx� (Ry�) units of group X (Y) that have not failed are withdrawn from the experiment atTx� (Ty� ). Then the number of failures from both groups are rx = ∑p

�=1 sx� and ry = ∑q

�=1 sy� . The NPI lower probability for the event

Xnx+1<Yny+1 in this situation is

P =q∑

�=1

⎧⎪⎨⎪⎩

sy�∑i�=1

P(Xnx+1<y�i�)MY (y�

i�, y�

i�+1) + P(Xnx+1<Ty� )MY (Ty� , y

�+11 )

⎫⎪⎬⎪⎭


where

P(Xnx+1< ·) =p∑

�=1

sx�∑i�=0

1(x�i�+1< ·)PX(x�

i�, x�

i�+1)

and the corresponding NPI upper probability is

P =q∑

�=1

sy�∑i�=0

P(Xnx+1<y�i�+1)P

Y (y�i�, y�

i�+1)

where

P(Xnx+1< ·) =p∑

�=1

⎧⎨⎩

sx�∑i�=0

1(x�i�< ·)MX(x�

i�, x�

i�+1) + 1(Tx� < ·)MX(Tx�, x�+11 )

⎫⎬⎭

As mentioned before, detailed justification of these results follows the same lines as the proof in the previous subsection. Thespecial case where such progressive censoring is only applied to one of the two groups also follows straightforwardly, and willbe briefly illustrated in the example in Section 4.

3.3. Type-II progressively hybrid censoring

Under this scheme of progressive censoring, which was also introduced in Section 1, one only observes the J failure timeswhich occur prior to time T, and at failure time xi (i=1, . . . , J) Ri units that have not failed are removed, and finally the experimentis ended at time T, at which time the RT remaining units are removed from the experiment. For this progressive censoring scheme,we can use the same M-functions as given in (1) and (2) for the intervals (xi, xi+1) and (x+

i , xi+1), where i = 0, 1, . . . , J, x0 = 0 andxJ+1 = ∞. However, for the additional interval (T, ∞), the M-function value is

MX(T,∞) = 1n + 1

[RT

n − J − ∑Ji=1Ri − RT + 1

] J∏k=1

n − k − ∑k−1l=1 Rl + 1

n − k − ∑kl=1Rl + 1

This also leads to the same formula (3) being appropriate for the probability PX(xi, xi+1), for i = 0, 1, . . . , J − 1, while for the lastinterval we have PX(xJ , ∞) = MX(xJ , ∞) + MX(x+

J , ∞) + MX(T, ∞).NPI comparison of two groups, X and Y, under such Type-II progressively hybrid censoring with (Rx1,R

x2, . . . ,R

xJx,RxTx ) and

(Ry1,Ry2, . . . ,R

yJy,RyTy ), respectively, is again based on the NPI lower and upper probabilities for the direct comparison of the next

future observations from each group, so for the event Xnx+1<Yny+1. For this censoring scheme, these NPI lower and upperprobabilities are

P =Jy∑j=0

P(Xnx+1<yj)PY (yj, yj+1) + {P(Xnx+1<Ty) − P(Xnx+1<yJy )}MY (Ty,∞)

where

P(Xnx+1< ·) =Jx∑i=0

1(xi+1< ·)PX(xi, xi+1)

and

P =Jy∑j=0

P(Xnx+1<yj+1)PY (yj, yj+1) + P(Xnx+1<∞)MY (Ty,∞)

where

P(Xnx+1< ·) =Jx∑i=0

1(xi < ·)PX(xi, xi+1) + {1(Tx < ·) − 1(xJx < ·)}MX(Tx,∞)

Detailed justification of these results is again similar as the proof given for the progressive Type-II censoring scheme, and alsothis case is illustrated in the example in Section 4.


Table 1Breakdown times of two samples of an insulating fluid.

Group Breakdown times

X 0.49 0.64 0.82 0.93 1.08 1.99 2.06 2.15 2.57 4.75Y 1.34 1.49 1.56 2.10 2.12 3.83 3.97 5.13 7.21 8.71

4. Example

In this example, we illustrate the above presented NPI approach to the comparison of two groups of lifetime data underseveral progressive censoring schemes. We use a subset of Nelson's dataset (Nelson, 2004) on breakdown times (in minutes) ofan insulating fluid that is subject to high voltage stress. The data are given in Table 1, for both groups there are 10 units involvedin the experiment, hence nx = ny = 10.

Ng and Balakrishnan (2005) used this example to illustrate the weighted precedence andweightedmaximal precedence tests,when progressive Type-II censoring is assumed to be applied to group Y. In their example, under the scheme Ry = (3, 0, 0, 0, 2),only five breakdown times from group Y are observed and other units are censored. They assume that the three units withactual observed breakdown times 2.10, 3.83 and 3.97 are instead removed from the experiment at the first breakdown time(y1 = 1.34), and that the two units with the largest actual breakdown times, 7.21 and 8.71, are removed from the experimentat the fifth breakdown time, which then is y5 = 5.13. So for all units of group X the actual breakdown times are observed, andsuch times are observed for five units from group Y, at times 1.34, 1.49, 1.56, 2.12 and 5.13. Ng and Balakrishnan (2005) derivedthe weighted precedence test statistic as equal to 67, with p-value 0.009 for the test of the null-hypothesis that both groups'breakdown times are equally distributed, and the weighted maximal precedence test statistic is equal to 50 with correspondingp-value 0.006. Therefore, they conclude that there is strong evidence to reject this null-hypothesis, even with this specific resultof the progressive censoring applied on the Y group. Their analysis therefore concludes that there is substantial evidence in thedata to support a claim that breakdown times for group Y tend to be significantly larger than for group X.

Below we present the NPI results for this example, applying different progressive censoring schemes. We consider severalcases with (mostly) progressive censoring, some in which it is applied only to group Y as done by Ng and Balakrishnan (2005),and some cases with such censoring applied to both groups. The results presented in Section 3 were for this latter, most general,situation, with the former as straightforward special cases. We present the NPI lower and upper probabilities that group Y isbetter than group X, as before in the direct predictive sense by comparing single next future observations from both groups, X11and Y11 in this example. Of course, the appropriate assumptions rc-A(n) are again made per group, and it is assumed that thegroups are fully independent.

Case A: Progressive Type-II censoring applied to group Y: Consider the same setting as used by Ng and Balakrishnan (2005) anddescribed above, with three units withdrawn from the experiment at the first observed breakdown time for group Y (at y1 =1.34),and two units for this groupwithdrawn at the last observed breakdown time, y5=5.13, sowith Ry=(3, 0, 0, 0, 2). It is also assumedthat all breakdown times for the units from group X are observed. So, with yc denoting a right-censored observation at time y,the data actually used in this case are

X : 0.49, 0.64, 0.82, 0.93, 1.08, 1.99, 2.06, 2.15, 2.57, 4.75

Y : 1.34, 1.34c, 1.34c, 1.34c, 1.49, 1.56, 2.12, 5.13, 5.13c, 5.13c

For this specific situation, the corresponding NPI lower and upper probabilities, as presented in Section 3.1, are P(Y11>X11) =0.6139 and P(Y11>X11) = 0.8052. These values could be interpreted as pretty strongly supporting the explicit event of interesthere, namely that if we would get one future value for each of these two groups, under exchangeability assumed per group, thenthe lower probability that Y11 would be greater than X11 will be substantially larger than 0.5, which might be interpreted asreflecting strong evidence in favour of this event. This conclusion is in line with the test results by Ng and Balakrishnan (2005)for exactly the same case. As this conclusion actually turns out to follow in each of the cases below (this is not necessarily thecase in general, of course), it is not repeated nor further discussed there, and the NPI results are just given for illustration withoutfurther detailed discussion.

Case B: Progressive Type-II censoring applied to groups X and Y: Suppose that the progressive Type-II censoring scheme is appliedto both groups X and Y, with Rx = (3, 1, 1, 0, 0) and Ry = (3, 2, 0, 0, 0) and resulting in the following data,

X : 0.49, 0.49c, 0.49c, 0.49c, 0.64, 0.64c, 0.93, 0.93c, 2.06, 2.15

Y : 1.34, 1.34c, 1.34c, 1.34c, 1.56, 1.56c, 1.56c, 2.10, 3.83, 7.21

If we calculate the test statistics proposed by Balakrishnan et al. (2008a,b), using notation as introduced in Section 1, we haveP(3) = 4, P(5) = 5, T(5) = 70 and Q (5) = 4. And by using the near 5% critical values and the exact level of significance summarized inBalakrishnan et al. (2008a,b), thenwewill not reject the null hypothesis for P(3), P(5) and Q (5) at significance level 5%; however we


reject the null hypothesis for T(5) at significance level 5%. The NPI results for the comparison of these two groups of breakdowntimes are P(Y11>X11) = 0.5448 and P(Y11>X11) = 0.8678.

Case C: Type-II progressively hybrid censoring applied to groups X and Y: In this example, a progressive Type-II censoring schemeis applied to groups X and Y, with Rx = (2, 1, 0, 1, 0, 0) and Ry = (1, 2, 0, 3). However, the experiment will be ended at T = 2.11,making this a Type-II progressively hybrid censoring scheme as discussed in Section 1. Suppose that the resulting data from thisexperiment are as follows:

X : 0.49, 0.49c, 0.49c, 0.64, 0.64c, 0.93, 1.99, 1.99c, 2.06, 2.11c

Y : 1.34, 1.34c, 1.49, 1.49c, 1.49c, 2.10, 2.11c, 2.11c, 2.11c, 2.11c

Then the NPI lower and upper probabilities are P(Y11>X11) = 0.5148 and P(Y11>X11) = 0.8744.

Case D: Progressive Type-I censoring applied to group Y: In this case, some units of group Y are removed from the experimentbefore breakdown, at different times, say at T = (T1, T2, T3) = (1.5, 3.5, 5.5). Suppose that one unit is removed at T1 = 1.5, three atT2 = 2.5, and one at T3 = 5.5, and let us assume that this leads to the following data for group Y : 1.34, 1.49, 1.5c, 1.56, 2.10, 3.5c,3.5c, 3.5c, 3.83 and 5.5c. We assume that no progressive censoring is applied to the X group. The corresponding NPI lower andupper probabilities for the comparison of groups X and Y are P(Y11>X11) = 0.6364 and P(Y11>X11) = 0.8244.

Case E: Progressive censoring applied to groups X and Y, according to a `throw away scheme': The `throw away scheme', aspresented by Cohen (1963), is a rather straightforward special case of progressive Type-II censoring, in which a fixed number ofunits is withdrawn from the experiment at each observed breakdown time. Suppose that this scheme is applied to both groupsX and Y, with one unit withdrawn each time, hence Rx = (1, 1, 1, 1, 1) and Ry = (1, 1, 1, 1, 1). Suppose further that the actuallyobserved breakdown times (and the corresponding right-censoring times) under this scheme are as follows:

X : 0.49, 0.49c, 0.64, 0.64c, 0.93, 0.93c, 1.08, 1.08c, 2.06, 2.06c

Y : 1.34, 1.34c, 1.56, 1.56c, 2.10, 2.10c, 3.83, 3.83c, 7.21, 7.21c

Then the corresponding NPI lower and upper probabilities are P(Y11>X11) = 0.5333 and P(Y11>X11) = 0.9291.

Case F: Precedence testing: Precedence testing can be considered as a special case of progressive censoring, as briefly explainedat the end of Section 2, the corresponding NPI results for this approach are presented by Coolen-Schrijner et al. (2009). Supposethat the breakdown of insulating fluids experiment is terminated as soon as the fifth breakdown from group Y is observed, i.e. attime y5 = 2.12. Then the breakdown times of five units from group Y are right-censored at that time, together with three unitsfrom group X. Then P(Y11>X11) = 0.5289 and P(Y11>X11) = 0.8264. Coolen-Schrijner et al. (2009) presented some nice resultsfor NPI-based precedence testing, including the attractive fact that if one increases the end-time of the experiment, such an NPIlower (upper) probability for comparison of two groups never decreases (increases).

Case G: Complete data: Let us end this example by considering NPI comparison of these two groups of breakdown data usingthe complete data as presented in Table 1, so without any (progressive) censoring scheme applied. NPI for such a comparison ofcomplete data from two groups was already presented by Coolen (1996), and is also easily derived from the results in this paperby obvious choices for the censoring schemes, namely Rxi = Ryj = 0 for all i and j, and hence rx = nx and ry = ny. For this situation,

the NPI results are P(Y11>X11) = 0.6364 and P(Y11>X11) = 0.8099.

5. Concluding remarks

The results presented in this paper show how NPI can be applied to a variety of problems addressed, mostly from classicalfrequentist perspective, in the statistics literature, and which are of great relevance in many practical applications. Most suchproblems can be seen as `multiple comparisons problems', and generally NPI provides exciting opportunities for such problems,from the novel perspective of comparison via explicit focus on future observations per group.We have restricted attention to twogroups, but the methods presented here are quite easily generalized to multiple groups, along the lines of the NPI methods forselection presented by Coolen and van der Laan (2001). Although the ideas for such a generalization are indeed straightforward,deriving analytical expressions of the corresponding NPI lower and upper probabilities becomes somewhat tedious, it is moreattractive to develop software routines that perform such calculations for any specific M-functions specified per group, and forany number of groups.

From NPI perspective, one does not have to restrict attention to specific (progressive) censoring schemes as presented inthis paper, as censoring can take place at any time without causing problems for the NPI approach, as long as the censoringmechanism is independent of the lifetime random quantities, and as long as one can reasonably assume full independence of thegroups being compared.

In such comparison problems, as indeed in statistics more generally, we strongly advise the use of a variety of differentstatistical approaches and methods, as it is often quite confusing to see to what extent conclusions are actually based on aspectsof the data or on assumptions underlying the methods used, which can often be hidden for non-expert users of such methods.


Due to the rather limited assumptions underlying NPI, it does provide an attractive opportunity to indicate the effect of strongerassumptions underlying other methods. Of course, the main difference with most classical methods lies in the fact that NPI ispurely predictive, whereas classical methods for multiple comparisons tend to focus on hypothesis testing, which can lead toapparently different conclusions.

In addition to applications to a wider variety of problems, there are many research challenges for the further developmentof NPI. These include the option to base inferences on more than one future observation per group, which is conceptually easyalthough one must not forget to take account of the fact that these future observations are inter-dependent (Coolen, 2006). Aninteresting challenge is the requirement to formulate appropriate predictive events of interest for a variety of inferential problems,which in this paper was a rather straightforward comparison of the single next observations per group. Often, however, suchpredictive inferences may be more in line with intuition than established statistical methods such as hypothesis testing. Moregenerally, the development of NPI for multivariate situations, including data with covariates, is a key challenge that promisesexciting research opportunities, the results of which are strongly needed to enhance the wide applicability of NPI.

Acknowledgement

The authors would like to thank an anonymous referee for making them aware of some recent results on precedence-typetesting based on progressively censored samples.

Appendix

In order to prove (1)–(3), we will use the results in Coolen and Yan (2004, pp. 30, 42), where

MX(xi, xi+1) = 1n + 1

∏{k:ck<xi}

nck + 1nck

and nck is the number of units at risk at ck.We can write the observations, both failure times and right-censoring times, of n units from group X as given below, in which

we assume that all observations are different values for the ease of presentation, but for tied right-censored observations one canderive the exact NPI results as limiting situationwith the difference between such right-censoring times becoming infinitesimallysmall. We follow Coolen and Yan (2004) in assuming, which is also standard in the wider literature, that coinciding failure andright-censoring times are actually such that the latter is slightly larger than the failure time. Let the data be

0<x1<c11< · · ·<c1R1 <x2<c21 < · · ·<c2R2 <x3< · · ·< · · ·<xi <ci1< · · ·<ciRi < xi+1< · · ·<xr <cr1< · · ·<crRr <∞

For the setting considered in this paper, cili is actually the right-censoring time of the lith unit censored at xi, for i = 1, . . . , r andli = 1, . . . ,Ri.

For any block k (k = 1, . . . , r), xk <ck1< · · ·<cklk < · · ·<ckRk <xk+1, ncklkis the number of units at risk at cklk , that is ncklk

= n − k −(lk − 1) − ∑k−1

l=1 Rl. Then

MX(xi, xi+1) = 1n + 1

∏{k:ck<xi}

nck + 1nck

= 1n + 1

i−1∏k=1

Rk∏lk=1

ncklk+ 1

ncklk

= 1n + 1

i−1∏k=1

n − k − ∑k−1l=1 Rl + 1

n − k − ∑kl=1Rl + 1

similar

PX(xi, xi+1) = 1n + 1

∏{k:ck<xi+1}

nck + 1nck

= 1n + 1

i∏k=1

n − k − ∑k−1l=1 Rl + 1

n − k − ∑kl=1Rl + 1

since MX(x+i , xi+1) = PX(xi, xi+1) − MX(xi, xi+1), then

MX(x+i , xi+1) = 1

n + 1

⎡⎣ i−1∏k=1

n − k − ∑k−1l=1 Rl + 1

n − k − ∑kl=1Rl + 1

⎤⎦ [

Rin − i − ∑i

l=1Rl + 1

]


References

Augustin, T., Coolen, F.P.A., 2004. Nonparametric predictive inference and interval probability. Journal of Statistical Planning and Inference 124 (2), 251–272.Bairamov, I., Eryilmaz, S., 2006. Spacings, exceedances and concomitants in progressive Type II censoring scheme. Journal of Statistical Planning and Inference

136 (3), 527–536.Balakrishnan, N., Aggarwala, R., 2000. Progressive Censoring: Theory, Methods, and Applications. Birkhauser, Boston.Balakrishnan, N., Ng, H.K.T., 2006. Precedence-Type Tests and Applications. Wiley, Hoboken, NJ.Balakrishnan, N., 2007. Progressive censoring methodology: an appraisal. Test 16 (2), 211–259.Balakrishnan, N., Tripathi, R.C., Kannan, N., 2008a. On the joint distribution of placement statistics under progressive censoring and applications to precedence

test. Journal of Statistical Planning and Inference 138 (5), 1314–1324.Balakrishnan, N., Tripathi, R.C., Kannan, N., Ng, H.K.T., 2008b. Precedence-type test based on progressively censored samples. Working Paper 0054, College of

Business, University of Texas at San Antonio. URL 〈http://ideas.repec.org/p/tsa/wpaper/0050.html〉.Cohen, A.C., 1966. Query 18: life testing and early failure. Technometrics 8 (3), 539–545.Cohen, A.C., 1963. Progressively censored samples in life testing. Technometrics 5 (3), 327–339.Coolen, F.P.A., 1996. Comparing two populations based on low stochastic structure assumptions. Statistics and Probability Letters 29 (4), 297–305.Coolen, F.P.A., 2006. On nonparametric predictive inference and objective Bayesianism. Journal of Logic, Language and Information 15 (1–2), 21–47.Coolen, F.P.A., van der Laan, P., 2001. Imprecise predictive selection based on low structure assumptions. Journal of Statistical Planning and Inference 98 (1–2),

259–277.Coolen, F.P.A., Yan, K.J., 2003. Nonparametric predictive comparison of two groups of lifetime data. In: Bernard, J.M., Seidenfeld, T., Zaffalon, M. (Eds.), ISIPTA'03:

Proceedings of the Third International Symposium on Imprecise Probabilities and their Applications. Proceedings in Informatics, vol. 18, Carleton Scientific,Waterloo, Canada, pp. 148–161.

Coolen, F.P.A., Yan, K.J., 2004. Nonparametric predictive inference with right-censored data. Journal of Statistical Planning and Inference 126 (1), 25–54.Coolen-Schrijner, P., Maturi, T.A., Coolen, F.P.A., 2009. Nonparametric predictive precedence testing for two groups. Journal of Statistical Theory and Practice 3

(1), 273–287.De Finetti, B., 1974. Theory of Probability: A Critical Introductory Treatment. Wiley, London.Hill, B.M., 1988. De Finetti's theorem, induction and An , or Bayesian nonparametric predictive inference (with discussion). In: Bernardo, J.M., DeGroot, M.H.,

Lindley, D.V., Smith, A.F.M. (Eds.), Bayesian Statistics, vol. 3. Oxford University Press, Oxford, pp. 211–241.Hill, B.M., 1968. Posterior distribution of percentiles: Bayes' theorem for sampling from a population. Journal of the American Statistical Association 63 (322),

677–691.Hill, B.M., 1993. Parametric models for A(n): splitting processes and mixtures. Journal of the Royal Statistical Society, Series B 55 (2), 423–433.Montanari, G.C., Cacciari, M., 1988. Progressively-censored aging tests on XLPE-insulated cablemodels. IEEE Transactions on Electrical Insulation 23 (3), 365–372.Nelson, W., 2004. Applied Life Data Analysis. Wiley, Hoboken, NJ.Ng, H.K.T., Balakrishnan, N., 2005. Weighted precedence and maximal precedence tests and an extension to progressive censoring. Journal of Statistical Planning

and Inference 135 (1), 197–221.Shafer, G., 1976. A Mathematical Theory of Evidence. Princeton University Press, Princeton, USA.Walley, P., 1991. Statistical Reasoning with Imprecise Probabilities. Chapman & Hall, London.Weichselberger, K., 2001. Elementare Grundbegriffe einer allgemeineren Wahrscheinlichkeitsrechnung I. Intervalwahrscheinlichkeit als umfassendes Konzept.

Physika, Heidelberg.

http://ideas.repec.org/p/tsa/wpaper/0050.html

nonparametric predictive comparison of lifetime data under progressive censoring

Documents