statistical methods for determining the accuracy of quantitative polymerase chain reaction-based...

STATISTICS IN MEDICINEStatist. Med. 2007; 26:895–902Published online 9 May 2006 in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/sim.2563

Statistical methods for determining the accuracy ofquantitative polymerase chain reaction-based tests

Barbra A. Richardson1;2;∗;†, James P. Hughes1 and Sarah Benki3

1Department of Biostatistics; University of Washington; Seattle; WA; U.S.A.2Division of Public Health Sciences; Fred Hutchinson Cancer Research Center Seattle; WA; U.S.A.

3Division of Human Biology; Fred Hutchinson Cancer Research Center Seattle; WA; U.S.A.

SUMMARY

Quantitative polymerase chain reaction (PCR)-based tests are used in several di�erent scienti�c �elds todetermine levels of a target DNA sequence of interest (the target molecule). The accuracy of quantitativePCR-based tests can be assessed by using the assay to determine the number of copies of the targetmolecule in a sample with a known concentration of the target molecule. For example, a sample witha known concentration of a target DNA sequence is serially diluted into replicate aliquots and theseare tested to determine if the observed quantity of the target is close to the expected quantity (giventhe concentration in the original sample and the dilution). Statistical methods that are conventionallyused to assess the accuracy of these assays do not take into account the variability in the number oftarget molecules in each aliquot from the original sample. We develop methods that take into accountthis extra variation and which determine the accuracy of quantitative PCR-based tests in estimating thenumber of target molecules based on a set of assays of serial dilutions from an original sample witha known concentration of target molecules. These methods are applied to data from an experiment totest the accuracy of a real-time PCR assay at low HIV-1 DNA copy levels. Copyright ? 2006 JohnWiley & Sons, Ltd.

KEY WORDS: binomial; zero-in�ated binomial; sensitivity

1. INTRODUCTION

Real-time polymerase chain reaction (PCR)-based assays, as well as other nucleic acidampli�cation assays, are used in a variety of areas including clinical microbiology andoncology, and are increasingly being used for several di�erent purposes [1–3]. For example,in the microbiological setting, real-time PCR can be used for rapid diagnostic screening,

∗Correspondence to: B. A. Richardson, Department of Biostatistics, Box 357232, University of Washington, Seattle,WA, 98195, U.S.A.

†E-mail: [email protected]

Contract=grant sponsor: National Institute of Health; contract=grant numbers: AI32518, AI29168.

Received 9 June 2005Copyright ? 2006 John Wiley & Sons, Ltd. Accepted 27 February 2006

896 B. A. RICHARDSON, J. P. HUGHES AND S. BENKI

assessing the relationship between pathogen level and clinical outcome, monitoring theemergence of drug-resistant variants, and screening for multiple strains of a pathogen in asingle multiplex assay [4]. These assays are particularly amenable to a high-throughput formatbecause ampli�cation and detection occur in a single tube. Further, they o�er a broad dynamicrange for quantitation, as well as high sensitivity, reproducibility, and accuracy [4, 5].As in conventional PCR, real-time PCR assays involve the generation and exponential

ampli�cation of a target DNA sequence of interest (target molecule). The accuracy of theseassays is generally assessed using data from spiking experiments. In these experiments, anassay is run on aliquots drawn from a spiked sample containing a targeted concentration ofDNA molecules. This process is repeated at lower and lower target concentrations by seriallydiluting the original spiked sample [6, 7]. The statistical methodology that has been used todetermine the accuracy of these assays varies, but in all cases the variation in the numberof target molecules that is induced by aliquoting samples is ignored [8, 9]. For instance, ameasure of the observed accuracy can be obtained by simply regressing the outcome of theassay (number of target molecules observed) against the expected number of target moleculesin the assays (i.e. the target concentration)—this simple method does not distinguish betweenthe inherent variability of the aliquoted samples and the true accuracy of the assay.Hughes and Totten [10], and Ou et al. [11] developed statistical methods to assess the

accuracy of qualitative PCR assays incorporating the variation in the number of DNA moleculesin each assay due to aliquoting. Hughes and Totten [10] de�ned the sensitivity of a qualitativeassay through use of a ‘sensitivity curve’—a plot of the probability of a positive test giventhe number of DNA molecules in a sample—and the speci�city—the probability of a negativetest given no DNA molecules in a sample. Hughes and Totten [10] estimated these curvesusing parametric, non-parametric and semi-parametric models, while Ou et al. [11] assessedthe performance of each of these methods for various values of the true sensitivity of the testand under the assumption of 100 per cent speci�city.In the current paper, we develop methodology to determine the accuracy of quantitative

PCR-based tests in estimating the number of target molecules (based on a set of assays of se-rial dilutions from an original sample with a known concentration of target molecules) that alsoaccounts for the variation in the number of target molecules due to aliquoting. Throughout weassume that the probability that we observe a target molecule when it is not truly present in analiquot is zero (speci�city =100 per cent). Scienti�cally, this is justi�ed because the accumu-lation of PCR product requires binding between a target DNA molecule and PCR primers de-signed to speci�cally match the target DNA sequence. Moreover, for the real-time PCR assaydescribed below, detection of PCR product requires binding between a target DNA moleculeand a sequence speci�c probe also ensuring high speci�city of the assay [5]. In Section 2, wedescribe our method for assessing the accuracy of these assays and goodness of �t, and inSection 3, we apply our methods to an experiment to assess the accuracy of a real-time quan-titative HIV-1 DNA PCR assay. Finally, in Section 4, we summarize and discuss our results.

2. METHODS

In a serial dilution experiment to assess the accuracy of a PCR-based quantitative test, let�i denote the average concentration of a target molecule (mean target concentration) of theith dilution (i=1; : : : ; M). For each i; ki aliquots are sampled and quantitative PCR testing is

Copyright ? 2006 John Wiley & Sons, Ltd. Statist. Med. 2007; 26:895–902

STATISTICAL METHODS FOR QUANTITATIVE PCR 897

performed. Denote the observed number of target molecules as xi where xi¿0. Let n denotethe true number of copies of the target molecule in a sample. Then, we de�ne the accuracyof the assay as the function f(xi|n)—the probability of observing xi copies of the targetmolecule from the output of the assay, given that exactly n copies were in the sample tested.Because the exact number of copies of the target molecule in a particular aliquot is

unknown, f(xi|n) cannot be directly estimated from the data. However, if we assume func-tional forms for (1) Pr(n|�i)—the true number of copies of the target molecule conditionalon the mean target concentration—and (2) f(xi|n), we can de�ne the observed accuracy ofa quantitative assay in terms of f and �i for each observed value xi:

h(xi|�i)=⎧⎨⎩

∞∑n=0Pr(n|�i)f(xi|n) for xi¿0

0 for xi¡0

2.1. Functional form for Pr(n|�i)Assuming there is no clumping of the target molecules in the sample from which aliquots aretaken, several others have assumed that Pr(n|�i), follows a Poisson distribution with mean�i : Pr(n|�i)=�ni exp(−�i)=n! [10, 12–14]. Given this functional form of Pr(n|�i),

h(xi|�i) =⎧⎨⎩

∞∑n=0

�ni exp(−�i)n!

f(xi|n) for xi¿0

0 for xi¡0(1)

2.2. Functional form for f(xi|n)To de�ne the observed accuracy of the assay, h(xi|�i), we also need to assume a functionalform for f(xi|n). Below, we consider two di�erent options for the distribution of f, thebinomial and the zero-in�ated binomial.

2.2.1. Binomial form for f(xi|n). Let � denote the sensitivity of the assay and de�ne it asthe probability that a single target molecule is detected given that the molecule is truly there.Assume that the detection of each target molecule is a Bernoulli trial with probability � andis independent of the detection of the other target molecules present. Then the probability ofobserving xi target molecules given that n¿0 target molecules are truly present (f(xi|n)) canbe represented by a binomial distribution with parameters � and n (Bin(�; n)):

Pr(xi|n)=

⎧⎪⎨⎪⎩(n

xi

)� xi(1− �)n−xi for xi6n and xi¿0

0 for xi¿n or xi¡0

Note, that we have implicitly assumed 100 per cent speci�city of the assay, since the proba-bility of observing more target molecules than were truly in the sample (Pr(xi¿n)) is zero.Given that f(xi|n) is Bin(�; n); h(xi|�i) in equation (1) reduces to:

h(xi|�i)= (�i�)xi exp(−�i�)xi!

(2)



Thus, under these distributional assumptions, the probability of observing xi¿0 targetmolecules in an aliquot taken from a solution with a mean target concentration of �i fol-lows a Poisson distribution with mean �i�, where � is the probability that a single targetmolecule is detected given that the molecule is truly there (sensitivity).Given the observed numbers of target molecules xi from assays on ki aliquots at the

i=1; : : : ; M mean target concentrations �i; the maximum likelihood estimate (MLE) of thesensitivity of the assay is �=

∑Mi=1

∑kij=1 xij

/∑Mi=1 ki�i. Similarly, an estimate of the variance

of the sensitivity estimate is given by VAR(�)=∑M

i=1

∑kij=1 xij

/(∑Mi=1 ki�i

)2allowing for

estimation of con�dence intervals for �. Alternatively, given that estimates of � may to beclose to one of the boundaries of the parameter space (i.e. they most often will be close to 1),pro�le likelihood con�dence interval estimates may be preferable to con�dence intervals basedon asymptotic estimates of the variance [15].

2.2.2. Zero-in�ated binomial form for f(xi|n). Again let � denote the sensitivity of the assayand de�ne it as before. In PCR-based assays, as in most laboratory tests, it is possible thatthe assay fails altogether (e.g. ampli�cation of all target molecules fails), giving xi=0. Letp equal the probability that the assay fails and we observe zero target molecules. Then, ifthere are truly n target molecules in an aliquoted sample, the probability of observing zeromolecules with the PCR-based assay is equal to the sum of the probability of the failureof the assay (p) and the probability of the assay working, but failure of product detection((1− p)(1− �)n). In addition, the probability of detecting 0¡xi6n target molecules is thenequal to the probability that the assay works (1 − p) times the probability of detecting thetarget molecules that are truly there (Bin(�; n)). Thus, to take into account the greater proba-bility of observing xi=0, we can assume that the functional form for f(xi|n) follows a zeroin�ated binomial with parameters �; p, and n(ZIBin(�; p; n)):

Pr(xi|n)=

⎧⎪⎪⎪⎪⎨⎪⎪⎪⎪⎩

p+ (1− p)(1− �)n for xi=0

(1− p)(n

xi

)� xi(1− �)n−xi for 0¡xi6n

0 for xi¿n or xi¡0

Again, note that implicit in the assumption of f(xi|n) following a ZIBin(�; p; n) distribution isthe assumption of 100 per cent speci�city of the assay. Given this functional form of f(xi|n)then, h(xi|�i) in (1) reduces to:

h(xi|�i)=

⎧⎪⎨⎪⎩p+ (1− p) exp(−�i�) for xi=0

(1− p)(�i�)xi exp(−�i�)xi!

for xi¿0(3)

Thus, under these distributional assumptions, the probability of observing xi target molecules inan aliquot taken from a solution with a mean target concentration of �i follows a zero in�atedPoisson distribution with mean �i�, and in�ation parameter p, where � is the probability thata single target molecule is detected given that the molecule is truly there (sensitivity) and pis the probability that the assay fails altogether.



Let Iij=1 if we observe no target molecules in aliquot number j at the ith mean targetconcentration, and let Iij=0 otherwise. Then, the log likelihood for the data is

l=M∑i=1

ki∑j=1

[{Iij log(p+(1−p) exp(−�i�))}+

{(1−Iij) log

((1−p)(�i�)

xi exp(−�i�)xi!

)}](4)

Numerical maximization of (4) must be used to get MLEs for of p and �. Given the proximityto the parameter space boundaries that are likely for these parameter estimates (i.e. estimates ofp will usually be close to 0 and estimates of � will usually be close to 1), pro�le likelihoodcon�dence interval estimates are preferable to estimates based on asymptotic estimates ofvariance [15].

2.3. Assessing binomial versus zero-in�ated binomial assumption

As outlined above, the assumption of a Bin(�; n) distribution for f(xi|n) results in aPoisson(�i�) distribution for h(xi|�i), while the assumption of a ZIBin(�; p; n) distributionfor f(xi|n) results in a zero-in�ated Poisson distribution with mean �i�, and in�ation parame-ter p for h(xi|�i). Hence, the Bin(�; n) assumption becomes a special case of the ZIBin(�; p; n)with p set to zero. Given this, the necessity of the ZIBin(�; p; n) assumption compared tothe simpler Bin(�; n) assumption can be tested by calculating a pro�le likelihood con�denceinterval for p under the ZIBin(�; p; n) assumption, and determining if the con�dence intervalcontains zero. If so, the simpler Bin(�; n) can be assumed.

2.4. Assessing goodness of �t

For each target dilution, �i, calculate qi =∑M

i=1(Pr( j|�i)=M), for j=0; 1; : : : ;∞ using theappropriate model for h ((2) or (3)). Let N( j|�i) denote the number of times xi= j whenthe target dilution was �i; and let q′

j=∑M

i=1 N( j|�i)=(kiM) for j=0; 1; : : : ;∞. Then, qj and q′j

can be thought of as the estimated mean probability of the number of target molecules beingequal to j, and the observed mean probability of the number of target molecules being equalto j, respectively. Goodness of �t of the model h, can be assessed graphically by plotting q′

jversus qj. A good �t is represented by this graph showing points falling closely around the45◦ line that goes through the origin.In addition, formal hypothesis testing of the goodness of �t of the model can be done

through a chi-square test. Let N( j) denote the number of times we observe j target moleculesin all of the assays performed, and let E( j)=

∑Mi=1 (Pr( j|�i) ∗ ki denote the number of times

we expect to observe j target molecules in all of the assays performed using the appropriatemodel for h ((2) or (3)). Then, choosing J such that it is the largest number for which noE( j) is less than 1, and no more than 20 per cent of all E( j) are less than 5, the statisticS=

∑Jj=0 (N( j)−E( j))2=E( j) has a chi-square distribution with J − 2 degrees of freedom if

(2) is used for h, and J − 3 degrees of freedom if (3) is used for h, and can be used to testfor goodness of �t of the model [16].

3. EXAMPLE=APPLICATION

In an experiment to test the accuracy of a real-time PCR assay at low DNA copy numbers,data were obtained from multiple PCR assays performed using serial dilutions of genomic



Figure 1. Mean observed probability versus mean expected probability of detecting j target molecules forj = 0; 1; : : : ; 15 assuming f(x|n) follows a binomial distribution.

DNA isolated from an HIV-1-infected cell line [17]. The real-time PCR assay was used toquantify the HIV-1 proviral copy number in each assay [18]. Serial dilutions of genomic DNAwere generated such that the expected mean number of target molecules was 10; 5; 4; 3; 2, and1 copies. Four such sets of serial dilutions were generated, and six aliquots were taken fromeach dilution. For one set of serial dilutions each aliquot was tested once using the real-timePCR assay (giving six assays per copy level), while for the other three sets of serial dilutionseach aliquot was tested twice using the real-time PCR assay (giving 3× 12=36 assays percopy level). Thus, a total of 42 (6 + 36) assays were performed per copy level.Using the binomial distributional assumption (i.e. assuming a probability of zero that the

assay completely fails), we found that the probability of detecting a single target molecule thatis truly present in an aliquot was 0.880 with a 95 per cent pro�le likelihood con�dence intervalof (0.826, 0.942). Figure 1 shows the goodness of �t graph under the binomial assumption.Note, that the location of the point for j=0 falls below and to the right of the 45◦ line. Thisindicates that more zeroes were observed in the data than would be expected by the assumedmodel, suggesting that the zero-in�ated binomial assumption for f should be considered. Inaddition, using the binomial assumption gives S=26:03 with 13 degrees of freedom, and ap-value of 0.02 indicating that the model does not �t well.Using the zero-in�ated binomial distributional assumption, we found that the estimated

probability of assay failing altogether (p) in this PCR-based assay is 0.0405 with a 95per cent pro�le likelihood con�dence interval of (0.0064, 0.0882). This exclusion of zero in thecon�dence interval indicates that the zero-in�ated binomial assumption for f isappropriate for these data, and that the probability of complete assay failure was small, butsigni�cantly greater than zero at a 2.5 per cent signi�cance level. In addition, we found thatwhen the assay did not fail, the probability of detecting a single target molecule that is trulypresent in an aliquot is high at 0.883 with a 95 per cent pro�le likelihood con�dence intervalof (0.847, 0.973). Figure 2 shows the goodness of �t graph under the zero-in�ated binomialassumption. Note that the location of the point for j=0 falls much closer to the 45◦ line than



Figure 2. Mean observed probability versus mean expected probability of detecting j target molecules forj = 0; 1; : : : ; 15 assuming f(x|n) follows a zero-in�ated binomial distribution.

when p was assumed equal to zero, indicating a better �t. In addition, using the zero-in�atedbinomial distributional assumption gives S=18:40 with 12 degrees of freedom, and a p-valueof 0.10 indicating that we cannot reject the hypothesis that the model �ts.

4. DISCUSSION

Real-time PCR-based assays are increasingly being used on a large scale in a wide varietyof �elds to quantify di�erent target molecules. Knowledge regarding the accuracy of theseassays, gained through serial dilution experiments, is necessary prior to using the results ofthe assay for clinical purposes or for laboratory research. However, the standard methods forassessing accuracy do not take into account the variation in the number of target moleculesdue to aliquoting. Here we have developed statistical methodology for determination of theaccuracy of these assays that (1) allows for this variation due to aliquoting and (2) allowsfor a non-zero probability of altogether failure of the assay.The methodology developed here is generally applicable for assessment of the ability of

real-time PCR-based assays to accurately quantify the number of target molecules. However,it is probably most useful in the situation where one wants to detect very low copy numbersof the target molecule (e.g. 610), since the variability due to aliquoting samples is moreapparent in low copy aliquots. In fact, when applied to data from an experiment to test theaccuracy of a real-time PCR assay at low DNA copy levels, we found a good �t of our modelto the data, and found that the additional assumption of a non-zero probability of failure ofthe assay is necessary.

ACKNOWLEDGEMENTS

This work was supported by National Institute of Health grants AI32518 and AI29168.



REFERENCES

1. Ginzinger DG. Gene quanti�cation using real-time quantitative PCR: an emerging technology hits themainstream. Experimental Hematology 2002; 30:503–512.

2. Klein D. Quanti�cation using real-time PCR technology: applications and limitations. Trends in MolecularMedicine 2002; 8:257–260.

3. Neisters HGM. Clinical virology in real time. Journal of Clinical Virology 2002; 25:S3–S12.4. Mackay IM, Arden KE, Nitsche A. Real-time PCR in virology. Nucleic Acids Research 2002; 30:1292–1305.5. Heid CA, Stevens J, Livak KJ, Williams PJ. Real time quantitative PCR. Genome Research 1996; 6:986–994.6. Elbeik T, Alvord WG, Trichavaroj R, de Souza M, Dewar R, Brown A, Cherno� D, Michael NL, Nassos P,Hadley K, Ng VL. Comparative analysis of HIV-1 viral load assays on subtype quanti�cation: Bayer VersantHIV-1 RNA 3.0 versus Roche Amplicor HIV-1 Monitor Version 1.5. Journal of Acquired Immune De�ciencySyndrome 2002; 29:330–339.

7. Johanson J, Abravaya K, Caminiti W, Erickson D, Flanders R, Leckie G, Marshall E, Mullen C, Ohhashi Y,Perry R, Ricci J, Salituro J, Smith A, Tang N, Vi M, Robinson J. A new ultrasensitive assay for quanti�cationof HIV-1 RNA in plasma. Journal of Virological Methods 2001; 95:81–92.

8. Haque KA, Pfei�er RM, Beerman MB, Struewing JP, Chanock SJ, Bergen AW. Performance of high-throughputDNA quanti�cation methods. BMC Biotechnology 2003; 28:20.

9. Weiss J, Wu H, Farrenkopf B, Schultz T, Song G, Shah S, Siegel J. Real time TaqMan PCR detection andquantitation of HBV genotypes A-G with the use of an internal quantitation standard. Journal of ClinicalVirology 2004; 30:86–93.

10. Hughes JP, Totten P. Estimating the accuracy of polymerase chain reaction-based tests using endpoint dilution.Biometrics 2003; 59:505–511.

11. Ou SS, Hughes JP, Richardson BA. Methods for assessing the accuracy of PCR-based tests: comparisons andextensions. Statistics in Medicine 2005; 24:1755–1764.

12. Shapiro DS. Quality control in nucleic acid ampli�cation methods: use of elementary probability theory. Journalof Clinical Microbiology 1999; 37:848–851.

13. Sharrock CEM, Kaminski E, Man S. Limiting dilution analysis of human T cells: a useful clinical tool.Immunology Today 1990; 11:281–286.

14. Sykes PJ, Neoh SH, Brisco MJ, Hughes E, Condon J, Morley AA. Quantitation of targets for PCR by use oflimiting dilution. Biotechniques 1992; 13:444–449.

15. Self SG, Liang K-Y. Asymptotic properties of maximum likelihood estimators and likelihood ratio tests undernonstandard conditions. Journal of the American Statistical Association 1987; 82:605–610.

16. Kotz S, Johnson N (eds). Encyclopedia of Statistical Sciences, vol. 3. Wiley: New York, 1983; 453–454.17. Clouse KA, Powell D, Washington I, Poli G, Strebel K, Farrar W, Barstad P, Kovacs J, Fauci AS, Folks TM.

Monokine regulation of human immunode�ciency virus-1 expression in a chronically infected human T cell.Journal of Immunology 1989; 142:431–438.

18. Rousseau CM, Nduati RW, Richardson BA, John-Stewart GC, Mbori-Ngacha DA, Kreiss JK, Overbaugh J.Association of levels of HIV-1-infected breast milk cells and risk of mother-to-child transmission. Journal ofInfectious Diseases 2004; 190:1880–1888.


statistical methods for determining the accuracy of quantitative polymerase chain reaction-based...

Documents