comparison of alternative measurement methods (kuttatharmmakul)

Comparison of alternative measurement methods

Siriporn Kuttatharmmakul, D. Luc Massart, Johanna Smeyers-Verbeke*

ChemoAC, Pharmaceutical Institute, Vrije Universiteit Brussel, Laarbeeklaan 103, B-1090, Brussel, Belgium

Received 19 February 1998; received in revised form 14 July 1998; accepted 14 July 1998

Abstract

A procedure to compare the performance (precision and bias) of an alternative measurement method and a reference method

has been extensively described. It is based on ISO 5725-6 which has been adapted to the intralaboratory situation. This means

that the proposed approach does not evaluate the reproducibility, but considers the (operatorinstrumenttime)-differentintermediate precision and/or the time-different intermediate precision. A 4-factor nested design is used for the study. The

calculation of different variance estimates from the experimental data is carried out by ANOVA. The Satterthwaite

approximation is included to determine the number of degrees of freedom associated with the compound variances. Taken into

account the acceptable bias, the acceptable ratio between the precision parameters of the two methods, the significance level and the probability to wrongly accept an alternative method with an unacceptable performance, the formulae to determinethe number of measurements required for the comparison are given. For the evaluation of the bias, in addition to the point

hypothesis testing, the interval hypothesis testing is also included as an alternative. Two examples are given as an illustration

of the proposed approach. # 1999 Elsevier Science B.V. All rights reserved.

Keywords: Comparison; Alternative measurement method; Bias; Precision; Repeatability; Time-different intermediate precision;

(Operatorinstrumenttime)-different intermediate precision; Nested design; ANOVA; Satterthwaite approximation; Interval hypothesistesting

1. Introduction

When a laboratory wants to replace an existing

analytical method by a new method (e.g. because

the latter is cheaper or easier to use) it has to show

that the new method performs at least as good as the

existing one. A comparison of the performance (pre-

cision and bias) of both methods has therefore to be

performed. One of the most advanced guidelines for

the comparison of two methods can be found in ISO

5725-6 [1]. However the ISO guideline is based on

interlaboratory studies and is therefore not applicable

in the intralaboratory situation. Indeed within a single

laboratory, the reproducibility, as evaluated by ISO,

cannot be determined but intermediate precision con-

ditions, such as changes in operator, equipment and

time should be considered since they contribute to the

variability of measurements performed in the labora-

tory.

In the ISO guideline the reference method is an

international standard method that was studied in an

interlaboratory test program and its precision (2) isassumed to be known. This assumption is reasonable

since the precision is obtained from a large number of

measurements. In the intralaboratory situation a

Analytica Chimica Acta 391 (1999) 203225

*Corresponding author. Tel.: +32-2477-4737; fax: +32-2477-

4735; e-mail: asmeyers@vub.vub.ac.be

0003-2670/99/$ see front matter # 1999 Elsevier Science B.V. All rights reserved.PII: S 0 0 0 3 - 2 6 7 0 ( 9 9 ) 0 0 1 1 5 - 4

laboratory has developed a first method and later on

wishes to compare a new method to the older already

internally validated method. For the latter, referred to

as the reference method, only an estimate of the

precision (s2) will be available since the precision is

determined from a rather limited number of measure-

ments. This of course determines the statistical tests to

be used in the comparison of the performance char-

acteristics of both methods.

Moreover, the ISO standard is meant to show that

both methods have similar precision and/or trueness

whereas a laboratory that performs a method compar-

ison study is interested to evaluate whether the new

method is at least as good as the reference method.

This implies that some two-sided statistical tests

included in the ISO guideline are not appropriate

for the comparison of two methods in a single labora-

tory, where example in the evaluation of the precision

one-sided tests have to be considered.

In the decision making concerning the new alter-

native method it is important (i) not to reject an

alternative method which in fact is appropriate, and

(ii) not to accept an alternative method which in fact is

not appropriate. The former is related to the a-error ofthe statistical tests used in the comparison and is

controlled through the selection of the significance

level. The latter is related to the b-error and when it isconsidered it is generally taken into account by includ-

ing sample size calculations. This approach is also

included in the ISO guideline.

In this article we propose an adaptation of the ISO

guideline to the intralaboratory comparison of two

methods. It is also applicable to the situation in which

two laboratories of, e.g., the same organisation are

involved, each laboratory being specialized in one of

the methods. For the evaluation of the bias, in addition

to the point hypothesis testing, interval hypothesis

testing [2] in which the probability of accepting a

method that is too much biased is controlled, is also

included.

Due to the specified acceptance criteria for the

alternative method, the proposed approach might lead

to a large number of measurements to be performed.

An alternative approach (which will be described in a

next article) is to perform the method comparison with

a user-defined number of measurements and to eval-

uate the probability that a method with an unaccep-

table performance will be accepted.

2. Methods

All symbols and abbreviations used in this paper are

defined in Table 1.

2.1. Experimental design

A 4-factor nested experimental design is used

[37]. This design is also one of the designs recom-

mended by ISO [3]. The schematic layout of the

design is given in Fig. 1. The four factors represent

four sources of variation that contribute to the varia-

bility of the measurements within one laboratory. The

factors considered are operator, instrument, time, and

random error. The experimental approach can be

described as follows. For each analytical method,

the sample is analysed by m operators. Each operator

performs, on each of q instruments, n replicated

measurements on each of p different days. To avoid

an underestimation of the day effect, the set of p

different days during which the measurements are

performed on each of the q instruments must be

different, i.e. two instruments cannot be operated on

the same day.

Fig. 1. Schematic layout for the 4-factor nested experimental

design applied. Only the nested structure under the ith operator, jth

instrument and the kth day is shown here. The nested structure

under other operators, instruments and days has the same pattern.

(instruinstrument, repreplicate).

204 S. Kuttatharmmakul et al. / Analytica Chimica Acta 391 (1999) 203225

Table 1

Definition of symbols and abbreviations applied in the document

d Absolute difference between the grand means obtained with two methods

D Component of day effect in a test result

E Random error component occurring in every test result

FI(OIT) Calculated F-value obtained from the comparison of (operatorinstrumenttime)-different intermediate precision (variance)FI(T) Calculated F-value obtained from the comparison of time-different intermediate precision (variance)

Fr Calculated F-value obtained from the comparison of repeatability variance

FB ;A Value of the F-distribution with B degrees of freedom associated with the numerator and A degrees of freedom associated withthe denominator; represents the portion of the F-distribution to the right of the given F-value

FA ;B Value of the F-distribution with A degrees of freedom associated with the numerator and B degrees of freedom associated withthe denominator; represents the portion of the F-distribution to the right of the given F-value

I Component of instrumental effect in a test result

m Number of operators

M General mean (expectation) of the test results

MS Mean squares

n Number of replicates performed on each day

n Average number of replicates performed on each dayN Total number of measurements

O Component of operator effect in a test result

p Number of days

q Number of instruments

s Estimate of

s2 Estimate of 2

tcal Calculated t-value obtained from the comparison of the means obtained with two methods

t/2 Two-sided tabulated t-value at significance level and degrees of freedom

t One-sided tabulated t-value at significance level and degrees of freedom

UCL Upper confidence limit

y Test resulty Grand mean of test resultsyi Arithmetic mean of the test results obtained from the ith operatoryij Arithmetic mean of the test results obtained from the ith operator and the jth instrumentyijk Arithmetic mean of the test results obtained from the ith operator, the jth instrument and the kth dayyijkL Particular test result related to the Lth replicate of the kth day, the jth instrument and the ith ope