Analyst, March 1996, Vol. 121 (275-278) 275

What Exactly is Fitness for Purpose in Analytical Measurement?

Michael Thompson" and Tom Fearnb a Department of Chemistry, Birkbeck College, University of London, Gordon House, 29 Gordon Square, London, UK WCIH OPP h Department of Statistical Science, University College London, Gower Street, London, UK WCI E 6BT

Fitness for purpose is the principle universally accepted among analytical scientists as the correct approach to obtaining data of appropriate quality. Yet few analytical scientists or end-users of data are in a position to specify exactly what quality of data is required for a specific task. A definition of fitness for purpose based on minimal expected loss is proposed in this paper. This idea enables one to develop optimal strategies for apportioning resources between sampling and analysis, and for balancing technical costs with end-user losses due to error. Keywords: Fitness for purpose; loss functions; quality; sampling

Introduction

A definition of fitness for purpose (FFP) is 'the property of data produced by a measurement process that enables a user of the data to make technically correct decisions for a stated purpose'.' FFP therefore refers to the magnitude of the uncertainty associated with a measurement in relation to the needs of the application area. In some fields of analysis only a very small relative uncertainty can be tolerated. For instance, a very high accuracy would be called for in an analysis used for negotiating a price for a consignment of scrap gold. In contrast, in geochemical prospecting for gold, where the purpose is essentially to distinguish locations that contain low but interesting concentrations of gold from those that do not, a relative uncertainty of 20% is probably all that is required. In the latter case a highly accurate analysis would be not only unnecessary but also prohibitively expensive. Two general conclusions can therefore be drawn: first, FFP demands sufficient but only necessary accuracy in analysis, and second, purely scientific requirements may be constrained by financial considerations.

The uncertainty used to express FFP is strongly related to all aspects of data quality in analytical science. The relationships among these concepts and practices are shown schematically in Fig. 1. A capability for expressing FFP quantitatively and a method for estimating the appropriate value are evidently prerequisites for conducting effective proficiency test$ inter- nal quality control3 and method performance studies (collab- orative trials).4>5

In some areas of analysis this requirement for quantitative information has already been addressed. For example, in applied geochemistry and environmental science, minimal requirements for quality of sampling and analysis have been formulated in relation to the needs of interpretation, although financial constraints have yet to be included in the models.6-10 In clinical biochemistry, empirical rules on data quality have been related to variations between and within patients.11.12 In

other areas the interdependence between costs and the quality of data has been noted but not developed into a useful too1.l3,I4 A minimal expected loss approach has been touched upon but not developed.15 At present, however, there seems to be no systematic approach to estimating the uncertainty that specifies FFP. In common practice, FFP criteria are based simply on professional experience. While the value of that cannot be gainsaid, a demonstrably correct estimate is more likely to dispel contention.

This paper investigates the possibility of a rational approach to FFP in some typical situations involving end-user decisions based on data obtained by the chemical analysis of a sample.

General Approach to Fitness for Purpose Costs of Sampling and Analysis Typically when we analyse a material, we first take a sample and then analyse it. Consider the result x of such a process, which can be broken down as follows:

x = true value + sampling error + analytical error

from which we have the uncertainty of x expressed as a variance v given by

v = v,,, + v, where vSam and vm are the variances of the uncertainties associated with sampling and analysis, respectively. What is the optimum relationship between the values of v,, and v, in the

Fig. 1 Relationship between fitness for purpose expressed as uncertainty and other concepts and practices relating to quality in analytical science.

Publ

ishe

d on

01

Janu

ary

1996

. Dow

nloa

ded

by G

ral U

nive

rsid

ad S

evill

a on

28/

10/2

014

09:0

8:00

. View Article Online / Journal Homepage / Table of Contents for this issue

276 Analyst, March 1996, Vol. 121

context of end-user requirements? This relationship is akin to a well known problem in sample survey statistics.16

The answer depends in part on the relative costs of sampling and analysis. First, we need to consider the costs of improving precision. Higher precision implies higher costs for both the analyst and the sampler. Suppose the cost of an analysis by a particular method characterised by Oan = 1 is B (where Oan = G). What is the cost of a result with double that precision, i.e., (3, = O S ? We could always achieve the improved precision by using the mean of four separate results obtained by the original method at a cost of 4B. In practice, we would be more likely to modify or change the method to achieve the required precision. Nevertheless, at worst we do not need to spend more than 4B. Consequently, we can say that the cost of analysis L, is limited by

L, = B/v, (1)

L,, = A/vs, (2) The constants A and B would be established at some fixed value of variance in a particular method. Eqns. (1) and (2) could be called the ‘cost rules’ for sampling and analysis.

To put these costs into context, we have to consider the financial losses Lend that would be incurred by the end-user of the data as a result of the total error involved in obtaining the measurement. That would depend on the specific purpose to which the data would be put, but in general we could anticipate that increased errors resulting from higher uncertainty variances would have a financial penalty for the end-user. Therefore, there must be optimum values of v,, and Van at which the loss function L given by

Likewise, the cost of sampling is limited by

L = Lend + L a m + Lm is minimized. We suggest that this minimum defines values of v,, and van that are ‘fit for purpose’.

Optimal Division of Resources Between Sampling and Analysis This optimization can be considered in two parts: first, what choice of v = v,, + v, balances cost with potential loss?, and second, what is the best division of resources between sampling and analysis at that particular value of v? Initially, we consider the second question, which can be posed as finding

min ($+$ (3)

given a fixed v = v,, + v,. This kind of constrained minimization can be effected by use of a Lagrange multiplier h. If we define

f = A/vsa, + B/van + h(vS, + van - V )

the minimum is found by solving the simultaneous equations

After some algebra, we find at the minimum

which gives I

A “-=v‘- van B

The combined cost of this optimal sampling and analysis is D/v, where D = ( f l + f l ) 2 . Eqn. (5) is an important general

result applicable to allocating resources between sampling and analysis in a wide range of situations.

The first part of the optimization can now be defined as minimizing the loss function

L = Lend + D/v with respect to the variance v, which is then subsequently apportioned between v,, and v , according to Eqn. (4). Some examples of this procedure follow. (Readers should note that the symbol L is used generically throughout the paper to indicate the particular loss function under consideration at that point.)

Establishing the Value of a Commodity by Analysis Consider the case where a lot of a material is being valued by assay of the principal constituent. This is a common reason for conducting analysis. As a general approach, we consider a situation where the loss to the end-user due to the total error E in obtaining the measurement increases with the magnitude of the error in some way. As a specific example we could postulate that

Lend = CE2

where E = x - c, c is the true concentration of the analyte and C is a constant. This is called the ‘cost function’, and is illustrated in Fig. 2. The cost function adopted here implies that errors in either direction cause a loss to the end-user and it markedly penalizes large errors. The total cost is therefore

L = C E ~ + D / V

In any particular instance, the value of E will be unknown but its distribution may be known or assumed. Then we may consider the expected cost (the prior expected loss), which is the expectation of L over the distribution of E, i.e.,

E(L) = C E ( E ~ ) + D/v

If the sampling and analytical methods are unbiased, E will have a mean of zero and a variance v, so that E ( E ~ ) is just v, and

E(L) = Cv+ D/u

This is minimized when

which gives v = e

v,, = v m v , = V I K

whence

Thus, by adopting a particular cost function we are able to establish optimum (fit for purpose) values for sampling and analytical uncertainties.

Error ( E )

Fig. 2 combined sampling and analytical error.

A hypothetical quadratic cost function based on the magnitude of

Publ

ishe

d on

01

Janu

ary

1996

. Dow

nloa

ded

by G

ral U

nive

rsid

ad S

evill

a on

28/

10/2

014

09:0

8:00

. View Article Online

Analyst, March 1996, Vol. 121 277

An alternative and reasonably realistic type of loss function that could be considered is

Lend = c I E I

illustrated in Fig. 3. Then the total loss would be L = C l ~ l +Dlv

To find the expected loss, we need further assumptions about the distribution of E. In particular, if E - N(O,a*), then E( I E I ) = a m , so that

I

E(L) = C - '," +D/v

Again, this is minimized by setting the derivative to zero, which gives

v = [ 2 4 $ ] ": v,, = v 6 v, = v JT Hence, when these simple cost functions are used, algebraic solutions to the minimization are obtainable. Not all simple cost functions provide algebraic solutions, however.

Use of an Empirical Cost Function Rather than use a general symmetrical loss function as above, we could construct an empirical function that refers to specific end-user requirements. For example, the end-user might specify a maximum tolerable loss due to errors in the process of making the measurement. Loss would be occasioned by the total error E having a positive value and, as a consequence, the purchaser paying for more of the analyte than was actually present in a consignment of the commodity. A tolerable loss could, for instance, be defined in relation to a limit E~ on the error. Of course, the average error would be zero, but the loss function used here represents an extreme aversion to overpayment. The loss function investigated takes the form

0 if E < EI

C if E b E~ I Loss = L

illustrated in Fig. 4. The expected cost function would be

E(L) = CPr(E b ~ 1 ) + Dlv and if we assume E - N(O,v), then

where Q, is the standard normal cumulative distribution function.

Error (E)

Fig. 3 of the combined sampling and analytical error.

A hypothetical cost function proportional to the absolute magnitude

This expression cannot be minimized analytically, but a variety of other methods are available to obtain a solution. For example, given the parameters A = &1000, B = &loo, C = S10000 and ~1 = I , simply plotting the cost L as a function of a = fi (Fig. 5 ) shows a minimum of about &3300 at a total standard deviation of about 1.2 units. Moreover, the plot shows the rapidly escalating costs of using a smaller uncertainty than necessary compared with the costs of above-optimal un- certainty. The indication, perhaps unexpected, is that we should in this instance err, if at all, on the side of higher uncertainty. That is probably sufficient information for most purposes.

Manufacture of a Material with an Upper Bound for an Impurity Consider the manufacture of a material that has an upper limit y 1 for some undesirable but ubiquitous constituent. This is another common reason for conducting analysis. If a batch of the material is prepared and found by analysis to be above the limit, then it must be discarded. It is possible to construct a hypothetical system with these characteristics that illustrates the estimation of a fitness for purpose criterion.

Let us assume that the cost L,f, of the manufacture of a batch varies with the intended level ( c ) of the contaminant: the lower the intended level the greater the cost of production, according to the equation

L,fr = R + SIC

where R and S are constants. As before, the cost of sampling and analysis is given by

Ltot = L,,, + L, = D/v

so the prima facie cost of preparing a batch of the product is L' = L,f, + Lto, = R + SIC + D/v

El Error ( E )

Fig. 4 an arbitrary limit E ~ .

An empirical loss function where loss occurs only if error exceeds

6000 1

3000 I I I I

0.0 0.5 1 .o 1.5 2.0

Total standard deviation

Fig. 5 The loss associated with the empirical loss function in Fig. 4, as a function of combined standard deviation of sampling and analysis, showing the magnitude and shape of the function near the minimum.

Publ

ishe

d on

01

Janu

ary

1996

. Dow

nloa

ded

by G

ral U

nive

rsid

ad S

evill

a on

28/

10/2

014

09:0

8:00

. View Article Online

278 Analyst, March 1996, Vol. 121

/“=2 /c:4.5

5 f j- 20

0 1 2 3 4 5 6 7 8 9 1 0 1 1

Concentration unlts

Fig. 6 Hypothetical system balancing manufacturing costs against costs of sampling plus analysis. When the targetted concentration of a contaminant c = 2 units there is virtually no chance of batch rejection, given the distribution of the measured concentration y’. For the optimal solution, when c = 4.5 and the combined standard deviation of sampling and analysis 0 = 1.4 units, there is a probability of about 0.07 that the resulting measurement exceeds the limit y 1 - 20.

Suppose, however, that the level of contaminant actually achieved (y) varies around the intended mean according to the distribution y - N(c,o,?) where oJc = 0.25. A proportion of the batches will be wasted if they are rejected by the analytical test. If p is the probability of rejection for a single batch, then the long-run proportion of batches accepted is 1 - p and the effective cost per accepted batch is

R + S/c + D/v L’ E(L) = - - - 1 - P 1-P

The value of p is determined by both the distribution of the batch concentration and the uncertainty of sampling and analysis (Fig. 6). A rational policy would be to retain the batch so long as the analytical measurement was below the specifica- tion limit for the analyte y1 by at least two standard deviations of the measurement error. Hence, writing o = 6, we have

R + S/c + 0 1 0 2

1 - Pr(y’ > y1 - 20) E(L) =

where y’ is the measured value of y. If the distribution of y’ given c is

then the expected loss is

E(L) = R + S/c + D/a2

y1- 20 - c @ ( v m ) This cost function can be minimized numerically with respect to the intended concentration c and the uncertainty variance. For a given choice of parameters, contours of cost can be plotted as a function of c and 0. For example, if we choose R = &200, S = &1000, D = 2200 and y1 = 10 concentration units, the cost function shown in Fig. 7 is obtained. This shows a minimum batch cost of &561 at 0 = 1.4 concentration units with a target concentration of the contaminant at c = 4.5 units. Under the optimum conditions, about 7% of the batches are rejected.

It would be possible to elaborate the cost function further by considering the possible penalties associated with false accep- tance of a batch of product that in fact did not comply with requirements. However, that consideration is beyond the scope of this paper.

0

3 2

I I I I

0 1 2 3 4

CT

Fig. 7 Production costs as a function of the targeted concentration c of the contaminant and the combined standard deviation of sampling and analysis u. shown as contour lines with a well defined minimum.

sampling and analysis should be appropriate, given the purpose of the measurement and the likely consequences of error. A natural way to formalize this idea is to set up optimization situations of the type presented above. However, the point of this paper is not to recommend the particular loss functions in the examples but to demonstrate (i) that there are well established methodologies for dealing with this type of problem and (ii) that the solutions are often fairly simple.

In practice, it would often be difficult to quantify costs and losses exactly. Further, it may not be possible to implement an optimal solution exactly-we would tend to use the available methods of analysis and sampling that best approximated the optimum requirement. Even so, a knowledge of where the optimum lies and the shape of the minimum, given a reasonable guess at the costs, will at least serve as a warning when the proposed sampling and analytical methods are not appropriate. This will not always be because methods of smaller uncertainty should be used-sometimes the warning will be that expensive methods are not justified by the purpose for which they are proposed.

References

5 6

7

8 9

10 11 12

13 14 15

16

Thompson, M., and Ramsey, M. H., Analyst, 1995, 120, 261. Thompson, M., and Wood, R., Pure Appl. Chem., 1993,65,2123. Thompson, M., and Wood, R., Pure Appl. Chem., 1995,67,649. Accuracy (Trueness and Precision) of Measurement Methods and Results, IS0 5725 : 1994, International Standards Organization, Geneva, 1994. Horwitz, W., Pure Appl. Chem., 1995, 67, 332. Handbook of Exploration Geochemistry, Vol. 2: Statistics and Data Analysis in Geochemical Prospecting, ed. Howarth, R. J., Elsevier, Amsterdam, 1983, pp. 92-107. Ramsey, M. H., Thompson, M., and Hale, M., J. Geochem. Explor., 1992, 44, 23. Ramsey, M. H., Appl. Geochem., 1993,2, 149. Ramsey, M. H., in Sampling of Environmental Materials for Trace Analysis, ed. Markert, B., VCH, Weinheim, 1994, pp. 93-108. Thompson, M., and Maguire, M., Analyst, 1993, 118, 1107. Tonks, D., Clin. Chem., 1963, 9, 217. Fraser, C. G., Petersen, P. H., Ricos, C., and Haekel, R., Eur. J . Clin. Chem. Clin. Biochem., 1992, 30, 31 1. Lundell, G. E. F., Znd. Eng. Chem., Anal. Ed., 1933, 5, 221. Horwitz, W., J. Assoc. Off. Anal. Chem., 1983, 66, 1295. Massart, D. L., Vanderginste, B. G. M., Deming, S. N., Michotte, Y., and Kaufman, L., Chemometrics: a Textbook, Elsevier, Amsterdam,

Hansen, M., Sample Survey Methods and Theory, Wiley, New York, 1988, pp. 141-147.

1953, pp. 221-223.

Conclusions If fitness for purpose is to mean anything concrete, the concept must be based on the premise that the resources allocated to

Paper .5/06045A Received September 13, I995

Accepted November 9, I995

Publ

ishe

d on

01

Janu

ary

1996

. Dow

nloa

ded

by G

ral U

nive

rsid

ad S

evill

a on

28/

10/2

014

09:0

8:00

. View Article Online