Accred Qual Assur (2()()() 5: 95-99 © Springer-Verlag 2()()O

Jean Pauwels Adriaan van der Veen Andree Lamberty Heinz Schimmel

Presented at: EURACHEM Workshop on Efficient Methodology for the Evaluation of Uncertainty in Analytical Chemistry, Helsinki, Finland 14-15 June 1999

J. Pauwels (181) . A. Lamberty H. Schimmel I nstitute for Reference Materials and Measurements, EC-JRC-IRMM, 244() Geel, Belgium e-mail: jean.pauwels@irmm.jrc.be Tel.: + 32-14-571722 Fax: +32-14-590406

A. van der Veen Nederlands Meetinstituut, P.O. Box 654, 26()() AR Delft, The Netherlands

Introduction

Evaluation of uncertainty of reference materials

Abstract Certification of reference materials is far more than just characterisation of a selected ho­mogeneous batch of material. From the perspective of the ISO Guide on the Expression of Uncer­tainty in Measurement (GUM) all uncertainty sources relevant to the user of an individual certified ref­erence material (CRM) sample at a moment in time should be part of the CRM uncertainty. This not only includes the full uncertainty of the batch characterisation (rath­er than the statistical variation), but also all uncertainties related to possible between-bottle variation, instability upon long-term storage

and instability during transport to the customer.

Key words Certified reference materials' Uncertainty· Characterisation . Uncertainty analysis

The accurate and traceable determination of a mean value of a quantity (content, amount) in a sample or a batch of material can be obtained in various ways, such as carrying out a number of independent repetitions us­ing a primary method of analysis [1], comparing the re­sults of a limited number of reference methods, or com­paring the results of various independent methods ap­plied in a series of laboratories. These three different methods are used by various producers to certify the values assigned to their reference materials (RMs), whereby this assignment is done using quite similar statements, but these statements may sometimes have very different meanings. Moreover, it must also be real­ised that the certification of a RM is much more than just carrying out a series of precise and accurate meas­urements traceable to the SI or to any other system of

units, to written or agreed standards or to an artefact, such as, e.g. the primary WHO materials to which sev­eral clinical RMs are traceable. The certification of a RM involves, in the first instance, the preparation of a larger number of homogeneous, stable and adequately packaged samples which are all representative of the complete batch, as well as the proper assessment of their homogeneity and stability. Ignoring this is not only one of the main reasons why problems occur with certified reference materials (CRMs), but also why they are the subject of needless discussions about primary, secondary, consensus, working, etc. RMs. This distic­tion in classes of RMs mainly exists in the mind of some metrologists, but is fully absent in the existing ISO­REMCO Guides. The latter only differentiate between (just) RMs and CRMs, whereby a RM is defined as "a material or substance one or more of whose property values are sufficiently homogeneous and well estab­lished to be used for the calibration of an apparatus,

30 1. Pauwels et al.

the assessment of a measurement method, or for assign­ing values to materials", and a CRM is just "a RM with a certificate in which the certified values are accompa­nied by an uncertainty at a stated level of confidence" [2].

What is a CRM user interested in?

CRMs are sometimes forced into a hierarchical system depending on the fact that there certified val~es were determined using a primary method of analysIs or are based on "less traceable" measurements obtained in a laboratory intercomparison. In reality, such a differen­tiation is meaningless, considering that very often the uncertainty component which originates from the c~ar­acterisation of the RM is dominated by uncertaInty components originating from several oth~r sources su.ch as insufficient guarantee of absence of InhomogeneIty and/or instability. Therefore, it is not correct when pro­ducers certify their RMs just considering the results of their accurate and traceable determinations of the mean value of the content of the CRM batch, knowing that their customers (users) are only interested in the mean value of the single bottle they ordered on condi­tion that it is received on the day of dispatch.

The ongoing revision of the ISO-Guide 35 [3] -which constitutes a complete rewriting - is therefore a unique opportunity to reconsid~r the pro~uction of CRMs. It will consider productIOn as an Integrated process of correct preparation, positive demonstration of homogeneity and stability, and accurate and tracea­ble characterisation, and thus of full implementation of the principles laid down in the Guide to the Expression of Uncertainty in Measurement (GUM) [4]. ThIS means that all components of uncertainty of "the sample on the desk of the user" should be properly evaluated and accounted for. Thereby, it must be strongly emphasis~d that the inability to demonstrate between bottle vana­tion or instability during storage or transportation, as well as confining the uncertainty of the batch ch.ar~cte~­isation to the statistical between-laboratory vanatIOn IS no longer acceptable. Ignoring this is one of the major causes of the so-called "Jorhem paradox" discussed at BERM-7 [5] where it was (rightly) found unacceptable that "results found to be unacceptable for user labora­tories are good enough to be used in the certification of the CRM", even if it is statistically just logic [6]! The consequence is, however, that one will h~ve.to acce~t­just as was the case for testing laboratones IntrodUCIng GUM - that uncertainties of CRMs will increase "from fiction to reality": an idea which is apparently difficult for many analysts to become accustom to, and which, moreover, may confuse those who tend to c~mpare the quality of the CRMs of various producers Just on the basis of the quoted uncertainty.

Uncertainty analysis in the preparation of a CRM

From the reasoning given above, it becomes apparent that the certification of a RM includes far more than just the characterisation of the material. This step, of~­en carried out as a collaborative study between multI­ple laboratories, is crucial for the quality of the material as a CRM, but it is generally insufficient.

From the perspective of EURACHEM Guide [7] as well as from GUM [4], a producer should include all uncertainty sources that are relevant to the package sold to the customer. Internal consistency of the uncer­tainty analysis requires the inclusion of the (residual) uncertainty from the experiments carried our for ho­mogeneity and stability testing. So, ev~n if th~ prod~~er cannot demonstrate any inhomogeneIty or InstabIlIty, there is still a (small) uncertainty budget to be included. Usually, this budget will be small, but in cases where only poorly repeatable methods of m~as~~ement are available, this contribution may be of sIgmfIcance.

A further consequence of this is that it really "pays off" in terms of uncertainty if a sufficient number of replicate measurements is carried out in ~omogene~ty and stability testing. The use of methods WIth good lIn­earity, selectivity and repeatability will also greatly co~­tribute to reducing the uncertainty from these expen­ments. These factors are all in the hands of the produc­er. Implementing them correctly and consistently will reduce the costs of "after sales" of a CRM producer, not to speak of the subsequent damage due to wrongly certified RMs.

This way of thinking may seem new, but those who have already gained experience with inhomogeneous and/or instable RMs have already developed ways to deal with these aspects. A CRM producer should in­clude in an uncertainty statement everything that "rea­sonably attributes" (GUM) to the uncertai~t~ of th.e measurand, i.e. the property value to be certIfIed. T.hIS ends where accidents and incidents start: if somethIng happens to a CRM during transport that goes bey?nd what can be foreseen, it is not part of an uncertaInty statement, as the information on the certificate will sti­pulate under what conditions the certificate (and the CRM) are valid.

What is important in the preparation of a CRM?

Good measurements carried out on bad quality candi­date RMs are a nonsense and a complete waste of time and money! Therefore, extreme care should be taken not only to prepare a stable and homogeneous base material, but also to sample it in a tight and inert con­tainment [8]. Matrix CRMs require in gen.eral to b~ clean and dry, to be transformed into an optImal phYSI-

cal and chemical form, and to be stored at the correct temperature from a very early stage in the production process. In general, microbiological degradation can be minimised by reducing the water content of the materi­al to a level between 1 and 3%. Packaging is best car­ried out in an atmosphere of argon - not under vacuum as this may become a source of leaks - whereby all pre­cautions must be taken to guarantee absolute tightness. This can be achieved using bottles with inserts, penicil­lin vials or ampoules, whereby it must be stressed that all three solutions have failed in the past: bottles and vials due to insufficiently tight or retracting inserts (e.g. due to ageing or freeze-temperature effects) or am­poules due to cracks appearing during storage as a con­sequence of stresses present in the glass.

What is important in homogeneity testing?

Homogeneity testing addresses a double problem: What is the variation in mean value which exists be­tween the various units of a batch of candidate RM? And, how inhomogeneous is the material contained in a bottIe?

The first problem is of utmost importance to the user as he/she will, in general, buy just one bottle, and will not care about the other ones! Therefore, between­units variation is an important component of uncertain­ty which must be included in the certified value of the CRM. The determination of the between-units varia­tion is carried out by measuring the value of a signifi­cant number of units. As the result of such measure­ments is a combination of two effects, the between-bot­tle variability [Shh] and the measurement repeatability [smcas]

(1)

the variation between the mean value of the bottles can only be obtained from measurements carried out with the highest repeatability: i.e. that each bottle must be analysed, using a highly repeatable method, on sample intakes of optimal size and carrying out a number of repetitions which is sufficient to obtain a measurement uncertainty which is negligible compared to the varia­tion between the bottles, i.e. s~cas < < S~h' Usually, this is however not the case. Then, U~h should, as far as pos­sible, be corrected for s~cas to obtain the best estimate of S~h [9].

To evaluate the inhomogeneity of the material con­tained in a bottle, within-bottle measurements have to be carried out. Also here, the result of such measure­ments is a combination of two effects, the within-bottle inhomogeneity [Sinh] and the method repeatability [Smclh]

(2)

Evaluation of uncertainty of reference materials 31

and the variation between the different samples within a bottle can only be obtained from measurements car­ried out using a highly repeatable method so that the method repeatability is negligible compared to the var­iation between the samples in a bottle, i.e. S~Clh« S?nh'

In this case, sample intakes must however be minimal, as the contribution of S~Clh to U?nh becomes negligible when extrapolating S?nh from smaller (m) to larger (M) sample sizes according to:

[S?nh]M = [S?nh]m' m/M == [U?nh]m' m/ M (3)

It must be emphasised that Sinh is irrelevant for the CRM uncertainty, provided the minimum representa­tive sample intake is properly determined. The value of Sinh is, however, of prime importance to estimate this minimum representative sample intake correctly [10].

In both cases it should be noted that: - Not correcting Uhh or Uinh for Smcas or Smclh is not real­

ly a problem, but leads to (too) conservative CRM uncertainty estimates.

- Corrected Shh or Sinh values may never be taken smaller than their respective combined uncertainties, i.e. U(Shh) and U(Sinh) [9].

What to do with stability data?

Stability testing at higher temperatures simulating pos­sible transport conditions and conditions of long-term storage are often part of procedures describing the pro­duction of CRMs [11]. In most cases they do, however, not give quantitative information on presumed instabil­ity, mainly as a consequence of insufficient measure­ment reproducibility and of an insufficient number of replicates. With the upcoming requirements of fixing expiry dates [12], it will be mandatory that not only quantitative data be available, but that their quality is such that high precision extrapolations can be made. This requires however that data are produced with measurement reproducibilities (or repeatabilities when isochronous measurements are carried out [13]) which are negligible compared to the certified uncertainty. An extrapolation method was recently proposed by Pau­wels et al. [14] to determine the time for which the cer­tified value of a CRM remains valid, based on the de­termination of the intersection of the lower 95% confi­dence bound with the lower limit of the certified confi­dence interval (see Fig. 1). Such calculations show how­ever that, with the levels of uncertainty presently cer­tifed, either unrealistically high precisions are required, or that shelf-lifes must be reduced to unrealistically short periods of time, even if one considers that further stability monitoring during the lifetime of the CRM makes regular re-evaluation and updating of the shelf­life possible. Therefore, in many cases, it may become necessary to re-evaluate the certified uncertainties of

32 1. Pauwels et at.

1.1

1.08

1.06

1.04

,--._--_.

• • • ----I

,; 1.02

~ 1

~ 0.98

~ 0.96 :>

0.94

0.92

0.9

o

-

• 10

• ...

- :;-• • • •

20

, •

:

i • i

• I

---------:

------------I

• --- : ""1

30 40 50 60

time (months)

Fig.l Example of determination of the long-term stability of cer­tified reference material (CRM): Cr in CRM 27RR (mussel tissue)

RMs taking into account a realistic stability uncertainty. Possibly, other approaches may be found to solve this extremely important problem, such as the one pro­posed by a group of experts working in the framework of a "Standards, Measurements and Testing Accompa­nying Measure" under the co-ordination of LGC (s. Burke, personal communication), consisting in extrapo­lating the certified value to mid-way of an arbitrarily chosen life-time and calculating the associated supple­mental uncertainty.

A similar reasoning may be appropriate for possible degradation of the CRM during transportation to the customer.

The characterisation of a homogenous batch of material

The estimation of the mean value of a quantity of a CRM batch using: (1) a primary method of analysis, or (2) by comparing the results of a limited number of ref­erence methods, or (3) the results of various indepen­dent methods applied in a series of laboratories should, in fact, only be variants of one and the same philoso­phy. The third characterisation method, however, re­quires that a number of analyses are carried out by one or more techniques in one or more laboratories, where­by each series of measurements is carried out with maxi-

References

mal guarantees of accuracy and traceability, and must be documented by a full uncertainty budget. For each set of determinations an expanded standard uncertainty ac­cording to GUM should then be calculated. The final estimation of the uncertainty of the characterisation of the batch (Uchar) should then take into account all these standard uncertainties, considering that those uncer­tainties which have been repeatedly determined in an independent way, decrease proportionally with the square root of the number of degrees of freedom. A proposal to handle this problem was published by Pau­wels et al. [15]. It is based on a separate consideration of three types of standard uncertainties: - Those which are exclusively laboratory dependent. - Those which are common to all laboratories, such as

the effect of between-bottle variation or the use of a common calibrant.

- Those which are common to groups of laboratories, e.g. those using the same measurement procedure. In this context it should be noted that matrix CRMs

are generally certified for mass fractions related to dry matter, i.e. that not only the amount of substance but also the dry sample mass has to be assessed and its un­certainty evaluated: a problem that is ignored and/or underestimated by many analytical chemists and a po­tential source of significant errors and unaccounted un­certainties in CRMs.

The CRM uncertainty according to GUM

The final uncertainty of a CRM according to GUM should consider all sources of uncertainty described above:

[ 2 2 2 2 ] 1/2 UCRM = Uchar + U oo + Ults + Usts , (4)

whereby Its and sts refer to long-term stability (upon storage) and short-term stability (during transport), re­spectively.

It is good practice to quantitatively determine all sources of uncertainty, be they significant or not. In the latter case they will anyhow disappear in the rounding­off of the calculation, but it will: - Avoid the risk of overlooking sources of uncertainty

due to ignorance. - Demonstrate to users that they have been considered

and what is their magnitude.

1. Quinn TJ (1997) Metrologia 34:61-65 2. ISO Guide 30 (19R1) Terms and defi­

nitions used in connection with refer­ence materials. ISO, Geneva, Switzer­land

3. ISO Guide 35 (19R9) Certification of reference materials - General and statistical principles. ISO, Geneva, Switzerland

4. ISO (1995) Guide to the expression of uncertainty in measurement. ISO, Geneva, ISBN 92-67-101 RR-9

5. Jorhem L (199R) Fresenius J Anal Chern 306: 370 373

6. Pauwels J (1 YYY) In: Fajgeli A, Parka­ny M (eds) The use of matrix refer­ence materials in environmental ana­lytical processes. The Royal Chemical Society, London, pp 31-45

7. EURACHEM (lYY5) Quantifying un­certainty in analytical measurement. EURACHEM, London, ISBN O-Y4SY26-0S-2

S. Kramer GN, Pauwels J (lYY6) Mikro­chim Acta 123:S7 -Y3

Evaluation of uncertainty of reference materials 33

Y. Pauwels J, Lamberty A, Schimmel H (lYYS) Accred Qual Assur 3:51-55

10. Pauwels J, Vandecasteele C (lYY3) Fresenius J Anal Chern 345:121-123

11. European Commission: DG XII-C-5 - document BCRIOlIY7 (1 YY7) Guide­lines for the production and certifica­tion of BCR reference materials. Eu­ropean Commission, Brussels

12. ISO Guide 31 (1 YYS) Reference ma­terials - Contents of certificates and labels (draft). ISO, Geneva, Switzer­land

13. Lamberty A, Schimmel H, Pauwels J (1 YYS) Fresenius J Anal Chern 360: 35Y-361

14. Pauwels J, Lamberty A, Schimmel H (1 YYS) Fresenius J Anal Chern 361:3Y5-3YY

15. Pauwels J, Lamberty A, Schimmel H (IYYS) Accred Oual Assur 3:1S0-1S4