This article was downloaded by: [Varun Ramamohan]On: 03 October 2012, At: 14:17Publisher: Taylor & FrancisInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,37-41 Mortimer Street, London W1T 3JH, UK

IIE Transactions on Healthcare Systems EngineeringPublication details, including instructions for authors and subscription information:http://www.tandfonline.com/loi/uhse20

A Monte Carlo approach to the estimation & analysisof uncertainty in clinical laboratory measurementprocessesVarun Ramamohan a , Vishal Chandrasekar a , Jim Abbott b , George G. Klee c & YuehwernYih aa School of Industrial Engineering, Purdue University, 315 N Grant St., West Lafayette, IN,47907, USAb Clinical Support Group, Roche Diagnostics Corporation, Indianapolis, IN, USAc Department of Laboratory Medicine & Pathology, Mayo Clinic, Rochester, MN, USA

Accepted author version posted online: 17 Feb 2012.Version of record first published: 09May 2012.

To cite this article: Varun Ramamohan, Vishal Chandrasekar, Jim Abbott, George G. Klee & Yuehwern Yih (2012): A MonteCarlo approach to the estimation & analysis of uncertainty in clinical laboratory measurement processes, IIE Transactions onHealthcare Systems Engineering, 2:1, 1-13

To link to this article: http://dx.doi.org/10.1080/19488300.2012.665153

PLEASE SCROLL DOWN FOR ARTICLE

Full terms and conditions of use: http://www.tandfonline.com/page/terms-and-conditions

This article may be used for research, teaching, and private study purposes. Any substantial or systematicreproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form toanyone is expressly forbidden.

The publisher does not give any warranty express or implied or make any representation that the contentswill be complete or accurate or up to date. The accuracy of any instructions, formulae, and drug doses shouldbe independently verified with primary sources. The publisher shall not be liable for any loss, actions, claims,proceedings, demand, or costs or damages whatsoever or howsoever caused arising directly or indirectly inconnection with or arising out of the use of this material.

IIE Transactions on Healthcare Systems Engineering (2012) 2, 1–13Copyright C© “IIE”ISSN: 1948-8300 print / 1948-8319 onlineDOI: 10.1080/19488300.2012.665153

A Monte Carlo approach to the estimation & analysis ofuncertainty in clinical laboratory measurement processes

VARUN RAMAMOHAN1,∗, VISHAL CHANDRASEKAR1, JIM ABBOTT2, GEORGE G. KLEE3

and YUEHWERN YIH1

1School of Industrial Engineering, Purdue University, 315 N Grant St., West Lafayette, IN, 47907 USAE-mail: vramamoh@purdue.edu2Clinical Support Group, Roche Diagnostics Corporation, Indianapolis, IN, USA3Department of Laboratory Medicine & Pathology, Mayo Clinic, Rochester, MN, USA

Received November 2010 and accepted February 2012.

Clinical laboratory testing is a vital component of many stages of the medical decision making process, and therefore informationabout the quality of the measurement process is critical to the medical decision-making process. A statement of uncertainty of theresult of a laboratory test provides this information. To obtain this information, the clinical laboratory measurement process isconceptualized as a self-contained system, the concept of process phases is introduced, and a broadly applicable algorithm describingthe modeling and estimation of uncertainty of such processes is developed. The article discusses how performance specifications forindividual components can be used to characterize their uncertainty, and uses Monte Carlo simulation to integrate these individualcomponent uncertainties into a net system uncertainty. The proposed approach is illustrated by developing a mathematical modelof the serum cholesterol assay analysis procedure. The uses of the model are to: 1) simulate, evaluate and optimize quality controlpolicies without resorting to conducting controlled experiments, 2) obtain performance targets for the measurement process by usinguncertainty estimates from the simulation, 3) estimate the contribution of each source of uncertainty to the net system uncertainty,and 4) study the effects of varying the parameters of the system on the net system uncertainty are illustrated with examples.

Keywords: Clinical laboratory testing, laboratory measurement processes, uncertainty estimation, Monte Carlo simulation, serumcholesterol assay

1. Introduction and literature review

A statement of uncertainty associated with the result of ameasurement describes the quality of the measurement pro-cedure. Information about the quality of the measurementresult becomes even more important in the case of clinicallaboratory test results, since clinical test results inform ev-ery stage of medical decision-making, including diagnosis,prognosis and drug dosage prescription. This need has beenrecognized by the United States Congress by its passageof the Clinical Laboratory Improvement Amendments in1988, which requires laboratories to validate laboratoryprocedures and establish valid quality control systems. Es-tablishing valid quality control systems requires knowledgeof the current level of quality being delivered by the labora-tory, which is provided by estimates of measurement uncer-tainty. In addition, accreditation to ISO/IEC 17025 (1999)also requires testing laboratories to provide estimates ofmeasurement uncertainty. Currently, many laboratories

∗Corresponding author

perform minimal testing to verify measurement perfor-mance in terms of the possible range of values that canbe reported by the laboratory test, measurement accuracywith respect to known standard samples, and reproducibil-ity using one lot of reagents and one operator, which doesnot adequately describe the system performance. Thisstudy presents a methodology that enables the analyticestimation of the uncertainty associated with a clinicallaboratory measurement system via the development ofa simulation model of the measurement system. Thisinformation, used in conjunction with the performance in-formation collected in the laboratory, can be used to designquality control systems that bridge the disparity betweenclinically required performance and that observed in thelaboratory. This study was carried out in collaboration withRoche Diagnostics Corporation based in Indianapolis, IN(USA) and the Mayo Clinic at Rochester, MN (USA).

This methodology involves the development of amathematical model of clinical laboratory measurementprocesses, and utilizes the Monte Carlo simulation tocharacterize the behavior of such a model with uncertaintyin its variables. General guidelines for the development of

1948-8300 C© 2012 “IIE”

Dow

nloa

ded

by [

Var

un R

amam

ohan

] at

14:

17 0

3 O

ctob

er 2

012

2 Ramamohan et al.

the mathematical model are presented, and these guidelinesare implemented to model the propagation of uncertaintythrough the blood serum cholesterol concentration mea-surement procedure. The sources of uncertainty associatedwith the serum cholesterol laboratory measurementprocedure are identified and specifications obtained fromthe literature and the instrument manufacturer are usedto characterize their variation. A functional relation-ship is established between the measurand—cholesterolconcentration—and the sources of uncertainty. The usesof this model to estimate the contribution of the varioussources of uncertainty to the net system uncertainty, and tosimulate and evaluate quality control policies are illustratedwith examples. This model will be useful for instrumentmanufacturers as the model quantifies the contribution ofeach source of uncertainty and therefore provides guidanceto direct their design efforts. As will be shown in Section4 of the article, clinicians can use this model to evaluatethe feasibility of implementing new laboratory practicesor quality control protocols as an alternative or aid toconducting controlled experiments in the laboratory.

The concept of uncertainty and the mathematicalrules for analytically estimating uncertainty for relativelysimple systems were first formally introduced in theISO/BIPM/OIML/IUPAC Guide to the Expression ofUncertainty in Measurement in 1993 (GUM, 1993) andlater revised in its subsequent editions and companion pub-lication (JCGM 100, 2008). The ‘uncertainty’ associatedwith the quantity to be measured or measurand refers to aparameter used to characterize the dispersion of values thatcan be attributed to the measurand (GUM, 1993). Whilethere have been publications (Alexandrov, 2001; Grabe,2001) criticizing the GUM (1993), particularly for its dis-carding of the notions of ‘true value’ and random andsystematic error, it is now widely regarded as the definitiveguide on the description of measurement variability. In thispaper, any component that is subject to variability is char-acterized by a probability distribution with the expectedvalue and the standard deviation as parameters, with thestandard deviation used as the parameter describing theuncertainty of measurement.

A typical clinical laboratory measurement process isdivided into three stages: the pre-analytical stage, whichdeals with patient sample collection, transportation, stor-age, preparation, etc; the analytical stage, which involvesanalysis of the patient sample using a calibrated instrument;and the post-analytical stage, which deals with recording,reporting, and interpreting the result of the measurement.Uncertainties are associated with each of above stages, andthe uncertainty of each stage arises from multiple sourcesof variation operating within that stage. Identifying all thesources of variation associated with each stage and hencewith the entire measurement process is often impractical(Linko et al., 2002); however, a list of important sourcesof variation associated with each stage is provided inTable 1. The methodology presented in this study focuses

Table 1. Laboratory measurement process stages

MeasurementProcess Stage Sources of Variation

Preanalytical Patient identificationPatient preparationVenipuncture site selectionSite preparationTourniquet application and timeProper venipuncture techniqueOrder of drawProper tube mixingCorrect specimen volumeProper tube handling and specimen

processingCentrifugationSpecial handling for blood specimensStorage and transportation temperature

stabilityAnalytical Calibrators or Certified Reference Materials

ReagentsInstrumentation

Post-analytical Measurement result recordingMeasurement result reportingMeasurement result interpretation

on identifying and characterizing the sources of uncertaintyassociated with the analytical stage of the process, and es-timates the uncertainty of the analytical stage of the labo-ratory measurement process via the Monte Carlo method.As can be seen from Table 1, there are numerous sources ofuncertainty associated with the pre-analytical phase, andquantifying the variation of these sources requires a sepa-rate discussion in its own right, and is beyond the scopeof this paper. However, as will be shown in Section 5,the simulation model can be used to estimate the effectof the pre-analytical stage on net system uncertainty fora range of values of pre-analytical uncertainty. The post-analytical stage typically does not contribute to measure-ment uncertainty; however, human errors may occur in thereporting and interpretation of results. These errors are, bydefinition, out of the scope of this study.

Various attempts have been made to estimate theuncertainty associated with specific analytical processes(Kristiansen et al., 1996; Kallner and Waldenstrom, 1999;Petersen et al., 2001; Linko et al., 2002; Patriarca et al.,2004; Burns and Valdivia, 2006; Leung et al., 2007; Fuentes-Arderiu and Dot-Bach, 2009). A broadly applicablemethodology proposed by Kallner (1999) suggests apply-ing the modeling process described in GUM (1993) and usesthe laws of uncertainty propagation for characterizing theuncertainty of the system. This study utilizes uncertaintyestimates for the purpose of setting quality specificationsfor such a measurement system. The publication of theCITAC/EURACHEM ‘Quantifying Uncertainty in Ana-lytical Measurement, 2nd Edition’ in 2000 (EURACHEM,

Dow

nloa

ded

by [

Var

un R

amam

ohan

] at

14:

17 0

3 O

ctob

er 2

012

Monte Carlo approach in measurement processes 3

2000) provides useful guidelines on implementing theGUM (1993) principles of uncertainty estimation inanalytical chemistry. Fuentes-Arderiu (2000) identifiedsome general elements of variation in clinical laboratoriesand raised some concerns regarding the implementationof the GUM (1993) principles in clinical laboratories suchas the quantification of pre-analytical uncertainty anduncertainty associated with the calibrators. Kristiansen(2001) proposed a generally applicable model for theevaluation of the uncertainty associated with clinicallaboratory measurement systems, based on the guidelinesproposed in GUM (1993), and applied the model forthe estimation of uncertainty of prolactin in humanserum. Burns (2004) published a comprehensive reviewof measurement uncertainty in bio-analytical chemistryand concluded that very few disciplines within the fieldimplement a full uncertainty budget associated with theirresults. The author goes on to suggest that the complexnature of implementing uncertainty estimation principlesbe structured into a more systematic basis. The systemsengineering perspective employed in this study attempts tolend precisely such structure to the uncertainty modelingprocess in clinical laboratories.

The concept of applying a systems engineering approachto model analytical methods in chemical laboratories haspreviously been suggested by Aronsson, de Verdier, andGroth (1974), and Krouwer (2002). In this article, wepresent guidelines for the development of a mathemati-cal model of a clinical laboratory measurement processfrom the systems engineering viewpoint. This involves con-ceptualizing the different stages of the clinical laboratorymeasurement process as a self-contained system. The con-cept of process phases of the analytical stage of a measure-ment process is introduced, and the principal componentsinvolved in a general clinical laboratory measurement pro-cess are identified. The input to the system, represented bythe patient sample, is processed by the system components,and a property of the sample is quantified and convertedinto the output quantity, represented by the result of themeasurement.

The Monte Carlo method is used to characterize thebehavior of the model under conditions of uncertainty inits components. As stated in the GUM Annex 1 (JCGM101, 2008), the rule of uncertainty propagation proposedin the GUM becomes unsuitable when: 1) the mathemati-cal model involved is non-linear in nature; 2) the behaviorof the measurand of the system is not Gaussian in nature;and 3) estimating the degrees of freedom for the sources ofuncertainty is not possible, particularly for Type B charac-terizations (GUM, 1993). The third reason is particularlyrelevant here, as the statistical characterization (Type A) ofthe various sources of uncertainty using relevant data is of-ten not practically possible in the laboratory. For instance,in the implementation of the model for the serum choles-terol assay described in Section 2.3, all the sources of uncer-tainty were characterized by Type B methods. The method

proposed by Kragten (1994) also becomes unsuitable forprecisely the above reasons, as the Kragten spreadsheetmethod is an automation of the GUM rule of propagationof uncertainty. The primary advantage of the Monte Carlomethod is that it requires the development of a simulationmodel of the process; which, as discussed earlier, grantssignificant flexibility in terms of analyzing the system un-certainty and modeling various laboratory practices. Forcomplex systems, the mathematical model itself will not bewritten out in its entirety (JCGM 101, 2008); expressionsrepresenting each step of the process are usually provided,thereby further encouraging the use of simulation for char-acterizing the behavior of such systems under uncertaintyin its components.

2. Process modeling

2.1. System description and process phases

In the analytical stage of any clinical measurement system,the measuring instrument analyses the patient sample andprovides the value of the measurand. A distinction mustbe made here between the terms analyte and measurand:the analyte is the substance or chemical constituent that isdetermined in an analytical procedure, whereas measurandrefers to the amount of the analyte in the patient sample.There are typically two types of measurands: direct measur-ands, wherein the amount of analyte is directly measuredby the instrument, and indirect measurands, wherein theinstrument quantifies a property of the analyte, and thismeasured property is converted into the measurand valuevia the calibration equation. In most measurement systems,the instrument first has to analyze known standards, anda calibration function has to be established. This phaseof the analytical process, wherein a calibrator or CertifiedReference Material (CRM) is analyzed by the instrumentto establish the calibration function, is called the calibra-tion phase. Three major components of the system are in-volved in this phase: the calibrator or reference material,the measuring instrument, and the reagents. In the contextof uncertainty estimation, there are three principal compo-nents of uncertainty associated with this phase: calibratoruncertainty, reagent uncertainty and instrument uncertainty.These combine together to form the uncertainty associatedwith this phase, calibration uncertainty.

The calibrated instrument is then used to analyze thepatient sample and provide the value of the measurand assystem output. This phase of the analytical process—wherethe patient sample analysis takes place—is called the mea-surement phase. There are three principal components ofthe system involved in this phase, the patient sample, thereagents and the measuring instrument. In terms of un-certainty estimation, the uncertainty associated with thepatient sample is called specimen uncertainty, and the un-certainties associated with the reagents and the instrument

Dow

nloa

ded

by [

Var

un R

amam

ohan

] at

14:

17 0

3 O

ctob

er 2

012

4 Ramamohan et al.

Fig. 1(a). System conceptualization.

are, as before, reagent uncertainty and instrument uncer-tainty. These combine together to form the uncertaintyassociated with this phase, measurement uncertainty. Cali-bration uncertainty and measurement uncertainty combineto form the net system uncertainty, which is the uncertaintyassociated with the measurand. The division of the mea-surement process into phases provides intuitive structureto the modeling process and facilitates an understandingof the contribution of different stages of the measurementprocess to the net system uncertainty. Such a conceptual-ization of the clinical laboratory measurement process isshown in Figure 1(a).

There are a few important points that must be men-tioned here: 1) Even though certain principal componentsplay a part in both phases, the uncertainty associated witheach phase is unique. 2) Specimen uncertainty representsuncertainty that changes the concentration of the analytein the patient specimen before analysis on the instrument,and hence represents the uncertainty associated with thepre-analytical stage. 3) While a majority of commonly usedlaboratory tests involve a chemical reaction that requiresthe use of reagents, there are some laboratory measure-ment processes that do not involve a chemical reaction;and hence do not have reagents as a principal component.

Fig. 1(b). P-module measurement system. (Color figure availableonline.)

2.2. Model development guidelines

The first step towards estimating the uncertainty associ-ated with a clinical laboratory measurement system is toidentify the different components of the system. As statedabove, the principal components of most clinical labora-tory measurement systems are: 1) the calibrator/CRM,2) the measuring instrument, 3) the reagents, and 4) thepatient sample. Once the components of the system havebeen identified, the next step is to identify the factors thatcontribute to the variation inherent in each component.To this end, the subcomponents of each principal com-ponent, if any, need to be identified. For instance, themeasuring instrument may have individual subcomponentswith attendant uncertainties. These factors, which con-tribute to the uncertainty of each component (and henceto the net system uncertainty), are the sources of uncer-tainty. After the sources of uncertainty have been identi-fied, the next step is to characterize the variation of thesesources with an appropriate distribution. Here, a Type Aor a Type B characterization of the sources of uncertaintymay be carried out (GUM, 1993). The Type A methodinvolves characterizing the variation of a source of un-certainty by finding the distribution with the best fit todata available for that particular source. However, in manycases, it is not practical to obtain such data for individ-ual sources, in which case the Type B method is used tocharacterize such sources. This method involves charac-terizing such sources based on expert professional judg-ment, and/or using specifications supplied by the manu-facturer.

The next step deals with establishing a mathematicalmodel that accurately describes the relationship betweenthe measurand and the system components. The follow-ing steps are generally required to develop the mathemati-cal model: 1) understanding the role that each componentof the system plays in determining the value of the mea-surand, 2) understanding the chemical reaction involved,where applicable, and 3) understanding the calibration pro-cess. The calibration function is a convenient starting pointfor the development of the mathematical model because,as explained below, each term in the calibration func-tion represents the uncertainty associated with a specificstage of the measurement process. The calibration functionrelates to the process phases as follows: the uncertaintyassociated with the parameters represents calibration un-certainty; the uncertainty associated with the independentvariable(s) represents the measurement uncertainty, includ-ing pre-analytical uncertainty; and the net uncertainty asso-ciated with the system is that associated with the dependentvariable. Once the mathematical model has been developed,the measurand can be evaluated for different values of eachsource of uncertainty. The behavior of the measurand un-der uncertainty can thus be characterized and estimates ofuncertainty can be developed.

Dow

nloa

ded

by [

Var

un R

amam

ohan

] at

14:

17 0

3 O

ctob

er 2

012

Monte Carlo approach in measurement processes 5

3. Model implementation

The measurement of cholesterol concentration in bloodserum was used as the test bed for the implementationof the methodology described in the previous section. To-tal cholesterol concentration in the blood serum is part ofthe complete lipid profile recommended by the third AdultTreatment Panel (ATP III) of the National Cholesterol Ed-ucation Program (NCEP) (NCEP, 2002), for the purposesof coronary heart disease assessment and patient manage-ment. The serum cholesterol assay measures the concen-tration of total cholesterol in blood serum in terms of mil-ligrams of cholesterol per liter of blood. In this section weoutline the development of the mathematical model usedto estimate the uncertainty associated with total choles-terol measurement using the Roche Diagnostics P-Modularmeasurement system, shown in Figure 1(b).

As described in section 2.1, the system includes the fol-lowing main components: 1) the measuring instrument,2) the calibrators, 3) the reagents, and 4) the patient sam-ples. The measurand in this system is the cholesterol con-centration of the sample, and it is an indirect measurand.The measurement system operates on the spectrophoto-metric principle. The P-module instrument contains a re-action disk consisting of multiple reaction cells. A sam-pling mechanism pipettes the sample into the reaction celland a reagent pipetting mechanism pipettes the reagentinto the reaction cell. A stirrer paddle facilitates the chem-ical reaction via mixing, which results in the productionof a molecule—called the chromophore—that absorbs thelight passed by the photometer through the reaction cell.This quantity, called the optical absorbance of the reactionmixture, is measured by the photometer. The optical ab-sorbance of the reaction mixture at a given wavelength λ isdefined as:

Aλ = log10

(I0

I

)

Here, where I is the intensity of light at a specified wave-length λ that has passed through a sample (transmittedlight intensity) and I0 is the intensity of the light beforeit enters the sample or incident light intensity (or power).Absorbance is dimensionless, and hence is expressed onlyas a number. Absorbance measurements are often carriedout in analytical chemistry since the absorbance of a sam-ple is proportional to the concentration of the absorbingspecies in the sample. In the case of cholesterol, the relation-ship between absorbance and cholesterol concentration isa linear one; that is, it can be expressed as: C = mA +I . Here C denotes the cholesterol concentration; m and Iare the parameters of the calibration function and A is theabsorbance reading obtained from the instrument.

For the purpose of generating random values of ab-sorbance for the simulation, an average reference line is es-tablished. The P-module system calibration data obtainedfrom Mayo Clinic is used to establish the average line. The

data consists of absorbance readings generated from cali-brator solutions with cholesterol concentrations of 0 mg/L(water blank) and 1650 mg/L obtained over 7 days. Themeans of these absorbance values at both concentrationlevels are computed. From the data, the mean absorbancereading for the water blank was calculated to be 1410.8.The mean absorbance reading at a cholesterol concentra-tion of 1650 mg/L was calculated to be 4546.8. The averagereference line is then characterized using these two points.

3.1. Calibration phase

In the calibration phase, the linear relationship between ab-sorbance and cholesterol concentration is characterized byestimation of the parameters of the function. The calibra-tion line is characterized by measuring the absorbance oftwo calibrator solutions; one with a high level of cholesterolconcentration and other being a water blank (a cholesterolconcentration of zero). Two principal components of thesystem are involved in this process: the calibrator solutionsand the measuring instrument. Consequently, two principalsources of uncertainty associated with these components,the calibrator uncertainty and instrument uncertainty.

Three factors contribute to the uncertainty of the cal-ibrator solution. The first relates to the uncertainty thatoccurs while preparing the calibrator solution; that is, itis the uncertainty in assigning a particular concentrationto the solution. It is called as the calibrator set-point un-certainty (denoted as uc1). The second source describes thedifference in concentration that occurs when different vialsof the calibrator solution are prepared. This is referred to asvial-to-vial variability (uc2). The third source describes thestability of the calibrator solution over time as the calibra-tor is reconstituted each time it is used. This time-dependentsource of uncertainty is referred to as calibrator reconsti-tuted stability (uc3(t)).

Since data for each of the above sources of uncertaintywere not available, it was decided to use specifications ob-tained from the manufacturer and from the literature tocharacterize the uncertainties of these sources. These spec-ifications were expressed in terms of percentage error. Forinstance, the specifications for the calibrator set-point un-certainty were obtained as ± 1.2%, from a study conductedby Duewer et al. (2009), demonstrating the comparabilityof certified reference materials. After discussion with boththe clinicians and the instrument manufacturer, it was de-cided to represent the uncertainties as truncated Gaussiandistributions with specific limits set for the maximum devia-tions. For instance, for the calibrator set-point uncertainty,the mean of the distribution was set at 0% with a stan-dard deviation of 0.6%. The specification limit of 1.2% isused to truncate the distribution at two standard deviationsfrom the mean. Thus, due to calibrator set-point uncer-tainty, a calibrator solution with a target concentration of1650 mg/L will vary between ± 1.2% of 1650 mg/L; thatis, the actual calibrator concentration will lie in the interval

Dow

nloa

ded

by [

Var

un R

amam

ohan

] at

14:

17 0

3 O

ctob

er 2

012

6 Ramamohan et al.

Table 2. Sources of uncertainty

Uncertainty Component Notation

Distribution(expressed as%

error)Mean

(% Error)

StandardDeviation(% Error)

MaximumError Range

(%) Notes

Components of Calibrator UncertaintyCalibrator Set Point

Uncertaintyuc1 Gaussian 0 0.6 ± 1.2

Vial-to-vial variability uc2 Gaussian 0 1.5 ± 4.5Calibrator Reconstituted

Stabilityuc3 Gaussian −1.25 per day 0.42 − 2.5 per day Modeled as a deterioration

of sample per day; alsotime dependent

Components of Instrument UncertaintySample Pipetting

Uncertaintyum1 Gaussian 0 1.5 ± 4.5

Reagent PipettingUncertainty

um2 Gaussian 0 4 ± 12

Photometer Uncertainty um3 Gaussian 0 0.45 ± 0.9

[1650 ± 19.8] mg/L. Similar specifications obtained fromthe manufacturer for the other sources of uncertainty, andthese were used to characterize their variation in a similarmanner. The details of the characterization are summarizedin Table 2. The notation used for each uncertainty compo-nent is also given in the table. This method of characteriz-ing the uncertainty associated with a system component iscalled ‘Type B’ estimation, as described in the GUM (1993).

These sources of calibrator uncertainty are combined se-rially in the order in which they are introduced into the cal-ibrator preparation process. Let C denote the desired valueto be assigned to the calibrator. Calibrator reconstitutedstability is a time dependent quantity, so the correspondingdistribution has to be sampled for each specified time pe-riod. Therefore, the equation for calibrator concentrationwhen these sources of uncertainty are introduced one afterthe other becomes:

C′ = C ∗ (1 + uc1) ∗ (1 + uc2) ∗N∏

t=1

(1 + uc3(t)) (1)

Here, C′

denotes the calibrator concentration after allcomponents of calibrator uncertainty have been intro-duced. The absorbance reading A1, which correspondsto a cholesterol concentration of C

′, is calculated us-

ing the average line. The absorbance of this calibra-tor, with calibrator uncertainty incorporated into it, isthen measured by the P-module instrument. The mea-surement of the absorbance of the calibrator C

′will

introduce instrument uncertainty into the model. Thisprocess is described below.

There are three sources of uncertainty associated withthe measuring instrument, corresponding to the sample,reagent and the photometer. The first is that associated withthe sample pipetting mechanism, and it refers to the uncer-tainty in the volume of sample pipetted out into the reactioncell. It is referred to as sample pipetting uncertainty (um1).

The absorbance of the final reaction mixture is linearly pro-portional to the number of molecules of the chromophorein the reaction mixture, which in turn is proportional tothe number of cholesterol molecules in the sample. Samplepipetting uncertainty changes the volume and not the con-centration of the sample, and therefore changes the numberof cholesterol molecules in the sample. Consequently, an x%change in the sample volume added will produce the same(x)% change in the absorbance. The second uncertainty isthat associated with the reagent pipetting mechanism, andit refers to the uncertainty in the volume of reagent pipet-ted out into the reaction cell. It is referred to as reagentpipetting uncertainty (um2). Since this chemical reaction isan end-point reaction, the reagent is present much in ex-cess of what is required. Therefore, a change in the volumeof the reagent would not have a significant impact on theamount of chromophore produced, and therefore wouldnot substantially affect the absorbance. This is an exampleof a system component that, while possessing significantuncertainty in itself, does not contribute to the uncertaintyof the measurand. The third source of uncertainty is thatassociated with the photometer. This is referred to as pho-tometer uncertainty (um3). These sources of uncertainty arecharacterized in much the same way as the components ofcalibrator uncertainty, and they are listed in Table 2.

These components are combined serially in the orderthat they affect the desired absorbance value. Following thereasoning used to determine calibrator uncertainty, we havethe following expression for absorbance after photometeruncertainty is introduced.

A′ = A1 ∗ (1 + um1) ∗ (1 + um3) (2)

Equations (1) and (2) denote the final values for calibratorconcentration and the corresponding absorbance afterall the sources of uncertainty involved in the calibrationprocess have been incorporated into the model. This

Dow

nloa

ded

by [

Var

un R

amam

ohan

] at

14:

17 0

3 O

ctob

er 2

012

Monte Carlo approach in measurement processes 7

process is repeated for the two different calibrator concen-tration levels. The corresponding concentration values andabsorbance readings are used to estimate the values of thecalibration line parameters m and I . Thus these param-eters incorporate the uncertainty due to all the differentcomponents of calibrator and instrument uncertainty. Inthis particular case, this process was implemented for cal-ibrator concentration levels of 0 mg/L (water blank) and1650 mg/L. Calibrator uncertainty and sample pipettinguncertainty is meaningless in case of the water blank,so only the photometer uncertainty contributes to thecalibration uncertainty.

3.2. Measurement phase

Once the calibration line has been established, the processmoves into the measurement phase. In terms of the cali-bration line, the uncertainty associated with this phase isthat associated with the independent variable A; or thatassociated with the sample absorbance reading. There aretwo major components of uncertainty associated with thisphase, the specimen (patient sample) uncertainty and theinstrument uncertainty. As explained earlier, specimen un-certainty is a measure of pre-analytical uncertainty, andcan be included in the model on a high-level basis withoutenumerating all the sources of uncertainty within the pre-analytical stage that make up pre-analytical uncertainty.This is done by introducing a random variable us that rep-resents pre-analytical uncertainty, and is characterized inmuch the same way as the sources of instrument and cali-brator uncertainty.

Instrument uncertainty has been dealt with previouslyin section 3.1. If we denote the ‘true’ concentration of thesample as Cs, and the corresponding absorbance as As, theabsorbance obtained after the incorporating sample andinstrument uncertainty is expressed as:

As′ = As ∗ (1 + us) ∗ (1 + ∗um1) ∗ (1 + um2) (3)

Here As′ denotes the absorbance after going through the

measurement phase, and us denotes the random realiza-tion of the sample uncertainty distribution. When this ab-sorbance value is input into the calibration line, we get thesystem output- the cholesterol concentration of the sample-as:

Cs′ = m ∗ As

′ + I (4)

The value of Cs′

is recorded for a large number of real-izations of all the uncertainty component distributions andthen the net system uncertainty is calculated as the standarddeviation of these recorded values of Cs

′.

4. Results

The net system uncertainty estimated by the model—thatassociated with the measurement result—is determined by

the configuration of the system. For instance, consider asystem that is calibrated once every 100 measurements.In order to estimate the average uncertainty associatedwith a measurement result from such a configuration, thesimulation must also be organized similarly: a calibrationline is established, and measurements taken from 100 sam-ples of known cholesterol concentration are recorded fora given calibration line. The standard deviation of the dis-tribution fitted to the 100 measurements recorded aboverepresents the net system uncertainty at that cholesterolconcentration associated with this particular configura-tion. As another example, if 100 measurements are takenfrom the same patient sample, the configuration of thesystem changes accordingly. Two types of system config-urations are studied in this paper, based on the organiza-tion of the data used to validate the model, as explainedbelow.

The preferred method of validation for the model wouldbe to conduct controlled experiments and compare the ex-perimental data with the simulation data. However, sinceconducting controlled experiments was currently not pos-sible, the simulation output was compared against uncer-tainty estimates calculated from quality control data. Eightsets of quality control data were provided by the MayoClinic, and each data set consisted of measurement resultsat two levels of cholesterol concentration. Two sets con-tained measurement data at cholesterol concentration lev-els of 1050 mg/L and 2500 mg/L. The remaining six datasets contained measurement data for two different levelsof cholesterol concentration—1070 mg/L and 2550 mg/L.The measurement data in each data set is organized as fol-lows: the instrument is calibrated, say, n times and for eachcalibration an average of, say, k cholesterol samples are ana-lyzed and concentration measurements are recorded. Thuseach data set contains a total of (n∗k) cholesterol concen-tration measurements. Each such data set is referred to asan experimental run, and the uncertainty of the run is cal-culated as the standard deviation of these (n∗k) data points.A set of k measurements recorded using a single calibrationline is referred to as a calibration run. The value of n differsfor each data set, but the value of k is approximately 20,averaged across the available data sets.

The uncertainty estimates calculated from the QC datasets as well as the simulation model are summarized inTable 3. The simulation model estimates uncertainty in amanner similar to that for the QC data sets, with n cali-brations and k samples run within each calibration. In thecase of the simulation, the value of n varies with each dataset and k = 20. It is seen from Table 3 that in all cases,uncertainty estimates from the QC data are significantlysmaller than those obtained from the simulation. This in-dicates that: 1) no important sources of uncertainty havebeen omitted from the model, and 2) the simulation pro-vides worst-case estimates of uncertainty for the system,since the model is built on the basis of specifications pro-vided by the manufacturer.

Dow

nloa

ded

by [

Var

un R

amam

ohan

] at

14:

17 0

3 O

ctob

er 2

012

8 Ramamohan et al.

Table 3. Comparison of simulation output with quality control data

Data Set #Concentrationlevel (mg/L)

QC Data Uncertainty(mg/L)

Simulationoutput (mg/L)

Concentrationlevel (mg/L)

QC Data Uncertainty(mg/L)

Simulationoutput (mg/L)

1 1050 24.8 36.9 2500 41.4 76.32 1050 20.1 36.9 2500 40.6 76.33 1070 11.0 37.5 2550 31.8 77.54 1070 13.1 37.5 2550 34.5 77.55 1070 13.3 37.5 2550 29.9 77.56 1070 14.2 37.5 2550 30.8 77.57 1070 18.3 37.5 2550 38.0 77.58 1070 14.5 37.5 2550 34.3 77.5

The effect of each source of uncertainty listed inTable 2 on the net system uncertainty for a experimen-tal run is quantified by simply removing that source fromthe model and re-estimating the system uncertainty with-out that particular source. These are summarized in Table4a. It can be seen from Table 4a that the components ofcalibrator uncertainty do not have a significant contribu-tion to the net system uncertainty. However, the removal ofsample pipetting uncertainty and photometer uncertaintyfrom the uncertainty model result in significant decreasesin uncertainty, to the tune of 71% and 5% respectively. Inaddition, we can also estimate the contribution of both theprincipal components of the system as well as that of theprocess phases to the net system uncertainty, as listed inTable 4b. Once again, this is accomplished by removingthe sources of uncertainty associated with each principalcomponent and/or phase and re-estimating the net systemuncertainty. It can be seen from Table 4b that measurementuncertainty has the largest contribution to the net systemuncertainty (close to 35%), as opposed to calibration un-certainty (approximately 25%). It is interesting to note thatwhile measurement uncertainty consists entirely of instru-ment uncertainty, calibration uncertainty comprises bothcalibrator and instrument uncertainty, but its contributionto net system uncertainty is less than that of measurementuncertainty. Understanding this disparity in contributionsbetween phases and system components requires a separatediscussion in its own right; and is beyond the scope of thispaper.

An important advantage of using a simulation approachis that it removes the necessity of running the simulationto estimate uncertainty for each concentration level. In thecase of the serum cholesterol assay, this is accomplishedby generating uncertainty estimates for different levels ofcholesterol concentration, and plotting uncertainty againstconcentration, as shown in Figure 2. The uncertainty profilethus generated is used to provide uncertainty estimates forpatient samples with unknown analyte concentrations.

Another important use of a simulation model is the factthat it can be used to evaluate various quality control poli-cies and study the effect of each policy on the net systemuncertainty as well as its efficacy in identifying false posi-tives and negatives. Also, different standard operating pro-cedures (SOPs) for calibrating and operating the equipmentcan be evaluated. Three different SOPs are evaluated andtheir effects on reducing the combined system uncertaintyare studied in this paper. Each of these SOPs is utilized insome clinical laboratories. The first SOP involves calibra-tion of the instrument by measuring the absorbance of astandard with known cholesterol concentration only onceand accepting this as the calibration line. The second SOPcalibrates using only one measurement, but immediatelyruns a control specimen and accepts the calibration only ifthe resulting concentration value is less than one standarddeviation from the mean, otherwise the system is calibratedagain. The third policy involves the use of multiple calibra-tors/ CRMs at the higher and lower concentration levelsto establish the calibration line. An example of this pol-icy is one wherein the absorbance values of two calibrator

Table 4a. Uncertainty analysis

Source of UncertaintySystem Uncertainty (mg/L at

1050 mg/L)System Uncertainty (with current

source of uncertainty removed); (mg/L)Percentage Contribution to

Net System Uncertainty

Calibrator Set PointUncertainty

36.90 36.70 –0.54

Vial-to-vial Variability 36.90 36.50 –1.08Reconstituted Stability 36.90 36.70 –0.54Sample Pipetting Uncertainty 36.90 10.70 –71.00Photometer Uncertainty 36.90 35.10 4.88

Dow

nloa

ded

by [

Var

un R

amam

ohan

] at

14:

17 0

3 O

ctob

er 2

012

Monte Carlo approach in measurement processes 9

Table 4b. Phase and component contribution

Source of UncertaintySystem Uncertainty (mg/L at

1050 mg/L)System Uncertainty (with current

component/phase removed); (mg/L)Percentage Contribution to

Net System Uncertainty

Calibration PhaseCalibrator Uncertainty 36.90 36.50 –1.08Instrument Uncertainty 36.90 28.10 –23.85Calibration Uncertainty 36.90 27.80 –24.66

Measurement PhaseInstrument Uncertainty 36.90 3.70 –89.97

solutions with the same target concentration level are mea-sured and averaged to obtain the absorbance value atthat particular level of cholesterol concentration. The ab-sorbance values at the higher and lower level of cholesterolconcentration are computed accordingly and the calibra-tion line is established. Using the simulation model, the in-teraction of the multiple policies may also be studied. Forinstance, the effect of implementing the one standard devia-tion check policy on a calibration line established using twocalibrator solutions at the higher and lower level concen-trations levels can be studied using the simulation model.

In this study, we studied the effect of using one, twoand three calibrator solutions at the higher and lower levelof cholesterol concentrations on the combined uncertaintyof an experimental run of the system. All simulation ex-periments were conducted for a cholesterol concentrationlevel of 1050 mg/L. Also, the effect of implementing theone standard deviation check policy on each one of theabove scenarios is also studied. Since each additional cali-brator corresponds to an increase in the cost of calibration,this enables the evaluation of such a policy without con-

ducting controlled experiments. Also, the introduction ofeach additional calibrator may eventually add to the netuncertainty after a certain limit. The results from evalu-ating these policies are summarized in Table 6. As can beseen from the table, the maximum decrease in uncertaintyis achieved when three calibrators at the high and low levelof cholesterol concentration are used along with the onestandard deviation check policy to establish the calibrationline.

The simulation model can also be used to investigatethe behavior of the system when the parameters of thesystem are varied. Since sample pipetting uncertainty andphotometer uncertainty are the largest contributors to netsystem uncertainty, the effects of varying the parametersassociated with these sources of uncertainty on the uncer-tainty of an experimental run were studied. The standarddeviations of the distributions characterizing these sourcesare varied from 0% to twice the given standard deviation(0% to 3% for sample pipetting uncertainty and 0% to 0.9%for photometer uncertainty). The resulting net system un-certainty, in terms of coefficient of variation, is plotted

Fig. 2. Serum cholesterol assay uncertainty profile. (Color figure available online.)

Dow

nloa

ded

by [

Var

un R

amam

ohan

] at

14:

17 0

3 O

ctob

er 2

012

10 Ramamohan et al.

Table 5. Quality control policy simulation results

No. of Data PointsNet uncertainty with 1

SD control (mg/L)

Percentage improvementwith each additional data

point (with control)Net uncertainty without

1 SD control (mg/L)

Percentage improvementwith each additional datapoint (without control)

1 30.3 17.9 36.9 —2 29.9 19.0 32.6 11.73 29.7 19.5 31.0 16.0

Fig. 3(a). Parameter variation: sample pipetting uncertainty. (Color figure available online.)

Fig. 3(b). Parameter variation: photometer uncertainty. (Color figure available online.)

Dow

nloa

ded

by [

Var

un R

amam

ohan

] at

14:

17 0

3 O

ctob

er 2

012

Monte Carlo approach in measurement processes 11

Fig. 4. Parameter variation: calibration slope. (Color figure available online.)

against the respective component uncertainty, as shownin Figures 3(a) and 3(b). In both cases, the system uncer-tainty increases linearly with an increase in the correspond-ing component uncertainty (R2 > 95% in both cases). Asexpected, the increase in calibration run uncertainty withincrease in photometer uncertainty is lower than in thecase of sample pipetting uncertainty, due to its significantlylower contribution to net system uncertainty. However, adecrease of 50% in sample pipetting uncertainty producesa corresponding decrease of approximately 45% in net sys-tem uncertainty. This capability of the simulation model isparticularly useful in cases where the change in system un-certainty either stops or decreases significantly after vary-ing the parameter under consideration beyond a certainthreshold level.

The effect of varying the parameters of the calibrationline on the system uncertainty of a calibration run was alsostudied. All experiments were carried out at a cholesterolconcentration level of 1050 mg/L. The standard deviationof the slope (denoted by σ ) under random simulation con-ditions was quantified and with the mean (denoted by µ)fixed at the slope of the average reference line, the slopewas varied from (µ − 3σ ) to (µ + 3σ ) in fixed increments.This range represents close to 100% of the possible varia-tion of the slope under random simulation conditions. Thenet system uncertainty at each increment was recorded andplotted against the corresponding slope. This is shown inFigure 4. In the case of varying the calibration slope, thenet system uncertainty increased from a CV of 2.13% to2.88% across the range of variation.

5. Discussion

The model estimates calibration run uncertainty for theserum cholesterol measurement procedure at 2.66% at1050 mg/L and at 2.02% at 2500 mg/L. Average calibrationrun uncertainty estimates from the laboratory are 1.41% at1050 mg/L and 1.24% at 2500 mg/L; however, the maxi-mum calibration run uncertainty estimates from the labora-tory are 2.75% at 1050 mg/L and 2.44% at 2500 mg/L. TheNCEP established precision goals of ± 3% and total errorgoals of ± 8.9% for total cholesterol (Bachorik and Ross1995), to prevent misclassification of patients due to mea-surement error. Total error (TE) is calculated as follows:

TE (%) = Max. Allowable Bias (%) + 1.96∗ (Max. Allowable Imprecision in %)

The average calibration uncertainty estimated by the modelis relatively close to the NCEP imprecision goal of 3%,and the maximum calibration run uncertainty estimatesfrom the laboratory QC data sets are also close to themaximum allowable imprecision goals established by theNCEP. Reducing instrument uncertainty, which has beenshown in the previous section to have the largest contribu-tion to system uncertainty, can be done to reduce the netsystem uncertainty and thereby further reduce the prob-ability that the imprecision estimates do not exceed theNCEP goals. It should be noted that bias was not estimatedin this study since the specifications used to characterize

Dow

nloa

ded

by [

Var

un R

amam

ohan

] at

14:

17 0

3 O

ctob

er 2

012

12 Ramamohan et al.

individual sources of uncertainty do not provide any infor-mation about bias. However, this is unlikely to be the caseif individual sources of uncertainty are characterized usingType A methods; a non-zero expected value for the sourcedistribution is most likely to be obtained. It is not usefulto estimate the bias associated with data from the QC setsince an experimental run consists of a large number ofdata points; hence the estimated bias will typically be lowerthan that actually seen in the laboratory. Based on averagecalibration run imprecision/uncertainty estimates from themodel, the minimum value for the maximum allowable biasrequired to exceed the 8.9% total error goals established bythe NCEP would have to be 3.7%, which is unlikely to es-cape detection by standard quality control tools such ascontrol charts. In conclusion, the uncertainty observed inthe laboratory or that estimated by the model is unlikely tolead to misclassification of patients.

There are a few important issues that need to be kept inmind while developing the model. First, it is vital to the ac-curate estimation of net system uncertainty that all sourcesof uncertainty are identified and characterized by distribu-tions that adequately describe the variation of each source.The most important component of uncertainty not in-cluded in this study is pre-analytical uncertainty. There havebeen a few studies that have included pre-analytical uncer-tainty in their analytical estimates of uncertainty (Kallner& Waldenstrom, 1999; Linko et al., 2002); both studiesconclude that pre-analytical uncertainty contributes lessthan 10% to the net system uncertainty. The advantageof a simulation is the flexibility it provides; hence the ef-fect of pre-analytical uncertainty on the uncertainty of acalibration run is estimated by including it in the model.As discussed in Section 3.2, pre-analytical uncertainty isrepresented in the model by the variable us and includedin Equation (4). The effect of pre-analytical uncertainty onnet system uncertainty for an experimental run is estimatedfor a range of values from 0 – 10%. The results of this studyare summarized in Table 6. As can be seen from Table 6, thecontribution of pre-analytical uncertainty exceeds 10% at a

net pre-analytical uncertainty of approximately 4%. Thereare other possible sources of uncertainty that, although notincluded currently in the study, require investigation; anexample being the uncertainty associated with the averagereference line.

If analytical uncertainty models are to be used meaning-fully, it is vital that they undergo validation with experi-mental data. Ideally, the assumptions made in the modelshould be validated via controlled experiments designedspecifically to test said assumptions. However, such anexperimental validation requires significant investment oftime and capital; also, part of the rationale behind build-ing such models involves precluding the need to conductexperiments. Therefore, it is entirely possible that com-prehensive experimental validation may not be carriedout in many cases where such models are used. In thesecases, comparison with experimental data should be car-ried out as a minimum level of validation for the model.The comparison should ideally result in analytical esti-mates being higher than estimates from experimental data,thereby indicating that all major sources of uncertaintyhave been included in the model. In the case of this study,due to access and resource limitations, it was not possibleto conduct such experimental validation; hence the com-parison with data from the clinical laboratories at MayoClinic.

Also, it is worth noting that system uncertainty needs tobe defined within the context of the system configuration;for instance, the uncertainty of an experimental run is sig-nificantly different from that of a system calibrated onlyonce. Developing a mathematical model that is accurateand truly representative of the clinical laboratory testingsystem is a critical part of the process of modeling such asystem. A close working relationship both with cliniciansand the instrument manufacturers is required. In conclu-sion, the primary advantage of the methodology presentedin this paper lies in the flexibility granted by the simulationmodel to estimate the contribution of each source of un-certainty and simulate the effect of introducing changes in

Table 6. Effect of preanalytical uncertainty

Net system uncertainty (mg/L)

Net Preanalytical Uncertainty(%) Without Preanalytical With Preanalytical

Percentage Contribution to NetSystem Uncertainty

1.0 28.0 28.3 1.12.0 28.0 28.9 3.23.0 28.0 29.8 6.44.0 28.0 31.3 11.85.0 28.0 33.1 18.16.0 28.0 35.0 25.07.0 28.0 37.1 32.58.0 28.0 39.5 41.19.0 28.0 41.9 49.6

10.0 28.0 44.5 58.9

Dow

nloa

ded

by [

Var

un R

amam

ohan

] at

14:

17 0

3 O

ctob

er 2

012

Monte Carlo approach in measurement processes 13

the process on the net system uncertainty as an alternativeto conducting controlled experiments.

References

Alexandrov, Y. I. (2001) Uncertainty of measurement. Twenty yearsafterwards. Fresenius Journal of Analytical Chemistry, 370, 690–693.

Aronsson, T., de Verdier, C-H., and Groth, T. (1974) Factors influencingthe quality of analytical methods – A systems analysis, with use ofcomputer simulation. Clinical Chemistry, 20(7), 738–748.

Bachorik, P. S. and Ross, J. W. (1995) National Cholesterol Educa-tion Program Recommendations for measurement of low-densitylipoprotein cholesterol: Executive summary. Clinical Chemistry,41(10), 1414–1420.

BIPM, IEC, IFCC, ISO, IUPAC, IUPAP and OIML. (1993) Guide to theExpression of Uncertainty in Measurement. Geneva, ISO.

Burns, M. (2004) Current practice in the assessment and control of mea-surement uncertainty in bio-analytical chemistry. Trends in Analyt-ical Chemistry, 23(5), 393–398.

Burns, M. and Valdivia, H. (2006) A procedural approach for the identifi-cation of sources of uncertainty associated with GM quantificationand real-time quantitative PCR measurements. European Food Re-search and Technology, 226, 7–18.

Duewer, D. L., Lippa, K. L., Long, S. E., Murphy, K. E., Sharpless, K. E.,Sniegoski, L. T., Welch, M. J., Tani, W., and Umemoto, M. (2009)Demonstrating the comparability of certified reference materials.Analytical and Bioanalytical Chemistry, 395, 155–169.

EURACHEM/CITAC Guide. (2000) Quantifying Uncertainty in Ana-lytical Measurements, 2nd Edition. Prague, Eurachem.

Fuentes-Arderiu, X. (2000) Uncertainty of measurement in clinical lab-oratory sciences. Clinical Chemistry, 46(9), 1437–1438.

Fuentes-Arderiu, X. and Dot-Bach, D. (2009) Measurement uncertaintyin manual differential leukocyte counting. Clinical Chemistry andLaboratory Medicine, 47, 112–115.

Grabe, M. (2001) Estimation of measurement uncertainties – An alter-native to the ISO Guide. Metrologia, 38, 97–106.

ISO/IEC 17025. (1999) General Requirements for the Competence of Cal-ibration and Testing Laboratories. Geneva, ISO.

Joint Committee for the Guides in Metrology 100. (2008) Evaluationof Measurement Data – Guide to the Expression of Uncertainty inMeasurement, http://www.bipm.org/utils/common/documents/jcgm/JCGM 100 2008 E.pdf (accessed July 2009).

Joint Committee for the Guides in Metrology 101. (2008) Evaluation ofMeasurement Data – Supplement 1 to the “Guide to the Expressionof Uncertainty in Measurement” – Propagation of Distributions Us-ing a Monte Carlo Method, http://www.bipm.org/utils/common/documents/jcgm/JCGM 101 2008 E.pdf (accessed September2008).

Joint Committee for the Guides in Metrology 104. (2009) Evalua-tion of Measurement Data – An Introduction to the “Guide tothe Expression of Uncertainty in Measurement” and Related Doc-uments, http://www.bipm.org/utils/common/documents/jcgm/JCGM 104 2009 E.pdf (accessed September 2008).

Kallner, A. (1999) Quality specifications based on the uncertainty ofmeasurement. Scandinavian Journal of Clinical & Laboratory Inves-tigation, 59, 513–516.

Kallner, A. and Waldenstrom, J. (1999) Does the uncertainty of com-monly performed glucose measurements allow identification of indi-viduals at high risk for diabetes? Clinical Chemistry and LaboratoryMedicine, 37, 907–912.

Kragten, J. (1994) Calculating standard deviations and confidence inter-vals with a universally applicable spreadsheet technique. Analyst,119, 2161–2165.

Kristiansen, J. (2001) Description of a generally applicable model for theevaluation of uncertainty in clinical chemistry. Clinical Chemistryand Laboratory Medicine, 39, 220–231.

Kristiansen, J., Christensen, J. M., and Nielsen, J. L. (1996) Uncertaintyof atomic absorption spectrometry: application to the determinationof lead in blood. Mikrochimica Acta, 123, 241–249.

Krouwer, J. (2002) Setting Performance and Evaluating Total Analyti-cal Error for Diagnostic Assays. Clinical Chemistry and LaboratoryMedicine, 48(6), 919–927.

Leung, G. N. W., Ho, E. N. M., Kwok, W. H., Leung, D. K. K., Tang,F. P. W., Wan, T. S. M., Wong, A. S. Y., Wong, C. H. F., Wong, J.K. Y., and Yu, N. N. (2007) A bottom-up approach in estimatingthe measurement uncertainty and other important considerationsfor quantitative analyses in drug testing for horses. Journal of Chro-matography A, 1163, 237–246.

Linko, S., Ornemark, U., Kessel, R., and Taylor, P. D. P. (2002) Evaluationof uncertainty of measurement in routine analytical chemistry –Applications to determination of the substance concentration ofcalcium and glucose in serum. Clinical Chemistry and LaboratoryMedicine, 40(4), 391–398.

National Cholesterol Education Program. (2002) Detection, evaluationand treatment of high blood cholesterol in adults (Adult treat-ment panel III), http://www.nhlbi.nih.gov/guidelines/cholesterol/atp3full.pdf (accessed September 2002).

Patriarca, M., Castelli, M., Corsetti, F., and Menditto, A. (2004) Estimateof uncertainty of measurement from a single-laboratory validationstudy: application to the determination of lead in blood. ClinicalChemistry, 50, 1396–1405.

Petersen, P. H., Brandslund, I., Jorgensen, L., Stahl, L., de Fine Olivarius,N., and Borch-Johnsen, K. (2001) Evaluation of systematic andrandom factors measurements of fasting plasma glucose as the basisfor analytical quality specifications in the diagnosis of diabetes. 3.Impact of the new WHO and ADA recommendations on diagnosisof diabetes mellitus. Scandinavian Journal of Clinical and LaboratoryInvestigation, 61, 191–204.

Dow

nloa

ded

by [

Var

un R

amam

ohan

] at

14:

17 0

3 O

ctob

er 2

012