CLIN. CHEM. 35/8, 1701-1 705 (1989)

CLINICALCHEMISTRY, Vol.35, No. 8, 1989 1701

Sources of Imprecision in Laboratories Screening for Congenital Hypothyroidism:Analysis ofNine Years of Performance DataSandra E. Jewell, William J. Slazyk, S. Jay Smith, and W. Harry Hannon

We evaluated sources of imprecision in screening assays forcongenital hypothyroidism performed between 1979 and1987 by more than 45 laboratories participating in the na-tional Neonatal Screening Standardization Program offeredby the Centers for Disease Control. Reported concentrationswere available from approximately 70 000 enriched dried-blood spot samples for thyroxin and thyrotropin, the primaryand secondary screening assays, respectively. The mostimportant sources of variation in assayed concentrations arewithin-laboratory imprecision and the methods or kits used.The importance of each of these sources depends on theanalyte and its concentration. In most cases, the greatestsource of measurement variation is attributable to within-laboratory imprecision (coefficient of variation 13% to 26%);however, kits are the largest source of variation at the lowerconcentrations of thyroxin. For both analytes,the greatest

relative imprecision occurred at the low concentration, whichwas within the critical range of the cutoff values for detectionof presumptive positive specimens. Imprecision in this rangeincreases the risk of misciassifications. Minimizing within-laboratory imprecision and differences among kits seems tobe the best way to improve overall efficiency of screeninglaboratories.

Additional Keyphrases: neonatal screening heritable disor-ders monitoring assay stability blood-spot samplesinterlaboratoiy performance “kit” methods intra/aboratoryperformance

Congenital hypothyroidism is a metabolic disorder witha nationwide incidence of 1 in 4500 newborns (1). It causesirreversible brain damage if the disease is not diagnosedand the infant treated soon after birth (2, 3). In the U.S.,neonatal screening programs for congenital hypothy-roidism are mandatory in most states (4).

In 1979, the Centers for Disease Control (CDC) institutedan external quality-assurance program, with co-sponsor-ship by the Health Resources and Services Administration,designed to assist both the laboratories that are operatingscreening programs for congenital hypothyroidism and themanufacturers of diagnostic products used in screening.The voluntary program offered each laboratory an oppor-tunity to monitor the long-term stability of its assays forthyroxin (T4) and thyrotropin (thyroid-stimulating hor-mone, TSH) by providing a six-month supply of each lot ofdried blood-spot quality control reference materials, a datareport sheet for the first 10 analytical runs, and then

Division of Environmental Health Laboratory SciencesCenterfor Environmental Health and Injury Control, Centers for DiseaseControl, Public Health Service, U.S. Department of Health andHuman Services, Atlanta, GA 30333.

Use of trade names is for identification only and does notconstitute endorsementby the Public Health Serviceor the U.S.Dept. of Health and Human Services.

Received February 16, 1989;accepted June 2, 1989.

providing the statistical analysis of the results derived.These external quality control materials allowed the labo-ratories to maintain the high-volume screening workloadand rapid turnaround of data with a minimum of analyticaldowntime owing to changes in routine quality controlspecimens, screening assays, reagents, and personnel. Be-tween 1979 and 1987, more than 45 laboratories-includ-ing CDC, state, private, and manufacturer-participated inthe program and assayed more than 70000 referencespecimens for T4 and TSH. The laboratories most activelyinvolved in the screening program were state public-healthlaboratories, and they produced about three-quarters of thedata.

Imprecision in the screening process can produce eitherfalsely positive values-which reduce the cost-effectivenessof screening, increase and misdirect the workload for fol-low-up activities, and produce anxiety for affected par-ents-or falsely negative values-which result in the fail-ure to detect the condition and have potential legal rami-fications. The purpose of our analyses was to determine thequality of laboratory performance and the sources andrelative importance of the factors influencing analyticalvariabilityin the screening process to better direct effortsfor laboratory improvement.

Methods and Materials

Materials

A quality control production lot consisted of specimensheets of whole-blood spots on filter paper. The blood spotswere extensively evaluated for stability and homogeneity(5, 6). Each specimen sheet contained five rows each ofthree concentrations of T4 or TSH and was numbered witha six-digit series. A random number table was used toselect sheets from the total production lot to send to eachlaboratory. The T4 specimens were divided among 33 pro-duction lots at three concentrations-low, medium, andhigh. For TSH, there were 24 production lots at threetarget concentrations. Participants received one sheet fromthe previous lot with each new specimen set, to provideanalytical continuity. The blood-spot sheets were packagedfor distribution in plastic bags containing desiccant andmailed semiannually to participating laboratories. Speci-mens were to be refrigerated on arrival. These shipmentsincluded instructions for analysis of each assay, along withthe data-collection forms, which were to be completed andreturned to CDC. The laboratories were asked to documentthe methods used in their assays. After receiving thecompleted form, CDC statistically analyzed the data, andresults were then returned to the participant laboratories.

Statistical Methods

Participating laboratories were assigned an identifica-tion number that designated their type-either state, man-ufacturer, or private. The data for production lots of thesame enriched concentration were aggregated into “pools”according to analyte. A preliminary histogram analysis

AIALTTE.t4A

CONCENTRATION YEI

ANAYI EI SH

HIGH

60

50

a 40>

30

20

B

LOW 5001101 HIGH

1702 CLINICAL CHEMISTRY, Vol. 35, No. 8, 1989

was done on each pool that identified outlying values. Toremove the extreme effects of these few points on estimatedstatistics, we deleted measurements exceeding the 99thpercentile.

The enrichment preparation of the whole-blood poolswith analytes for the dried-blood spot materials was per-formed, with an identical protocol for each new lot, semi-annually over a nine-year period (6). To eliminate theconfounding effect that among-lot concentration differenceswould have introduced, we standardized all data so thatindividual lot means equaled the overall consensus meanfor each analyte and concentration. This adjustment al-lowed our analysis to focus on laboratory sources of impre-cision.

Analysis of variance was used to quantiQy and categorizevariability. Variability was also expressed in terms ofcoefficients of variation (CVs), percentages of total vari-ance, and standardized differences from the mean value.“Percentage of total variance” refers to the amount ofvariation attributable to each component factor in theanalysis. Within-laboratory imprecision, the term usedhere, is also known as “replication error.” Throughout, werefer to the screening assay methods used by the laborato-ries as “kits.” The kits included in this analysis were madeeither in-house or by Becton Dickinson Immunodiagnostics(Orangeburg, NY 10962), Diagnostic Products Corp. (LosAngeles, CA 90045), Kallestad Laboratories-only T4-(Cheska, MN 55318), Meloy Laboratories, Inc. (Springfield,VA 22151), Micromedic System (Horsham, PA 19044),Neometrics, Inc. (E. Northport, NY 11731), Nichols Insti-tute (San Juan Capistrano, CA 92675), or Nuclear MedicalSystems-only TSH-(now Biomerica, Newport Beach, CA92663). In Figure 3, the character symbols denoting kitmanufacturers were assigned alphabetically according tothe name of the manufacturer. For both analytes, thesymbol “C” represents the in-house method.

Standardized deviations were used to compare lot and kitmeans within a given concentration and to display them ina uniform format. Because the supplementations and con-sensus means varied slightly from lot to lot, absolutevalues could not be used for this purpose. Standardizeddeviations were calculated by first determining the overallmean value and standard deviation for the analyte. Then,the overall mean value was subtracted from each value forthe analyte and divided by the overall standard deviation.The results expressed the standard deviations of the indi-vidual values from a mean value of zero. This is referred toas “standardizing to zero.” A standardized deviation of zerowould indicate that the value of the individual observationwas the same as the overall mean value for the variableanalyzed. A standardized deviation value of 1 would showthat the individual observation was one standard deviationaway from the overall mean value.

Results

We initially considered five factors as possible sources ofvariation in laboratory measurements: (a) frequency ofparticipation in the program; (b) the type of laboratoryperforming the assay; (c) within-laboratory imprecision; (d)imprecision among laboratories; and (e) the kits used. Toexamine whether a laboratory’s frequency of participationcontributed significantly to its performance, the laborato-ries were divided into those that sent results annuallyduring the period 1979-1987 and those that did not. Wefound no statistically significant differences (P >0.05) at-

ONCENTRATION�

Fig. 1. Percentof total variancedue to kits,among-laboratory,andwithin-laboratoryimprecision,by concentrationCodes:K = kits (methods);L = among-laboratory;R = within-laboratory

tributable to the frequency of laboratory participation andconcluded that this factor was not an important discrimi-nating variable in the precision of results.

Similarly, when results were analyzed according to thetype of laboratory-state, manufacturer, or private-per-forming the assay, state laboratories gave the highest CVsat the low and medium concentrations of both T4 and TSH,but there were no statistically important differences inmean values. We concluded that data from different labo-ratories could be analyzed as a single factor without cate-gorizing by type.

Analysis of variance results for the remaining factors-kits, among-laboratory imprecision, and within-laboratoryimprecision-are reported in Figure 1, A and B. The resultsshow that the relative importance of the sources of vari-ability for reported concentrations of the two analytes wasquite different. For example, the overall patterns show thata larger proportion of the variance was attributable to kitsand among-laboratory imprecision and a smaller propor-tion to within-laboratory variability for T4 than for TSH.This result may be due in part to the smaller meanconcentration values for T4 and the poor discriminatingcapabilities of some kits at very low concentrations. Thegreatest source of imprecision for TSH, within-laboratoryreplication, accounted for more than 50% of the totalvariance at each concentration. Kits accounted for thehighest percentages of variance at the low and mediumconcentrations of T4, 51% and 39%, respectively. Within-laboratory replication produced the greatest amount oftotal variance, 44%, at the high concentration of T4. Theproportion of total T4 variation ascribable to kits declined

M.ALTIt.I4. tim Llmctai,,A,i,m

50

AHALYIt.14.5(0110 COI1CLHIRAT 10

25

0

75

1

LOT

-25

*TIHLYI(.ISH. 1501101 COHCLHTRATION

Law.,CimMr,.LI4-50

LOT

801 502 811 812 514 834 537 402 455 802 705

AHALTT(.T4,111THCOIICINIRATIOS

Cr14.,,,(-I

so

40

20

0

-20

-40

50.

Uppo,4dr,. 25L114

0.

-25

-50

-75

75

I

805 808 810 831 412 512 112

LOT

AIIAITIt-ISH. HICIl COHCtNIRAI los

I--Cr111.,,.04.

1Jr0.,00,44...,.

U.,.

I.,,-CiWdrc.1=,

801 802 811 812 814 835 835 403 406 803703

LOT LOT

Fig. 2. Percentage bias from the aggregate mean with 95% confidence limits by lotEnclosed rectangle, 50% ofrunmeans;horizontal ilneinsiderectangle,lot mean;extending ilnes, minimumandmaximumvalues.Lotsare plotted inchronologicalorder

805 808 810 832 413 513 713

CLINICAL CHEMISTRY, Vol. 35, No. 8, 1989 1703

TabAnalyte

le 1. CoefficIent of Varia tion and Deg14

rees of Freedom for Reported Con cent ration by SoTSH

urce

Concentration Low Medium High Low Medium High

Statistic CV df CV df CV df CV dl CV df CV df

KitsAmong-labs.Within-labs.

312023

79727

14104

161216

79728

14153

111013

79728

14211

201526

53514

7350

171120

53514

7498

121016

53514

7496

as mean concentrations increased, from 51% at the lowestconcentration to 29% at the highest. For TSH, the percent-age of total variance accounted for by kits was relativelystable at 29% to 34% over the three concentrations. Thepercent of total variance attributable to among-laboratoryimprecision was between 22% and 27% forT4 and between15% and 20% for TSH. The greatest source of variation forboth analytes was within-laboratory imprecision, except atthe low and medium concentrations of T4, where most ofthe imprecision was attributable to kits.

The relative variability, indicated by the CVs given inTable 1, shows that within-laboratory replication had thehighest CVs, 26% to 16%, among the three concentrationsfor TSH. Kits used in TSH assays produced CVs thatdeclined from 20% to 12% with increasing concentration.Variation among laboratories was associated with thelowest CVs for TSH, between 15% at the low concentrationand 10% at the high. For T4, kits had the highest CVs at

the low concentration, 31%, and among-laboratories hadthe lowest CVs at the medium and high concentrations. Atthe high concentration of T4, within-laboratory imprecisionaccounted for the highest CVs.

When mean values for blood-spot production lots werestandardized to zero and plotted in chronological order, thewithin-lot variability showed no marked improvement inprecision over time for either analyte (Figure 2, A-F. Theconfidence limits were widest for the lowest concentrationsbut narrowed as concentrations increased for each analyte.At each concentration of T4, most of the lot mean values fellslightly below the mean value for the given concentration.This finding indicated that a source of systematic variabil-ity unrelated to lots affected the laboratory results. Themean values for TSH materials generally varied above andbelow the overall mean value at each concentration.

Table 2 shows the consensus mean values from thelaboratories, the enriched values, and the assayed mean

Enriched value8Consensus valueMean assayed

value (COC)

X SD nEnriched valuesb 24 - -

Consensus valueMean assayed

value (COG)

21 5

27 7

7924

X SD

39 -

33 7

Endogenousconcentrations measured <8 og/L.

b Endogenousconcentrations<2.5 milli-int. unitslL.

n

8072

239 44 7

x so79 -

68 11

241 82 12

n

8070

241

Table 2. Blood-Spot Values, Means, and Standard Deviations by Analyte and Concentration

1704 CLINICALCHEMISTRY, Vol. 35, No. 8, 1989

Thyrexin, 1ig/L

Low Medium High

X SD n X SD fl X SD n21 - - 49 - - 77 - -

23 5 14970 49 8 14970 76 10 15028

31 6 344 64 8 342 96 14 342

Thyrotropin,mull-mt.units/L

Low Medium High

values from CDC for the blood spots at the three concen-trations for T4 and TSH. Total measured analyte concen-trations should reflect the enriched concentration for eachpool plus the endogenous concentrations, which by analyt-ical measurement are usually very low: <8 p.g/L for T4 and2.5 micro-mt. unitsfL for TSH. The consensus value for T4was below the expected (enriched-plus.endogenous) valueat each concentration by 11% to 21%. For TSH, the consen-sus value was consistently 17% to 22% below the expectedvalue. The mean assayed values (CDC) were alwaysslightly greater than the expected values for both analytes.

Figure 3, A-D, shows the mean values of the kits ex-pressed as standardized deviations from the overall meanvalues for the three concentrations for T4 and TSH. Ana-lytical kits, identified by Codes A through K, had a system-atic influence on the results both in terms of mean valuesand variability. The nearly parallel lines for both meanvalues and CVs in these figures show that the bias intro-duced by methods is usually independent of concentration.For example, Kit A yielded a medium-range concentrationand a high within-laboratory CV for T4 at all concentra-tions, whereas Kit H consistently gave a low mean and alow CV. The kits that performed closest to the consensusmean for T4 were Kit A at the low and medium concentra-tions and Kit F at the high concentration. Kit H consis-tently had the smallest imprecision. For TSH, Kit F pro-duced a low mean and a high CV, and Kit K showed a highmean value, and Kit A a low CV for all concentrations. KitH gave results for TSH that were close to the consensusmean at all concentrations and, along with Kit A, showedless imprecision than others. Few kit means or CVs felloutside of one standard deviation of the overall mean valueand no results were greater than two standard deviationsfrom the mean.

DiscussionThe precision of laboratory measurements is an impor-

tant factor in public-health programs. In neonatal screen-ing, we have found that the primary determinants ofprecision are within-laboratory variability and the differ-ences among kits. Although our data indicate that a fewkits give more-precise results for some concentrations thanfor others, the performance of most kits remained rela-

tively stable for mean values and CVs at all concentrationsfor both T4 and TSH. This consistency indicates that kitsmust be regarded as an independent source of systematicbias in the screening process.

The consensus mean values for the dried-blood spotmaterials were lower than the expected values for T4 andTSH at all concentrations. The values for enriched samplesgiven in Table 2 for the blood-spot pools do not include theendogenous concentrations in these pools. The mean assayvalues determined at CDC for blood-spot preparationsindicate excellent analytical recoveries for the expectedvalues, but they are consistently higher than the consensusvalues. The World Health Organization’s InternationalReference Preparations (68/38 and 80/558) are used toenrich the blood pools for preparation of the TSH dried-blood spots. When calibrators traceable to this referencematerial are used, the recoveries obtained at CDC for thedistributed blood spots are excellent. The differences be-tween the consensus value and the expected values arewider across the concentrations for TSH than T4, althoughan international reference calibration material is availableonly for TSH. The availability of the TSH reference mate-rial makes the bias difficult to explain. In a 1981 study (5)the TSH values for the low- and medium-concentrationpools were found to be close to the expected value, but thehigh-concentration pool was about 10% above. This con-trasts with our data and may indicate that there has beena change in the TSH calibration for kits.

An analysis of the ranges and standard deviations oflaboratory values submitted over the nine-year periodshowed that there was no obvious improvement in precisionat any concentration for either analyte as a function of time.The fact that the T4 and TSH results are used for screeninginstead of diagnostic purposes may have obscured the needto improve analytical precision. For both analytes, the great-est relative imprecision occurs at the low concentration,which is in the critical range of the cutoff values for detectionof presumptive positive specimens.

Although we have presented evidence of the degree ofimprecision involved in the neonatal screening process,theimpact of measurement error is difficult to quantify. Ana-lytical methods did not, up to 1985, contribute directly to amissed case of congenital hypothyroidism (7, 8). Measure-

2

01

01

C

E0

-I.2

01

-2

C

-I

2

Lw MEDIUMCONCENTRATION

ANALTTE’14

- -- ___5--44------.. -

i#{246}w MEDIUM

CONCENTRAIION

ANALTIE TSR

2

LOS MEDIUM

COOICENTRATION

ANAL VT(=1SR

CLINICAL CHEMISTRY, Vol. 35, No. 8, 1989 1705

ment error for the primary screen, T4, is handled on a dailybasis by most screening laboratories, usually by settingfloating cutoffs to include many false positives and tovirtually preclude false-negative specimens. Therefore, theactual impact of T4 imprecision is measured less in terms offalse negatives than in the cost-effectiveness of each screen-ing program.

Because TSH assays are used with a fixed cutoff, within-laboratory imprecision will have a direct impact on thenumber of assay results misclassified owing to this second-

ANALYTE-14

- - .1<

- - - - = .8

-

.8

.0-_.__ ____.,___.,,_.z___t- -E--’’0

-t

5coNc(NrRATION�F

ig.3 .D eviationsf romt hes tandardizedconsensusmeana ndconsensuscoefficientofv ariation,byk it( seeMaterials)a ndconcent-ration�

ary screen. For example, given the TSH average 26% CV atthe low concentration and a presumptive positive decisionlevel of 20 milli-int. unitsfL, a third of assays with a truevalue of 20 milli-int. unitsfL would be expected to registerresults outside the 15-25 range and another 5% would falloutside the 10-30 range. Actual congenital hypothyroidismcases usually register at least 40 milli-int. unitsfL but canshow TSH values as low as 25 (2). It follows, therefore, thatimprecision in a screening laboratory can lead to missedcases, but more typically will mean needless anxiety forparents and that the follow-up staff is unnecessarily bur-dened with falsely positive values.

A One of the 10-year public-health objectives issued in1980 for the nation (9) stated: “By 1990, virtually allnewborns should be screened for metabolic disorders forwhich effective and efficient tests and treatment are avail-able (e.g., PKU and hypothyroidism).” Given the manysources of imprecision in the screening process, we believethat our data show generally good performance by thescreening laboratories and assay kits. Multiple sources ofvariability notwithstanding, screening results are largelyhomogeneous; e.g., few results fell outside of one standard

HIGH deviation of the mean concentration or of the mean CVvalue, whether the variability was quantified according to

B kits, among-laboratory or within-laboratory imprecision.Overall the data indicate that the 1990 objective is beingmet with effective screening tests, although efforts to im-prove precision will contribute to increased operationalefficiency.

HIGH

This studywas supported in part by an interagency agreementwith the Health Resources and ServicesAdministration. We thankLinnard Taylor, TheodosiaPrater, and Charlie Banks for prepar-ing materials, and Roberta Jensen, Theresa Rogers, BarbaraAdam, and Dr. JamesE. Myrick for technical assistance.

References1. Fisher DA, Foley BL, Mitchell M. Problems and pitfalls of

C newborn screening programs based onthe experience in Californiaand New England. In: Andrews LB, ed. Legal liability and qualityassurance in newborn screening.Chicago:American Bar Founda-tion, 1985:38-43.2. Smith DW, Klein AH, Henderson JR. Congenital hypothy-roidism: signs and symptoms in the newborn period. J Pediatr1975;87:958-62.3. Walfish PG. Evaluation of three thyroid function tests fordetecting neonatal hypothyroidism. Lancet 1976;i:1208-22.4. Andrews LB. The rationale behind the recommendationsforquality assurance in newborn screening.Chicago:American Bar

HIGH Foundation, 1985:82-118.5. Hearn TL, Hannon WH. Interlaboratory surveysof the quanti-

tation of thyroxin and thyrotropin in dried-blood spots. Clin Chem28:1982;2022-5.6. Slazyk WE, Phillips DL, Therrell Jr BL, Hannon WH. Effectoflot-to-lot variability in filter paper on the quantification of thy-roxin, thyrotropin, and phenylalanine in dried-blood specimens.Clin Chem 1988;34:53-8.7. Report of the New England regional screening programs andthe New England congenital hypothyroidism collaborative. Pit-falls in screening for neonatal hypothyroidism. Pediatrics1982;70:16-20.8. Holtzman C, Slazyk WE, CorderoJF, Hannon WI!. Descriptiveepidemiologyof missedcasesof phenylketonuria and congenital

HIGH hypothyroidism. Pediatrics 1986;78:553-8.9. Public Health Reports.Public health service implementationplansfor attaining the objectivesfor the nation. Washington, DC:U.S. Govt. Printing Office, September-October (Supplement)1983:38-9.