JOURNAL OF CLINICAL MICROBIOLOGY, June 1982, p. 1111-11190095-1137/82/061111-09$02.00/0

Vol. 15, No. 6

Accuracy and Precision of the Autobac System for RapidIdentification of Gram-Negative Bacilli: a Collaborative

EvaluationA. L. BARRY,"* T. L. GAVAN,2 P. B. SMITH,3 J. M. MATSEN,4 J. A. MORELLO,5 AND B. H.

SIELAFF6tUniversity of California-Davis Medical Center, Sacramento, California 958171; The Cleveland Clinic

Foundation, Cleveland, Ohio 441062; Centers for Disease Control, Atlanta, Georgia 303333; University ofUtah School of Medicine, Salt Lake City, Utah 841324; University of Chicago Medical Center, Chicago,

Illinois 606375; and Pfizer Inc., Groton, Connecticut 063406

Received 23 November 1981/Accepted 28 January 1982

Gram-negative bacilli were identified within 3 to 6 h by determining susceptibil-ity to 18 different antibacterial agents in the Autobac I system and by applying atwo-stage quadratic discriminant analysis to the susceptibility patterns. TheAutobac system was compared with standard reference methods for identifyingglucose nonfermenters and glucose fermenters. Intralaboratory and interlabora-tory precision of the Autobac system was comparable to that of the referencemethods. Sensitivity (accuracy) and specificity of the two systems were alsocomparable, although there were some differences with certain species. Autobacresponses were considered to be equivocal (needing additional tests) if the relativeprobability of an accurate identification was <0.70. Only 5% of 2,889 strainsproduced such equivocal results; a similar number of strains gave low probabilitylevels with the reference methods. When the two systems disagreed, an indepen-dent reference laboratory arbitrated, confirming 49% of the Autobac responsesand 36% of the reference identifications. With equivocal responses excluded, theoverall accuracy of the Autobac system was 95.3% compared with 95.9% for thereference method. The respective accuracy estimates would be 93.8% and 93.1%if all first-choice identifications were evaluated.

In general, previous attempts to automate ormechanize identification of microorganismshave been accomplished by simply adaptingtraditional methods for use in a mechanized testsystem. In 1973, Friedman and MacLowry (5)reported a computer-assisted system for identifi-cation of bacteria based on an analysis of thepattern of susceptibility to common antimicrobi-al agents. Sielaff et al. (8) mechanized thisapproach by using susceptibility test results ob-tained after 3 to 5 h with the Autobac I system(Pfizer Diagnostics, Groton, Conn. [now theproperty of General Diagnostics, Morris Plains,NJ 07950]). By applying a quadratic discriminantfunction technique for data analysis, they ob-tained a 97% correlation with conventional iden-tification procedures. Matsen et al. (personalcommunication) have examined the possibilityof utilizing a wide variety of antibacterial agents,other than the common chemotherapeuticagents, for the purpose of bacterial identifica-tion. After extensive screening of potentiallyuseful agents, 18 different antibacterial agents

t Present address: Minnesota Mining and ManufacturingCo., St. Paul, MN 55144.

were selected because of their discriminatorycapabilities. Sielaff et al. (9) describe a systemfor rapid (3- to 6-h) identification of gram-nega-tive bacilli, based on analyzing the patterns ofsusceptibility to 18 different antibacterial agents.This system used the Autobac I system fittedwith a programmed computer which will per-form a two-stage quadratic discriminant analysisof the susceptibility patterns.The present report documents the accuracy

and precision of the proposed Autobac systemfor rapid identification of gram-negative bacilli.For this evaluation, standard reference methodswere used to confirm the identification of eachisolate. The accuracy of the reference methodsselected for this collaborative study was alsoestimated by further testing selected strains atthe Centers for Disease Control, Atlanta, Ga.The precision of each identification system wasevaluated by testing 92 isolates in triplicate ineach of five independent laboratories.

MATERIALS AND METHODSThe study consisted of two phases. The first phase

was designed to document precision (reproducibility)of the reference methods and of the Autobac systems.

1111

on May 23, 2020 by guest

http://jcm.asm

.org/D

ownloaded from

1112 BARRY ET AL.

The second phase consisted of testing fresh clinicalisolates and stock cultures to document the specificityand sensitivity of the Autobac system, in comparisonwith reference methods.

Bacterial isolates. For the first phase of this study, 92stock cultures were selected at the Centers for DiseaseControl. The identification of each isolate was recon-firmed, and each strain was suspended in rabbit blood,dispensed into 15 different Durham tubes, and frozenat -70°C. The cell suspensions were identified onlywith randomly selected code numbers. Each of thefive participating laboratories received 276 coded cellsuspensions representing three tubes of each stockculture. Personnel performing these tests were notaware of the initial identifications or of which suspen-sions were duplications. The frozen suspensions sentto each testing laboratory were thawed and subcul-tured just before being identified with the referencemethods and with the Autobac system. Subsequently,both interlaboratory and intralaboratory reproducibili-ty of both identification systems was evaluated. Thisculture collection included 3 Escherichia coli, 2 Kleb-siella pneumoniae, 3 Klebsiella ozaenae, 3 Proteusmirabilis, 3 Proteus vulgaris, 4 Morganella morganii,3 Providencia stuartii, 2 Providencia rettgeri, 4 Salmo-nella enteritidis, 2 Arizona hinshawii, 3 Shigella son-nei, 3 Serratia marcescens, 4 Enterobacter aerogenes,3 Enterobacter cloacae, 3 Enterobacter agglomerans,3 Hafnia alvei, 3 Citrobacter freundii, 3 Citrobacterdiversus, 3 Edwardsiella tarda, 3 Yersinia enterocoli-tica, 3 Yersinia pseudotuberculosis, 3 Acinetobactercalcoaceticus subsp. anitratus, 3 Aeromonas hydro-phila, 2 Moraxella lacunata, 2 Flavobacterium menin-gosepticum, 2 Alcaligenes odorans, 3 Pseudomonasaeruginosa, 3 Pseudomonas maltophilia, 3 Pseudomo-nas cepacia, 2 Pseudomonas putidalfluorescensgroup, 3 Pseudomonas putrefaciens, and 3 Pseudomo-nas stutzeri strains. The laboratories failed to report all1,380 pairs of identifications: 1,253 reference identifi-cations and 1,240 Autobac identifications were avail-able for analysis. The unreported identifications wererandomly distributed, not likely to introduce signifi-cant bias to the data.The second phase of this study involved tests with

recent clinical isolates and a few stock cultures, in-cluded as representatives of the less common species.The four clinical laboratories participating in this studytested 2,889 isolates with standard reference methodsand with the Autobac system. For each isolate, bothtest systems were initiated at the same time, and theresults of all tests were stored in a central computerfacility. Subcultures of 653 strains were submitted tothe Centers for Disease Control for arbitration: 327strains were identified as belonging to the same spe-cies with the Autobac and reference test systems and326 strains had discrepant identifications with the twoindependent systems. The former 327 strains weresubmitted for arbitration because the initial programgave discrepant results, but they were found to be inagreement when the modified program that is de-scribed in this report was applied to the data. The firstexperimental program is not described because itproved to be inaccurate.Ten quality control strains were tested by both

identification systems at approximately biweekly in-tervals during the second phase of this study. Thecontrol strains included E. coli, K. pneumoniae, P.

vulgaris, M. morganii, S. marcescens, C. diversus, P.aeruginosa, P. putrefaciens, M. lacunata, and A.calcoaceticus subsp. antitratus. These strains wereselected to provide at least one positive response andone negative response to each of the individual testsincluded in both identification systems.Autobac identification system. The test system evalu-

ated in this study has been described in detail bySielaff et al. (9). Briefly, each isolate was subculturedto blood agar and MacConkey agar plates. The nextday, the following information was recorded: (i)growth on MacConkey agar, (ii) lactose fermentationon MacConkey agar, (iii) presence of precipitated bilearound colonies on MacConkey agar, (iv) spot oxidasetest (6) results, (v) spot indole test (10) results, and (vi)swarming growth on blood agar. These six pieces ofinformation were entered into the computerized identi-fication system before the Autobac cuvettes wereread. To inoculate a cuvette, one or more isolatedcolonies were suspended in phosphate-buffered saline,and the suspension was then adjusted to a standardturbidity (ca. 107 colony-forming units per ml), byusing the Autobac photometer. For each 13-chambercuvette, 2 ml of inoculum suspension was added to 18ml ofEugon broth (BBL Microbiology Systems, Cock-eysville, Md.). In the present study, two 13-chambercuvettes were inoculated, and 24 different elutiondisks were tested. After our data were analyzed, the 18elution disks described by Sielaff et al. (9) wereselected, and the identifications presented here arebased only on the results of tests with those 18 disks.A 19-chamber cuvette will soon be available for testingthe 18 elution disks, providing one control chamber.Once inoculated, the cuvettes were allowed to incu-

bate at 35°C in an Autobac incubator-shaker. Thecuvettes were all read after 3 h of incubation. Ifsufficient growth was not initiated in the control cham-ber, the cuvettes were reincubated and read at hourlyintervals for the next 3 h. If sufficient growth had notbeen obtained after 6 h of incubation, the test wasaborted, and the strain was retested the next day.Repeated failure to grow within 6 h was a very rareoccurrence and when this did happen, the strain wasremoved from our study. As soon as satisfactorygrowth was obtained in the control chamber, turbidityin each test chamber was measured, and the resultswere automatically entered into the computer alongwith the preliminary test data noted earlier. The pro-gram then provided a first- and second-choice identifi-cation along with the relative probability (R.P.) thatexpresses the confidence with which one may accepteach reported identification. A low R.P. indicates thatadditional tests are needed to confirm the identifica-tion; it might represent a species that is not in the database.

Reference methods. Conventional tubed media wereused to obtain a reference identification. These media(Scott Laboratories, Fiskeville, R.I.) were essentiallyidentical to those used at the Centers for DiseaseControl for identification of Enterobacteriaceae (3) orby G. L. Gilardi (personal communications) for identi-fication of the glucose nonfermenters. The media usedto identify glucose fermenters and nonfermenters aredescribed more completely below. Most tests wereread after 24 h and, if negative, they were read againafter 48 h of incubation. o-Nitrophenyl-P-D-galacto-pyranoside tests were read after 20 min and 1, 2, 4, and

J. CLIN. MICROBIOL.

on May 23, 2020 by guest

http://jcm.asm

.org/D

ownloaded from

AUTOBAC ID SYSTEM 1113

48 h or until a positive reaction was seen. Voges-Proskauer, phenylalanine deaminase, and indole testswere performed after 48 h of incubation only, sinceexternal reagents had to be added. The results of alltests were entered into the computer, which wasprogrammed with percentage figures obtained fromthe appropriate sources (primarily from the Centersfor Disease Control, Enteric Bacteriology Section, andfrom G. L. Gilardi). The latter computer program wasdeveloped to identify only the genera or species in-cluded in the Autobac data base, using rather tradi-tional methods. When the two systems disagreed,many more discriminatory tests were performed, asneeded, for final identification of each isolate. Thisarbitration work was performed exclusively at theCenters for Disease Control by P. B. Smith, D. L.Rhoden, and A. 0. Esaias. Arbitration was performedwith 653 isolates for which discrepant identificationswere initially obtained by Autobac and referencemethods. The Autobac program was revised at the endof this study, and when the revised program was used,327 of the 653 Autobac identifications were in agree-ment with the initial reference identifications. Only therevised program is evaluated in this report.

Statistical analysis. For the purpose of this report,the term "accuracy" indicates agreement with a refer-ence identification (reference methods or arbitrationtests when available). "Precision" is used inter-changeably with "reproducibility" to indicate repeat-ability of a particular test response. Interlaboratoryand intralaboratory reproducibilities are each ex-pressed as "reproducibility index," rather than per-centage of responses in agreement with modes or withthe expected responses. This allowed comparison ofall possible pairs of data and expresses the proportionof data pairs that were in agreement (4, 7). Some datadid not permit the selection of a modal response to beused as an index of agreement; thus some data couldnot be analyzed in the traditional manner. Further-more, the reproducibility index could easily accommo-date missing data, i.e., if only two of three responseswere reported, one pair of data was available foranalysis. "Sensitivity" of an identification system isdefined as the percentage of strains within a givenspecies that were accurately identified. "Specificity,"on the other hand, examines the number of times agiven species identification was reported by the testsystem and expresses the percentage of times thosereports were in agreement with the reference identifi-cation (accurate).

C)C)Cx. _C.-0 C)

n 5C) . --

o. °. n

o_. F _

,-C U)C

C 0Xto) -I

0X -

C)oC

C) 0.

- U)

RESULTSPrecision of reference tests. Ten quality control

strains were each tested 29 and 41 different timesagainst the appropriate battery of referencetests, thus providing an opportunity to estimatethe precision of each of the individual test proce-dures. False-positive or false-negative reactionswere recorded with about 2 to 3% of most tests(Tables 1 and 2). Overall precision varied from93.5% for M. morganii to 99.2% for A. calcoace-ticus. Additional estimates of precision wereobtained from replicate tests that were per-formed with the larger collection of isolates

_,

II

w + A¢, + " w

0%-.

00

s.t-4

00

-+

+ +

( +

+ + +

- + +- + I l- +

I + + I -l

+ + +

w +

+I.-&4+

1-- t-i+

+ + +

I -+

+ +)+ t

+ ,_" +

'IO

A

I-

10%

4

t-

'Ow

o- + w -"

+ WJI

I + +..

+- + +

+

" I +- I

o-I +I. + Iw-+Wv+o,

+ w I I+ + +

w$ 00.. .1 PO -en~ >A ao w xo

VOL. 15, 1982

,0n

C)^

C2

ze

z

r'

C) cc

<i

_

c

° o),

c (1c

tir

C).

0

C)

0C)C_

co

C)

00

CD-U)

-9C)_.

C)_.

0z00O

C)

0C)3-9_.

C)3

-9a'

0-_.

cDA

r)DBBD3

rvi2

CY.w2D.3D

?I10

A

2D10a.(c

l< --I,

o O

I= F

C),.

0 0

o x1

a"-' 0.C) -CD ,2?. a'

C)

C) I

CD,

A' o10

w ~-+

+s

on May 23, 2020 by guest

http://jcm.asm

.org/D

ownloaded from

TABLE 2. Precision of reference methods used for identification of glucose-nonfermenting gram-negativebacilli'

Most common resultb and no. of errors recorded with: OverallControl strain

(no. of times tested) Mo- Oxi- Argi- H2S 10% 1% NO3 Growth precisiontility dase nine (KIA) Lactose Glucose to gas at 42°C (%)

P. aeruginosa (40) + + + - - + + +1 2 98.8

P. putrefaciens (33) + + - + - - +4 1 3 1 3 94.5

A. calcoaceticus (32) - - - - + + - +1 1 99.2

M. lacunata (14)d - + - - - - - -

2 98.2

Precision %C 95.8 100 99.2 100 100 96.6 96.6 95.0 97.9

a Replicate tests with four control strains tested in four laboratories; see Table 3 for description of tests.b Positive (+) or negative (-), "errors" are either false-positive or false-negative results.c Precision expressed as percentage of tests with expected results.d The control strain of M. lacunata as tested repeatedly in only two laboratories.

included in the first phase of this study. Table 3describes the individual tests performed and liststhe precision estimated on the basis of 15 repli-cate tests with each of 92 strains. In that seriesof tests, false-positive or false-negative resultswith one or more reference tests resulted in amisidentification (13.8%) of the 1,380 identifica-tions recorded or in an "unable to identify"response (3%); 83.2% of the 1,380 identificationswere considered correct. The relative probabili-ty of an accurate result (R.P. value) was low(<0.70) with 28% of incorrect responses butwith only 3% of the correct identifications.When low R.P. values are obtained, erroneousreference tests may be suspected, and additionaltesting is needed before a final identification canbe made with confidence.

Intralaboratory and interlaboratory reproduc-ibility. The precision of Autobac identificationswas established and compared with that of thereference methods by examining results of thefirst phase of this study. Both interlaboratoryand intralaboratory precisions were expressedas reproducibility indexes rather than percent-ages of responses in agreement with the modalresponse (as done in Tables 1 through 3). Withthis approach, all possible pairs of responses arecompared (4, 7). For example, when three re-sponses are being compared, there will be threepairs, i.e., first and second, first and third, andsecond and third. If the three identificationswere E. coli, E. coli, and E. cloacae, the repro-ducibility index would be 0.33 because only oneof the three pairs (33.3%) was in agreement. Inmost studies to date, such reports would havebeen considered 66.7% reproducible (two ofthree correct when compared with the mode).The reproducibility ratio is statistically morevalid because all randomly occurring situationsare considered. It permits inclusion of all data,

even when some data are missing or when theresults are such that a modal response cannot beestablished for estimating precision. However,the reproducibility index provides figures whichappear much lower than percent agreement fig-ures that are customarily quoted. Because theAutobac and reference results were analyzed inthe same way, the reader is urged to comparereproducibility indexes recorded for the twomethods and not to be concerned with theirabsolute magnitudes. Each reproducibility indexwas calculated by dividing the number of pairs inagreement by the total number of pairs beingcompared (Table 4).

Intralaboratory precision ranged from 0.67 to0.84 for the Autobac system compared with 0.58and 0.88 for the reference method. When datafrom all laboratories were combined, intralabor-atory reproducibility indexes were nearly com-parable. The chi-square test demonstrated nosignificant differences between the reproducibil-ities of the two methods (X2 = 1.62, P > 0.20).Interlaboratory reproducibility indexes were0.82 for both methods. Combined ratios, calcu-lated by comparing all possible pairs of respons-es, were 0.74 and 0.76 for the Autobac andreference methods, respectively. Because over8,800 pairs of responses were compared in thisanalysis, the difference between the two indexeswas statistically significant (P < 0.05). Howev-er, the difference is probably too small to be ofclinical importance.Accuracy of identifications. When the identifi-

cation obtained with the Autobac system dis-agreed with that obtained with the referencemethod, independent arbitration was used todetermine whether either of the methods wascorrect. Since entirely different approaches areused in the two systems to achieve an identifica-tion, both systems are not likely to be in error at

J. CLIN. MICROBIOL.1114 BARRY ET AL.

on May 23, 2020 by guest

http://jcm.asm

.org/D

ownloaded from

AUTOBAC ID SYSTEM 1115

TABLE 3. Description and estimated precision of conventional tests utilized to obtain referenceidentifications

Test Precision (%)M Test Precision (%)OGlucose nonfermenters Ornithine decarboxylase, MoellerH2S, Kligler iron agar (KIA) slants 98.1 basal medium.97.5Oxidase spot test, 1% tetramethyl-p- DNase, toluidine blue medium 97.2

phenylenediamine .............. 97.5 ONPG' test for P-D-galactosidaseArginine dihydrolase, Moeller basal activity.96.4

broth.96.2 Sucrose fermentation, peptone brothNitrate reduction to gas, indole with Andrade indicator.96.1

nitrate broth.95.6 Xylose fermentation, peptone broth10%o Lactose oxidation, purple agar with Andrade indicator.96.1

slants ...................... 95.4 Adonitol fermentation, peptoneMotility, hanging drop preparation. 93.7 broth with Andrade indicator .... 95.9Glucose, Hugh and Leifson OFb Arabinose fermentation, peptone

basal medium .................. 88.5 broth with Andrade indicator .... 95.4Growth at 42°C, tryptic soy agar Lactose fermentation, peptone broth

slants ................... ... 74.9 with Andrade indicator.......... 94.6Flagellum stains, selected strains Voges-Proskauer (VP) test, MR-VPe

only......................- c broth and Barritt reagent ........ 94.3Phenylalanine deaminase (PD),

Glucose fermenters phenylalanine agar slants.94.2Oxidase spot test, 1% tetramethyl-p- Salicin fermentation, peptone brothphenylenediamine .............. 99.4 with Andrade indicator.......... 92.5

Lysine decarboxylase, Moeller basal Motility test medium with triphenyl-medium ...................... 98.3 tetrazolium chloride............. 89.8

Indole production, peptone broth Malonate utilization, malonate agarand Kovac reagent.............. 98.1 slants ...................... 86.9

H2S, triple sugar iron (TSI) agar Urease activity, Christensen ureaslants ...................... 97.7 agar slants ..................... 86.3

a Precision of each conventional test expressed as the percentage of correct reactions noted when triplicatetests were performed with 92 strains (66 glucose fermenters and 26 nonfermenters) in each of five separatelaboratories. Fifteen test results were obtained with each strain, and the most common response was accepted asthe correct result for estimating precision.

b OF, Oxidation-fermentation.c, Insufficient data to estimate precision.d ONPG, o-Nitrophenyl-p-D-galactopyranoside test.' MR-VP, Methyl red-Voges Proskauer.

the same time, and if they are, they are unlikelyto provide the same erroneous identification.For that reason, we assumed that the referencemethod was correct when it confirmed the Auto-bac response, thus allowing us to evaluate theaccuracy of the reference method without arbi-tration of an excessive number of strains. Thedata in Table 5 confirm the validity of thisassumption, i.e., when both systems agreed, 323of 327 identifications were found to be accurate.When the two tests disagreed, arbitration con-firmed 49o of the Autobac responses and 36%of the reference identifications.With both test systems, the computer printout

provided an R.P. value which indicates theconfidence that may be attached to the first-choice identification. The response may be con-sidered equivocal if a low R.P. value is listed;i.e., additional tests are needed before the finalidentification can be made with confidence.When estimating the accuracy of a test system,equivocal responses should be excluded sincethey can be neither accurate nor inaccurate.

With other commercial systems, about 5 to 10%of the strains tested have been found to beequivocal (1, 2). The practical utility of a testsystem is diminished if more than 10% of re-sponses indicate a need for additional confirma-tory tests.The data in Table 6 were accumulated to

determine how the accuracy of both systemswas affected by excluding strains with variousR.P. values. With both test systems, eliminationof identifications with low R.P. values excludedmore erroneous responses than correct identifi-cations, thus increasing the overall accuracy.Both systems were approximately 97% accurateif all responses with R.P. <0.95 were excluded,but 11 and 19% of the strains would be consid-ered equivocal. With the Autobac system, only8.5% of the strains had R.P. values of <0.80,and 96.2% of the remaining identifications wereconsidered accurate. However, a significantnumber (about 25%) of Enterobacter spp. wouldbe excluded because the R.P. was <0.80. FewerEnterobacter spp. would be excluded if only

VOL. 15, 1982

on May 23, 2020 by guest

http://jcm.asm

.org/D

ownloaded from

TABLE 4. Intralaboratory and interlaboratory reproducibility' of Autobac identification compared with thatof identifications obtained with reference methods

Pairs in agreement/total no. of pairs Reproducibility indeXbComparison

Autobac Reference Autobac Reference

IntralaboratoryLab 1 211/257 220/255 0.82 0.86Lab 2 211/251 213/257 0.84 0.83Lab 3 162/211 125/214 0.77 0.58Lab 4 211/258 219/263 0.82 0.83Lab 5 177/263 232/264 0.67 0.88All laboratories 972/1240 1,009/1,253 0.78 0.81

Interlaboratoryc 675/826 699/857 0.82 0.82Combination' 6,516/8,809 6,733/8,823 0.74 0.76

a Five laboratories tested 92 strains, each in triplicate, yielding 1,380 responses for each method; fewerresponses were available for analysis because some laboratories failed to report all test results.bNumber of pairs in agreement divided by number of possible pairs.Comparing the most frequent identification reported by each of five laboratories; when one laboratory

reported three different responses, one was randomly selected for comparison.dOverall combination of intralaboratory and interlaboratory precision; each identification was compared with

all other identifications. For each strain, 105 possible pairs of responses were compared.

R.P. values of <0.70 were considered to be

equivocal. This would exclude only 5% of all of

the responses and would result in an overall

accuracy of 95.3%. Consequently, we recom-

mend that first-choice Autobac responses be

accepted only if the R.P. value is .-0.70. Whenwe did that, the overall accuracy of the Autobac

system was comparable to that of the reference

tests.

Sensitivity and specificity. The sensitivity of an

identification system may be defined as the

percentage of strains within a given species or

group of species that was accurately identified

(agreed with the reference identification). On the

other hand, specificity designates the confidence

that may be given to a particular species identifi-

cation reported by the system under evaluation.

For example, 94 strains of C. freundii were

tested with the reference tests, but only 55%

were accurately identified (sensitivity), whereas

94% of the C. freundii identifications obtained

TABLE 5. Result of arbitration tests compared with

first-choice identifications initially obtained with

reference and Autobac systems in the participatinglaboratories

Initial Autobac and referenceArbitration tests identification:

confirmed

Disagreed Agreed

Reference tests 118a _b

Autobac tests 159-

Neither 49 4

Both -323

Total 326 327

a Data presented as number of strains in each cate-

gory.

b-, None possible.

with the reference tests were accurate (specific-ity). Further, the reference methods were 97%sensitive but only 88% specific for A. calcoaceti-cus: all but 3 of the 101 A. calcoaceticus strainswere correctly identified, but other species werealso being misidentified as A. calcoaceticus. Thecalculated sensitivity and specificity for each ofthe microbial species included in the secondphase of this study are presented in Table 7. Allfirst-choice identifications obtained wth the ref-erence method may be contrasted with thoseobtained with the Autobac system. In addition,the percentages of strains within each speciesthat would be excluded because the R.P. values

TABLE 6. Estimated accuracy of reference andAutobac identifications: effect of excluding equivocalidentifications, based on the R.P. that each response

was accurate% Excluded at each % Accuratea after

R.P. R.P. level excluding testsexcluded

Referenceb Autobac Referenceb AutobacNonec -_c - 93.1 93.8<0.60 4.1 3.0 95.4 95.2<0.70 5.2 5.0 95.9 95.3<0.75 6.3 6.5 96.2 95.6<0.80 7.1 8.5 96.4 96.2<0.85 7.8 11.1 96.6 96.5<0.90 9.2 15.0 97.0 96.9<0.95 11.2 19.3 97.4 97.4a Accuracy was judged as agreement with the refer-

ence identifications or arbitration tests (when avail-able)."Eighteen strains that could not be identified with

the initial reference tests were excluded when calculat-ing accuracy.

cEvaluation of all first-choice identification, regard-less of the R.P. value given; -, none.

1116 BARRY ET AL. J. CLIN. MICROBIOL.

on May 23, 2020 by guest

http://jcm.asm

.org/D

ownloaded from

AUTOBAC ID SYSTEM 1117

TABLE 7. Sensitivity and specificity of the reference methods and of Autobac identifications (before andafter excluding equivocal results with R.P. below 0.70)

% Sensitivity (accuracy) of: % Specificityc of:No. of A

AutobacReference identification strains responses Reference Autobac responses Reference Autobac responses

tested excluded testb All Excluding testb All ExcludingR.P. < 0.70 methods tests R.P. < 0.70 methods tests R.P. < 0.70

A. calcoaceticus 101 0.0 97.0 97.0 97.0 88.2 97.0 98.0Aeromonas sp. 38 0.0 97.4 100.0 100.0 92.5 100.0 100.0Alcaligenes sp. 17 17.6 64.7 88.2 92.9 78.6 83.3 86.7C. diversus 56 3.6 98.2 96.4 98.1 91.7 96.4 98.1C. freundii 94 21.1 55.3 80.0 91.9 94.5 72.4 75.6E. tarda 7 0.0 85.7 100.0 100.0 100.0 58.3 70.0E. cloacae 229 14.8 89.4 83.0 83.9 96.7 90.9 94.7E. aerogenes 173 6.4 98.8 91.3 93.8 96.6 95.2 96.2E. agglomerans 33 15.2 87.5 75.8 75.0 43.8 64.1 70.0E. coli 414 4.3 97.6 95.4 97.5 97.6 97.8 99.2Flavobacterium sp. 13 0.0 38.5 100.0 100.0 83.3 86.7 86.7H. alvei 39 5.1 87.2 92.3 94.6 87.2 85.7 89.7K. pneumoniae 312 4.2 97.8 96.5 97.6 98.0 94.1 96.7Klebsiella (other species) 9 22.2 77.8 77.8 100.0 63.6 70.0 70.0Moraxella sp. 4 0.0 100.0 100.0 100.0 25.0 66.7 66.7M. morganii 100 3.0 96.9 97.0 97.9 90.5 94.2 94.9P. mirabilis 304 1.6 98.3 99.3 100.0 100.0 98.7 99.3P. vulgaris 64 1.6 96.8 89.1 88.9 90.9 100.0 100.0ProvidencialP. rettgeri 91 4.4 63.3 96.7 98.8 100.0 89.8 94.4P. cepacia 9 0.0 77.8 77.8 77.8 100.0 70.0 77.8P. maltophilia 57 3.5 71.9 94.7 98.2 95.3 93.1 93.1P. putidalfluorescens 30 6.7 96.6 93.3 92.9 84.8 96.6 96.3P. stutzeri 6 0.0 100.0 83.3 83.3 66.7 100.0 100.0P. aeruginosa 308 1.0 98.7 97.4 98.7 99.3 98.7 99.0Pseudomonas (other species) 10 0.0 50.0 66.7 60.0 41.7 50.0 75.0SalmonellalArizona 94 4.3 95.7 92.6 96.6 92.8 93.5 94.5Serratia sp. 185 1.6 98.9 97.3 97.8 95.8 100.0 100.0Shigella sp. 71 4.2 97.2 93.0 94.1 72.6 95.7 95.5Y. enterocolitica 18 16.7 94.1 88.9 93.3 72.7 69.6 77.8Y. pseudotuberculosis 3 33.3 33.3 33.3 50.0 33.3 100.0 100.0

Total 2,889 5.0 93.1 93.8 95.3a Autobac responses were evaluated with and without excluding equivocal results, R.P. < 0.70. The

percentage of strains within each species with equivocal responses is listed.b Excluding 18 strains that could not be identified by the initial reference tests, including all other first-choice

identifications.c Specificity indicates the confidence that can be placed upon a given species identification, i.e., the number of

correct results divided by the number of times each species was reported.

for Autobac responses were <0.70 are noted inTable 7. By excluding such equivocal identifica-tions, the sensitivity and specificity of the Auto-bac system were somewhat improved.

In addition to the clinical isolates reportedhere, tests were performed with 10 isolatesbelonging to species that are not included in theAutobac program (6 Achromobacter xylosoxi-dans, 3 Pasteurella multocida, and 1 Bordetellabronchiseptica isolate). The Achromobacter sp.isolates were all misidentified as Pseudomonassp., as was the Bordetella sp. The Pasteurellasp. isolates were misidentified as Moraxella sp.,E. agglomerans, and Edwardsiella sp. All ofthose misidentifications were reported with R.P.values of .0.70 and thus would not be consid-ered equivocal identifications.

DISCUSSION

The Autobac system for rapid identification ofgram-negative bacilli represents a unique ap-proach to bacterial taxonomy based on the pat-terns of susceptibility to various antibacterialagents. Continuing efforts to find non-chemo-therapeutic agents that can be used in such a

system should improve the reliability of thisapproach. Resistant variants might appear insome environments, leading to atypical suscepti-bility patterns whch might lead to misidentifica-tions. Because there should be no selectivepressure for variants that become resistant tothe non-chemotherapeutic agents, efforts shouldbe made to replace the therapeutic agents thatare currently included in the system.

VOL . 1 5, 1982

on May 23, 2020 by guest

http://jcm.asm

.org/D

ownloaded from

1118 BARRY ET AL.

The taxonomic groups that were used to de-velop the Autobac identification system werelimited to 30 species or groups of species that arelikely to be found in clinical specimens. Noattempt was made to identify species of Aetro-monas, Alcaligenes, Fl/a tobacteriuln, Morax-ella, Serri-atia, or Slhigella. In most clinical situa-tions, identifications to the generic level will besufficient. Serological confirmation of Sailinioniel-la spp. and Shigella spp. would be a necessarysupplement to this system. Sailmonella sp. andAr-izona sp. are not distinguished, but many holdthat Arizona sp. should be placed in the genusSallmonella. P. rettugeri is not distinguished fromP. stiuartii (urea positive and urea negative).Failure to make that distinction currently pre-sents no important clinical problem. K. pnieia-mnoniae and K. oxytoca are not separated, butthat could be accomplished by simply referringto the spot indole test (10). K. o.xvtoca beingindole positive. Furthermore, the Autobac com-puter program does not yet identify three recent-ly recognized species (Enterobacter sakaz-akii,E. gergo'ii, and Citrobacter amnalonaticis). Ourlimited experience with testing a few speciesthat are not included in the current programunderscores the possibility that other misidenti-fications can occur with uncommonly encoun-tered species.

In spite of these relatively minor limitations.the Autobac system can rapidly identify the vastmajority of gram-negative bacilli found in clini-cal material. The sensitivity. specificity. andprecision of the Autobac system are comparableto the corresponding characteristics of the refer-ence test system. However, the Autobac systemis much more rapid (3 to 6 h versus 48 h). and themechanization and computer-assisted interpre-tation minimize the technologist time required toobtain reliable results.

In most studies of this nature, a new testsystem is normally compared directly with areference system, and any discrepancies areassumed to represent errors on the part of thenew system. The present study provided aunique opportunity to evaluate the referencemethod, as well as the Autobac system. Whenthe two disagreed, the Autobac system was inerror about half of the time, and the referencemethods were in error a little more than half ofthe time (Table 5). Although every effort wasmade to standardize and to control the referencemethods, false-positive and false-negative reac-tions were obtained with about 2 to 3% of theindividual tests. Occasionally, the erroneoustest results were important enough to result in amisidentification and, subsequently, a disagree-ment with the Autobac system. For the samereason, false-positive or false-negative resultswith the Autobac system could result in errone-

ous identifications which disagree with the refer-ence methods. For that reason, it is easy tounderstand why correlation between two inde-pendent identification systems rarely exceeds 90to 95%, depending upon the precision of the twosystems. An even greater number of discrepan-cies might be expected if conventional testsrather than standard reference methods hadbeen used for evaluating the Autobac system.For example, conventional tests may representstandard tubed media read after overnight incu-bation, accepting some loss of precision andaccuracy for the sake of convenience. Suchprocedures are not appropriate reference meth-ods for evaluating new identification systems.By excluding tests with a low probability of an

accurate response, a significant proportion ofmisidentifications were eliminated. The overallaccuracy of both systems was as great as 97%when all responses with R.P. levels of <0.95were excluded. But that would have requiredsupplementary tests with 11 or 19% of the iso-lates included in the second phase of this study.We concluded that Autobac identifications withR.P. values of .0.70 could be accepted andthose of <0.70 should be confirmed with supple-mentary tests. Only 5% of the isolates includedin this study would require such confirmation,and the overall accuracy of both the Autobacand the reference methods was 95 to 96%.Sensitivity and specificity of the two methodsvaried somewhat with different species, but, ingeneral, the Autobac system was as sensitiveand specific as the reference method.

In summary, the Autobac system provides aunique approach to the rapid identification ofgram-negative bacilli. The overall results indi-cate that the system is just as sensitive, specific,and precise as the standard reference methods.The mechanized system, with computer-assistedinterpretation, requires a minimum amount oftechnologist time and provides reliable resultswithin 3 to 6 h.

ACKNOWLEDGMENTSWe express our sincere gratitude to the following microbiol-

ogists who performed the studies described in this report: R.Aaron. C. William Bacon. Robert E. Badal. Ann 0. Esaials,B. B. Gardner. Michael H. Graves. C. Knapp, Dwane L.Rhoden, and M. Telenson. We also acknowledge the invalu-able assistance provided by microbiologists in the EntericBacteriology Section of the Centers for Disease Control andby G. L. Gilardi, who provided certain unpublished data thatwere needed for establishing the data base used for interpreti-tion of the standard reference tests.

LITERATURE CITED1. Barry, A. L., and R. E. Badal. 1979. Rapid identification

of Enterobocterioaoea with the Micro-ID system versusAPI 20E and conventional media. J. Clin. Microbiol.10:293-298.

2. Barry, A. L., R. E. Badal, and L. J. Effinger. 1979. Identi-

J. Cl-IN. MICROBIOL

on May 23, 2020 by guest

http://jcm.asm

.org/D

ownloaded from

AUTOBAC ID SYSTEM 1119

fication of Enterobocteriaceae in frozen microdilutiontrays prepared by Micro-Media Systems. J. Clin. Micro-biol. 10:492-496.

3. Ewing, W. H., and B. R. Davis. 1975. Media and tests fordifferentiation of Enterobocterioceae. U.S. Department ofHealth, Education and Welfare. National CommunicableDisease Center. Atlanta. Ga.

4. Fleiss, J. L. 1971. Measuring nominal scale agreementamong many raters. Psychol. Bull. 76:378-382.

5. Friedman, R., and J. MacLowry. 1973. Computer identifi-cation of bacteria on the basis of their antibiotic suscepti-bility patterns. AppI. Microbiol. 26:314-317.

6. Kovacs, N. 1956. Identification of Pseilonionaos ae(lruinl-

osa by the oxidase reaction. Nature (London) 178:703.7. Light, R. J. 1971. Measures of response agreement for

qualitative data: some generalizations and alternatives.Psychol. Bull. 76:365-377.

8. Sielaff, B. H., E. A. Johnson, and J. M. Matsen. 1976.Computer-assisted bacterial identification utilizing antimi-crobial susceptibility profiles generated by Autobac 1. J.Clin. Microbiol. 3:105-109.

9. Sielaff, B. H., J. M. Matsen, and J. E. McKie. Novelapproach to bacterial identification that uses the Autobacsystem. J. Clin. Microbiol. 15:1103-1110.

10. Vracko, R., and J. C. Sherris. 1963. Indole-spot test inbacteriology. Am. J. Clin. Pathol. 39:429-432.

VOL. 15 1982

on May 23, 2020 by guest

http://jcm.asm

.org/D

ownloaded from