Internal Medicine Resident Satisfaction with a Diagnostic Decision SupportSystem (DXplain®) Introduced on a Teaching Hospital Service.

Brent A. Bauer, MD', Mark Lee, MD', Larry Bergstrom, MD', Dietlind L. Wahner-Roedler, MD', John Bundrick, MD', Scott Litin, MD', Edward Hoffer, MD2, Richard J.

Kim2, Kathleen Famiglietti2, G. Octo Barnett, MD2 Peter L. Elkin, MD'1. Area General Internal Medicine, Department of Medicine, Mayo Clinic

2. Laboratory of Computer Science, Massachusetts General Hospital

Abstract

Objective:The objective of this study was to determine whetherInternal Medicine residents wouldfind the use of anexpert system (i. e. Clinical Diagnostic DecisionSupport System) to be a satisfactory experience whenused during a clinical rotation in the hospital setting.Resident willingness to use the instrument was

considered to be ofparticular importance because ofgrowing concerns regarding the educationalexperience of residents in the hospital. Thus, our

first goal was to assess resident satisfaction with thetool, prior to widespread implementation.

Study Population:Residents on the General Medical Hospital Service atSt. Mary's Hospital, Rochester, Minnesota

Study Type:Prospective cohort study

Method:We provided unrestricted access DXplain, a Web-based Clinical Diagnostic Decision Support System,to five general medical teams in St. Mary's Hospital,Rochester, MN. All residents were particularlyencouraged to access the system during theevaluation of new admissions. Usage of the systemwas recorded electronically each time a user loggedon. At the conclusion of the 2 month study period, a

survey was sent electronically to each of theparticipating residents.

Results:During the study period, a total of 30 residents (GI=20, G3= 10) rotated on the five medical services.29/30 residents responded to the survey. 18/29indicated that they had used the service, while 11/29stated that they had not accessed the system. Theresident's logged on 117 times to enter a case duringthe study period, with several entering more than one

case per logon. The average number of log-ons peruser was 2.4/week. Of the 18 who used the system,15 found it useful (83.3%), 2 were unsure whether it

was useful (IL I%), and I (5.6%) did not think it washelpful. However, when asked how often the systemled the user to consider novel alternative diagnoses13/18 (72.2%) responded "almost always tofrequently" and 5/18 (27.8%) said "occasionally tosometimes". None of the users felt that the system"rarely" or "never" yielded additional diagnosticconsiderations. Seventeen out of eighteen users(94.4%) thought the system was "easy to use ". Whenasked if they would like to have such a systemregularly available 13/18 (72.2%) responded yes,while 4/18 (22.2%) were unsure. One resident saidthat they would not like to have DXplain available(5. 6%o).

Conclusion:We believe the data reflect a significant level ofsatisfaction with the system among residents. Theirrecognition that it frequently led them to considernovel diagnoses suggests it had a positiveeducational impact.

Introduction

Currently, the practice of medicine is undergoingsignificant changes, perhaps nowhere morenoticeably than in the hospital setting. Patients whoare admitted to a hospital service, especially at ateaching institution, generally are sicker and morecomplex than even a decade or two ago. Economicfactors constrain resources and time available forhospital evaluations and therapeutics. As a result,residents, perhaps more acutely than anyone, havebeen sensitized to the need to rapidly diagnose, treat,and dismiss patients from the hospital setting. One ofthe consequences of this approach to medicine is thatthere is often little time for reflection orcontemplation regarding a newly admitted patient.The goal is to determine a working diagnosis andproceed with haste to confirm the diagnosis, treat thepatient and return the patient to the outpatient setting.This becomes a risky strategy when applied toelderly, complex patients, with multi-organ systemdysfunction, whom are currently being seen withincreased frequency on our teaching hospital service.

AMIA 2002 Annual Symposium Proceedings3 31

In particular, the diagnostic data from the last twoyears indicates that the average number of significantdiagnoses per patient is greater than four. As a result,these increasing demands on the practice maytruncate attempts at generating a differentialdiagnosis, and can often lead to misidentification ofthe problem or elimination of secondary diagnosesfrom the problem list.'

This trend is troubling because the utility of thedifferential diagnosis is multifold.2 First, theclinicians' differential provides other physicians withinsight into what the admitting resident was thinkingat the time of admission. This can help provide theattending physician with insights into deficits in theresident's approach to the case. The differentialdiagnosis list also helps promote educational effortson rounds, as other care providers mentally mustconsider each stated potential diagnosis, and measureits fit with the case, as presented. Third, a robustdifferential diagnosis helps keep the care team alertto the potential for a change in diagnosis.3'4 Whenworking with a solitary diagnosis early on, thetendency is often to make the data fit the diagnosis,which has already been "chosen" (the "anchoring andadjustment phenomenon").2 Maintaining a viabledifferential diagnosis helps keep the medical teamalert to the possibility of a change in the diagnosticfocus of the care team, with potential for change intherapeutics, prognosis, etc.

However, the very time constraints which have givenrise to the concerns mentioned above, also makeresidents justifiably wary of any new demand on theirtime. Thus, before we proceeded to widespreadimplementation of a new tool (which we perceived tobe easy to use and time efficient) we wanted to test iton a small sample of the intended users. Wereasoned that even if the tool was helpful, residentswould still be dissatisfied if it was cumbersome orslow or otherwise contributed disproportionately totheir workload. Thus, we set out to evaluate theperceptions of residents actually using the system inthe clinical setting.

Preliminary Work

Diagnostic Decision Support Systems

DXplain, a computer-based medical education,reference and decision support system, wasdeveloped by the Laboratory of Computer Science(LCS) at Massachusetts General Hospital (MGH).56DXplain has the characteristics of an electronic

medical textbook, a medical reference system and adecision support tool. In the role of a medicaltextbook, DXplain can provide a comprehensivedescription and selected references for over 2,000different diseases, emphasizing the signs andsymptoms that occur in each disease, the etiology, thepathology, and the prognosis. As a decision supporttool, DXplain uses its knowledge base of the crudeprobabilities of approximately 5,000 clinicalmanifestations (History, PE findings, Lab data, Xraydata and elements of the PMHx) and generates fromit a differential diagnosis7'8 associated with individualdiseases. The system uses an interactive format tocollect clinical information and makes use of amodified form of Bayesian logic to produce a rankedlist of diagnoses which might be associated with theclinical manifestations. DXplain uses this sameknowledge base and logic to list other findings that, ifpresent, would support a particular disease, and alsolists what findings entered by the user are not usuallyfound in a particular disease. The system alsoprovides references and disease descriptions for eachof the diagnoses in its database.9

Over the past fifteen years ago DXplain has beenused by thousands of physicians and medicalstudents. Five years ago, LCS began to makeDXplain available over the Internet to hospitals,medical schools, and medical organizations."'

Methods

We provided unrestricted access to DXplain to fivegeneral medical teams in St. Mary's Hospital,Rochester, MN (Mayo Foundation's largest teachinghospital). Each service takes call every other day,with an average of 4 admissions per day. Teams arecomprised of two First Year residents, a Third Yearresident and a staff physician. Each team wasprovided with a one-hour introductory session inwhich basic features of DXplain were reviewed. Allresidents were particularly encouraged to access thesystem during the evaluation of new admissions. Aspecial link was created on the Residency home pageto allow easy linkage to the Web site containingDXplain. Usage of the system was recordedelectronically each time a user logged on. Remindersof the system's availability were sent out via e-mailevery two weeks. A survey (Figure A) was sentelectronically to each of the participating residents atthe conclusion of their rotation on the service.

Data analysis was performed using the BinomialExact Test to compare the response rates with a rate

32

of 50% to determine whether or not the results couldbe explained by a random effect.

Figure A: Survey for Resident Staff afterusing DXplain

1. Did you find the program Helpful? Y,N, Undecided, Did not use

2. Please comment on your answer in #1.3. How many times a week did you use

the program?4. Was the program more useful on your

admitting days or other days?5. Was the program easy to use? Y, N,

Undecided, Did not use6. Was the program as easy to use as you

expected? Easier, Just as Easy, Harderto use, Undecided, Did not use

7. Would you like a program such as thisone to be available routinely to InternalMedicine Residents at Mayo? Y, N,Undecided, Did not use

8. If there was a feature ofthe programwhich was particularly helpful pleasecomment.

9. Did the DXplain program help youthink of diagnoses, which you thoughtreasonable to consider for your patient'spresentation.(Always, Almost Always,Frequently, Sometimes, Occasionally,Rarely, Never)

Results

During a two-month period, a total of 30 residents(GI =20, G3= 10) rotated on the five medicalservices. During the two-month study period theresidents had 117 logons to DXplain where clinicalcases were entered (multiple cases may have beenentered per logon). 29/30 (96.7%) residentsresponded to the survey. 18/29 (62.1%) indicatedthat they had used DXplain, while 11/29 (37.9%)stated that they had not accessed the system. Theaverage number of log-ons per user was 2.4/week.Of the 18 who used the system, 15 (83.3%) found ituseful, 2 (11.1%) was unsure whether it was useful,and 1 (5.6%) did not think it was helpful (Mean94.4% with lower confidence interval of 64%(p=0.03, Compared with a coin flip, MethodBinomial Exact Test)). However, when asked howoften the system led the user to consider novelalternative diagnoses, 13/18 (72.2%) responded"always to frequently" and 5/18 (27.8%) said

"occasionally to sometimes". None of the users feltthat the system "rarely" (0/18) or "never" (0/18)yielded additional diagnostic considerations. 100%of the respondents felt that DXplain yieldedadditional diagnostic possibility at least"Occasionally" (with lower confidence interval of78% (p=0.003) Compared with a coin flip, MethodBinomial Exact Test). Seventeen out of eighteenusers thought the system was "easy to use" (94.4%with lower confidence interval of 72% (p=0.004Compared with a coin flip, Method Binomial ExactTest)). When asked if they would like to have such asystem regularly available 13/18 (72.2%) respondedyes, while 4/18 (22.2%) were unsure. 1/18 (5.6%)responded that they would not want to have DXplainavailable to their practice (94.4% did not say no (withlower confidence interval of 72% (p=0.004)Compared with a coin flip, Method Binomial ExactTest)).

Did you find the program Helpful?| Useful | Not Useful | Unsure

Resident Responses 115 1 * 2* To rule out that the system was "not useful" wereported a mean rate of 17/18 (94.4%) with lowerconfidence interval of 64% (p=0.03) Compared witha coin flip, Method Binomial Exact Test

Did the DXplain program help you think ofdiagnoses, which you thought reasonable toconsider for your patient's presentation?

Almost Sometimes - Rarely Neveralways- Occasionallyfrequently

Responses 13 5* 0 0* 100% responded at least Occasionally with lowerconfidence interval of78% (p=0.003) Compared witha coin flip, Method Binomial Exact Test.

Was the program easy to use?| Easy | Not easy | Unsure |

Resident Responses 117 | 0 | 194.4% responded that DXplain was easy to use, withlower confidence interval of 72% (p=0.004)Compared with a coin flip, Method Binomial ExactTest.

33

Would you like a program such as this one to beavailable routinely to Internal Medicine Residentsat Mayo?

Yes No UnsureResidentResponses |13 1* 14*72.2% yes, 94.4% did not say no with lowerconfidence interval of72% (p=0.004) Compared witha coin flip, Method Binomial Exact Test.

Conclusions

Residents were generally satisfied with the use ofDXplain as an adjunct to their usual evaluation ofnew admissions onto a hospital service. 83.3% of therespondents felt it was useful and 94.4% felt it waseasy to use. Also of importance, every respondent(100%) felt that the use of the system at least"occasionally" led to contemplation of previouslyunconsidered diagnoses. Even more impressive,72.2% felt that it "almost always or frequently" led tonew diagnostic considerations.

The strengths of this study include our ability toelectronically monitor each time the system wasaccessed. Thus, we did not have to rely on residentrecall of their usage of DXplain. Anecdotally, theresidents said that they felt that DXplain was usefulin recommending truly novel diagnoses. Forexample, one resident identified a case of "InclusionBody Myositis" after it was suggested by DXplain.

Acknowledgments

This work has been supported, in part, by a MayoClinician Engaged in Education Grant and the MayoClinic's Department of Internal Medicine. Theauthors greatly appreciate the efforts of the Residentson the General Medical Hospital Service at SaintMary's Hospital and our clinical colleagues in AreaGeneral Internal Medicine at the Mayo Foundationwho worked with DXplain during their HospitalService Rotations.

A weakness of this study was its small sample size.Survey response rates are notoriously low, especiallyamong physicians. Our response rate of 96.7% frombusy residents covering a rapidly changing hospitalpractice was an excellent response rate.

An intriguing finding is that 17 of 18 respondents feltthat the system was easy to use and brought newdiagnoses into consideration. We recognize thatindividual resident exposure to the tool was limitedand it is certainly possible that application to anincreasing number of clinical cases would lead torecognition of DXplain's greater utility, especially ifa novel diagnosis suggested by DXplain turned out tobe correct.

It is also disturbing that only 18 of the 29 surveyresponders actually tried the system. This mayreflect that time has become so constrained onInternal Medicine residency hospital programs thatresidents cannot even take the time to experimentwith a new tool that could potentially increase theireffectiveness as physicians. The authors concludethat residents on the Saint Mary's Hospital GeneralMedical Service generally found DXplain easy to useand useful for expanding their differential diagnoses.

Future research should seek to determine what factorslead to greater utilization of diagnostic decisionsupport systems.

References

1. Elkin PL, Tuttle M, Keck K, Campbell K, AtkinG, Chute CG. The Role of Compositionality inStandardized Problem List Generation. In: BCesnik, AT McCray, J-R Scherrer (eds).Proceedings of the 9h World Congress onMedical Informatics. MEDINFO 98 1998;1:660-664.

2. Tversky A, Kahneman D. Judgment underuncertainty: heuristics and biases. Science 1974;185:1124-31.

3. Chueh HC, Barnett GO. "Just In Time" ClinicalInformation. Academic Medicine. 1997;71(6):512-517.

34

4. Barnett GO. The computer and clinicaljudgement. New Eng J Med. 1982;307:493-494.

5. Barnett GO, Cimino JJ, Hupp JA, Hoffer EP.DXplain - an evolving diagnostic decision-supportsystenL JAMA. 1987;258:67-74.

6. "Dxplain - Important Issues in the Developmentof a Computer Based Decision Support System";Barnett GO, Hoffer EP, Elkin PL; IEEEProceedings of the 14th Annual Symposium onComputers and Medical Care 1990, p. 1013-17.

7. "Evolution of Dxplain: A Decision SupportSystem"; Packer MS, Hoffer EP, Bamett GO,Famiglietti KT, Kim RJ, McLatchey JP, ElkinPL, Cimino CC, Studney DR IEEE Proceedingsof the 13th Annual Symposium on Computersand Medical Care 1989, p. 949-53.

8. "Closing the Loop on Diagnostic DecisionSupport Systems"; Elkin PL, Bamett GO,Famiglietti KT, Kim RJ; Proceedings of the 14'hAnnual Symposium on Computers and MedicalCare 1990, p. 589-594.

9. Barnett GO, Famiglietti KT, Kim RJ, Hoffer EP,Feldman MJ. DXplain on the Internet.Proceedings of the 1998 AMIA Annual FallSymposium, 1998 Nov 7-11, p. 607-611.

10. www.lcs.mgh.harvard.edu

35