research objective

Development and Evaluation of a Cognitive Simulator for Improving the Diagnostic Proficiency of Generalist, Community-Based Pathologist for Malignant Melanoma

Dana Grzybicki, PhD, MD1, Rebecca Crowley, MD, MS2, Drazen Jukic, MD, PhD2,3, Stephen Raab, MD1, Olga Medvedeva, MS2, Eugene Tseytlin, MS2, Melissa Castine, BA2

Department of Pathology, University of Colorado Denver Health Sciences Center1, Center for Biomedical Informatics, University of Pittsburgh2, Department of Pathology, UPMC3

The research objective is to examine the effectiveness of cognitive simulation as a method for improving generalist community practice pathologists’ diagnostic proficiency for difficult pigmented skin lesions. We will determine whether an ITS improves patient safety by decreasing diagnostic errors. Elements of the current system have been previously evaluated and shown to produce dramatic learning gains among pathology residents and fellows3,4.

Research Objective

Background

A controlled clinical evaluation of the effectiveness of use of the modified intelligent tutoring system simulator as an educational intervention for improving diagnostic accuracy compared to the currently used conventional forms of continuing education (e.g. reviewing one or more current textbooks, reviewing some current medical literature, and working through publically available online review cases).

A group of 20 volunteer private practice, community generalist pathologists, matched for years in practice, estimated annual total number of surgical cases examined, and estimated annual total number of pigmented lesions examined will be randomly assigned to the experimental condition (simulation tutor group) or the control condition (self-study group).

Study Design

Preliminary Findings

1. Using Protégé OWL, a much more complex and comprehensive knowledge base was successfully created that can accurately represent difficult melanocytic skin lesions, including lesions previously associated with diagnostic errors.

2. As a result of information derived from pilot studies, extensive modifications to the previous cognitive simulator interface, case authoring tool, and tutor functionality were necessary in order to generate a simulation tool acceptable and user-friendly for practicing pathologists. The new system reflects the higher level of diagnostic expertise, different diagnostic workflow requirements, and changes in tissue section visual and cognitive assessments with increasing time in practice.

3. Based on results from the initial baseline testing scores of the volunteer subjects, a high level of variability in diagnostic expertise for melanocytic skin lesions exists among practicing community pathologists.

References

ConclusionsWe have been able to successfully develop and pilot test a cognitive simulator suitable for use by experienced, practicing pathologists for effectiveness testing as an educational tool to improve histopathologic diagnostic skills for one of the most litigated diagnostic errors in anatomic pathology: false positive or false negative diagnoses of malignant melanoma. General pathologists who have volunteered to serve as subjects in this study show a high level of variability in their level of diagnostic expertise for malignant melanoma.

Intelligent Tutoring Systems (ITS) are computer-based knowledge communication systems that provide individualized instruction by incorporating methods from artificial intelligence1. Despite many successes in other domains, few intelligent tutoring systems have been developed for use in medical fields2. However, through previous empirical studies we have successfully developed two different tutors, SlideTutor and ReportTutor, that have shown to improve learning among resident pathologists3,4. SlideTutor focuses on teaching the set of diagnostic findings that are needed in coming to a correct final diagnosis, whereas ReportTutor focuses on teaching the relevant prognostic factors that are required to report during the sign-out of a case. In our current system, we have integrated and modified SlideTutor and ReportTutor into a Pathology Simulation Tutor (SimTutor) that provides appropriate guidance and instruction for non-trainee, generalist pathologists with varying expertise in the diagnosis of melanocytic skin lesions.

1. Anderson JR, Corbett AT, Koedinger KR et al. Cognitive tutors: Lessons Learned. J Learning Sci 1995, 4: 167-207. 2. Lillehaug S-I and LaJoie SP. AI in medical education, another grand challenge for medical informatics. AIIM 1998 12: 197-225.3. Crowley RS, Legowski E, Medvedeva, OM, Tseytlin E, Roh E and Jukic D. Evaluation of an Intelligent Tutoring System in Pathology: Effects of External Representation on Performance Gains, Metacognition, and Acceptance. JAMIA 2007 14(2):182-190.4. Saadawi GM, Tseytin E, Legowski E, Jukic D, Castine M, and Crowley RS. A natural language intelligent tutoring system for training pathologists: implementation and evaluation. Adv Health Sci Educ Theory Pract. 2007 Oct 13.

Implications

Results of the study currently underway will provide a first summative evaluation of a medical cognitive tutoring system. The use of cognitive simulation could be a highly convenient and effective tool for both CME and proficiency testing purposes that could be expanded to many domains in Pathology.

Expert mode monitors your work in the background, providing hints and analyzing errors

A correct diagnosis allows you to move onto the next tab

Order additional levels and/or immunohistochemical stains

Ask for help and get a series of progressive hints that will walk you through the problem

Novice mode monitors your work and will give immediate feedback for any incorrect action

Critical diagnostic errors will immediately show up as red text and enter you into novice mode

Add multiple authored slides per case

Hint messages will help guide you how to properly evaluate and write the important prognostic features required for a given diagnosis

Communication between the diagnostic and prognostic parts of the tutor allow the final diagnosis and findings to be carried over to the report

The report tab of the tutor acts much like a word processor while parsing the text

Glass Slide Proficiency

Test

Digital SlidePre-test

Tutor Training

Study Session 14 hr Tutor

Use

Digital SlidePost-test 1


Use





Use


Use


Test


Test

Digital SlidePre-test

Study Session 14 hr Self

Study



Study





Study


Study


Test

Experimental Group Timeline

Control Group Timeline

Percent Correct of Glass Slide Pre-test (n=20 cases)

0%

20%

40%

60%

80%

100%

0 2 4 6 8 10 12 14

Subject

Pe

rce

nt

Co

rre

ct

research objective

Documents

diagnostic accuracy

litigated diagnostic

general pathologists

cognitive simulator

cognitive assessments

current system

high level of variability

anatomic pathology