1

Preclinical-Bench Testing IIUsing Human Observers to

Objectively Measure and Evaluate Imaging Performance of Digital WSI

Systems

Max Robinowitz, MD

FDA Center for Devices and Radiological Health

October 22, 2009

2

Object Object Related Related

DiagnosisDiagnosis

Please look at this picture

Identify all that you see

What do you think they represent?

What is your diagnosis?

3From Kayser K et al Diagnostic Pathology 2006

4

How Can We Objectively Evaluate If Image Quality Presented to Pathologists

is Fit-for-Purpose?•How can we quantify the contribution of the image quality to a final tissue-based diagnosis?•How can we quantify contribution added from the eye & brain of the pathologist?•How can we quantify the ability of pathologists to search & detect different objects rendered by OM & WSI?•How can we identify the contributions of the ergonomics and human factors of the different work stations?

5

Role of Preclinical-Bench Testing

Physical & biological phantoms may provide objective data for prediction of diagnostic performance of pathologists with the optical microscope & different WSI systems in the real world in a manner similar to x-ray phantomsThe general application of WSI to routine surgical pathology practice potentially involves all of clinical medicineAll clinically essential areas within the whole spectrum of surgical pathology may have to be tested

6

Human Observer Studies with Biologic & Human Histologic Specimens Phantoms

• Minimize the subjectivity of study pathologists by emphasizing each pathologist’s assessments of image quality and information content before the overlay of high-level intellectual diagnostic interpretation

• Targets can be constructed from biological materials of known size, number, fine morphological features & spatial distributions to quantify image quality---objectively

7

Controlling Subjectivity in Performance of Real World Tasks of Pathologists

• Pre-test tutorial to standardize criteria for the study pathologists.

• Pre-specify the features to observe, enumerate, using criteria that minimize requirements for higher-level interpretation by pathologists

• Use biological objects in study as the equivalent of a Snell eye chart to test eye glasses.

• Task: Recognition of shapes & sizes in images using description rather than diagnostic terms --- Have pathologist quantify the detailed features of the image

8

Diatoms = Algae with Known Size & 3D Textural Features Requiring 20 & 40X Objectives to View Details in Image

97 Diatoms ranging from about 2 to 100 micrometers in diameter. Note: fine 3D internal features that can be

objectively measured & enumerated

From Kemp K diatoms.uk.com

9

Objective “Open Book Exam” Study with Human Specimens

• Controlled, non-subjective evaluation of image quality rendered by OM versus the image quality rendered by WSI systems

• Reproducible, optimal preparations of normal and abnormal human specimens on TMA cores and whole slide

10

Critical objects & Features to evaluate

• Accuracy & precision of pathologists for identification of objects and features in targets

• Clinically significant objects, e.g., cancer microinvasion, nuclear chromatin structure, nucleoli, cytoplasmic vacuoles, inclusions, granules & striations, architectural growth patterns, etc.

11

Critical Variables to evaluate• Effects of tissue preparation, slide

thickness, different stains, different adjustments & settings of OM and WSI system

• Variability in planes of focus selected by WSI on different runs

• Variability in pathologists with different types of image features

12

Tasks for Study Pathologists That CanUse Cores or Whole Slide Specimens

• Task 1 Search & find/detect objects and textures to test image quality, focus and fidelity, dynamic range, color fidelity across whole slide

• Task 2 Enumerate number & measure objects

• Task 3 Classify detected objects

13

Task 4 for Study Pathologists is Limited to Whole Slide Specimens

• Task 4 Diagnose based on the enumeration and cumulative semi-quantitative scores from the objects and features from larger areas that exceed the size of TMA cores (mimics intended use)

– Quantify with Scoring system that is a composite of multiple simple features—some require the optical resolution at 40X & z-axis focusing

14

Tissue Microarrays with Cores of Normal & Abnormal Features Selected by Experts

TMA cores from ROIs of multiple specimens, in precise order of a grid on single slide, allows efficient viewing by study pathologists for only some of the fit-for-purpose testing

TMA Paraffin block

Individual core

9X9 grid

15

Tissue Microarray (TMA) Tissue CoresDiameter Sizes and Numbers/Slide

• Core sizes are 0.6, 1.0, 1.5 and 2.0  mm most common. Most popular is 1.0 mm.

•  0.6 mm = standard 40X objective field of view• 1.0 mm = standard 20X objective field of view•   Larger cores can be made but even 1.5 and 2.0

are very hard to work with, but are good for normal tissues where you want to show a number of features.

• 0.6 = 500 cores/slide; 1.0 = 200 cores/slide• 1.5 = 75 cores/slide; 2.0 = 35 cores/slide

16

Tissue Microarray (TMA) Tissue CoresApplications and Limitations

• Tissue microarrays are suitable to test pathologists’ ability for recognition of cellular objects (features) and some limited spatial tissue growth patterns

• TMAs are NOT suitable for testing the tasks of search & find/detection of ROI within the multiple different diagnostic areas on a whole slide specimen as required in the real-world of surgical pathology

17

Bench-Testing for Realistic Diagnostic Tasks: Combinations of Multiple Features

1. Pre-test tutorial with schematic with the features to be identified and compared with slide

2. Slide with multiple conditions to search, detect & describe according to their location on the whole slide and its image on WSI, e.g. upper right quadrant, lower right quadrant

3. Match & Describe each abnormality according to the schematics and pre-test “standardized terminology”

III

IVDCIS

18

Nottingham BrCa (Ordinal) ScoreControlled Task for OM & WSI

• Tubule Formation– Majority of tumor (>75%) 1– Moderate degree (10-75%) 2– Little or none (<10%) 3

• Mitotic Count– 0-9 mitoses/10 hpf 1– 10 -19 mitoses/10 hpf 2– 20 or > mitoses/10 hpf 3

• Nuclear Pleomorphism– Small regular uniform cells 1– Moderate nuclear size and

variation 2– Marked nuclear variation 3

• Combined Histologic Grade

• Low grade (I) 3-5• Intermediate grade

(II) 6-7• High grade (III) 8-9

This is just one of many clinically relevant examples

19

Heirarchy for Grading Dx Errors10 levels of increasing complexity of diagnostic

interpretation & final diagnosis nomenclature

Mireskandari M, Kayser G, Hufnagl P, Schrader T, Kayser K

Diagnostic Pathology 2004

Image Based

High-Order Intellectual Overlays

20

FDA Desires Panel Input

• What roles can bench-testing of OM and WSI serve for guiding the designs for clinical studies in real world surgical pathology practices?

• What safety and effectiveness concerns can be resolved with pre-clinical studies using selected histopathology specimens, such as TMAs and whole slides? What are the limitations?

• What concerns require clinical trials with representatives of the intended pathologists who must examine whole slides in the real world setting of surgical pathology practice?