Impacts of Artificial Intelligence and New Technology for Melanoma Detection

Professor of Dermatology

Director, Pigmented Lesion & Melanoma Program

Physician Leader, Cutaneous Oncology Program

Stanford University Medical Center and Cancer Institute

S U S AN M . S W E T T E R , M D

2 How do we know what we know?

• Unconscious pattern matching• Rules-based systems

Systems drive the way we think:System 1 – fast, intuitive, emotionalSystem 2 – slower, more deliberate and logical

Dual-process theory applied to Dermatologists

System 1 (intuitive) “visual intelligence” – rapid, operates below level of perceptible consciousness (“gut feeling”)

- focuses on pattern recognition: “blink, think” and “10-second rule” (Giuseppe

Argenziano, Naples)

- why “ugly duckling” rule discriminates melanoma more accurately than ABCD clinical warning signs (Gaudy-Marquest C et al. JAMA Dermatol 2017)

System 2 (analytical) – deliberate judgment, based on conscious applications of rules acquired through learning

Transition from intuitive to analytical reasoning can hinder clinical reasoning and increase diagnostic error

Pelaccia T et al. Med Educ Online 2011. Stolper E et al. BMC Fam Pract 2009. Norman GR, Eva KW. Med Educ 2010.

● Skin cancer - most common cancer in the US

● 1 in 5 Americans will develop skin cancer in their lifetime

● Latest estimate (2012): >5.4 million cases BCC/SCC treated in >3.3 million

persons in the US (Rogers HW et al. Arch Dermatol 2015)

● In 2019 –

● estimated >96K new cases of invasive melanoma and >95K melanomas in situ

in the US

● nearly 8000 melanoma-associated deaths

● Survival rate for melanoma is >95% if detected early

Skin Cancer Facts

Siegal RL et al. Cancer Statistics, 2019. CA Cancer J Clin 2019

“MELANOMA WRITES ITS MESSAGE IN THE SKIN WITH ITS OWN INK

AND FOR ALL OF US TO SEE”-Neville Davis, Queensland, Australia

…so why is early detection so hard?

Global Health Care Accessibility

Global access to quality health care has improved in the last two decades, but the gap between countries with the most and least effective treatments has grown

Barber R et al. The Lancet 2017

7 How do we democratize health care access?

Aneja S et al. Arch Dermatol 2012

▪ 6.3 billion smartphones

globally by 2021

▪ Data collection at scale

▪ Diagnostics at scale

▪ Diminish health disparities

Can we use AI to expand access to dermatologists?

8

AI has changed our world

• Driverless cars

• Translation capabilities

• Mortgage lending

• Financial markets

Convolutional Neural Networks

https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/

• Capture relationship of pixels to each other using filter as a matrix• Run algorithms over and over until errors are minimized across all images• Allows for profound level of pattern recognition beyond human brain capability

Multilayer neural networks: involve local connections, shared weights, pooling

10 Advances in AI/Machine Learning/Computer Vision

Breakthroughs in artificial

intelligence over past 7

years

Convolutional neural

networks + large

databases + processing

power = deep learning

Krizhevsky, Sutskever, Hinton et al. ImageNet Classification with Deep Convolutional Neural Networks, NIPS 2012

11 Algorithms train themselves to recognize what is important and what is not, without human intervention

Hsu et al, 2013

- ImageNet project: CNNs trained to classify 1.4 million high-resolution images into 1000 different classes with low error rates

- With sufficient examples of huskies, chihuahuas, basal cell carcinomas or melanomas, algorithms learn the relevant patterns of these categories

Constructing the dataset:

Nearly 130,000 images:- clinician-labeled and/or biopsy-proven from 18 different, open-access online repositories

- clinical data from Stanford Dermatology clinics

Biopsy proven images:- University of Edinburgh library- ISDIS ISBI challenge (dermoscopic)

Images comprised more than 2,000 dermatologic diseases

Clinical and Dermoscopic Images Spanning Breadth of Disease

129K images2K diseases

Visual Taxonomy – Disease Partitioning Algorithm

Our Objectives

Evaluate performance of deep learning algorithms

(namely, a single CNN) on classification of cutaneous

malignancies

Compare to dermatologists

2 main (binary) questions:

▪ Lesion benign or malignant?

▪ Biopsy/treat or reassure?

Methods

Recruited 21 board-certified dermatologists▪ Stanford University

▪ University of Pennsylvania

▪ Massachusetts General Hospital/Harvard

▪ University of Iowa

Three 100+ question testsa. Epidermal tumors (BCC/SCC vs SKs)

b. Clinical images of melanocytic lesions (melanoma vs benign nevi, excluded SKs)

c. Dermoscopic images of melanocytic lesions (separate dataset)

Training

Objective: build a system that could accommodate significant variation inherent in photographic images (e.g. lighting, zoom, angle) with no pre-processing or lesion segmentation

Goal: feed any captured skin image directly into the system and output a classification

Inception v3 CNN architecture reprinted from https://research.googleblog.com/2016/03/train-your-own-

image-classifier-with.html

Deep Convolutional Neural Network (CNN)

- Used GoogleNet Inception v3 CNN architecture pretrained on ImageNet dataset- Trained and fine-tuned our CNN using transfer learning- Resulted in probability distribution over clinical classes of skin- Applied a partitioning algorithm to our taxonomy to define training classes

Sample Test ImagesUsed new dermatologist-labeled dataset of >129K clinical images, including >3300 separate dermoscopic images

Algorithm ready for testing within 1 year

CNN outputs a malignancy probability for each image

Results for CNN vs Dermatologists

Esteva A et a. Nature. 2017;542:115-118.

CNN performed at least as well as dermatologists as a whole(Limitation of dermoscopy test as most dermatologists were not experts in

pigmented lesions or dermoscopy)

Limitations of our study

▪ Retrospective

▪ Potential spectrum bias

› Study design→ 2 disease categories

▪ Differences between in-person exam and telederm (“in silico”)

› We “blink, think, and compare” and use dermoscopy to aid diagnosis

▪ Transparency on pathology labels/ dermatopathology accuracy/ wide

variability for melanocytic neoplasms (Elmore JG et al. BMJ 2017)

▪ Opportunities for bias/confounding

› Need to ensure extensive representation by varied skin types

› Extensive additional work is needed

• Prospective clinical validation studies needed

23

Limitations of Deep Learning

• System is opaque; we don’t

know why it calls an image

benign or malignant

• Dots, rulers, marks, etc. may

introduce bias into dataset

• Investigators may not be aware

of them or extent

• Impact on MD cognition/learning?

Strengths

Computer can assess image data imperceptible to human eye▪ Only looking at specific lesions, not whole-body or sequential change detection

Broadly applicable across disciplines▪ Algorithms will improve with transfer learning

▪ Edges, shapes remain important

▪ Show a skin cancer app cat pictures and it will improve at classifying skin cancer

Capable of running on a smartphone device▪ App created to study this, prospective “real-time” validation

in clinical setting

What’s happened with AI in Dermatology since then?

▪ At least 8 publications pertaining to AI (CNNs) for:

› dermoscopy (with or without clinical images), dermatopathology (nod BCC, dermal

nevi, SK), melanocytic lesions, melanoma, acral melanoma dermoscopy,

nonmelanoma skin cancer (BCC/SCC), onychomycosis

▪ CNN consistently performs as well if not better than “expert”

clinicians

▪ Limitations: datasets lack full spectrum of skin phenotypes and

lesions (particularly less common banal lesions/disorders)

▪ No real-world prospective use/validation as yet

Gilmore SF. PLoS One 2018; Han SS. PLoS One 2018; Marchetti MA. J Am Acad Dermatol 2018; Yu C. PLoS One 2018; Haenssle HA. Ann Oncol 2018; Brinker TJ. Eur J Cancer 2019; Fujisawa Y. Brit J Dermatol 2019; Marka A. BMC Med Imaging 2019

Can AI be used to quantify and monitor skin disease severity?

Context matters

Dermatologist’s clinical impression based on factors beyond visual and dermoscopic examination of a lesion in isolation

- Sequential imaging, change detection, incorporation of clinical metadata?

Are we comfortable with AI as black box?

A trained neural net does not necessarily mimic the decision-making approach of

humans

Identifies its own criteria for informative patterns associated with a disease

Is AI a Pandora’s box?

• Probably not, and it won’t replace dermatologists!• Human decisions made based on machine output of probability and other factors!• Real vs imaginary dangers:• “AI is a fundamental existential risk for human civilization.” (Elon Musk, 2017)• “AI software will help us understand biology, understand how to intervene and improve lives very dramatically.”

(Bill Gates, 2018)

• Roles in health care and education • Regulation and liability need to be addressed proactively

Real-world, clinical validation of app in progress

“Smart dermoscope”

“Augmented clinician”

54.8

Can AI help dermatologists?

-35 Y female with atypical mole syndrome, seven cutaneous melanomas(3 melanomas in situ, 3 T1a invasive melanomas and 1 T2a SLNB-positive melanoma) and ten severely dysplastic nevi diagnosed since 2016

-negative for p16/CDKN2A mutation

-followed q 3 mos with total body photography and digital dermoscopy

A tool for surveillance? – NOT YET

New moderately dysplastic nevi with focal severe atypia

Can AI enhance clinical decision-making?

Janda M, Soyer HP. Brit J Dermatol. 2019:180: 247-248.

Mar VJ, Soyer HP. Ann Oncol. 2018;29:1625-1628. (Schematic from Smithers BM, Dunn J and

Soyer HP. Whither melanoma in Australia? Med J Aust 2017;207:330–331.)

Future of melanoma early detection

Targeted surveillance of high risk individuals with high resolution 3D imaging and integration of AI

AI can augment, but not replace decision-making and human interactions in medicine