dermatologist-level classification of skin cancer with ...€¦ · dermatologist 1 65.6%...

Dermatologist-level classification of skin cancer with deep neural networks

Enhancing the Expert

Andre EstevaPI: Sebastian ThrunStanford University

1

How can technology assist a human?

2

How can AI assist a dermatologist?

6

Skin Cancer

7

● 5.4M cases of non-melanoma skin cancer each year in US

Skin Cancer

8

● 5.4M cases of non-melanoma skin cancer each year in US● 20% of Americans will get skin cancer

Skin Cancer

9

● 5.4M cases of non-melanoma skin cancer each year in US● 20% of Americans will get skin cancer● Actinic Keratosis (pre-cancer) affects 58 million Americans

Skin Cancer

10

● 5.4M cases of non-melanoma skin cancer each year in US● 20% of Americans will get skin cancer● Actinic Keratosis (pre-cancer) affects 58 million Americans● 76,000 melanomas each year - 10,000 deaths

Skin Cancer

11

● 5.4M cases of non-melanoma skin cancer each year in US● 20% of Americans will get skin cancer● Actinic Keratosis (pre-cancer) affects 58 million Americans● 76,000 melanomas each year - 10,000 deaths● $8.1B in US annual costs for skin cancer

Skin Cancer

12


Skin Cancer

13Years

100

75

50

25

0

Sur

viva

l Pro

babi

lity

1 5 10 15

Stage I

Stage II

Stage III

Stage IV


Skin Cancer

Years

100

75

50

25

0

Sur

viva

l Pro

babi

lity

1 5 10 15

Stage I

Stage II

Stage III

Stage IV

14

0 1 2 3 4

Early detection is critical

17

6.3 billion smartphones

18

Skin Cancer Classification

19

~130,000 images of skin

2000 diseases


20

~130,000 images of skin

2000 diseases


21

Epidermal Lesions Melanocytic Lesions Melanocytic Lesions(Dermoscopy)

Benign

Malignant


22

Deep Convolutional Neural Network (Inception-v3)


23

Acral-lent. melanomaAmelanotic melanomaLentigo melanoma...

Blue nevusHalo nevusMongolian spot…

Training Classes (757)


Skin Lesion Image


Partitioning Algorithm 24

Acral-lent. melanomaAmelanotic melanomaLentigo melanoma...

Blue nevusHalo nevusMongolian spot…

Training Classes (757)


Inference Classes (varies by task)

92% Malignant

8% Benign

Skin Lesion Image


Partitioning Algorithm 25


P = 0.1

P = 0.05

P = 0.05

P = 0.1

P = 0.02 P = 0.03

P = 0.05Training Classes

Inference Classes

P = 0.4

26

Dermatologist-level performance

27

Validation set


28

Validation set Classifier Three-way accuracy

Dermatologist 1 65.6%


CNN 69.5%

CNN - PA 72.0%

Disease classes: three-way classification

0. Benign single lesions1. Malignant single lesions2. Non-neoplastic lesions


29

Validation set Classifier Three-way accuracy



CNN 69.5%

CNN - PA 72.0%

Classifier Nine-way accuracy



CNN 48.9%

CNN - PA 55.3%

Disease classes: nine-way classification

0. Cutaneous lymphoma and lymphoid infiltrates 1. Benign dermal tumors, cysts, sinuses2. Malignant dermal tumor3. Benign epidermal tumors, hamartomas, milia, and

growths4. Malignant and premalignant epidermal tumors5. Genodermatoses and supernumerary growths6. Inflammatory conditions7. Benign melanocytic lesions8. Malignant Melanoma

Disease classes: three-way classification

0. Benign single lesions1. Malignant single lesions2. Non-neoplastic lesions


30


Test set

31


Test set: Dermatologist Comparison (376 images)

32

Sensitivity

Carcinoma: 135 images

Spe

cific

ity

Sensitivity


Algorithm: AUC = 0.96Dermatologists (25)Average Dermatologist


33

Sensitivity Sensitivity Sensitivity

Carcinoma: 135 images Melanoma: 130 images Melanoma: 111 dermoscopy images

Spe

cific

ity

Spe

cific

ity

Spe

cific

ity







34


Carcinoma: 707 images Melanoma: 225 images Melanoma: 1010 dermoscopy images

Spe

cific

ity

Spe

cific

ity

Spe

cific

ity



Test set: Total (1942 images)

Algorithm: AUC = 0.96 Algorithm: AUC = 0.96 Algorithm: AUC = 0.94

35

How does the algorithm work?

36

37

T-SNE Visualization

Van der Maaten & Hinton, 2008

Epidermal BenignEpidermal MalignantMelanocytic BenignMelanocytic Malignant

38

Basal Cell Carcinomas

Squamous Cell Carcinomas

Melanomas

Nevi

Seborrheic Keratoses

T-SNE Visualization

Epidermal BenignEpidermal MalignantMelanocytic BenignMelanocytic Malignant

Basal Cell Carcinomas

Squamous Cell Carcinomas

Melanomas

Nevi

Seborrheic Keratoses

39

T-SNE Visualization

40

What is the network fixating on?

Malignant Melanocytic Lesion

41


Simonyan, Zisserman, 2014

Malignant Melanocytic Lesion

Malignant Epidermal Lesion

Malignant Dermal Lesion

Benign Melanocytic Lesion

Benign Epidermal Lesion

Benign Dermal Lesion

Inflammatory Condition

Genodermatosis

Cutaneous Lymphoma

42


43

What does the network misclassify?

CNN Dermatologist 1 Dermatologist 2

True

Lab

el

Predicted Label Predicted Label Predicted Label

012345678

0 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8 0

1

44

What does the network misclassify?

Dermatologist-level Classification of Skin Cancer with Deep Neural Networks

Andre Esteva*, Brett Kuprel*, Rob Novoa, Justin Ko, Susan Swetter, Helen Blau, Sebastian ThrunNature, 2017(Equal contribution authors*)

45

How can AI assist a dermatologist?

46

Community

47

Questions?

[email protected]@andreesteva

cs.stanford.edu/people/esteva

48

dermatologist-level classification of skin cancer with ...€¦ · dermatologist 1 65.6%...

Documents