dermatologist-level classification of skin cancer with ...€¦ · dermatologist 1 65.6%...

Dermatologist-level classification of skin cancer with deep neural networks

Enhancing the Expert

Andre EstevaPI: Sebastian ThrunStanford University

1

How can technology assist a human?

2

3

4

5

How can AI assist a dermatologist?

6

Skin Cancer

7

● 5.4M cases of non-melanoma skin cancer each year in US

Skin Cancer

8

● 5.4M cases of non-melanoma skin cancer each year in US● 20% of Americans will get skin cancer

Skin Cancer

9

● 5.4M cases of non-melanoma skin cancer each year in US● 20% of Americans will get skin cancer● Actinic Keratosis (pre-cancer) affects 58 million Americans

Skin Cancer

10

● 5.4M cases of non-melanoma skin cancer each year in US● 20% of Americans will get skin cancer● Actinic Keratosis (pre-cancer) affects 58 million Americans● 76,000 melanomas each year - 10,000 deaths

Skin Cancer

11

● 5.4M cases of non-melanoma skin cancer each year in US● 20% of Americans will get skin cancer● Actinic Keratosis (pre-cancer) affects 58 million Americans● 76,000 melanomas each year - 10,000 deaths● $8.1B in US annual costs for skin cancer

Skin Cancer

12

● 5.4M cases of non-melanoma skin cancer each year in US● 20% of Americans will get skin cancer● Actinic Keratosis (pre-cancer) affects 58 million Americans● 76,000 melanomas each year - 10,000 deaths● $8.1B in US annual costs for skin cancer

Skin Cancer

13Years

100

75

50

25

0

Sur

viva

l Pro

babi

lity

1 5 10 15

Stage I

Stage II

Stage III

Stage IV

● 5.4M cases of non-melanoma skin cancer each year in US● 20% of Americans will get skin cancer● Actinic Keratosis (pre-cancer) affects 58 million Americans● 76,000 melanomas each year - 10,000 deaths● $8.1B in US annual costs for skin cancer

Skin Cancer

Years

100

75

50

25

0

Sur

viva

l Pro

babi

lity

1 5 10 15

Stage I

Stage II

Stage III

Stage IV

14

0 1 2 3 4

15

16

Early detection is critical

17

6.3 billion smartphones

18

Skin Cancer Classification

19

~130,000 images of skin

2000 diseases

Skin Cancer Classification

20

~130,000 images of skin

2000 diseases

Skin Cancer Classification

21

Epidermal Lesions Melanocytic Lesions Melanocytic Lesions(Dermoscopy)

Benign

Malignant

Skin Cancer Classification

22

Deep Convolutional Neural Network (Inception-v3)

Skin Cancer Classification

23

Acral-lent. melanomaAmelanotic melanomaLentigo melanoma...

Blue nevusHalo nevusMongolian spot…

Training Classes (757)

Deep Convolutional Neural Network (Inception-v3)

Skin Lesion Image

Skin Cancer Classification

Partitioning Algorithm 24

Acral-lent. melanomaAmelanotic melanomaLentigo melanoma...

Blue nevusHalo nevusMongolian spot…

Training Classes (757)

Deep Convolutional Neural Network (Inception-v3)

Inference Classes (varies by task)

92% Malignant

8% Benign

Skin Lesion Image

Skin Cancer Classification

Partitioning Algorithm 25

Skin Cancer Classification

P = 0.1

P = 0.05

P = 0.05

P = 0.1

P = 0.02 P = 0.03

P = 0.05Training Classes

Inference Classes

P = 0.4

26

Dermatologist-level performance

27

Validation set

Skin Cancer Classification

28

Validation set Classifier Three-way accuracy

Dermatologist 1 65.6%

Dermatologist 2 66.0%

CNN 69.5%

CNN - PA 72.0%

Disease classes: three-way classification

0. Benign single lesions1. Malignant single lesions2. Non-neoplastic lesions

Skin Cancer Classification

29

Validation set Classifier Three-way accuracy

Dermatologist 1 65.6%

Dermatologist 2 66.0%

CNN 69.5%

CNN - PA 72.0%

Classifier Nine-way accuracy

Dermatologist 1 53.3%

Dermatologist 2 55.0%

CNN 48.9%

CNN - PA 55.3%

Disease classes: nine-way classification

0. Cutaneous lymphoma and lymphoid infiltrates 1. Benign dermal tumors, cysts, sinuses2. Malignant dermal tumor3. Benign epidermal tumors, hamartomas, milia, and

growths4. Malignant and premalignant epidermal tumors5. Genodermatoses and supernumerary growths6. Inflammatory conditions7. Benign melanocytic lesions8. Malignant Melanoma

Disease classes: three-way classification

0. Benign single lesions1. Malignant single lesions2. Non-neoplastic lesions

Skin Cancer Classification

30

Skin Cancer Classification

Test set

31

Skin Cancer Classification

Test set: Dermatologist Comparison (376 images)

32

Sensitivity

Carcinoma: 135 images

Spe

cific

ity

Sensitivity

Skin Cancer Classification

Algorithm: AUC = 0.96Dermatologists (25)Average Dermatologist

Test set: Dermatologist Comparison (376 images)

33

Sensitivity Sensitivity Sensitivity

Carcinoma: 135 images Melanoma: 130 images Melanoma: 111 dermoscopy images

Spe

cific

ity

Spe

cific

ity

Spe

cific

ity

Sensitivity Sensitivity Sensitivity

Skin Cancer Classification

Algorithm: AUC = 0.96Dermatologists (25)Average Dermatologist

Algorithm: AUC = 0.94Dermatologists (22)Average Dermatologist

Algorithm: AUC = 0.91Dermatologists (21)Average Dermatologist

Test set: Dermatologist Comparison (376 images)

34

Sensitivity Sensitivity Sensitivity

Carcinoma: 707 images Melanoma: 225 images Melanoma: 1010 dermoscopy images

Spe

cific

ity

Spe

cific

ity

Spe

cific

ity

Sensitivity Sensitivity Sensitivity

Skin Cancer Classification

Test set: Total (1942 images)

Algorithm: AUC = 0.96 Algorithm: AUC = 0.96 Algorithm: AUC = 0.94

35

How does the algorithm work?

36

37

T-SNE Visualization

Van der Maaten & Hinton, 2008

Epidermal BenignEpidermal MalignantMelanocytic BenignMelanocytic Malignant

38

Basal Cell Carcinomas

Squamous Cell Carcinomas

Melanomas

Nevi

Seborrheic Keratoses

T-SNE Visualization

Epidermal BenignEpidermal MalignantMelanocytic BenignMelanocytic Malignant

Basal Cell Carcinomas

Squamous Cell Carcinomas

Melanomas

Nevi

Seborrheic Keratoses

39

T-SNE Visualization

40

What is the network fixating on?

Malignant Melanocytic Lesion

41

What is the network fixating on?

Simonyan, Zisserman, 2014

Malignant Melanocytic Lesion

Malignant Epidermal Lesion

Malignant Dermal Lesion

Benign Melanocytic Lesion

Benign Epidermal Lesion

Benign Dermal Lesion

Inflammatory Condition

Genodermatosis

Cutaneous Lymphoma

42

What is the network fixating on?

43

What does the network misclassify?

CNN Dermatologist 1 Dermatologist 2

True

Lab

el

Predicted Label Predicted Label Predicted Label

012345678

0 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8 0

1

44

What does the network misclassify?

Dermatologist-level Classification of Skin Cancer with Deep Neural Networks

Andre Esteva*, Brett Kuprel*, Rob Novoa, Justin Ko, Susan Swetter, Helen Blau, Sebastian ThrunNature, 2017(Equal contribution authors*)

45

How can AI assist a dermatologist?

46

Community

47

Questions?

esteva@cs.stanford.edu@andreesteva

cs.stanford.edu/people/esteva

48