Feature-preserving Artifact Removal from Dermoscopy Images

Howard Zhou1, Mei Chen2,

Richard Gass2, James M. Rehg1,

Laura Ferris3, Jonhan Ho3, Laura Drogowski3

1School of Interactive Computing, Georgia Tech2Intel Research Pittsburgh

3Department of Dermatology, University of Pittsburgh

Skin cancer and melanoma Skin cancer : most common of all cancers

Skin cancer and melanoma Skin cancer : most common of all cancers

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Basal Cell Carcinoma

Hemangioma

Compound nevus Seborrheic keratosis

Skin cancer and melanoma Skin cancer : most common of all cancers Melanoma : leading cause of mortality

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Basal Cell Carcinoma

Hemangioma

Compound nevus Seborrheic keratosis

Melanoma

Melanoma

Skin cancer and melanoma Skin cancer : most common of all cancers Melanoma : leading cause of mortality Early detection significantly reduces mortality

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Basal Cell Carcinoma

Hemangioma

Compound nevus Seborrheic keratosis

Melanoma

Melanoma

[ Image courtesy of “An Atlas of Surface Microscopy of Pigmented Skin Lesions: Dermoscopy” ]

Clinical ViewDermoscopy view

Dermoscopy Skin surface microscopy Improve diagnostic accuracy by 30% for trained,

experienced physicians Requires 5 or more years of experience Computer-aided diagnosis (CAD) to assist less

experienced physicians

Clinical view Dermoscopy view

Artifacts in dermoscopy images Hair, air-bubbles,… Interfering with computer-aided diagnosis

[ Image courtesy of Grana et al. 2006]

Hair, air-bubbles,… Interfering with computer-aided diagnosis

[ Image courtesy of Grana et al. 2006]

Artifacts in dermoscopy images

Artifacts in dermoscopy images Hair, air-bubbles,… Interfering with computer-aided diagnosis

[ Image courtesy of Grana et al. 2006]

Hair lesion boundary

Artifacts in dermoscopy images Hair, air-bubbles,… Interfering with computer-aided diagnosis

[ Image courtesy of Grana et al. 2006]

Hair lesion boundary

Artifacts in dermoscopy images Hair, air-bubbles,… Interfering with computer-aided diagnosis

[ Image courtesy of Grana et al. 2006]

Hair lesion boundary Hair pigmented network

Previous work Hair detection and tracing

Fleming et al. 1998 Thresholding and averaging

“DullRazor”, Tim K. Lee et al. 1997 Schmid et al. 2003

Thresholding and inpainting Paul Wighton et al. 2008 (right here in the

conference)

Detection: thresholding Removal: morphological

operations

Schmid et al.

Schmid et al. Thresholding false

detection Accidental removal of

diagnostic features

Schmid et al. 2003

ThresholdingThresholding

Schmid et al. Morphological operation

(neighbors’ average) blurring

Morphological operationMorphological operation

Schmid et al. 2003

Feature-preserving Feature-preserving artifact removal artifact removal (FAR)(FAR)

Detection: Explicit curve modeling

Removal: Exemplar-based inpainting

Our method (FAR)Schmid et al. 2003

Our method (FAR)

FAR Curve modeling

more accurate hair detection

ThresholdingThresholding Curve modelingCurve modeling

Schmid et al. 2003

Our method (FAR)

FAR Exemplar-based

inpainting preserving features

ThresholdingThresholding Curve modelingCurve modelingMorphological operationMorphological operation Exemplar-based inpaintingExemplar-based inpainting

Schmid et al. 2003

Our method (FAR)

FAR Exemplar-based

inpainting preserving features

ThresholdingThresholding Curve modelingCurve modelingMorphological operationMorphological operation Exemplar-based inpaintingExemplar-based inpainting

Schmid et al. 2003

Our method (FAR)

FAR Exemplar-based

inpainting preserving features

Schmid et al. 2003

Our method (FAR)

FAR Exemplar-based

inpainting preserving features

Schmid et al. 2003

Our method (FAR)

FAR Exemplar-based

inpainting preserving features

Schmid et al. 2003

System overview

Threholding

Curve fitting & intersection analysis

Exemplar patches

Exemplar-based inpainting

Dermoscopy image

Hair removed

Luminance difference dark thin structure

Line points

Line segments

Parameterized curves

Mask

Line points linking

Line points Detection

Input dermoscopy image

Enhancing dark-thin structure Luminosity channel in CIE L*u*v* Difference b/a morphological closing

[ Schmid-Saugeona et al. 2003, “Towards a computer-aided diagnosis system for pigmented skin lesions” ]

Detecting line points

[ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]

Curve B(t)

Detecting line points

[ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]

n(t)

Curve B(t)Cross section

n(t)

f(x)

Detecting line points

[ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]

n(t)

Cross section

n(t)

f(x)

Curve B(t)

Detecting line points

[ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]

n(t)

Cross section

n(t)

f(x)

f’ = 0|f’’| large

Curve B(t)

Detecting line points

[ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]

n(t)

Cross section

n(t)

f(x)

f’ = 0|f’’| large

Curve B(t)

n(t) : direction ┴ curve B(t)

eigenvector corresponding to the maximum absolute eigenvalue of the local Hessian

Detecting line points

[ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]

n(t)

Detecting line points

[ Steger 1998, ”An Unbiased Detector of Curvilinear Structures” ]

Linking line points

Link the neighboring points to get line segments (sets of ordered line points)

Fitting polynomial curves

A set of ordered points Pi s

P

Fitting polynomial curves

A set of ordered points Pi s Parametric curveP

Fitting polynomial curves

A set of ordered points Pi s Parametric curve

B(t)

P

Fitting polynomial curves

B(t)

P A set of ordered points Pi s Parametric curve

Minimize sum of squared distance

Fitting polynomial curves

A set of ordered points Pi s Parametric curve

Minimize sum of squared distance

Linear system (can be solved by Gaussian elimination)

B(t)

P

Handling hair intersection

Configurations:

Hair intersection Line segments Intersection analysisLink Line segment

……

Before curve fitting and linking

Line segments

After curve fitting and linking

Parameterized curves

After curve fitting and linking

Parameterized curves

After curve fitting and linking

Hair mask

After curve fitting and linking

Hair mask

Exemplar-based inpainting

[ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

[ Image courtesy of Criminisi et al. 2003 ]

Fill in with patches from the image itself Patch ordering structure propagation.

Exemplar-based inpainting Fill in with patches from the image itself Patch ordering structure propagation.

[ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

Exemplar-based inpainting Fill in with patches from the image itself Patch ordering structure propagation.

[ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

Exemplar-based inpainting Fill in with patches from the image itself Patch ordering structure propagation.

[ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

Exemplar-based inpainting Fill in with patches from the image itself Patch ordering structure propagation.

[ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

Exemplar-based inpainting Fill in with patches from the image itself Patch ordering structure propagation.

[ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

Exemplar-based inpainting Fill in with patches from the image itself Patch ordering structure propagation.

[ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

Exemplar-based inpainting Fill in with patches from the image itself Patch ordering structure propagation.

[ Criminisi et al. 2003, “Object removal by exemplar-based inpainting” ]

Before FAR

After FAR

More results Explicit curve

modeling Exemplar-based

inpainting

Our method (FAR)Schmid et al. 2003

More results Explicit curve

modeling Exemplar-based

inpainting

Our method (FAR)Schmid et al. 2003

Our method (FAR)

FAR Exemplar-based

inpainting preserving features

Schmid et al. 2003

When is FAR not suitable ?

Oops, too much hair!

When is FAR not suitable ?

Too much hair Makes explicit

modeling difficult

Schemid et al. 2003 (DullRazor) Our method (FAR)

Conclusion

Automatic system that detects and removes curvilinear artifacts

Feature-preserving artifact removal: Explicit curve modeling Exemplar-based

inpainting

Future work

Speed up exemplar-based inpainting

Future work

Speed up exemplar-based inpainting Handle hair with arbitrary intensity

Future work

Speed up exemplar-based inpainting Handle hair with arbitrary intensity Extend to removing air bubbles

Questions ?

Additional results

Our method (FAR)Original Dermoscopy image