4/6/18

1

Learning Normalized Inputs for Iterative Estimation

in Medical Image Segmentation

Michal Drozdzal, Gabriel Chartrand, Eugene Vorontsov, Mahsa Shakeri, Lisa Di Jorio, An Tang, Adriana Romero,

Yoshua Bengio, Chris Pal, Samuel Kadoury

Medical imaging modalities - basicsEndoscopyElectron

MicroscopyComputed

TomographyMagnetic Resonance

Imaging

2D/3D

Temporaldimension

Signal scale

2D/3D 3D 3D 2D

No No No Yes

Grayscale RGBHounsfieldscale -

We will t

reat all

data a

s 2D d

ata.

4/6/18

2

Image segmentation

Electron microscopy

CT

Endoscopy

Medical imaging segmentation pipeline

Pre-processing Model Post-processing

4/6/18

3

Medical imaging segmentation pipeline

Pre-processing Model Post-processing

Modality & model specific3D for 2D model

FP reduction

Morphological operations2D/3D CRF

What is used?

Medical imaging segmentation pipeline

Pre-processing Model Post-processing

Fully Convolutional NetworkFCN8

UNET

Jon Long et. al. CVPR 2015Olaf Ronneberger et. al. MICCAI 2015

Modality & model specific3D for 2D model

FP reduction

Morphological operations2D/3D CRF

What is used?

4/6/18

4

Fully Convolutional Networks

Down sampling Up sampling Skip- Pooling- Strided convolutions

- Repeat + convolution- Transposed convolutions

- Concatenate- Sum

This can be ResNet, DenseNet, …This can be ResNet, DenseNet, …

Medical imaging segmentation pipeline

Pre-processing Model Post-processing

Modality specific(handcrafted)

Fully Convolutional NetworkFCN8

UNET

Jon Long et. al. CVPR 2015Olaf Ronneberger et. al. MICCAI 2015

Range normalizationValue clipping

StandardizationN4 (MRI)

Modality & model specific3D for 2D model

FP reduction

Morphological operations2D/3D CRF

What is used?

4/6/18

5

Examples of segmentation pipelines

Range normalizationValue clipping

StandardizationN4 (MRI)

Histogram equalizationGaussian smoothing

2D Unet2D FCN8

2D FC-ResNet3D Unet3D FCN8

3D FC-ResNet

2D CRF3D CRF

Morphological operations

Tools of medical imaging segmentation practitioner:Pre-processing Model Post-processing

Lung segmentation in CT:Lung segmentation in MRI:Liver segmentation in CT:

Standardization + 2DUnet N4 + 2DUNetValue clipping + 3DUnet + morphological operations

Examples of (hypothetical) segmentation pipelines:

Let’s design a model that can be trained with any imaging modality and does not require any pre-processing.

4/6/18

6

Let’s use ResNets

F(x)

x

F(x) + x+

F(x)

x

F(x) + convolution(x)+

convolution

*N *N

x has high impact on feature maps due to skip connections

ResNet uses initial convolution that can adapt input

Recent findings suggest that F() is a transformation close

to identity [1]

We found that FC-ResNetsare more susceptible to data

pre-processing than FCNs

[1] Veit at al. Residual Networks are Exponential Ensembles of Relatively Shallow Networks

Observations about ResNets:

Fully Convolutional

Network

Fully Convolutional

Residual Network

EM

CT

MRI

CT

MRI

EM

What do we propose?

4/6/18

7

Model

Fully Convolutional

Network

Fully Convolutional

Residual Network

Input[1xaxb]

Segmentation map[1xaxb]

Feature map[1xaxb]

+/- 1M parameters +/- 11M parameters

DataElectron Microscopy Computed Tomography Magnetic Resonance

Cell segmentation Lesion segmentation Prostate segmentationN=

30 training images30 testing images

N=105 training volumes30 testing volumes

N=50 training volumes30 testing volumes

4/6/18

8

Experimental setup

Data preparation:> No normalization!> Data augmentation

- Flips- Rotation- Shearing- Elastic transformations- Cropping

Optimization:> RMSprop> Weight decay

EM data results (as of mid 2017)

96.8 96

.9 97 97.1 97

.2

97.2 97

.3

97.7 97

.8 98 98.1

VRAND

Pyramid-LSTM FC-ResNet optree-idsa SCImotif IDSIA Unet CUMedVisionFusionNet IAL Ours

Comparison to published methods Qualitative results(test set)

4/6/18

9

CT data results (as of mid 2017)53

.5 57 61.7 71

.1

D ICE

FCN8 Unet FC-ResNet Ours

Comparison to standard FCNs

Image True segmentation Result

Qualitative results(test set)

MRI data results (as of mid 2017)

79.92 83

.02

74.17

82.39

86.65

2D FCN 3D FCN

Situs OursSRIBHME CAMP-TUM2CUMED

Comparison to published methodsQualitative results

(test set)

4/6/18

10

Fully Convolutional

Network

Fully Convolutional

Residual Network

EM

CT

MRI

CT

MRI

EM

EM

CT

MRI

EM

CT

MRI

Inpu

t int

ensit

y hist

ogra

m:

Norm

alize

d int

ensit

y hist

ogra

m:

Pre-processor effect

Can we quantify the pre-processing effect?

Pre-processor quantification

d( , ) > d( , ) d( , ) = d( , )

Mean Jensen–Shannon distance on validation set

2.99

3.57

5.71

2.98 3.3

5 3.89

2.48 2.

87

3.38

EM CT MRI

input data standardization pre-processor

d(0,0) d(1,0) d(2,0)

d(2,0)

d(1,0) d(1,1)

d(2,1) d(2,2)

d(2,1)

4/6/18

11

Wrap up

A low capacity FCN can serve as a learnable pre-processor.

Combining learnable pre-processor with FC-ResNet yields very good results on a variety of image modalities.

Single pipeline for all type of medical data!

No need to handcraft data pre-processing.

Want to work at the confluence of academia and industry?

MILA has open positions for:• Professors

• Software engineers• Director of software

• R&D & technology transfer• Linux sysadmins

is hiring!

https://tinyurl.com/mila-jobs

4/6/18

12

Learning Normalized Inputs for Iterative Estimation

in Medical Image Segmentation

Michal Drozdzal, Gabriel Chartrand, Eugene Vorontsov, Mahsa Shakeri, Lisa Di Jorio, An Tang, Adriana Romero,

Yoshua Bengio, Chris Pal, Samuel Kadoury