f eature -b ased a lignment of v olumetric m ulti - modal i mages matthew toews, lilla zöllei,...

FEATURE-BASED ALIGNMENTOF VOLUMETRIC MULTI-MODAL IMAGES

Matthew Toews, Lilla Zöllei, William Wells III June 29, 2013 IPMI

Challenges: Arbitrary subjects, anatomies (brain, body, …), modalities (MR, CT, …), pathology, lack of one-to-one homology, unknown initialization, DICOM errors, …

ROBUST IMAGE ALIGNMENT

Thoracic CTInfant Brain MR

TBI, MR

Brain MR, CT(tumor)

2

Robust clinical usage Initialize registration, segmentation routines

Large-scale data mining, e.g. Google style

APPLICATIONS

Thoracic CTInfant Brain MR

TBI, MR

Brain MR, CT(tumor)

3

FEATURE-BASED ALIGNMENT METHOD

Alignment via 3D scale-invariant feature correspondences

Efficient and robust model-to-image alignment using 3D scale-invariant features.Medical Image Analysis, 2013. Matthew Toews & William M. Wells III.

4

FEATURE-BASED ALIGNMENT METHOD

Strengths Robust: lack of one-to-one homology, disease, resection,

Globally optimal: no ‘capture radius’, initialization

Efficient: Memory, computation

Useful: Alignment, disease classification, prediction

Weaknesses Does not align different modalities Requires pre-aligned training data


5

CONTRIBUTIONS

Inverted local feature correspondence Extend scale-invariant feature representation to

multi-modal alignment

Group-wise feature-based alignment Remove requirement for pre-aligned training

data Multiple modalities

6

OVERVIEW

Scale-invariant feature representation

Inverted feature correspondence

Group-wise feature-based alignment

SCALE-INVARIANT FEATURES

Distinctive image patches Image-to-image

correspondence SIFT method: computer vision

Invariant to scaling, rotation, translation, illumination changes

Fast, efficient, robust Large-scale image search

Distinctive Image Features from Scale-Invariant KeypointsD. G. Lowe, IJCV, 2004.

9

3D Geometry S = {X, σ, Θ} Location X = (x, y, z) Scale σ Orientation

(axis: 3 unit vectors )

Appearance I Intensity descriptor

SCALE-INVARIANT FEATURES IN 3D


S I

)ˆ,ˆ,ˆ( 321

10

Difference-of-Gaussian scale-space extrema.

IDENTIFYING LOCATION X, SCALE σ

d

xdI ),()(xI

x

),()(),( xGxIxI

Feature locations, scales:blob-like image patterns

d

xdIx

),(localmax

,

iiX ,

11

Dominant local image gradient directions)( IH 3D gradient orientation histogram

I

I

Vote location (upon unit sphere)

IVote magnitude

ASSIGNING ORIENTATION Θ


)(argmaxˆˆ

1 IH

))ˆ(ˆ( argmaxˆ11

ˆ2

IH 213

ˆˆˆ

)(XII )( IH

12

Encode local image content For image correspondence / matching

Gradient orientation histogram (GoH) Quantization: 8 location x 8 orientation

bins 113 voxels → 64 bins (small size)


FEATURE DESCRIPTOR I

IGoHNormalized

Image PatchI

)(XI

13

OVERVIEW




MATCHING ACROSS MODALITIES

Joint Intensity relationship Globally multi-modal Locally linear

MP-RAGE

T2

negative localcorrelation

positive localcorrelation

Non-rigid registration of multi-modal images using both mutual information and cross-correlation.Medical Image Analysis, 2008. A. Andronache, M.V. Siebenthal, G. Szekely, P. Cattin.

15


Positive local correlation Conventional correspondence methods,

descriptor matching

MP-RAGE

T2positivecorrelation

16


Negative local correlation Conventional correspondence fails Inverted local gradient, orientations ,

descriptor

MP-RAGE

T2

negativecorrelation

21ˆ,ˆ

17


Inverted correspondence Rotate orientation, descriptor elements by -π

about Correspondence successful

MP-RAGE

T2

negativecorrelation

3̂

18


MP-RAGE, T2, intra-subject No conventional correspondences in GM / WM 22 inverted correspondences within GM / WM

MP-RAGE

T2

19


Infant T1 MR: newborn ↔ 2 years old GM / WM contrast inversion due to mylenation No conventional correspondences in GM / WM 4 inverted correspondences within GM / WM

Unbiased average age-appropriate atlases for pediatric studiesNeuroImage 2011. V.S. Fonov, A.C. Evans, D.L. Collins et al.

20

OVERVIEW




GROUP-WISE ALIGNMENT

Automatically align a set of subject images Arbitrary initialization, modalities

22

GROUP-WISE ALIGNMENT: MODEL

ji

iijiji TSIpTp,

)|,(})({

}{ iT

},{ ijij SI

Transform set: image i to atlas (similarity transform)

Feature descriptor, geometry set: image i, feature j

Bayes rule

Conditional feature independence

Marginalization over F

}{ ,lkfF Latent model feature set, feature k,conventional & inverted modes l={0,1}

}){|},({})({}),{|}({ iijijiijiji TSIpTpSITp

ji

lklk

ilkijiji fpTfSIpTp,

,,

, )(),|,(})({

Input features (Iij,Sij) are - Conditionally independent - Identically distributed according to a Gaussian mixture model

23

LIKELIHOOD: APPEARANCE DESCRIPTOR

lk

lkilkijijiijij fpTfSIpTSIp,

,, )(),|,()|,(

),|()|(),|,( ,,, ilkijlkijilkijij TfSpfIpTfSIp Conditional independence

Descriptor:Isotropic Gaussian density over descriptor elements.

Note: descriptor conditionally independent of alignment Ti,- Fast Matching, no search over Ti -

28

),|(),|(),|()|(),|,( ,,,,, ilkijilkijilkijlkijilkijij TfpTfpTfXpfIpTfSIp

LIKELIHOOD: GEOMETRY

Location: Isotropic Gaussian

lk


,, )(),|,()|,(

Scale:Gaussian in log σij

Orientation:Isotropic GaussianApproximate Von Mises

29

PRIOR

Latent feature probability: Discrete

lk


,, )(),|,()|,(

0, lkf

1, lkf

Conventional appearance mode

Inverted appearance mode

30

GROUP-WISE ALIGNMENT: ALGORITHM

Inputs:Volumetric images

Outputs:Alignment solutions: {Ti}

Feature-based model: {fk,l}, p(Iij, Sij| fk,l,Ti), p(fk,l)

31

GROUP-WISE ALIGNMENT: ALGORITHM

1) Feature extraction2) Initialization

Approximate {Ti}

3) Model LearningFixed {Ti}, vary {fk,l}, p(Iij, Sij| fk,l,Ti), p(fk,l)

Mixture model; density, probability parameter estimation.

4) Alignment / Model FittingFixed {fk,l}, p(Iij, Sij| fk,l,Ti), p(fk,l), vary {Ti}

Subject-to-model alignment.

Iterate between 3) & 4) until convergence, i.e. {Ti} no longer changes.

32

1) FEATURE EXTRACTION

Features extracted once from individual images

Note: algorithms use feature data only ~100X data reduction compared to original

image volumes. ~25 seconds for 2563-voxel image, standard PC. ~2K features per brain image

33

2) INITIALIZATION

Approximate image alignment Nearest neighbor descriptor matching Hough transform, similarity

Note: some initial misalignment OK A small subset of image should be aligned Misaligned features sets have negligible impact

34

3) MODEL ESTIMATION

Mixture modeling Estimate set {fk,l}, parameters of p(Iij, Sij| fk,l,Ti), p(fk,l) Robust feature clustering across subjects

Similar to mean-shift algorithm

Note Model conventional appearance Same structure, two distinct latent features, e.g.:

0,1 lkf0,2 lkf


35

4) MODEL FITTING

Maximum a-posteriori estimation Maximize Ti individually

Conditional independencei

ijijiijiji SITpSITp }),{|(}),{|}({

}),{|( argmax ijijiT

iMAP SITpTi

36

4) MODEL FITTING

Maximum a-posteriori estimation Maximize Ti individually

Two approaches Conventional

Multi-Modal (conventional & inverted modes) Assume

Conditional independencei

ijijiijiji SITpSITp }),{|(}),{|}({

}),{|( argmax ijijiT

iMAP SITpTi

)()( 0,1, lklk fpfp

0)( 1, lkfp

37

EXPERIMENTS

Group-wise Alignment: RIRE data set Modalities: T1, T2, PD, MP-RAGE, CT Brain: 9 subjects, 39 images All subjects exhibit brain tumors

Difficult problem: subject abnormality, no prior information used regarding modalities, initialization.

Compare conventional vs. multi-modal fitting

Comparison and evaluation of retrospective intermodality brain image registration techniquesJournal of Computer Assisted Tomography, 1997. J.B. West, J.M. Fitzpatrick et al.

38

RESULTS: ALIGNMENT

39

RESULTS

Conventional alignment: 3 failure cases (all CT)

Multi-Modal alignment: success

40

Failure case

Success

Failure

RESULTS

Model feature examples

41

DISCUSSION: INVERTED CORRESPONDENCE

Useful for fitting/matching between modalities Less useful once model has been learned May be more prone to false correspondences

Analogous to mutual information Useful when prior information is weak

42

A marginalized MAP approach and EM optimization for pair-wise registrationIPMI 2007. L. Zollei, M. Jenkinson, S. Timoner, W.M. Wells III

DISCUSSION: GROUP-WISE ALIGNMENT

Alignment of difficult multi-modal data Unknown initialization Also effective for infant MR, torso CT, lung CT

Fast 22 minutes (vs. 10-20 hours for group-wise

registration) Deformable alignment?

Global similarity Ti + local linear deformations about correspondences.

43

ACKNOWLEDGEMENTS

NIH grants: P41-EB-015902

P41-RR-013218R00 HD061485-03P41-EB-015898P41-RR-019703

44

f eature -b ased a lignment of v olumetric m ulti - modal i mages matthew toews, lilla zöllei,...

Documents