Visual classification of complicated plaques based onmultidimensional image fusion

A. Hennemuth1, A. Harloff2, T. Spehl2, N. Pavlov2, O. Friman1, D. Paul2, D. v. Elverfeldt2,C. Kuehnel1, S. Wirtz1, H. Goebel3, J. Mannheim4, H. K. Hahn1, B. Pichler4, J. Hennig2, M. Markl2

anja.hennemuth@fraunhofer.mevis.de

1Fraunhofer MEVIS, Bremen, 2University Hospital Freiburg,3University Hospital Cologne, 4University Hospital Tuebingen

Abstract: Multi-contrast MRI has shown to be suitable for plaque analysis, whichis important for the risk assessment of internal carotid artery stenoses. The purposeof this work was the development of software prototypes, which support the develop-ment of methods for the classification and exploration of plaque-morphology in 3Dand allow for a comparison with histological images and CT images. Complex ex-vivo plaque specimen were evaluated at a high field 9.4T MRI system with differentcontrasts (T1-fatsat-GRE, T2*-GRE, T2-RARE, PD-RARE) and high resolution (ca.100µm3). Plaque-images were processed to enhance specific plaque-components andcombined into RGB-images. The resulting images were interactively explored in cou-pled views of volume-rendering and user-defined MPRs. µCT images were also pre-processed for visualization. The user could then explore corresponding image regionsin MR images, histological images and CT images. Initial results showed a benefit forthe assessment of plaque-constitution and 3D-morphology.

1 Introduction

High-grade internal carotid artery (ICA) stenoses are a leading source of ischemic strokeand the assessment of the risk factors is a crucial task. The plaque constitution is animportant risk factor [SHT+07]. Especially complicated type VI plaques according tothe American Heart Association (AHA) classification contain thrombi, acute or old in-traplaque hemorrhage and/or ulceration of the fibrous cap [SCD+95]. While CT gives areliable detection and quantification of calcifications [dWOM+06], the analysis of histo-logical slices is the goldstandard for the analysis of other plaque constituents.Previous studies have shown that MRI exhibits excellent softtissue contrast and can beused to distinguish major components of carotid atherosclerotic plaque [YKF+02, CFU+05,FMS+08]. In addition, T1, T2 and proton-density(PD) weighted sequences, T2* and dif-fusion weighted sequences have been evaluated for plaque characterization [QRV+07].For the analysis of multicontrast MRI of plaques, there exist several approaches andmost of them focus on the segmentation of plaque images acquired with T1, T2 andPD weighted images [XHY02, AvdGW+04, ISM+04, CBH+06, ASH+07, KBV+09,LXF+06, SGL+08]. A major problem in the development of intensity-based classifica-tion algorithms for visual or automatic image analysis is the dependency of the intensity

distribution upon the imaging parameters. Thus, some approaches train classification al-gorithms using reference segmentations from histological slices and CT images to adaptthem to a given parameter set [LXF+06, CBH+06, ASH+07]. Other approaches based onfuzzy clustering start with a representative sample [ISM+04] or a priori-knowledge of thequantitative T2 map [SGL+08].For the training of classificators and the validation of results, the MR images have to becompared with the goldstandards CT (for calcifications) and/or histology. Histologicalimages normally consist of a small set of slices and the orientation may differ from theMR image slices. Furthermore, the preparation process leads to deformations and rup-tures in the loosely connected tissue. An accurate matching of the images is thereforedifficult. In previous studies, experts visually inspected the data and transferred the seg-mentation of the histology images to the region of the MR image they assumed as matching[ISM+04, ASH+07]. Clarke et al. apply Delauney triangle nonlinear registration to match

Figure 1: Histological images with deformations from the preparation procedure

the histological images with the MR images for an automatic per voxel comparison of theclassification [CBH+06].For the alignment of MRI and CT images linear registration is applied [DSC+06, CBH+06].Dey et al. compare in-vivo CT and T1-weighted MR images and thereto apply mutual in-formation based linear registration to the segmented vessels. Clarke et al. also use linearregistration to align µCT and MR images of ex-vivo plaques. Fig. 2 shows an example ofa µCT image and a corresponding MR image.

Previous studies have either focussed on the segmentation of ex-vivo images or on theanalysis of in-vivo images. Furthermore, different tools like Matlab, ImageJ and ENVI areapplied in the analysis process.The aim of the present work is to provide a set of software assistants that support theanalysis and comparison of histological images, CT images and multicontrast MRI ex-vivo and in-vivo. On the one hand, the common software environment should facilitatethe cooperative analysis of study data by pathologists, radiologists and neurologists, onthe other hand, this software should provide a framework for the evaluation of imagingsequences and classification algorithms for ex-vivo as well as in-vivo image data.

Figure 2: Examples of an ex-vivo T1-weighted MR image and the corresponding µCT image. Thefield of view and the orientation differs.

Contrast Sequence Res. [mm3] TE / TR [ms] Flip angle Turbo factor AveragesT1 GRE 0.1 2.4/13.4 20 N/A 12T2* GRE 0.1 7.9/12.8 20 N/A 10T2 RARE 0.1 40/2500 90-180 16PD RARE 0.1 9/1500 90-180 8

Table 1: Pulse sequence parameters for multi-contrast 3D ex-vivo plaque imaging using an ultra-highfield small animal system (9.4T).

2 Data

To date, in an ongoing study, plaque specimen of patients undergoing carotid endarterec-tomy for ICA stenosis≥ 70% (local degree of stenosis) are evaluated. Plaques are fixed in4% formalin for ≥24 hours after surgical resection, transferred into tubes containing freeoil (1,1,2-Trichlorotrifluoroethane) and evaluated at a high field 9.4T MRI system (BrukerBioSpin 94/20, Ettlingen, Germany). The MRI protocol provides 3D imaging with a spa-tial resolution of 100µm3 including T1-fatsat-GRE, T2*-GRE, T2-RARE, and PD-RAREimaging (see Table 1). Total scan time is 10 hours. After MR scanning the plaque speci-men are scanned with a µCT Scanner (Siemens µCAT II) before EVG-coloured histolog-ical images of single slices are produced. The in-vivo images were acquired prior to theintervention with a 3T MRI scanner (Siemens TrioTim, Erlangen, Germany). They con-tain a Time-of-Flight(TOF) image volume (0.52×0.52.×1.0mm3)and T1-, T2-, T2*- andPD-weighted images acquired at lower resolutions (ca. 0.54×0.54×4.0).

3 Methods

To support the analysis of the study data, two software prototypes are being developed. Thefirst one supports the manual classification of the histological slices by the pathologist.The user can interactively define contours and label the according regions. The secondprototype focusses on the processing of the volume images from CT and MRI and theexploration of the spatial and contentual correspondances of the different images. Figure3 shows an overview of the main functionalities of the actual software.

Figure 3: Overview of the image data and prototype functionality for the analysis of the ex-vivoplaques. The histological images are classified manually using a special prototype. CT and MRimages are preprocessed within the same prototype before they can be explored separately or fusedin 3D. The interactive definition of arbitrary slice planes in 3D supports the detection of the CT andMR image regions that correspond with the histological slices.

The plaque analysis software is implemented within the image processing and visualiza-tion R&D platform MeVisLab [BVUW+07]. CT and MR images are preprocessed toremove non-plaque structures and background noise.

3.1 Preprocessing

In the CT images, the cylinder that contains the plaque is clearly visible (Fig. 2). To restrictthe visualization to the interesting image region, the cylinder is removed automatically asshown in Fig. 4. First, an automatic thresholding is applied. The threshold is determinedby the analysis of the histogram of a central image slice. The cylinder component is thendetected. The mask that results from filling the cylinder component is then used to maskthe original image.

Figure 4: Removal of surrounding structures from µCT image. The leftmost image shows the ac-quired image with the plaque and the surrounding structures. The middle image shows the resultof the component analysis with the plaque and the containing cylinder identified as one component.The rightmost image shows the masked plaque image, where the structures surrounding the plaqueare removed.

To prepare the MR images for the combined visualization, an initial semi-automatic back-ground suppression is applied. Thereto the available contrasts are combined into a max-imum image, which is then processed with an Otsu Threshold [Ots79] and a subsequentmorphological closing. Based on this pre-segmentation the contours of the outer borderand the lumen are detected to further distinguish dark plaque regions and lumen throughthe contour lengths (Fig. 5). The resulting contours can be corrected interactively by theusers.

3.2 Visual Exploration of Multicontrast MR Images

Many of the approaches described in Section 1 use exactly three MR contrasts and it iscommon to fuse these contrast images into an RGB image for further processing and com-parison with histological slices[ISM+04, ASH+07]. This approach is extended here. Theuser can select the contrast for each channel and window each color channel separately toget the most meaningful representation of the fused image (Fig. 6).Previous studies have given estimates for the relative appearance of specific plaque com-ponents at different contrasts [FMS+08]. This knowledge can be applied to suppress orenhance components through arithmetic combinations. Figure 8 shows such a fusion that

Figure 5: Separation of plaque and background based on the maximum image of the given MRcontrasts. After the application of thresholding and morphological closing plaque and backgroundare separated by the detection of the contours.

combines the PD-weighted and the T2-weighted images to enhance the fibrous cap andthe lipid rich/necrotic core regions.

The 3D and MPR views are coupled: MPRs are defined by moving and rotating the cor-responding lines in the intersecting viewing planes and can be shown in the 3D volumerendering for orientation [LKP06]. The current combination image can also be overlaidonto the original images, which are shown in the 2D viewers on the left side (Fig. 9).

3.3 Comparison of Histological Images and Image Volumes

To compare the volumetric CT and MR images with the histological slices, it is importantto find the corresponding transsection plane. The user can reproduce the slice position byadjusting the MPR in the 3D viewing panel. The 2D image panel on the left can eithershow 2D views of the volume images or the EVG-colored histological slice images andthe manual segmentation results of the pathologist. To facilitate the interactive positionmatching the histological slice can also be rotated (Fig. 6).

3.4 Analysis of in-vivo Plaque Images

In addition to the low resolution due to the shorter scanning time, in-vivo plaque imagesare affected by patient motion. Thus, a combined analysis of the different MR contrastsrequires a prior registration. Thereto we apply a 3D registration based on normalized mu-tual information and affine transformations. To identify the region of interest and quantifythe vessel narrowing caused by the plaques, the vessels are first segmented in the TOFimage with a region growing approach [BRL+04]. The result of this segmentation canthen be used to identify the interesting image regions for the exploration and the compar-ison with the ex-vivo imaging results. Figure 7 shows a visualization result for registered

Figure 6: Comparison of the histological image and the corresponding image region in the RGBimage that results from the fusion of the PD-weighted, T2-weighted and T2*-weighted ex-vivo MRimages. The user can rotate the histological image and define the corresponding slice plane bymoving and rotation the slice markers in the orthogonal views.

color-combined in-vivo images.

4 Results and Discussion

In an ongoing study the described tools have been applied to three datasets. Histologicalslices have been classified successfully by the pathologist and results have been transferredto the neurological department, where the study results are evaluated.The visual exploration of the multicontrast MR images resulted in a quicker apprehensionof the plaque constitution than the conventional side-by-side comparision of the separateimage slices. The left part of figure 8 shows the results of a high resolution multi-contrastplaque acquisition as viewed in the standard analysis procedure. By comparing signalintensities in T1, T2, T2* and PD weighted images, different plaque components such ascalcification, hemorrhage, and lipid rich/necrotic core (LR/NC) are identified slicewise.

Applying the new analysis tool, the same data was used to create novel contrasts, as shownon the right side in Fig. 8 multiplying T2 and PD contrast of the spatially registered plaquedata. Note, that in this case, contrast manipulation provides a clearer delineation of thefibrous cap and the LR/NC compared to the standard view in left part of Fig. 8.

The use of different color channels for the combination of three contrasts into a singleimage is shown in Fig. 9 for another plaque specimen. The combination of T1 (red colorchannel), T2 (green color channel) and PD (blue color channel) illustrates the potential forimproved identification of plaque components. Particularly fibrous tissue (hyperintense in

Figure 7: Fused visualization of in-vivo multicontrast MR images. The vessels are segmented in theTOF image, which covers a larger region than the other images. The plaque is clearly visible on theRGB image that stems from the T1-, T2- and PD-weighted images.

T2 and thus green in Fig. 9), calcification (signal void), and necrosis (hyperintense in T1and thus red in Fig. 9) can clearly be identified.

An expert determined the image regions corresponding to the histological slices in theMR volumes. To finally transfer the classification results of the pathologist, a non-rigidregistration as proposed by Clarke et al. [CBH+06] will be necessary to compensate thedeformations from the preparation of the histological slices (Fig. 1).

The analysis of the in-vivo images has not been adressed in the study yet. However, firsttests with the given datasets indicate that the methods developed with the ex-vivo imagedata can be transferred to in-vivo images.

5 Conclusions

In the previous sections we presented a prototypical software for multi-modal inspectionof carotid plaque specimens. The software prototype offers functionalities for interactiveclassification of histological images, texploration of single MR contrasts or images fusedvia arithmetic operations or color channels. Furthermore, an interactive alignment of his-tological images and volumetric images is supported. In an ongoing study, first datasetshave been processed with the software. Initial results indicate the potential for improveddelineation of specific plaque components. 3D visualization of the resulting contrast orcolor maps allow to explore and understand the complex 3D plaque morphology. Futurework will focus on the development and evaluation of automatic analysis methods formulticontrast MR images ex-vivo and in-vivo.

Figure 8: Left:Original multi-contrast images of a complex ex-vivo plaque specimen with calcifi-cation, a necrotic core including old hemorrhage, and a fibrous cap separating the core and vessellumen. Right: 3D ex-vivo plaque analysis with multiplied T2 and PD contrast highlighting fibroustissue. Note, that the fibrous cap is considerably enhanced and easier to identify than in the standardview shown in Fig. 1. 3D coverage facilitates the evaluation of thickness and extent of cap andplaque core within the entire specimen.

References

[ASH+07] Russell W Anderson, Christopher Stomberg, Charles W Hahm, Venkatesh Mani,Daniel D Samber, Vitalii V Itskovich, Laura Valera-Guallar, John T Fallon, Pavel BNedanov, Joel Huizenga, and Zahi A Fayad. Automated classification of atheroscle-rotic plaque from magnetic resonance images using predictive models. Biosystems,90(2):456–466, 2007.

[AvdGW+04] I. M. Adame, R. J. van der Geest, B. A. Wasserman, M. A. Mohamed, J. H CReiber, and B. P F Lelieveldt. Automatic segmentation and plaque characterizationin atherosclerotic carotid artery MR images. MAGMA, 16(5):227–234, Apr 2004.

[BRL+04] Tobias Boskamp, Daniel Rinck, Florian Link, Bernd Kummerlen, Georg Stamm, andPeter Mildenberger. New vessel analysis tool for morphometric quantification andvisualization of vessels in CT and MR imaging data sets. Radiographics, 24(1):287–297, 2004.

[BVUW+07] I. Bitter, R. Van Uitert, I. Wolf, L. Ibanez, and J.-M. Kuhnigk. Comparison of FourFreely Available Frameworks for Image Processing and Visualization That Use ITK.13(3):483–493, 2007.

[CBH+06] Sharon E Clarke, Vadim Beletsky, Robert R Hammond, Robert A Hegele, andBrian K Rutt. Validation of automatically classified magnetic resonance images forcarotid plaque compositional analysis. Stroke, 37(1):93–97, Jan 2006.

[CFU+05] Baocheng Chu, Marina S Ferguson, Hunter Underhill, Norihide Takaya, JianmingCai, Michel Kliot, Chun Yuan, and Thomas S Hatsukami. Images in cardiovascularmedicine. Detection of carotid atherosclerotic plaque ulceration, calcification, andthrombosis by multicontrast weighted magnetic resonance imaging. Circulation,112(1):e3–e4, Jul 2005.

[DSC+06] Damini Dey, Piotr Slomka, Daisy Chien, David Fieno, Aiden Abidov, Rola Saouaf,Louise Thomson, John D Friedman, and Daniel S Berman. Direct quantitative in

Figure 9: Multi-contrast plaque analysis tool. Different contrasts are displayed via the RGB colorchannels: T1 (red), T2 (green) and PD (blue).

vivo comparison of calcified atherosclerotic plaque on vascular MRI and CT by mul-timodality image registration. J Magn Reson Imaging, 23(3):345–354, Mar 2006.

[dWOM+06] Thomas T de Weert, Mohamed Ouhlous, Erik Meijering, Pieter E Zondervan, Jo-hanna M Hendriks, Marc R H M van Sambeek, Diederik W J Dippel, and Aadvan der Lugt. In vivo characterization and quantification of atherosclerotic carotidplaque components with multidetector computed tomography and histopathologicalcorrelation. Arterioscler Thromb Vasc Biol, 26(10):2366–2372, Oct 2006.

[FMS+08] Sebastiano Fabiano, Stefano Mancino, Matteo Stefanini, Marcello Chiocchi,Alessandro Mauriello, Luigi Giusto Spagnoli, and Giovanni Simonetti. High-resolution multicontrast-weighted MR imaging from human carotid endarterectomyspecimens to assess carotid plaque components. Eur Radiol, 18(12):2912–2921, Dec2008.

[ISM+04] V.V. Itskovich, D.D. Samber, V. Mani, J.G.S. Aguinaldo, J.T. Fallon, C.Y. Tang,V. Fuster, and Z.A. Fayad. Quantification of human atherosclerotic plaques us-ing spatially enhanced cluster analysis of multicontrast-weighted magnetic reso-nance images. Magnetic Resonance in Medicine : Official Journal of the Societyof Magnetic Resonance in Medicine / Society of Magnetic Resonance in Medicine,52(3):515–523, 2004.

[KBV+09] Christof Karmonik, Pamela Basto, Kasey Vickers, Kirt Martin, Micheal J Reardon,Gerald M Lawrie, and Joel D Morrisett. Quantitative segmentation of principalcarotid atherosclerotic lesion components by feature space analysis based on mul-ticontrast MRI at 1.5 T. IEEE Trans Biomed Eng, 56(2):352–360, Feb 2009.

[LKP06] F. Link, M. Koenig, and H.-O. Peitgen. Multi-Resolution Volume Rendering withper Object Shading. In L. Kobbelt, T. Kuhlen, and R. Westermann, editors, VisionModelling and Visualization, pages 185–191, Berlin, Aachen, 2006. Aka.

[LXF+06] Fei Liu, Dongxiang Xu, Marina S Ferguson, Baocheng Chu, Tobias Saam, NorihideTakaya, Thomas S Hatsukami, Chun Yuan, and William S Kerwin. Automated invivo segmentation of carotid plaque MRI with Morphology-Enhanced probabilitymaps. Magn Reson Med, 55(3):659–668, Mar 2006.

[Ots79] Nobuyuki Otsu. A Threshold Selection Method from Gray-Level Histograms. IEEETransactions on Systems, Man and Cybernetics, 9(1):62–66, January 1979.

[QRV+07] Ye Qiao, Itamar Ronen, Jason Viereck, Frederick L Ruberg, and James A Hamil-ton. Identification of atherosclerotic lipid deposits by diffusion-weighted imaging.Arterioscler Thromb Vasc Biol, 27(6):1440–1446, Jun 2007.

[SCD+95] H. C. Stary, A. B. Chandler, R. E. Dinsmore, V. Fuster, S. Glagov, W. Insull, M. E.Rosenfeld, C. J. Schwartz, W. D. Wagner, and R. W. Wissler. A definition of ad-vanced types of atherosclerotic lesions and a histological classification of atheroscle-rosis. A report from the Committee on Vascular Lesions of the Council on Arte-riosclerosis, American Heart Association. Circulation, 92(5):1355–1374, Sep 1995.

[SGL+08] B. Sun, D.P. Giddens, R.J. Long, W.R. Taylor, D. Weiss, G. Joseph, D. Vega, andJ.N. Oshinski. Automatic plaque characterization employing quantitative and mul-ticontrast MRI. Magnetic Resonance in Medicine : Official Journal of the Societyof Magnetic Resonance in Medicine / Society of Magnetic Resonance in Medicine,59(1):174–180, 2008.

[SHT+07] T. Saam, T.S. Hatsukami, N. Takaya, B. Chu, H. Underhill, W.S. Kerwin, J. Cai,M.S. Ferguson, and C. Yuan. The vulnerable, or high-risk, atherosclerotic plaque:noninvasive MR imaging for characterization and assessment. Radiology, 244(1):64–77, 2007.

[XHY02] D. Xu, J.-N. Hwang, and C. Yuan. Segmentation of Multi-Channel Image withMarkov Random Field Based Active Contour Model. J. VLSI Signal Process. Syst.,31(1):45–55, 2002.

[YKF+02] Chun Yuan, William S Kerwin, Marina S Ferguson, Nayak Polissar, ShaoxiongZhang, Jianming Cai, and Thomas S Hatsukami. Contrast-enhanced high resolu-tion MRI for atherosclerotic carotid artery tissue characterization. J Magn ResonImaging, 15(1):62–67, Jan 2002.