MedGIFT projects in medical imaging

Henning Müller

Where we are

2

Who I am

•  Medical informatics studies in ���Heidelberg, Germany (1992-1997) •  Exchange with Daimler Benz research, USA

•  PhD in image processing, image retrieval, Geneva, Switzerland (1998-2002) •  Exchange with Monash University, Melbourne, AUS

•  Titular professor in radiology and medical informatics at the University of Geneva (2014-) •  Postdoc, assistant professor between 2002-2013

•  Professor in Computer Science at the ���HES-SO, Sierre, Switzerland (2007-)

3

Why working on image retrieval?

•  Much imaging data is produced

•  Imaging data is very complex •  And getting more complex

•  Imaging is essential in diagnosis ���and treatment planning

•  Images out of their context���loose most of their sense •  Clinical data is necessary

•  Diagnoses are often not precise

•  Evidence-based medicine & case-based reasoning 4

Khresmoi – retrieval

5

The informed patient

6

Steps for our retrieval system

7

Image pre- treatment

Visual feature extraction

Feature modeling

Multimodal fusion

Classification, detection, retrieval

Visualization, results

presentation

Resource creation

Diagnosis aid

8

Identifying user requirements •  Surveys among radiologists

•  Also GPs and patients

•  Observing diagnosis processes •  Analyzing search log files (Goldminer, PubMed, HON)

•  Eye tracking on a radiology viewing station

•  What are information needs and what are tasks that are hard and where help is needed?

•  Test the developed systems in user studies •  Analyze feedback

•  Record the system use for understanding problems 9

Eye tracking

10

Data used for ParaDISE

•  Scientific data of the biomedical literature •  600’000 articles and 1.6 mio figures of the open access

literature (>4 mio images if separating compound figures)

•  Public data source but only 2D data

•  Clinical data from the Vienna Medical University image archive •  5TB of data of two consecutive months

•  Radiology reports for each case (in German)

•  Private data source, so access only with password

•  Link medical cases with similar cases from the literature based on image data and text 11

Creation of the VISCERAL database

12

Annotations (20 organs, 55 landmarks)

13

Connecting different data levels

14

EHR, PACS

Classification of journal figures •  Most figures in articles���

are not diagnostic ���imaging

•  Captions do not always ���allow to identify the ���image type •  Visual information can help

•  All these image types are ���mapped to RadLex and ���UMLS/MeSH •  Allows reusing information and search in related terms

15

H Müller, J Kalpathy-Cramer, D Demner-Fushman, S Antani, Creating a classification of image types in the

medical literature for visual categorization, SPIE medical imaging, San Diego, USA, 2012.

Context is important (25 yo vs. 88 yo)!

16

Visual feature extraction

•  Colors & grey levels

•  Shapes after segmentations

•  Texture information •  In 2D, 3D, 4D

•  In several scales and directions

•  Local vs. global information extraction •  Finding interest points

•  Finding regions or volumes of interest

•  Combination of features is usually best 17

Visual feature modeling

•  Visual words instead of raw visual features •  Reducing the curse of dimensionality

•  Find models similar to text (synonyms, polysemy)

18 A Foncubierta, AG Seco de Herrera, H Müller, Medical Image Retrieval using a Bag of Meaningful Visual Words,

ACM MM workshop on medical multimedia retrieval, Barcelona, Spain, 2013.

Feature extraction and detection •  Learn combinations of Riesz wavelets as digital

signatures using SVMs •  Create signatures to detect small local lesions and

visualize them

19 A Depeursinge, A Foncubierta–Rodriguez, D Van de Ville, H Müller, Rotation–covariant feature learning ���

using steerable Riesz wavelets, IEEE Transactions on Image Processing, 2014.

Information fusion

•  Combine information from���text or structured data ���with visual information

•  Text data can be mapped���to semantics to understand���links •  Also language-independent

•  Early fusion

•  Late fusion •  Rank-based vs. score-based

20

1

1

M

N

t

tc

c

⎛ ⎞⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎝ ⎠

M

M SVM 1

M

t

t

⎛ ⎞⎜ ⎟⎜ ⎟⎜ ⎟⎝ ⎠

M

SVM

1

N

c

c

⎛ ⎞⎜ ⎟⎜ ⎟⎜ ⎟⎝ ⎠

MSVM

tq

cq

modN∏

( )t ip w

( )c ip w

Detection and retrieval of similar cases

21 A Depeursinge, D Van de Ville, A Platon, A Geissbuhler, PA Poletti, H Müller, Near-Affine-Invariant Texture Learning for Lung Tissue Analysis Using Isotropic Wavelet Frames, IEEE Transactions on Information Technology in Biomedicine, 16(4), 2012.

Khresmoi4radiology interface

22

Khresmoi4professionals interface

23

Semantic search, also for images

24

Khresmoi4everyone interface

25

Shambala – a simple web interface

26

Much involvement in benchmarking •  ImageCLEF

•  Has had a medical task since 2004 •  2013: modality classification, compound figure separation,

image-based and case-based retrieval •  2014: liver annotation

•  VISCERAL •  Organ segmentation and landmark detection (ISBI)

•  Lesion detection and retrieval task

•  Khresmoi LinkedIn group, …

27

Cloud-based evaluation in VISCERAL

28

Test

VISCERAL data

29

4D data analysis

30

OA Jimenez del Toro, A Foncubierta-Rodriguez, MI Vargas Gomez, H Müller, A Depeursinge, Epileptogenic lesion quantification in MRI using contralateral 3D texture comparisons, MICCAI 2013, Springer LNCS, Nagoya, Japan, 2013.

A Depeursinge, A Foncubierta-Rodriguez, A Vargas, D Van de Ville, A Platon, PA Poletti, H Müller, Rotation-covariant texture analysis of 4D dual-energy CT as an indicator of local pulmonary perfusion, ISBI 2013, San Francisco, USA, 2013.

•  Dual Energy CT for perfusion analysis in pulmonary embolism •  Collaboration with emergency radiology

•  Epileptogenic lesion detection in several MRI image series (T1, T2, DTI)

Material Attenuation Coefficient vs keV

0.1

1 10 100

40 50 60 70 80 90 100 110 120 130 140 Photon Energy (keV)

m(E

) (c

m2/

mg) Iodine

Water

80 keV 140 keV

4D visualization

31

•  Visualization of two (min and max) energy levels to visualize pulmonary embolisms

Another view on 4D

32

An infrastructure supporting the load

•  Small, fixed experiments are easy, large routine updates and use are difficult!! Big data is hard! •  Workflow for data re-indexation, maximum automation

•  Khresmoi: Private cloud •  All components in virtual machines connected with a

SOA infrastructure, reattribution of resources possible

•  Local computation •  Hadoop/MapReduce to distribute the computation

•  Needs some optimization

•  Cloud use when local resources are not sufficient

33

System overview

34

Infostructure in MD-Paedigree

35

Conclusions

•  Visual information retrieval has many interesting challenges in the medical field •  Many supporting techniques are required

•  Treating big data is a challenge and digital medicine is really big data •  Many techniques can and need to be used with image

analysis and machine learning as the basis

•  Digital medicine is a reality and more is yet to come … genetics, molecular imaging, …

•  We also need corresponding infrastructures

36

Contact and more information

•  More information can be found at •  http://khresmoi.eu/

•  http://visceral.eu/

•  http://medgift.hevs.ch/

•  http://publications.hevs.ch/

•  Contact: •  Henning.mueller@hevs.ch

37