1 + 1 > 2? getting more out of multi-modality imaging...multi-modality imaging examples pet-mr...
TRANSCRIPT
![Page 1: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/1.jpg)
1 + 1 > 2?Getting More Out of Multi-Modality Imaging
Matthias J. Ehrhardt
September 26, 2019
![Page 2: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/2.jpg)
Outline
1) Motivation: Examples of Multi-ModalityImaging (Why?)
2) Mathematical Models for Multi-ModalityImaging (How?)
3) Application Examples: Remote Sensingand Medical Imaging (1 + 1 > 2?)
![Page 3: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/3.jpg)
Motivation: Examples of Multi-Modality Imaging
![Page 4: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/4.jpg)
Multi-Modality Imaging Examples
PET-MR
PET-MR (and PET-CT, SPECT-MR, SPECT-CT)
Combine anatomical (MRI)and functional (PET) infor-mation
7 clinical scanners in UK
Currently images are justoverlayed
Challenge: Reduce scan-ning time, increase imagequality, lower dose
image: Sheth and Gee, 2012
![Page 5: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/5.jpg)
Multi-Modality Imaging Examples
PET-MR Multi MRI
Multi-Sequence MRIpre-contrastT1-weighted (a),dual-echo T2 (b, c)post-contrast2D T2 FLAIR (d, e),T1-weighted (f)
StandardizedMRI protocolfor multiple sclerosis
6 scans, total 30 minRovira et al., Nature Reviews Neurology, 2015
Challenge: Reduce scanning time
![Page 6: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/6.jpg)
Multi-Modality Imaging Examples
PET-MR Multi MRI Spectral CT
Spectral CT
CT spectral CT
images:Shikhaliev and Fritz, 2011
material decomposition
Acquisition: energy resolved measurements
Combination: material information
Challenge: Low dose / high noise in some channels
![Page 7: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/7.jpg)
Multi-Modality Imaging Examples
PET-MR Multi MRI Spectral CTHyper
+ optical
Image fusion in remote sensing
Acquisition: low resolution hyperspectral data (127 channels,1m × 1m) and high resolution photograph (0.25m × 0.25m)acquired on plane or satellite, e.g. by NERC AirborneResearch & Survey Facility
Challenge: get best of both worlds—high spatial andspectral resolution
![Page 8: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/8.jpg)
Multi-Modality Imaging Examples
PET-MR Multi MRI Spectral CTHyper
+ opticalX-ray
+ optical
X-ray separation for art restauration Deligiannis et al. 2017
Acquisition: photographs and x-ray images
Challenge: separate the x-rays of the doors
![Page 9: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/9.jpg)
Fairly Large FieldI Regular sessions at major conferences: Applied Inverse
Problems, SIAM ImagingI Symposium in Manchester in 3-6 Nov 2019I Special Issue in IOP Inverse Problems
I Collaborative Software Projects: CCPi (Phil Withers) andCCP PETMR
![Page 10: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/10.jpg)
Mathematical Models for Multi-Modality Imaging
![Page 11: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/11.jpg)
Image Reconstruction
Variational Approach:
u∗ ∈ arg minu
{D(Au, b) + αJ (u) + ıC (u)
}A forward operator (often but not always linear),
e.g. Radon transform
D data fit, e.g. least-squares D(Au, b) = 12‖Au − b‖2,
Kullback–Leibler divergenceD(Au, b) =
∫Au − b + b log(b/Ay)
J regularizer, e.g. total variationJ (u) = TV(u) :=
∑i |∇ui | Rudin et al., 1992
ıC constraints, e.g. nonnegativity
How to include information from other modalities?
![Page 12: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/12.jpg)
Image Reconstruction
Variational Approach:
u∗ ∈ arg minu
{D(Au, b) + αJ (u) + ıC (u)
}A forward operator (often but not always linear),
e.g. Radon transform
D data fit, e.g. least-squares D(Au, b) = 12‖Au − b‖2,
Kullback–Leibler divergenceD(Au, b) =
∫Au − b + b log(b/Ay)
J regularizer, e.g. total variationJ (u) = TV(u) :=
∑i |∇ui | Rudin et al., 1992
ıC constraints, e.g. nonnegativity
How to include information from other modalities?
![Page 13: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/13.jpg)
Modelling Structural Similarity
Definition: The Weighted Total Variation (wTV) of u is
dTV(u) :=∑i
wi‖∇ui‖, 0 ≤ wi ≤ 1
See e.g. Ehrhardt and Betcke ’16
I If c > 0, c < wi , then c TV ≤ wTV ≤ TV.
I If wi = 1, then wTV = TV.
I wi = η‖∇vi‖η , ‖∇vi‖2η = ‖∇vi‖2 + η2, η > 0
![Page 14: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/14.jpg)
Modelling Structural Similarity
Definition: The Weighted Total Variation (wTV) of u is
dTV(u) :=∑i
wi‖∇ui‖, 0 ≤ wi ≤ 1
See e.g. Ehrhardt and Betcke ’16
I If c > 0, c < wi , then c TV ≤ wTV ≤ TV.
I If wi = 1, then wTV = TV.
I wi = η‖∇vi‖η , ‖∇vi‖2η = ‖∇vi‖2 + η2, η > 0
![Page 15: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/15.jpg)
Modelling Structural Similarity
Definition: The Weighted Total Variation (wTV) of u is
dTV(u) :=∑i
wi‖∇ui‖, 0 ≤ wi ≤ 1
See e.g. Ehrhardt and Betcke ’16
I If c > 0, c < wi , then c TV ≤ wTV ≤ TV.
I If wi = 1, then wTV = TV.
I wi = η‖∇vi‖η , ‖∇vi‖2η = ‖∇vi‖2 + η2, η > 0
![Page 16: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/16.jpg)
Modelling Structural Similarity
〈∇u,∇v〉 = cos(θ)|∇u||∇v |
Definition: Two images u and v are said to have parallel levelsets or are structurally similar (denoted by u ∼ v) if θ = 0 orθ = π, i.e.
∇u ‖ ∇v i.e. ∃ α such that ∇u = α∇v .
I Dominant idea in this fieldI Parallel Level Set Prior, e.g. Ehrhardt and Arridge ’14
I Directional Total Variation, e.g. Ehrhardt and Betcke ’16
I Total Nuclear Variation, e.g. Knoll et al. ’16
I Coupled Bregman iterations, e.g. Rasch et al. ’18
I Others are: joint sparsity (e.g. wTV), joint entropy, ...
![Page 17: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/17.jpg)
Modelling Structural Similarity
〈∇u,∇v〉 = cos(θ)|∇u||∇v |
Definition: Two images u and v are said to have parallel levelsets or are structurally similar (denoted by u ∼ v) if θ = 0 orθ = π, i.e.
∇u ‖ ∇v i.e. ∃ α such that ∇u = α∇v .
I Dominant idea in this fieldI Parallel Level Set Prior, e.g. Ehrhardt and Arridge ’14
I Directional Total Variation, e.g. Ehrhardt and Betcke ’16
I Total Nuclear Variation, e.g. Knoll et al. ’16
I Coupled Bregman iterations, e.g. Rasch et al. ’18
I Others are: joint sparsity (e.g. wTV), joint entropy, ...
![Page 18: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/18.jpg)
Modelling Structural Similarity
〈∇u,∇v〉 = cos(θ)|∇u||∇v |
Definition: Two images u and v are said to have parallel levelsets or are structurally similar (denoted by u ∼ v) if θ = 0 orθ = π, i.e.
∇u ‖ ∇v i.e. ∃ α such that ∇u = α∇v .
I Dominant idea in this fieldI Parallel Level Set Prior, e.g. Ehrhardt and Arridge ’14
I Directional Total Variation, e.g. Ehrhardt and Betcke ’16
I Total Nuclear Variation, e.g. Knoll et al. ’16
I Coupled Bregman iterations, e.g. Rasch et al. ’18
I Others are: joint sparsity (e.g. wTV), joint entropy, ...
![Page 19: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/19.jpg)
Modelling Structural Similarity
〈∇u,∇v〉 = cos(θ)|∇u||∇v |
Definition: Two images u and v are said to have parallel levelsets or are structurally similar (denoted by u ∼ v) if θ = 0 orθ = π, i.e.
∇u ‖ ∇v i.e. ∃ α such that ∇u = α∇v .
I Dominant idea in this fieldI Parallel Level Set Prior, e.g. Ehrhardt and Arridge ’14
I Directional Total Variation, e.g. Ehrhardt and Betcke ’16
I Total Nuclear Variation, e.g. Knoll et al. ’16
I Coupled Bregman iterations, e.g. Rasch et al. ’18
I Others are: joint sparsity (e.g. wTV), joint entropy, ...
![Page 20: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/20.jpg)
Modelling Structural Similarity
〈∇u,∇v〉 = cos(θ)|∇u||∇v |
Definition: Two images u and v are said to have parallel levelsets or are structurally similar (denoted by u ∼ v) if θ = 0 orθ = π, i.e.
∇u ‖ ∇v i.e. ∃ α such that ∇u = α∇v .
I Dominant idea in this fieldI Parallel Level Set Prior, e.g. Ehrhardt and Arridge ’14
I Directional Total Variation, e.g. Ehrhardt and Betcke ’16
I Total Nuclear Variation, e.g. Knoll et al. ’16
I Coupled Bregman iterations, e.g. Rasch et al. ’18
I Others are: joint sparsity (e.g. wTV), joint entropy, ...
![Page 21: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/21.jpg)
Modelling Structural Similarity
〈∇u,∇v〉 = cos(θ)|∇u||∇v |
Definition: Two images u and v are said to have parallel levelsets or are structurally similar (denoted by u ∼ v) if θ = 0 orθ = π, i.e.
∇u ‖ ∇v i.e. ∃ α such that ∇u = α∇v .
I Dominant idea in this fieldI Parallel Level Set Prior, e.g. Ehrhardt and Arridge ’14
I Directional Total Variation, e.g. Ehrhardt and Betcke ’16
I Total Nuclear Variation, e.g. Knoll et al. ’16
I Coupled Bregman iterations, e.g. Rasch et al. ’18
I Others are: joint sparsity (e.g. wTV), joint entropy, ...
![Page 22: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/22.jpg)
Modelling Structural Similarity
〈∇u,∇v〉 = cos(θ)|∇u||∇v |
Definition: Two images u and v are said to have parallel levelsets or are structurally similar (denoted by u ∼ v) if θ = 0 orθ = π, i.e.
∇u ‖ ∇v i.e. ∃ α such that ∇u = α∇v .
I Dominant idea in this fieldI Parallel Level Set Prior, e.g. Ehrhardt and Arridge ’14
I Directional Total Variation, e.g. Ehrhardt and Betcke ’16
I Total Nuclear Variation, e.g. Knoll et al. ’16
I Coupled Bregman iterations, e.g. Rasch et al. ’18
I Others are: joint sparsity (e.g. wTV), joint entropy, ...
![Page 23: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/23.jpg)
Directional Total VariationI Note that if ‖∇v‖ = 1, then
u ∼ v ⇔ ∇u − 〈∇u,∇v〉∇v = 0
Definition: The Directional Total Variation (dTV) of u is
dTV(u) :=∑i
‖[I− ξiξTi ]∇ui‖, 0 ≤ ‖ξi‖ ≤ 1
Ehrhardt and Betcke ’16, related to Kaipio et al. ’99, Bayram and Kamasak ’12
I If c > 0, ‖ξi‖2 ≤ 1− c , then c TV ≤ dTV ≤ TV.I If ξi = 0, then dTV = TV.I ξi = ∇vi
‖∇vi‖η , ‖∇vi‖2η = ‖∇vi‖2 + η2, η > 0
0π
![Page 24: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/24.jpg)
Application Examples
![Page 25: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/25.jpg)
Multi-Modality Imaging Examples
PET-MR Multi MRI Spectral CTHyper
+ opticalX-ray
+ optical
Multi-Sequence MRIEhrhardt and Betcke, SIAM J. Imaging Sci., vol. 9, no. 3, pp. 1084–1106, 2016.
Joint work with:Computer Science: M. Betcke (UCL)
![Page 26: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/26.jpg)
Multi-Sequence MRI Results
gr. truth
side info
no prior TV
wTV dTV
sampling
![Page 27: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/27.jpg)
Multi-Sequence MRI Results
gr. truth
side info
no prior TV
wTV dTV
sampling
![Page 28: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/28.jpg)
Multi-Sequence MRI Results
gr. truth
side info
no prior TV
wTV dTV
sampling
![Page 29: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/29.jpg)
Multi-Sequence MRI Results
gr. truth
side info
no prior TV
wTV dTV
sampling
![Page 30: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/30.jpg)
Quantitative Results
T1 T2
70
80
90
100
SS
IM[%
]
no prior
TV
wTV
dTV
mean
median
I Range (min, max), mean and median over 12 data sets
![Page 31: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/31.jpg)
Multi-Modality Imaging Examples
PET-MR
PET-MREhrhardt et al., Phys. Med. Biol. (in press), 2019
Ehrhardt et al., Proceedings of SPIE, vol. 10394, pp. 1–12, 2017
Joint work with:Mathematics: A. Chambolle (Ecole Polytechnique, France), P. Richtarik(KAUST, Saudi Arabia), C. Schonlieb (Cambridge)Medical Physics: P. Markiewicz (UCL),Neurology: J. Schott (UCL)
![Page 32: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/32.jpg)
PET-MR ResultsReconstruction model:
minu
{KL(Au + r ; b) + λJ (u) + ı≥0(u)
}Total Variation, J = TV
Directional Total Variation (using MRI), J = dTV
![Page 33: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/33.jpg)
PET-MR ResultsReconstruction model:
minu
{KL(Au + r ; b) + λJ (u) + ı≥0(u)
}Total Variation, J = TV
Directional Total Variation (using MRI), J = dTV
![Page 34: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/34.jpg)
Multi-Modality Imaging Examples
PET-MR Multi MRI Spectral CTHyper
+ opticalX-ray
+ optical
Image fusion in remote sensingBungert et al., Inverse Probl., vol. 34, no. 4, p. 044003, 2018
Joint work with:Mathematics: L. Bungert (Erlangen, Germany), R. Reisenhofer (Vienna,Austria), J. Rasch (Berlin, Germany), C. Schonlieb (Cambridge),Biology: D. Coomes (Cambridge)
![Page 35: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/35.jpg)
Standard regularization versus image fusion
Reconstruction model:
minu
{12‖S(u ∗ k)− v‖2 + λJ (u) + ı≥0(u)
}data standard, J = TV fusion, J = dTV
data standard, J = TV fusion, J = dTVdata standard, J = TV fusion, J = dTVdata standard, J = TV fusion, J = dTV
![Page 36: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/36.jpg)
Blind versus non-blind image fusion
Blind
reconstruction model:
minu
,k
{12‖S(u ∗ k)− v‖2 + λJ (u) + ı≥0(u)
+ ıS(k)
}data fusion
blind fusion
data fusion
blind fusion
data fusion
blind fusion
data fusion
blind fusion
![Page 37: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/37.jpg)
Blind versus non-blind image fusion
Blind reconstruction model:
minu,k
{12‖S(u ∗ k)− v‖2 + λJ (u) + ı≥0(u) + ıS(k)
}data fusion blind fusion
data fusion blind fusiondata fusion blind fusiondata fusion blind fusion
![Page 38: 1 + 1 > 2? Getting More Out of Multi-Modality Imaging...Multi-Modality Imaging Examples PET-MR Multi MRI Multi-Sequence MRI pre-contrast T 1-weighted (a), dual-echo T 2 (b, c) post-contrast](https://reader034.vdocuments.mx/reader034/viewer/2022050522/5fa5a48cadd0f53c944dcfe4/html5/thumbnails/38.jpg)
Conclusions and Outlook
Summary:
I Multi-Modality Imaging examples:PET-MR, multi-sequence MRI, spectralCT, Hyper + optical,X-ray + optical
I Mathematical Models to exploitsynergies between modalities
I Examples: indeed often 1 + 1 > 2!
Future:
I Which modalities complement eachother best?
I Multi-modality imaging can help tolower dose, increase resolution ...
I Expertise in image / video processing,compressed sensing, machine learning ...
independent
synergistic