Machine Learning for Tomographic ImagingGe WANG, PhD; wangg6@rpi.edu

Biomedical Imaging CenterAI-based X-ray Imaging System (AXIS) LabCBIS/BME, Rensselaer Polytechnic Institute

University of Minnesota Institute of Mathematics, October 16, 2019

Machine Learning for Tomographic ImagingGθ WANG, PhD; wangg6@rpi.edu

Biomedical Imaging CenterAI-based X-ray Imaging System (AXIS) LabCBIS/BME, Rensselaer Polytechnic Institute

October 16, 2019

Machine Learning for Tomographic ImagingGθ WGAN, PhD; wangg6@rpi.edu

Biomedical Imaging CenterAI-based X-ray Imaging System (AXIS) LabCBIS/BME, Rensselaer Polytechnic Institute

October 16, 2019

Acknowledgment

Roadmap for Deep Recon

IEEE Trans. Medical Imaging Special Issue

AcknowledgmentWe are supported by NIH, General Electric, Hologic, IBM, & NVIDIA. We are seeking collaborative opportunities

Outline

Theoretical Exploration Translational Efforts Teaching Innovation

Various Types of Cells

Quadratic Neurons as Fuzzy Gates & Factors

OutputInput

𝑥𝑥𝑛𝑛

𝑦𝑦

… …

1

𝑥𝑥1

Fuzzy Logic Interpretation

Fuzzy Logic System

https://microcontrollerslab.com/fuzzy-logic-system-working-example/

Deep Fuzzy Logic System

From Data-driven to Rule-based

http://dailynous.com/2018/06/25/creating-semantic-network-history-philosophy/

Universal Approximation via Factorization

Width- versus Depth-Efficiency

Width: n+4 & Depth: Deep

Deep Depth: Univariate Polynomial of Order N

Log Operation: Loudness & Brightness in dB

Wide Width: Univariate Polynomial of Order N

Quadratic Neurons as Factors (with Real coefficients, According to the Algebraic Fundamental Theorem)

• Duality in Depth & Width of the Architecture

• Depth & Width Convertible• Diverse Networks for the

Same Task• Same Network Performs

Different Tasks

Depth & Width Balanced… …… …… …… …

… … …

… …

……

Particle/Factor Mathematics

Kolmogorov

Outline

Theoretical Exploration Translational Efforts Teaching Innovation

Initial Deep Network for Low-dose CT

WGAN Network for Low-dose CT

IEEE Trans. Medical Imaging 37:1522-1534, 2018 (for details, see https://arxiv.org/abs/1802.05656 or https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8353466)

Low-quality CT Scan(1/4 Radiation Dose)

High-quality CT Scan(Standard Radiation Dose)

Machine Learning Turns Low-quality CT Image into High-quality

Counterpart

CT Denoising Neural Network by RPI,Sichuan University, & Harvard University

GE Healthcare for Low-dose CT (RSNA’18)

IR Methods vs MAP DL for Low-dose CT

Commercial Iterative Recon (IR) Algorithms in This Study

Our MAP Network-based Deep Learning (DL) with Optimized Depth

CLONE CLONE CLONE。。。

Sparse-data CT De-artifacts: “LEARN”

tNx…1st

0x

Iteration-inspired Layer

…

Under-sampled Data Reconstruction

2nd t-th(T-1)-

th T-th

-1 -1( )t T tA Ax yλ −

1tx − tx

Σ-

-

Conv+ReLU

Conv

Chen H, Zhang Y, Chen YJ, Zhang WH, Sun HQ, Lv Y, Liao PX, Zhou JL, Wang G:LEARN: Learned Experts’ Assessment-based Reconstruction Network for Sparse-data CT. IEEE Trans. Medical Imaging, June 2018

AUTOMAP with 𝑶𝑶 𝑵𝑵𝟒𝟒 Complexity

Zhu B, Liu JZ, Cauley SF, Rosen BR, Rosen MS: Image reconstruction by domain-transform manifold learning. Nature 555:487–492, 2018

𝟑𝟑 × 𝟑𝟑k-space Data

ReshapeTo a Vector

FC Layers to All Pixels

𝟑𝟑 × 𝟑𝟑 Image

𝑶𝑶 𝑵𝑵𝟑𝟑 × 𝑵𝑵𝒗𝒗 Complexity𝑵𝑵: # of Data or Pixels per Row 𝑵𝑵𝒗𝒗: # of views

Intelligent CT Network (iCT-Net)

iCT Network

Li, Yinsheng & li, ke & Zhang, Chengzhu & C. Montoya, Juan & Chen, Guang-Hong. (2019). Learning to Reconstruct Computed Tomography (CT) Images Directly from Sinogram Data under A Variety of Data Acquisition Conditions. IEEE Transactions on Medical Imaging. PP. 10.1109/TMI.2019.2910760.

iCTNet with 𝑶𝑶 𝑵𝑵𝟐𝟐 ×𝑵𝑵𝒄𝒄 Complexity

*Li, Y., Li, K., Zhang, C., Montoya, J. and Chen, G., “Learning to Reconstruct Computed Tomography (CT) Images Directly from Sinogram Data under A Variety of Data Acquisition Conditions,” IEEE Transactions on Medical Imaging, 1–1 (2019).

4 × 3sinogram

4𝛼𝛼 × 3sinogram

FC

FC

3 × 3image

3 × 3image

3 × 3 final output

Dual Network Architecture (DNA)

Few-view CT

Intermediate FBP Image

Few-view Sinogram

FBPConvNet

Jin KH, McCann MT, Froustey E, Unser M: Deep Convolutional Neural Network for Inverse Problems in Imaging. IEEE Transactions on Image Processing 26(9):4509–4522, 2017

Reconstructed Image

Adversarial DNA Architecture & High Efficiency

DNA: Generator 1

Filtration Backprojection Refinement

Backprojection with 𝑶𝑶 𝑵𝑵 Complexity

Smart Backpropagation with Learned Weights Restricted

to a Single X-ray Path

Learned Filtered Projection

Backprojection with 𝑶𝑶 𝑵𝑵𝟐𝟐 Complexity

Smart Backpropagation with Learned Weights Restricted

to a Single X-ray Path

Learned Filtered Projection

Comparative Study

Truth FBP LEARN [1] Sino-U-net [2] iCTNet [3] DNA

[1] Chen, H., Zhang, Y., Chen, Y., Zhang, J., Zhang, W.,Sun, H., Lv, Y., Liao, P., Zhou, J. and Wang, G.,“LEARN: Learned Experts’ Assessment-BasedReconstruction Network for Sparse-Data CT,”IEEE Transactions on Medical Imaging 37(6),1333–1347, 2018

[2] Lee, H., Lee, J., Kim, H., Cho, B. and Cho, S., “Deep-Neural-Network-Based Sinogram Synthesis forSparse-View CT Image Reconstruction,” IEEETransactions on Radiation and Plasma MedicalSciences 3(2), 109–119, 2019

[3] Li, Y., Li, K., Zhang, C., Montoya, J. and Chen, G.,“Learning to Reconstruct ComputedTomography (CT) Images Directly fromSinogram Data under A Variety of DataAcquisition Conditions,” IEEE Transactions onMedical Imaging, 1–1, 2019)

Koning Breast CT via Deep Recon

AI-based Breast Slices at 1/3 X-ray Dose

Current Breast Slices at 1/3 X-ray Dose

Current Breast Slices at Full X-ray Dose

Outline

Theoretical Exploration Translational Efforts Teaching Innovation

Medical Imaging Course at RPI

Tue Topic Fri Topic08/30 Introduction

09/03 Monday Schedule 09/06 Hands-on 1: Colab & MNIST09/10 Network & Backpropagation 09/13 Hands-on 2: MNIST Finish09/17 Architectures & Interpretability 09/20 Image Quality: Resolution, Noise, & SSIM09/24 CT Physics (Recorded) 09/27 CT Reconstruction10/01 Deep CT Reconstruction 10/04 Hands-on 3: CT Networks10/08 More CT Stuff 10/11 MRI Physics10/15 Spin-Echo 10/18 K-Space & More10/22 Review, Quiz, Q & A 10/25 Deep MRI Reconstruction10/29 Hands-on 4: MRI Networks 11/01 Nuclear Physics11/05 Nuclear Imaging 11/08 US Imaging11/12 Optical Imaging 11/15 Multimodality Imaging11/19 Image Quality: Task-specific 11/22 Q & A11/26 Hands-on 5: More 11/29 Nov. 27-29 Thanksgiving12/03 Q & A 12/06 Project Presentation12/10 Academic/Industrial Outlook Dec. 11 Last Day

Schedule for Fall 2019

First Book on Deep Recon (eBook & Hardcopy)

Theme & Coauthors

Theme ofThis Book:

TomographicImage

Reconstruction

TomographicData

Acquisition

Reconstructed

Image

Image Analysis,One of Successful

Applications of Artificial

Intelligence & Machine Learning

DataAs Tomographic

Features

Coauthors of the Book:

Yi Zhang, Xiaojing Ye, Xuanqin Mou

Outline of Our Book

Recognizing Your Digits

Read an image fileNormalize it

Resize toa desired size

Pass into the network

Typical Result

GAN: One of Greatest Ideas

Optimal Discriminator

Optimal Discriminator = 𝑃𝑃1𝑃𝑃1+𝑃𝑃2

P1 P2 If a new datum, if discriminator function is greater than 0.5, the datum should belong to Class 1 (blue); otherwise, it comes more likely from Class 2 (green)

Optimal Generator

https://towardsdatascience.com/understanding-generative-adversarial-networks-gans-cd6e4651a29

Adversarial Learning

Data

Freq

uenc

y1.0

0.5

Data/Target/Real Distribution

Synthetic/Source/Fake Distribution

Adversarial Learning: Initial InterplayFr

eque

ncy

1.0

Data

Like

lihoo

d

Data

1.0

0.5 0.5

Data

1.0

0.5

Improve Generator

Improve Discriminator

Adversarial Learning: Further ImprovementFr

eque

ncy

1.0

Data

Like

lihoo

d

Data

1.0

0.5 0.5

Data

1.0

0.5

Improve Generator

Improve Discriminator

Simultaneous CT-MRI

https://www.ncbi.nlm.nih.gov/pubmed/26429262

Joint CT-MRI Reconstruction

MA: CT

Patches

Update

Update

Recon

Fast MRI Scan

Few-view CT Scan

MA: MRI

Deformation Map

Recon

Patches

Multi-modality Atlas (MA):CT Atlas + MRI Atlas

Deformed Atlas

Deformed Atlas

CurrentMRI Image

CurrentCT Image

FusedCT-MRIImage

SimultaneousCT-MRI Scan

https://www.ncbi.nlm.nih.gov/pubmed/26672028

Ensemble Learning Improves MRI Resolution

Dynamic Cardiac MRI

Sunnybrook Cardiac DataNormal (35 slices , 20 phases) Cine steady state free precession (SSFP) MR short axis (SAX) images

• Slice thickness=8mm• Gap=8mm• FOV=320mm by 320mm• Matrix=256 by 256• 1.5mm x 1.5 mm resolution

100-view Recon ReferenceAI-based Recon

Excitement = MRI & CT Coupled (E=MC2)