medical imaging informatics:medical imaging … · 2016. 6. 30. · – picture archiving and...

MEDICAL IMAGING INFORMATICS:MEDICAL IMAGING INFORMATICS:Lecture # 1

Basics of Medical Imaging Informatics:g gEstimation Theory

Norbert SchuffProfessor of Radiology

VA Medical Center and UCSFVA Medical Center and [email protected]

UCSF VAMedical Imaging Informatics 2011 NschuffCourse # 170.03Slide 1/31

Department of Radiology & Biomedical Imaging

What Is Medical Imaging Informatics?• Signal Processing

– Digital Image Acquisition – Image Processing and Enhancement

• Data Mining• Data Mining– Computational anatomy– Statistics– Databases– Data-mining– Workflow and Process Modeling and SimulationWorkflow and Process Modeling and Simulation

• Data Management– Picture Archiving and Communication System (PACS) – Imaging Informatics for the Enterprise – Image-Enabled Electronic Medical Records – Radiology Information Systems (RIS) and Hospital Information Systems (HIS)Radiology Information Systems (RIS) and Hospital Information Systems (HIS)– Quality Assurance – Archive Integrity and Security

• Data Visualization– Image Data Compression – 3D, Visualization and Multi-media3D, Visualization and Multi media – DICOM, HL7 and other Standards

• Teleradiology– Imaging Vocabularies and Ontologies– Transforming the Radiological Interpretation Process (TRIP)[2]– Computer-Aided Detection and Diagnosis (CAD).

UCSF VADepartment of Radiology & Biomedical Imaging

Co pute ded etect o a d ag os s (C )– Radiology Informatics Education

• Etc.

What Is The Focus Of This Course?Learn using computational tools to maximize information for

knowledge gain:

Improve

Pro-active

ImageMeasurements Model knowledge

pData

collectionRefine Model

Image Model g

Extract information

Compare with

modelRe-active

UCSF VAMedical Imaging Informatics 2009, NschuffCourse # 170.03Slide 3/31


Challenge: Maximize Information Gain

1. Q: How can we estimate quantities of interest from a i t f t i ( i ) t ?given set of uncertain (noise) measurements?

A: Apply estimation theory (1st lecture today)

2. Q: How can we measure (quantify) information?A: Apply information theory (2nd lecture next week)A: Apply information theory (2 lecture next week)



Estimation Theory: Motivation Example IGray/White Matter Segmentation

1.0

Hypothetical Histogram

0.6

0.8

0.0

0.2

0.4

Intensity

GM/WM overlap 50:50;GM/WM overlap 50:50;Can we do better than flipping a coin?



Estimation Theory: Motivation Example IIEstimation Theory: Motivation Example IIGoal: Capture dynamic signal on a

static background1

High signal to noise

-1

0

0 1000 2000 3000 4000 5000 6000

Time

-2

P i l t i

1

3

5Poor signal to noise

D. Feinberg Advanced MRI Technologies, Sebastopol, CA

5

-3

-1



0 1000 2000 3000 4000 5000 6000

Time

-5

Estimation Theory: Motivation Example IIIEstimation Theory: Motivation Example III

Goal:Diffusion Imaging

Goal:Capture directions of fiber bundles

•Sensitive to random motion of water•Probes structures on a microscopic scale

Microscopic tissue sample

Dr. Van Wedeen, MGH



Quantitative Diffusion Maps

Basic Concepts of Modeling

: target of interest and unknown

: measurement

: Estimator - a good f b d

guess of based on measurements

Cartoon adapted from: Rajesh P N Rao Bruno A Olshausen Probabilistic Models of the Brain



Cartoon adapted from: Rajesh P. N. Rao, Bruno A. Olshausen Probabilistic Models of the Brain. MIT Press 2002.

Deterministic Model

N = number of measurementsM = number of states, M=1 is possibleUsually N > M and |noise||2 > 0

noise Hθ

The model is deterministic, because discrete values of are solutions.

M noise N NHθ

Note:1) we make no assumption about 2) Each value is as likely as any

another value

What is the best estimator under thesecircumstances?



Least-Squares Estimator (LSE)

The best what we can do is minimizing noise:

LSEˆ 0

M Mnoise

N

N θ

Hθ

H

LSEˆ 0 T T

N θH H H

1 1

LSE

T TnH H Hθ

•LSE is popular choice for model fittingU f l f b i i d i i•Useful for obtaining a descriptive measure

But •LSE makes no assumptions about distributions of data or parameters•Has no basis for statistics “deterministic model”



•Has no basis for statistics deterministic model

Prominent Examples of LSE

100

150 Mean Value: 1

1ˆN

meanj

jN

0

50

100

Inte

nsiti

es (Y

)

1j

Variance 2

var1

1ˆ ˆ1

N

iance meanj

jN

100 300 500 700 900

Measurements (x)

-50

11 jN

100

200

ity

Amplitude:1̂

Frequency:2̂

-100

0Inte

ns

2

Phase:3̂

Decay:4̂



100 300 500 700 900

Measurements

Likelihood Model

Pretend we know something about

We perform measurements for all possible

Likelihood |L p

We perform measurements for all possiblevalues of

We obtain the likelihood function of given our measurements given our measurements

Note: is random is a fixed parameter is a fixed parameterLikelihood is a function of both the unknown and known



Likelihood Model (cont’d)

|L p N

N G lNew Goal:Find an estimatorwhich gives the most likely probability distributionprobability distribution underlying L



Maximum Likelihood Estimator (MLE)

Goal: Find estimator which gives the most likely probability distribution underlying xN.

M lik lih d f ti max |MLE p N

Max likelihood function

ln | 0d pd

N

MLE can be found by taking the derivative of Likelihood F

MLE

d Nθ θ



Example I: MLE Of Normal DistributionNormal distribution

222

1

1| , exp2

N

jp j

N 0.8

1.0

Normal Distribution

12 j

0.2

0.4

0.6

a2

log of the normal distribution (normD)

100 300 500 700 900a1

0.0 222

1

1ln | ,2

N

j

p j

N

Log Normal Distribution

MLE of the mean (1st derivative):

2

1ln 0ˆ4 MLE

Nd p jd

-5

0

21ˆ4 MLE

EL jM

p jd

1 N

MLE jN

-10



1j

MLE jN

100 300 500 700 900a1

-15

Example II: MLE Of Binominal Distribution(Coin Toss)( )

Distribution function f(y|n,w):n= number of tossesn number of tossesw= probability of success

0.7f(y|n=10 w=7)

0 1

0.2

f(y|n=10,w=7)

0.0

0.1

y 0.3f(y|n=10,w=3)

0 0

0.1

0.2



1 2 3 4 5 6 7 8 9 10N0.0

y

MLE Of Coin Toss (cont’d) Goal:Given the observed data f (y|w=0.7, n=10), find the parameter MLE that most likely produced the data.

( | 7, 10)MLEL y n

most likely produced the data.

For a fair coin 0.5MLE



MLE Of Coin Toss (cont’d)

!| 1! !

n yynL yy n y

0.20

0.25

ood

Likelihood function of coin tosses

! !y n y

0 00

0.05

0.10

0.15

Like

liho

What is the likelihood of observing 7 heads given that we tossed a fair coin 10 times

0.1 0.3 0.5 0.7 0.9W

0.00

10 7710!| 10, 7 0.5 1 0.5 0.127! 10 7 !

L n w

10 7710!

unfair coin =0.6

log likelihood function-1

10 7710!| 10, 7 0.6 1 0.6 0.217! 10 7 !

L n w

ln |!ln ln ln 1

L w yn y w n y w

-3

-2



ln ln ln 1! !

y w n y wy n y

0.1 0.3 0.5 0.7 0.9

MLE Of Coin Toss

ld L

Evaluate MLE equation (1st derivative)

ln0

1ML MMLE E LE

d L n yyd

0(1 )

MLEMLE

MLE MLE

y n yn

According to the MLE principle, the distribution f(y/n) for a given n is the most likely distribution to have generated the observed datais the most likely distribution to have generated the observed dataof y.



Relationship between MLE and LSE is independent of noiseNMLE and noiseN have the same distribution

Assume:

noiseN is zero mean and gaussian

| |noisep p θ N N H

p(|) is maximized when LSE is minimized



Bayesian Model

Now, the daemon comes into play, but we know

Prior knowledge

The daemon’s preferencesfor (prior knowledge).

New Goal:

prior p

New Goal:Find the estimator which gives the most likely probability distribution of probability distribution of given everything we know.



Bayesian Model

|Nposterior C L p



Maximum A-Posteriori (MAP) Estimator ( )Goal: Find the most likely MAP (max. posterior density of ) given .

max |MAP NL p N

Maximize joint density

MAP can be found by taken the partial derivative

ln | ln | ln 0d L p L pd

N N



Example III: MAP Of Normal Distribution

Th l f MAP iThe sample mean of MAP is:

2

2 21

N

jj

T

MAP

1jT

If we do not have prior information on , inf or T inf

MAP MLˆ ˆ ˆ, LSEμ μ μ



Posterior Distribution and Decision Rules

p(|)p(|)

MSE



X MAPMSE

Decision Rules

Measurements Likelihood Posterior function

Prior

Distribution

Gain

Result

Prior Distribution

GainFunction



Some Desirable Properties of Estimators I:

Unbiased: Mean value of the error should be zero

- 0E

Consistent: Error estimator should decrease asymptotically as number of

2- 0 for large NMSE E

Consistent: Error estimator should decrease asymptotically as number of measurements increase. (Mean Square Error (MSE))

0 for large NMSE E

What happens to MSE when estimator is biased?2 2 - - b bMSE E E

pp



variance bias

Some Desirable Properties of Estimators II:Some Desirable Properties of Estimators II:Efficient: Co-variance matrix of error should decrease asymptotically to itsminimal value for large N

- - . . .T

i ki kE some very small value θC

minimal value for large N



Example:P ti Of E ti t M d V iProperties Of Estimators Mean and Variance

1 1N

1

1 1ˆj

E E j NN N

Mean:

The sample mean is an unbiased estimator of the true meanThe sample mean is an unbiased estimator of the true mean

21 1N

2

22 22 2

1

1 1ˆN

j

E E j NN N N

Variance:

The variance is a consistent estimator becauseIt approaches zero for large number of measurements.



Properties Of MLEp

• is consistent: the MLE recovers asymptotically the true y p yparameter values that generated the data for N inf;

• Is efficient: The MLE achieves asymptotically the minimum error (= max. information)



SummarySummary

• LSE is a descriptive method to accurately fit data to a p ymodel.

• MLE is a method to seek the probability distribution that k th b d d t t lik lmakes the observed data most likely.

• MAP is a method to seek the most probably parameter value given prior information about the parameters andvalue given prior information about the parameters and the observed data.

• If the influence of prior information decreases, i.e. many measurements, MAP approaches MLE



Some Priors in ImagingSome Priors in Imaging

• Smoothness of the brain• Anatomical boundaries • Intensity distributions• Anatomical shapes• Physical models

P i t d f ti– Point spread function– Bandwidth limits

• Etc.



Estimation Theory: Motivation Example IGray/White Matter Segmentation

1.0

Hypothetical Histogram

0.6

0.8

0.0

0.2

0.4

Intensity

What works better than flipping a coin?

Design likelihood functions based onanatomyco-occurance of signal intensitiesothers



Determine prior distributionpopulation based atlas of regional intensitiesmodel based distributions of intensitiesothers

Estimation Theory: Motivation Example IIEstimation Theory: Motivation Example IIGoal: Capture dynamic signal on a

static background5

Poor signal to noise

1

3

-5

-3

-1

0 1000 2000 3000 4000 5000 6000

Time

Improvements to identify the dynamic signal:

Design likelihood functions based on

D. Feinberg Advanced MRI Technologies, Sebastopol, CA

Design likelihood functions based onauto-correlationsanatomical information

Determine prior distributions fromi l t



serial measurementsmultiple subjectsanatomy

Estimation Theory: Motivation Example IIIEstimation Theory: Motivation Example III

Goal:Capture directions

Diffusion Spectrum Imaging – Human Cingulum Bundle

Capture directions of fiber bundles

Improvements to identify tracts:p o e e ts to de t y t acts

Design likelihood functions based onsimilarity measures of adjacent voxelsfiber anatomy

Determine prior distributions fromanatomyfiber skeletons from a populationothers

Dr. Van Wedeen, MGH

others



MAP Estimation in Image Reconstructions ith Ed P i P iwith Edge-Preserving Priors



Dr. Ashish Raj, Cornell U

MAP in Image Reconstructions with Edge-Preserving PriorsPreserving Priors

For DTI, use the fact that coregistered DTI images have common edge features:images have common edge features:



Dr. Justin Haldar Urbana-Champaign

MAP Estimation In Image Reconstruction

Human brain MRI. (a) The original LR data. (b) Zero-padding interpolation. (c) SR with box-PSF. (d) SR with Gaussian-PSF.p ( ) ( )

From: A. Greenspan in The Computer Journal Advance Access published February 19, 2008



Improved ASL Perfusion Results p

zDFT = zero-filled DFT

UCSF VABy Dr. John Kornak, UCSF

Bayesian Automated Image SegmentationBayesian Automated Image Segmentation

Bruce Fischl MGH



Bruce Fischl, MGH

Segmentation Using MLE

A: Raw MRIB: SPM2C: EMSD: HBSA

fromHabib Zaidi, et al, NeuroImage 32 g(2006) 1591 – 1607



Population Atlases As Priorsp



Dr. Sarang Joshi, U Utah, Salt Lake City

Population Shape Regressions Based Age-S l ti P iSelective Priors

Age = 29 33 37 41 45 49



Age 29 33 37 41 45 49Dr. Sarang Joshi, U Utah, Salt Lake City

Imaging Software Using MLE And MAPImaging Software Using MLE And MAPPackages Applications Languages

VoxBo fMRI C/C++/IDLVoxBo fMRI C/C++/IDLMEDx sMRI, fMRI C/C++/Tcl/Tk SPM fMRI, sMRI matlab/C iBrain IDLiBrain IDLFSL fMRI, sMRI, DTI C/C++

fmristat fMRI matlab BrainVoyager sMRI C/C++BrainVoyager sMRI C/C

BrainTools C/C++ AFNI fMRI, DTI C/C++

Freesurfer sMRI C/C++Freesurfer sMRI C/CNiPy Python



Literature2 Probability Distributions 672 Probability Distributions 672.1 Binary Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . 682.1.1 The beta distribution . . . . . . . . . . . . . . . . . . . . . 712.2 Multinomial Variables . . . . . . . . . . . . . . . . . . . . . . . . 742.2.1 The Dirichlet distribution . . . . . . . . . . . . . . . . . . . 762.3 The Gaussian Distribution . . . . . . . . . . . . . . . . . . . . . . 782.3.1 Conditional Gaussian distributions . . . . . . . . . . . . . . 852.3.2 Marginal Gaussian distributions . . . . . . . . . . . . . . . 882.3.3 Bayes’ theorem for Gaussian variables . . . . . . . . . . . . 902.3.4 Maximum likelihood for the Gaussian . . . . . . . . . . . . 932.3.5 Sequential estimation . . . . . . . . . . . . . . . . . . . . . 942.3.6 Bayesian inference for the Gaussian . . . . . . . . . . . . . 972.3.7 Student’s t-distribution . . . . . . . . . . . . . . . . . . . . 1022 3 8 Periodic variables 1052.3.8 Periodic variables . . . . . . . . . . . . . . . . . . . . . . . 1052.3.9 Mixtures of Gaussians . . . . . . . . . . . . . . . . . . . . 1102.4 The Exponential Family . . . . . . . . . . . . . . . . . . . . . . . 1132.4.1 Maximum likelihood and sufficient statistics . . . . . . . . 1162.4.2 Conjugate priors . . . . . . . . . . . . . . . . . . . . . . . 1172.4.3 Noninformative priors . . . . . . . . . . . . . . . . . . . . 1172.5 Nonparametric Methods . . . . . . . . . . . . . . . . . . . . . . . 120p2.5.1 Kernel density estimators . . . . . . . . . . . . . . . . . . . 1222.5.2 Nearest-neighbour methods . . . . . . . . . . . . . . . . . 124Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1273 Linear Models for Regression 1373.1 Linear Basis Function Models . . . . . . . . . . . . . . . . . . . . 1383.1.1 Maximum likelihood and least squares . . . . . . . . . . . . 1403 1 2 Geometr of least sq ares 1433.1.2 Geometry of least squares . . . . . . . . . . . . . . . . . . 1433.1.3 Sequential learning . . . . . . . . . . . . . . . . . . . . . . 1433.1.4 Regularized least squares . . . . . . . . . . . . . . . . . . . 1443.1.5 Multiple outputs . . . . . . . . . . . . . . . . . . . . . . . 1463.2 The Bias-Variance Decomposition . . . . . . . . . . . . . . . . . . 1473.3 Bayesian Linear Regression . . . . . . . . . . . . . . . . . . . . . 1523.3.1 Parameter distribution . . . . . . . . . . . . . . . . . . . . 153



3.3.1 Parameter distribution . . . . . . . . . . . . . . . . . . . . 1533.3.2 Predictive distribution . . . . . . . . . . . . . . . . . . . . 1563.3.3 Equivalent kernel . . . . . . . . . . . . . . . . . . . . . . . 1573.4 Bayesian Model Comparison . . . . . . . . . . . . . . . . . . . . . 1613.5 The Evidence Approximation . . . . . . . . . . . . . . . . . . . . 165

Literature

Estimation Theory 774.1 Basic concepts 784.2 Properties of estimators 804.3 Method of moments 844.3 Method of moments 844.4 Least-squares estimation 864.4.1 Linear least-squares method 864.4.2 Nonlinear and generalized least squares * 884.5 Maximum likelihood method 904.6 Bayesian estimation * 944.6.1 Minimum mean-square error estimator 944.6.2 Wiener filtering 964.6.3 Maximum a posteriori (MAP) estimator 974.7 Concluding remarks and references 99Problems 1015 Information Theory 1055 Information Theory 1055.1 Entropy 1055.1.1 Definition of entropy 1055.1.2 Entropy and coding length 1075.1.3 Differential entropy 1085.1.4 Entropy of a transformation 1095.1.4 Entropy of a transformation 1095.2 Mutual information 1105.2.1 Definition using entropy 1105.2.2 Definition using Kullback-Leibler divergence 110



LiteraturePart I: Theories of Perception and LearningChapter 1: Bayesian Modelling of Visual Perception, by P. Mamassian, M. Landy and L. MaloneyChapter 2: Vision, Psychophysics, and Bayes, by P. Schrater and D. KerstenChapter 3: Visual Cue Integration for Depth Perception, by R. JacobsChapter 4: Velocity Likelihoods in Biological and Machine Vision, by Y. Weiss and D. FleetChapter 5: Learning Motion Analysis by W Freeman J HaddonChapter 5: Learning Motion Analysis, by W. Freeman, J. Haddon and E. PasztorChapter 6: Information Theoretic Approach to Neural Coding and Parameter Estimation: A Perspective, by J.-P. NadalChapter 7: From Generic to Specific: An Information Theoretic Perspective on the Value of High-Level Information, by A. Yuillep g , yand J. CoughlanChapter 8: Sparse Correlation Kernel Reconstruction and Superresolution, by C. Papageorgiou, F. Girosi and T. Poggio



Literature



Literature

Mathematical• H. Sorenson. Parameter Estimation – Principles and Problems.p

Marcel Dekker (pub)1980. Signal Processing• S Kay Fundamentals of Signal Processing – Estimation TheoryS. Kay. Fundamentals of Signal Processing Estimation Theory.

Prentice Hall 1993.• L. Scharf. Statistical Signal Processing: Detection, Estimation, and

Time Series Analysis Addison-Wesley 1991Time Series Analysis. Addison Wesley 1991. Statistics:• New Directions in Statistical Signal Processing. From Systems to

Brain Ed S Haykin MIT Press 2007Brain. Ed. S. Haykin. MIT Press 2007.



medical imaging informatics:medical imaging … · 2016. 6. 30. · – picture archiving and...

Documents