information signal processingjao/talks/invitedtalks/isp2.pdf · information signal processing j. a....

Information Signal ProcessingInformation Signal Processing

Joseph A. OJoseph A. O’’SullivanSullivan

Electronic Systems and Signals Research Laboratory

Center for Security TechnologiesDepartment of Electrical and Systems Engineering

Washington [email protected]

http://essrl.wustl.edu/~jaoSupported by: ONR, NSF, NIH, Boeing Foundation, DARPASpecial thanks to Naveen Singla

J. A. O’Sullivan. 11/21/2003Information Signal Processing

2

CollaboratorsCollaborators

Jasenka BenacMichael D. DeVoreAndrew (Lichun) LiClayton MillerLee MontagninoRyan Murphy Natalia SchmidNaveen SinglaBrandon WestoverShenyu Yan

G. James BlaineRoger ChamberlainMark FranklinDaniel R. FuhrmannRonald S. IndeckChenyang LuPierre Moulin, UIUCMarcel MullerRobert PlessDavid G. PolitteChrysanthe PrezaAndrew Singer, UIUC Donald L. SnyderBruce R. WhitingJeffrey F. Williamson, VCULihao Xu

Faculty Current and Former Students


4

OutlineOutline

• DSP ISP

- Data models, computational models,

algorithms

- Central role of information

• Graphical Data Models- X-ray CT imaging

- Iterative decoding

• Message Passing EM

Algorithms

• Applications Revisited

• Speculation on Trends• Conclusions


5

Signal Processing

Information TheoryComputation and

Communication

FFTFFT

MultiresolutionMultiresolutionanalysisanalysis


6

X(0)

X(1)

X(2)

X(3)

X(4)

X(5)

X(6)

X(7)

x(0)

x(4)

x(2)

x(6)

x(1)

x(5)

x(3)

x(7)

FFT


7

Signal Processing

Information Theory

Numerical analysisNumerical analysis

Processors: parallel, Processors: parallel, ASIC, etc.ASIC, etc.

Systolic architecturesSystolic architectures

FFTWFFTW

Computation and

Communication


8

Signal Processing

Information Theory

Numerical analysisNumerical analysis

Processors and Processors and architecturesarchitectures

FFT, FFTWFFT, FFTW

Transversal filtersTransversal filters

MRAMRA

Complexity theoryComplexity theory

Graphical modelsGraphical models

KalmanKalman filtersfilters

CompressionCompression

Computation and

Communication


9

Computation and

Communication

Information Signal ProcessingInformation Signal Processing

Signal Processing

Information Theory

ComplexityComplexity--constrained processingconstrained processing

Signal processing on graphsSignal processing on graphs

Distributed signal processingDistributed signal processing

Distributed information theoryDistributed information theory

Distributed computation and Distributed computation and communicationcommunication

Optimal information extraction, Optimal information extraction, communication, computationcommunication, computation


10

Distributed sensing,

communication, computation““The architecture for a The architecture for a

fully netted maritime forcefully netted maritime force””


11

Wireless Sensor NetworksWireless Sensor Networks

http://www.greatduckisland.net/index.php

Great Duck Island Habitat


14

OutlineOutline

• DSP ISP


algorithms





Algorithms




15

Graphical Data ModelsGraphical Data Models

•• Model 1:Model 1: y = Hsy = Hs– y is n × 1, s is m × 1,

and H is n × m

– yj depends on sk if hjk 0

– Defines a graphical model

•• Model 2:Model 2:

– Neighborhood structure

– Bipartite graph model

•• Model 3:Model 3:– RVs on edges of graph

k

j

j

kj jksypp ))(,|()|( sy

k kj

kjk sxpp)(

)|()|( sx

j

jkj jkxypp ))(,|()|( xy


16

Tomography

S

D

Nonrandom Graphs

Line integrals through patient

Quantization point spread function

weights on edges of graph

Helps organize computations

Siemens Somotom Emotion


17

Computational ModelsComputational Models

System model accounts for:

• Information extraction

problem definition

• Compression of sensor data

• Network throughput

• Processor cycles per

instruction

• Size of processor local

memory

• Communication bandwidth

of each link

• etc.


18

Computational ModelsComputational Models

• Local resources plus remote

• Communicate observation as well as classification- Human in the loop

- Remote contribution to classification when available

• Dynamic resource availability

• Sequence of partial classifications (an, n)


19

Progress: Computational Graph Progress: Computational Graph Same as Data Model GraphSame as Data Model Graph

•• Message passing algorithmsMessage passing algorithms– Pearl’s belief propagation

– Iterative decoding

» Turbo-codes, parallel concatenated codes

» Low density parity check codes

» Repeat-accumulate codes, serial concatenated codes

– Iterative equalization and decoding

•• ExpectationExpectation--Maximization (EM) AlgorithmsMaximization (EM) Algorithms– Graphical models

– General problem

– Gaussian, Poisson (emission tomography, transmission tomography)

– Abstract examples on random graphs


20

10011010101000

01010101010001

10100001010101

01100110010100

10010101001010

00101000101011

01001010100110

H Regular (3,6) n=14

Random Graphs

Comment: LDPC parity check matrix


21

001010100000

100100000010

010001001000

000100010001

100000001100

001000100010

010001010000

000010000101

H Regular (2,3) n=12

Random Graphs


22

10011010101000

01010101010001

10100001010101

10101010010100

10010101001010

00101000101101

01001001010110

H Irregular n=14

Random Graphs


24

k

j

jk

jk

),(

),(

)1(

'

)1(

)(

'

)1(

stateg

statef

m

kjk

m

jk

m

jkj

m

jk


27

Iterative Decoding Message PassingIterative Decoding Message Passing

kjk

m

jk

m

jk

\)('

)(

'

)1(

jkj

m

kjz

m

jk k

\)('

)1(

'

)1(

2tanh)1(

2tanh

Codeword Bit Nodes

zk

xj

Check Nodes

jk

jk


28

Iterative Decoding Message PassingIterative Decoding Message Passing

-6

-5

-4

-3

-2

-1

0

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8

SNR [dB]

Bit e

rro

r ra

te,

Lo

g B

ase

10

ISI-free

BIAWGN Capacity

[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix


29

OutlineOutline

• DSP ISP


algorithms





Algorithms




30

ML Problem: xsxxys

dpp )|()|(lnmax

EM Algorithm: )|()|(

),|(ln),|(minmin

sxxy

syxsyx

s pp

EM AlgorithmEM Algorithm

sk

yj

Hidden Data

Incomplete Data

xjk


31

VariationalVariational RepresentationsRepresentations

•• Convex Decomposition LemmaConvex Decomposition Lemma. Let f be convex. Then

• Special Case: f is ln

• Basis for EM; see also De Pierro, Lange, Fessler

i

ii

i i

ii

rii

rr

xfrxf

0,1

)()(1

i

ii

i i

ii

i

iq

q

1,0:

lnminln

P

P


32

EM AlgorithmEM Algorithm

')|'()'|(

)|()|(),|(

)(

)()()1(

xsxxy

sxxysyx

dpp

ppm

mmm

xsxsyxss

dp, mmm )|(ln)|(argmax )()1()1(

Assume the factorizations:

k kj

kjk sxpp)(

)|()|( sx

j

jkj jkxypp ))(,|()|( xy

These computations become local and thus message passing

In general, these are global computations


33

Message Passing EM AlgorithmMessage Passing EM Algorithm

)('

'

)('

)(

'''

'),('

'

)('

)(

'''

1(

)|())(',|(

)|())(',|(

),|(

jk

jk

jk

m

kjkjkj

kkjk

jk

jk

m

kjkjkj

(m)

jjk

)m

dxsxpjkxyp

dxsxpjkxyp

yx

KK

KK

K

K

s

jkkjk

m

jjk

m

s

m

k dxsxp,yxsk

)|(ln)|(argmax )()1()1(s

sk

yj

Input Data

Measured Data

xjk

)( jkx

ks


34

GaussianGaussian--MAPMAP

)2

,0(~

),0(~

;

0 Iw

Is

wHsy

NN

PN

)(

)(

0

)1( ˆ

|)(|2|)(|

ˆkj

l

jk

l

k x

j

NkP

Ps

)('

)1(

'

)1()1( ˆ|)(|

1ˆˆ

jk

l

kj

l

k

l

jk syj

sx

sk

yj

Input Data

Measured Data

xjk

jkx

ks


35

GaussianGaussian--MAPMAP



36

Emission TomographyEmission Tomography

y ~ Poisson(H )

: Mean of emitted photons

)(

)()(

)1( ˆ|)(|

ˆˆ

kj

m

j

m

km

k qk

)('

)(

'

)(

ˆˆ

jk

m

k

jm

j

yq

k

yj

Pixels

Measured Data

jq

kˆ


37




38




39

Transmission TomographyTransmission Tomography

y ~ Poisson(I0exp(-H ))

: photon attenuation )(

)(

)()()1( ln1ˆˆ

kj

m

j

kj

j

m

k

m

kq

y

z

)(

)(

0

)( ˆexpjk

m

k

m

j Iq

k

yj

Pixels

Measured Data

jq

kˆ


40


Transmission TomographyTransmission Tomography


41

Comments on DetailsComments on Details

•• Information geometry basisInformation geometry basis

•• Easily extended to arbitrary Easily extended to arbitrary HH

•• Low density Low density sparsesparse

•• Constraints in iterative decoding vs. forward Constraints in iterative decoding vs. forward modelmodel


42

OutlineOutline

• DSP ISP


algorithms





Algorithms

• Applications Revisited- Iterative decoding

- X-ray CT imaging



43

Science and technologyScience and technologypotentially yield potentially yield

6 Tb/in6 Tb/in22

courtesy R. S. courtesy R. S. IndeckIndeck

Patterned Magnetic Media


44

Advanced Recording MediaAdvanced Recording MediaBluBlu--Ray DiscRay Disc

Next-generation Optical Disc Video Recording Format


45

2D Intersymbol Interference2D Intersymbol Interference

1111

11

11

11

1111

21

21

11211

kkkk

k

xxx

x

xxx

1111110

110

12

1111110

100100

kkkkkk

kkkkkk

k

kk

k

rrrr

rrrr

r

rrrr

rrrw(i,j)

25.05.0

5.01h

Includes

Guard Band

jijijijijiji wxxxxr ,1,11,,1,, 25.05.05.0

Singla et al., “Iterative decoding and equalization for 2-D recording channels,” IEEE Trans.

Magn., Sept. 2002.


46

Full Graph Message PassingFull Graph Message Passing

Measured Data Nodes (r)

Codeword Bit Nodes (x)

Check Nodes (z)

jijijijijiji wxxxxr ,1,11,,1,, 25.05.05.025.05.0

5.01h


49

Full Graph Message Passing

-6

-5

-4

-3

-2

-1

0

0 0.5 1 1.5 2 2.5 3

SNR [dB]

Bit e

rro

r ra

te, L

og

Ba

se

10

ISI-free

Full Graph_50

Full Graph Message PassingFull Graph Message Passing



50

Full Graph AnalysisFull Graph Analysis

Length 4 cycles present which degrade performance Length 4 cycles present which degrade performance of messageof message--passing algorithm passing algorithm

x(i+2,j) x(i+2,j+1)

x(i+1,j) x(i+1,j+1)

x(i,j) x(i,j+1)

x(i+2,j+2)

x(i+1,j+2)

x(i,j+2)

r(i+1,j+1)

r(i,j+1)r(i,j)

r(i+1,j)

From

Check

Nodes

Kschischang et al., “Factor graphs and the sum-product algorithm,” IEEE Trans. Inform.

Theory, Feb. 2001.


51

Ordered Subsets Message Passing Ordered Subsets Message Passing

From Imaging From Imaging –– Data set is grouped into subsets to Data set is grouped into subsets to increase rate of convergence of image increase rate of convergence of image reconstruction algorithmsreconstruction algorithms

For Decoding For Decoding –– Measured data is grouped into Measured data is grouped into subsets to eliminate short length cycles in the subsets to eliminate short length cycles in the Channel ISI graphChannel ISI graph

H. M. Hudson, and R. S. Larkin, “Accelerated image reconstruction using ordered subsets

of projection data,” IEEE Trans. Medical Imaging, Dec. 1994


52

Labeling of data nodes into 4 subsetsLabeling of data nodes into 4 subsets

For each iteration use data nodes of one label onlyFor each iteration use data nodes of one label only

Labeled ISI GraphLabeled ISI Graph

J. A. O’Sullivan, and N. Singla, “Ordered subsets message-passing,” Int’l Symp. Inform.

Theory, Yokohama, Japan 2003.


53

Ordered Subsets Message Passing

-6

-5

-4

-3

-2

-1

0

0 0.5 1 1.5 2 2.5 3

SNR [dB]

Bit e

rror

rate

in log10

ISI-free

Ordered Subsets_200

Full Graph_50


Ordered Subsets Message PassingOrdered Subsets Message Passing


54

CT Imaging in Presence of High CT Imaging in Presence of High Density Attenuators (J. Williamson, PI)Density Attenuators (J. Williamson, PI)

Brachytherapy applicators

After-loading colpostats

for radiation oncology

Cervical cancer: 50% survival rate

Dose prediction important

Object-Constrained Computed

Tomography (OCCT)


55

Filtered Back ProjectionFiltered Back Projection

Truth FBP

FBP: inverse Radon transform


56

Transmission TomographyTransmission Tomography• Source-detector pairs indexed by y; pixels indexed by x

• Data d(y) Poisson, means g(y: ), log likelihood function

• Mean unattenuated counts I0, mean background

• Attenuation function (x,E), E energies

• Maximize over or ci; equivalently minimize I-divergence

)(),(),(exp),():(

):():(ln)()):(|(

0 yExxyhEyIyg

ygygydgdl

E x

y

X

Y

I

i

ii ExcEx1

)()(),(


57

MaximumMaximum--LikelihoodLikelihoodMinimum IMinimum I--divergencedivergence

• Poisson distribution

• Poisson distributed data loglikelihood function

• Maximization over equivalent to minimization of I-divergence

kk

kkI

kkkNP

ek

kNPk

ln)||(

!lnln)(ln

!)(

)(),(),(exp),():(

):()():(

)(ln)()):(||(

):():(ln)()):(|(

0 yExxyhEyIyg

ygydyg

ydydgdI

ygygydgdl

E x

y

y

X

Y

Y


58

Maximum Likelihood Maximum Likelihood Minimum IMinimum I--DivergenceDivergence

Difficulties: log of sum, sums in exponent

)()()(),(exp),():(

):()():(

)(ln)()):(||(

):():(ln)()):(|(

1

0 yExcxyhEyIyg

ygydyg

ydydgdI

ygygydgdl

E x

I

i

ii

y

y

X

Y

Y


59

Interpretation: Compare predicted data to measured data

via ratio of backprojections

Update estimate using a normalization constant

Comments: choice for constants; monotonic convergence;

linear convergence; fixed points satisfy Kuhn-

Tucker conditions; constraints easily incorporated

)(~

)(ˆln

)(

1)(ˆ)(ˆ

)(

)()()1(

xb

xb

xZxcxc

li

li

i

li

li

New Alternating Minimization AlgorithmNew Alternating Minimization Algorithmfor Transmission Tomographyfor Transmission Tomography


60David G. Politte

October 31, 2002

Mini CT, AM Iteration 0000001


61David G. Politte

October 31, 2002



62David G. Politte

October 31, 2002



63David G. Politte

October 31, 2002



64David G. Politte

October 31, 2002



65David G. Politte

October 31, 2002



66David G. Politte

October 31, 2002



67David G. Politte

October 31, 2002



68David G. Politte

October 31, 2002



69David G. Politte

October 31, 2002



70David G. Politte

October 31, 2002



71David G. Politte

October 31, 2002



72David G. Politte

October 31, 2002



73David G. Politte

October 31, 2002



74David G. Politte

October 31, 2002



75David G. Politte

October 31, 2002



76David G. Politte

October 31, 2002



77David G. Politte

October 31, 2002



78David G. Politte

October 31, 2002



79

Our Plans in CT ImagingOur Plans in CT Imaging

•• CTCT MultirowMultirow SinogramSinogram data:data:14081408 ×× 768768 ×× nndd ×× nnzz

– where nd is the number of detector rows and nz is the number of gantry rotations

•• Fully 3Fully 3--D Implementations for D Implementations for Quantitative CTQuantitative CT

•• SpeedSpeed--up: Ordered Subsets, up: Ordered Subsets, MultigridMultigrid Methods, Parallel Methods, Parallel Implementations on Clusters Implementations on Clusters of PCsof PCs

•• Future: PETFuture: PET--CTCT Siemens Somotom Emotion


80

Slide and data from R. Laforest and M. Mintun.

PETCT-211


81

PETCT-165

Slide and data from R. Laforest and M. Mintun.


82

Additional Algorithm/DetectorAdditional Algorithm/DetectorModel DevelopmentModel Development

•• RegularizationRegularization

•• Energy integrating detectors Energy integrating detectors

•• Finite detector size, better source modelFinite detector size, better source model

•• Finite pixel, Finite pixel, voxelvoxel sizesize

•• Average integral or average exponentialAverage integral or average exponential(arithmetic vs. geometric average)(arithmetic vs. geometric average)

•• Partial volume effectsPartial volume effects

•• MotionMotion

•• ScatteringScattering

•• Limited angle tomographyLimited angle tomography

•• Region of interestRegion of interest

•• Scanner implementations: beam hardening Scanner implementations: beam hardening correction, sampling, etc.correction, sampling, etc.

),( EyEdN


83

Computation and

Communication

Signal Processing

Information Theory

ComplexityComplexity--constrained processingconstrained processing

Signal processing on graphsSignal processing on graphs

Distributed signal processingDistributed signal processing

Distributed information theoryDistributed information theory

Distributed computation and Distributed computation and communicationcommunication

Optimal information extraction, Optimal information extraction, communication, computationcommunication, computation


84

Limits of Information TheoryLimits of Information Theory

•• Information theory provides bounds on performance Information theory provides bounds on performance of communication, compression, and data analysisof communication, compression, and data analysis

– Channel coding theorem (capacity)

– Entropy, rate-distortion theory

– Fisher information

•• Open Problems in Information TheoryOpen Problems in Information Theory

– Broadcast channel p(y1,y2|x) capacity region of achievable (R1,R2)

» Depends only on p(y1|x) and p(y2|x); degraded channel known

– Distributed source compression achievable (R1,R2, D1,D2)

•• Algorithmic information theory (Algorithmic information theory (KolmogorovKolmogorovcomplexity)complexity)

p(x,yp(x,y))

XXnn

YYnn

XXnn

ffyy

ffxx

YYnngg ^

^


85

Speculation on FutureSpeculation on Future

•• Distributed compressionDistributed compression– Compression with side information

– Analogies to information embedding

– Reduced communication rates

•• Broadcast channel models Broadcast channel models – Appropriate for motes

– Communication at different rates using a common signal

– Reduced communication rates

•• Tradeoffs in communication and computationTradeoffs in communication and computation

•• Mobile computing: cheap Mobile computing: cheap expensiveexpensive cheapcheap– mobile base station + network mobile


86

ConclusionsConclusions

•• Information Signal ProcessingInformation Signal Processing

—— DSP, Information Theory, Computation andDSP, Information Theory, Computation and

CommunicationCommunication

•• Role of Graphical ModelsRole of Graphical Models

•• Message Passing EM AlgorithmsMessage Passing EM Algorithms

•• Iterative Equalization and DecodingIterative Equalization and Decoding

•• XX--Ray CT ImagingRay CT Imaging


87

Future WorkFuture Work

Signal Processing

Information Theory

Fast algorithmsFast algorithms

Optimal communicationOptimal communication

Distributed information Distributed information theorytheory

Computation and

Communication

information signal processingjao/talks/invitedtalks/isp2.pdf · information signal processing j. a....

Documents