information signal processingjao/talks/invitedtalks/isp2.pdf · information signal processing j. a....

Information Signal ProcessingInformation Signal Processing

Joseph A. OJoseph A. O’’SullivanSullivan

Electronic Systems and Signals Research Laboratory

Center for Security TechnologiesDepartment of Electrical and Systems Engineering

Washington Universityjao@ee.wustl.edu

http://essrl.wustl.edu/~jaoSupported by: ONR, NSF, NIH, Boeing Foundation, DARPASpecial thanks to Naveen Singla

J. A. O’Sullivan. 11/21/2003Information Signal Processing

CollaboratorsCollaborators

Jasenka BenacMichael D. DeVoreAndrew (Lichun) LiClayton MillerLee MontagninoRyan Murphy Natalia SchmidNaveen SinglaBrandon WestoverShenyu Yan

G. James BlaineRoger ChamberlainMark FranklinDaniel R. FuhrmannRonald S. IndeckChenyang LuPierre Moulin, UIUCMarcel MullerRobert PlessDavid G. PolitteChrysanthe PrezaAndrew Singer, UIUC Donald L. SnyderBruce R. WhitingJeffrey F. Williamson, VCULihao Xu

Faculty Current and Former Students

OutlineOutline

• DSP ISP

- Data models, computational models,

algorithms

- Central role of information

• Graphical Data Models- X-ray CT imaging

- Iterative decoding

• Message Passing EM

Algorithms

• Applications Revisited

• Speculation on Trends• Conclusions

Signal Processing

Information TheoryComputation and

Communication

FFTFFT

MultiresolutionMultiresolutionanalysisanalysis

Signal Processing

Information Theory

Numerical analysisNumerical analysis

Processors: parallel, Processors: parallel, ASIC, etc.ASIC, etc.

Systolic architecturesSystolic architectures

FFTWFFTW

Computation and

Communication

Signal Processing

Information Theory

Numerical analysisNumerical analysis

Processors and Processors and architecturesarchitectures

FFT, FFTWFFT, FFTW

Transversal filtersTransversal filters

MRAMRA

Complexity theoryComplexity theory

Graphical modelsGraphical models

KalmanKalman filtersfilters

CompressionCompression

Computation and

Communication

Computation and

Communication

Information Signal ProcessingInformation Signal Processing

Signal Processing

Information Theory

ComplexityComplexity--constrained processingconstrained processing

Signal processing on graphsSignal processing on graphs

Distributed signal processingDistributed signal processing

Distributed information theoryDistributed information theory

Distributed computation and Distributed computation and communicationcommunication

Optimal information extraction, Optimal information extraction, communication, computationcommunication, computation

Distributed sensing,

communication, computation““The architecture for a The architecture for a

fully netted maritime forcefully netted maritime force””

Wireless Sensor NetworksWireless Sensor Networks

http://www.greatduckisland.net/index.php

Great Duck Island Habitat

OutlineOutline

• DSP ISP

algorithms

Algorithms

Graphical Data ModelsGraphical Data Models

•• Model 1:Model 1: y = Hsy = Hs– y is n × 1, s is m × 1,

and H is n × m

– yj depends on sk if hjk 0

– Defines a graphical model

•• Model 2:Model 2:

– Neighborhood structure

– Bipartite graph model

•• Model 3:Model 3:– RVs on edges of graph

kj jksypp ))(,|()|( sy

kjk sxpp)(

)|()|( sx

jkj jkxypp ))(,|()|( xy

Tomography

Nonrandom Graphs

Line integrals through patient

Quantization point spread function

weights on edges of graph

Helps organize computations

Siemens Somotom Emotion

Computational ModelsComputational Models

System model accounts for:

• Information extraction

problem definition

• Compression of sensor data

• Network throughput

• Processor cycles per

instruction

• Size of processor local

memory

• Communication bandwidth

of each link

• etc.

Computational ModelsComputational Models

• Local resources plus remote

• Communicate observation as well as classification- Human in the loop

- Remote contribution to classification when available

• Dynamic resource availability

• Sequence of partial classifications (an, n)

Progress: Computational Graph Progress: Computational Graph Same as Data Model GraphSame as Data Model Graph

•• Message passing algorithmsMessage passing algorithms– Pearl’s belief propagation

– Iterative decoding

» Turbo-codes, parallel concatenated codes

» Low density parity check codes

» Repeat-accumulate codes, serial concatenated codes

– Iterative equalization and decoding

•• ExpectationExpectation--Maximization (EM) AlgorithmsMaximization (EM) Algorithms– Graphical models

– General problem

– Gaussian, Poisson (emission tomography, transmission tomography)

– Abstract examples on random graphs

10011010101000

01010101010001

10100001010101

01100110010100

10010101001010

00101000101011

01001010100110

H Regular (3,6) n=14

Random Graphs

Comment: LDPC parity check matrix

001010100000

100100000010

010001001000

000100010001

100000001100

001000100010

010001010000

000010000101

H Regular (2,3) n=12

Random Graphs

10011010101000

01010101010001

10100001010101

10101010010100

10010101001010

00101000101101

01001001010110

H Irregular n=14

Random Graphs

stateg

statef

Iterative Decoding Message PassingIterative Decoding Message Passing

2tanh)1(

Codeword Bit Nodes

Check Nodes

Iterative Decoding Message PassingIterative Decoding Message Passing

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8

SNR [dB]

ISI-free

BIAWGN Capacity

[10000,5000] regular (3,6) matrix[10000,5000] regular (3,6) matrix

OutlineOutline

• DSP ISP

algorithms

Algorithms

ML Problem: xsxxys

dpp )|()|(lnmax

EM Algorithm: )|()|(

),|(ln),|(minmin

syxsyx

EM AlgorithmEM Algorithm

Hidden Data

Incomplete Data

VariationalVariational RepresentationsRepresentations

•• Convex Decomposition LemmaConvex Decomposition Lemma. Let f be convex. Then

• Special Case: f is ln

• Basis for EM; see also De Pierro, Lange, Fessler

lnminln

EM AlgorithmEM Algorithm

')|'()'|(

)|()|(),|(

)()()1(

sxxysyx

xsxsyxss

dp, mmm )|(ln)|(argmax )()1()1(

Assume the factorizations:

kjk sxpp)(

)|()|( sx

jkj jkxypp ))(,|()|( xy

These computations become local and thus message passing

In general, these are global computations

Message Passing EM AlgorithmMessage Passing EM Algorithm

)|())(',|(

kjkjkj

dxsxpjkxyp

k dxsxp,yxsk

)|(ln)|(argmax )()1()1(s

Input Data

Measured Data

)( jkx

GaussianGaussian--MAPMAP

)1( ˆ

|)(|2|)(|

)1()1( ˆ|)(|

jk syj

Input Data

Measured Data

GaussianGaussian--MAPMAP

Emission TomographyEmission Tomography

y ~ Poisson(H )

: Mean of emitted photons

)1( ˆ|)(|

Pixels

Measured Data

Transmission TomographyTransmission Tomography

y ~ Poisson(I0exp(-H ))

: photon attenuation )(

)()()1( ln1ˆˆ

)( ˆexpjk

Pixels

Measured Data

Transmission TomographyTransmission Tomography

Comments on DetailsComments on Details

•• Information geometry basisInformation geometry basis

•• Easily extended to arbitrary Easily extended to arbitrary HH

•• Low density Low density sparsesparse

•• Constraints in iterative decoding vs. forward Constraints in iterative decoding vs. forward modelmodel

OutlineOutline

• DSP ISP

algorithms

Algorithms

• Applications Revisited- Iterative decoding

- X-ray CT imaging

Science and technologyScience and technologypotentially yield potentially yield

6 Tb/in6 Tb/in22

courtesy R. S. courtesy R. S. IndeckIndeck

Patterned Magnetic Media

Advanced Recording MediaAdvanced Recording MediaBluBlu--Ray DiscRay Disc

Next-generation Optical Disc Video Recording Format

2D Intersymbol Interference2D Intersymbol Interference

1111110

100100

kkkkkk

rrrw(i,j)

25.05.0

Includes

Guard Band

jijijijijiji wxxxxr ,1,11,,1,, 25.05.05.0

Singla et al., “Iterative decoding and equalization for 2-D recording channels,” IEEE Trans.

Magn., Sept. 2002.

Full Graph Message PassingFull Graph Message Passing

Measured Data Nodes (r)

Codeword Bit Nodes (x)

Check Nodes (z)

jijijijijiji wxxxxr ,1,11,,1,, 25.05.05.025.05.0

Full Graph Message Passing

0 0.5 1 1.5 2 2.5 3

SNR [dB]

ISI-free

Full Graph_50

Full Graph Message PassingFull Graph Message Passing

Full Graph AnalysisFull Graph Analysis

Length 4 cycles present which degrade performance Length 4 cycles present which degrade performance of messageof message--passing algorithm passing algorithm

x(i+2,j) x(i+2,j+1)

x(i+1,j) x(i+1,j+1)

x(i,j) x(i,j+1)

x(i+2,j+2)

x(i+1,j+2)

x(i,j+2)

r(i+1,j+1)

r(i,j+1)r(i,j)

r(i+1,j)

Kschischang et al., “Factor graphs and the sum-product algorithm,” IEEE Trans. Inform.

Theory, Feb. 2001.

Ordered Subsets Message Passing Ordered Subsets Message Passing

From Imaging From Imaging –– Data set is grouped into subsets to Data set is grouped into subsets to increase rate of convergence of image increase rate of convergence of image reconstruction algorithmsreconstruction algorithms

For Decoding For Decoding –– Measured data is grouped into Measured data is grouped into subsets to eliminate short length cycles in the subsets to eliminate short length cycles in the Channel ISI graphChannel ISI graph

H. M. Hudson, and R. S. Larkin, “Accelerated image reconstruction using ordered subsets

of projection data,” IEEE Trans. Medical Imaging, Dec. 1994

Labeling of data nodes into 4 subsetsLabeling of data nodes into 4 subsets

For each iteration use data nodes of one label onlyFor each iteration use data nodes of one label only

Labeled ISI GraphLabeled ISI Graph

J. A. O’Sullivan, and N. Singla, “Ordered subsets message-passing,” Int’l Symp. Inform.

Theory, Yokohama, Japan 2003.

Ordered Subsets Message Passing

0 0.5 1 1.5 2 2.5 3

SNR [dB]

in log10

ISI-free

Ordered Subsets_200

Full Graph_50

Ordered Subsets Message PassingOrdered Subsets Message Passing

CT Imaging in Presence of High CT Imaging in Presence of High Density Attenuators (J. Williamson, PI)Density Attenuators (J. Williamson, PI)

Brachytherapy applicators

After-loading colpostats

for radiation oncology

Cervical cancer: 50% survival rate

Dose prediction important

Object-Constrained Computed

Tomography (OCCT)

Filtered Back ProjectionFiltered Back Projection

Truth FBP

FBP: inverse Radon transform

Transmission TomographyTransmission Tomography• Source-detector pairs indexed by y; pixels indexed by x

• Data d(y) Poisson, means g(y: ), log likelihood function

• Mean unattenuated counts I0, mean background

• Attenuation function (x,E), E energies

• Maximize over or ci; equivalently minimize I-divergence

)(),(),(exp),():(

):():(ln)()):(|(

0 yExxyhEyIyg

ygygydgdl

ii ExcEx1

)()(),(

MaximumMaximum--LikelihoodLikelihoodMinimum IMinimum I--divergencedivergence

• Poisson distribution

• Poisson distributed data loglikelihood function

• Maximization over equivalent to minimization of I-divergence

ln)||(

!lnln)(ln

)(),(),(exp),():(

):()():(

)(ln)()):(||(

):():(ln)()):(|(

0 yExxyhEyIyg

ygydyg

ydydgdI

ygygydgdl

Maximum Likelihood Maximum Likelihood Minimum IMinimum I--DivergenceDivergence

Difficulties: log of sum, sums in exponent

)()()(),(exp),():(

):()():(

)(ln)()):(||(

):():(ln)()):(|(

0 yExcxyhEyIyg

ygydyg

ydydgdI

ygygydgdl

Interpretation: Compare predicted data to measured data

via ratio of backprojections

Update estimate using a normalization constant

Comments: choice for constants; monotonic convergence;

linear convergence; fixed points satisfy Kuhn-

Tucker conditions; constraints easily incorporated

)(ˆln

1)(ˆ)(ˆ

)()()1(

xZxcxc

New Alternating Minimization AlgorithmNew Alternating Minimization Algorithmfor Transmission Tomographyfor Transmission Tomography

60David G. Politte

October 31, 2002

Mini CT, AM Iteration 0000001

61David G. Politte

October 31, 2002

62David G. Politte

October 31, 2002

63David G. Politte

October 31, 2002

64David G. Politte

October 31, 2002

65David G. Politte

October 31, 2002

66David G. Politte

October 31, 2002

67David G. Politte

October 31, 2002

68David G. Politte

October 31, 2002

69David G. Politte

October 31, 2002

70David G. Politte

October 31, 2002

71David G. Politte

October 31, 2002

72David G. Politte

October 31, 2002

73David G. Politte

October 31, 2002

74David G. Politte

October 31, 2002

75David G. Politte

October 31, 2002

76David G. Politte

October 31, 2002

77David G. Politte

October 31, 2002

78David G. Politte

October 31, 2002

Our Plans in CT ImagingOur Plans in CT Imaging

•• CTCT MultirowMultirow SinogramSinogram data:data:14081408 ×× 768768 ×× nndd ×× nnzz

– where nd is the number of detector rows and nz is the number of gantry rotations

•• Fully 3Fully 3--D Implementations for D Implementations for Quantitative CTQuantitative CT

•• SpeedSpeed--up: Ordered Subsets, up: Ordered Subsets, MultigridMultigrid Methods, Parallel Methods, Parallel Implementations on Clusters Implementations on Clusters of PCsof PCs

•• Future: PETFuture: PET--CTCT Siemens Somotom Emotion

Slide and data from R. Laforest and M. Mintun.

PETCT-211

PETCT-165

Slide and data from R. Laforest and M. Mintun.

Additional Algorithm/DetectorAdditional Algorithm/DetectorModel DevelopmentModel Development

•• RegularizationRegularization

•• Energy integrating detectors Energy integrating detectors

•• Finite detector size, better source modelFinite detector size, better source model

•• Finite pixel, Finite pixel, voxelvoxel sizesize

•• Average integral or average exponentialAverage integral or average exponential(arithmetic vs. geometric average)(arithmetic vs. geometric average)

•• Partial volume effectsPartial volume effects

•• MotionMotion

•• ScatteringScattering

•• Limited angle tomographyLimited angle tomography

•• Region of interestRegion of interest

•• Scanner implementations: beam hardening Scanner implementations: beam hardening correction, sampling, etc.correction, sampling, etc.

),( EyEdN

Computation and

Communication

Signal Processing

Information Theory

ComplexityComplexity--constrained processingconstrained processing

Signal processing on graphsSignal processing on graphs

Distributed signal processingDistributed signal processing

Distributed information theoryDistributed information theory

Distributed computation and Distributed computation and communicationcommunication

Optimal information extraction, Optimal information extraction, communication, computationcommunication, computation

Limits of Information TheoryLimits of Information Theory

•• Information theory provides bounds on performance Information theory provides bounds on performance of communication, compression, and data analysisof communication, compression, and data analysis

– Channel coding theorem (capacity)

– Entropy, rate-distortion theory

– Fisher information

•• Open Problems in Information TheoryOpen Problems in Information Theory

– Broadcast channel p(y1,y2|x) capacity region of achievable (R1,R2)

» Depends only on p(y1|x) and p(y2|x); degraded channel known

– Distributed source compression achievable (R1,R2, D1,D2)

•• Algorithmic information theory (Algorithmic information theory (KolmogorovKolmogorovcomplexity)complexity)

p(x,yp(x,y))

YYnngg ^

Speculation on FutureSpeculation on Future

•• Distributed compressionDistributed compression– Compression with side information

– Analogies to information embedding

– Reduced communication rates

•• Broadcast channel models Broadcast channel models – Appropriate for motes

– Communication at different rates using a common signal

– Reduced communication rates

•• Tradeoffs in communication and computationTradeoffs in communication and computation

•• Mobile computing: cheap Mobile computing: cheap expensiveexpensive cheapcheap– mobile base station + network mobile

ConclusionsConclusions

•• Information Signal ProcessingInformation Signal Processing

—— DSP, Information Theory, Computation andDSP, Information Theory, Computation and

CommunicationCommunication

•• Role of Graphical ModelsRole of Graphical Models

•• Message Passing EM AlgorithmsMessage Passing EM Algorithms

•• Iterative Equalization and DecodingIterative Equalization and Decoding

•• XX--Ray CT ImagingRay CT Imaging

Future WorkFuture Work

Signal Processing

Information Theory

Fast algorithmsFast algorithms

Optimal communicationOptimal communication

Distributed information Distributed information theorytheory

Computation and

Communication

information signal processingjao/talks/invitedtalks/isp2.pdf · information signal processing j. a....

Documents

benjamin c.kuo signal flow graph

graph signal processing - data science reading group 1 graph...

graph signal processing for machine learning: a review and

signal flow graph solution of deterministic and stochastic

graph signal processing techniques for analyzing aviation

graph convolutional neural networks• introduction to graph...

notes 13 signal flow graph analysis

graph signal analysis: imaging, learning,...

signal flow graph (diajarkan)

information-theoretic signal and image...

notes 15 - signal flow graph analysis

temporal graph signal decomposition

graph signal processing using deep neural networks · 2017....

graph signal processing: an introductory...

graph-based models and transforms for signal/data...

example signal flow graph analysis

submitted to proceedings of the ieee 1 graph signal

understanding the basis of graph signal processing via an

the factor graph approach to model-based signal processing

block diagrams & signal flow graphs lectures 5 & 6...