Besov
Bayes
Chomsky
Plato
Richard Baraniuk Rice Universitydsp.rice.edu
Joint work with Hyeokho ChoiJustin RombergMike Wakin
Multiscale Geometric Image Analysis
Low-Level Image Structures
• Smooth/texture regions
• Edge singularities along smooth curves (geometry)
geometry
texture
texture
Computational Harmonic Analysis
• Representation
Fourier sinusoids, Gabor functions, wavelets, curvelets, Laplacian pyramid
basis, framecoefficients
Computational Harmonic Analysis
• Representation
• Analysis study through structure of might extract features of interest
• Approximation uses just a few termsexploit sparsity of
basis, framecoefficients
Multiscale Image Analysis
• Analyze an image a multiple scales
• How? Zoom out and record information lost in wavelet coefficients
info 1 info 2 info 3
…
Wavelet-based Image Processing
• Standard 2-D tensor product wavelet transform
Transform-domain Modeling
• Transform-domain modeling and processing
transform
coefficient model
Transform-domain Modeling
• Transform-domain modeling and processing
transform vocabulary
coefficient model
Transform-domain Modeling
• Transform-domain modeling and processing
• Vocabulary + grammar capture image structure
• Challenging co-design problem
transform vocabulary
coefficient grammar model
Nonlinear Image Modeling
• Natural images do not form a linear space!
• Form of the “set of natural images”?
+ =
Set of Natural Images
Small: N x N sampled images comprise an
extremely small subset of RN2
Set of Natural Images
Small: N x N sampled images comprise an
extremely small subset of RN2
Set of Natural Images
Small: N x N sampled images comprise an
extremely small subset of RN2
Complicated: Manifold structure
RN2
Wedge Manifold
Wedge Manifold
3 Haar wavelets
Wedge Manifold
3 Haar wavelets
Stochastic Projections
• Projection
= “shadow”
Higher-D Stochastic Projections
• Model (partial) wavelet coefficient joint statistics
• Shadow more faithful to manifold
Manifold Projections …
Multiscale Grammars for
Wavelet Transforms
Wavelet Statistical Properties
Marginal Distribution
0
• manysmall Scoefficients
• smooth regions
Marginal Distribution
0
• a fewlarge Lcoefficients
• edgeregions
Wavelet Mixture Model
state: S or L
wc: Gaussian withS or L variance
Magnitude Correlations
• Persistenceof wc’s onquadtree
• S S
• L L
Wavelet Hidden Markov Tree
state: S or L
wc: Gaussian withS or L variance
A
Wavelet Statistical Models
• Independent wc’s w/ generalized Gaussian marginal – processing: thresholding (“coring”)– application: denoising
• Correlated wc magnitudes– zero tree image coder [Shapiro]
– estimation/quantization (EQ) model [Ramchandran, Orchard]
– JPEG2000 encoder– graphical models:
• Gaussian scale mixture [Wainwright, Simoncelli]
• hidden Markov process on quadtree (HMT)• processing: tree-structured algorithms
EM algorithm, Viterbi algorithm, …
– applications: denoising, compression, classification, …
Barbara
Barbara’s Books
Denoising Barbara’s Books
nois
y b
ooks
WT t
hre
shold
ing
DW
T H
MT
Denoising Barb’s Books
Wave
lets a
nd Subband C
oding
Vetterli
and K
ovace
vic
My
Life
as
a D
og
Pari
s H
ilton
Numeric
al Analys
is of W
avelet
Methods
Albert
Cohen
HMT Image Segmentation
HMT Image Segmentation
Zero Tree Compression
• Idea: Prune wavelet subtrees in smooth regions
– tree-structured thresholding
– spend bits on (large) “significant” wc’s
– spend no bits on (small) wc’s…
zerotree symbol Z = {wc and all decendants = 0}
Z
New Multiscale Vocabularies
• Geometrical info not explicit
• Modulations around singularities (geometry)
• Inefficient -large number of significant wc’s cluster around edge contours, no matter how smooth
Wavelet Modulations
• Wavelets are poor edge detectors• Severe modulation effects
Wavelet Modulations
• Wavelet wiggles…so its inner product with a singularity wiggles
L
S
L
S
scale
signal
Wavelet Modulation Effects
• Wavelet transform of edge
Wavelet Modulation Effects
• Wavelet transform of edge
• Seek amplitude/envelope
• To extract amplitude need coherent representation
1-D Complex Wavelets [Grossman, Morlet, Lina, Abry, Flandrin, Mallat, Bernard, Kingsbury, Selesnick, Fernandes, …]
• real waveleteven symmetry
imaginary waveletodd symmetry
• Hilbert transform pair(complex Gabor atom)
• 2x redundant tight frame• Alias-free; shift-invariant• Coherent wavelet representation (magnitude/phase)
2-D Complex Wavelets[Lina, Kingsbury, Selesnick, …]
• Even/odd real/imag symmetry• Almost Hilbert transform pair
(complex Gabor atom)• Almost shift invariant• Compute using 1-D CWT
-75 +75 +45 +15 -15 -45
real
imag
• 4x redundant tight frame• 6 directional subbands
aligned along 6 1-D manifold directions
• Magnitude/phase
Wavelet Image Processing
Coherent Wavelet Processing
real part
imaginarypart
+i
Coherent Wavelet Processing
|magnitude|
x exp(i phase)
Coherent Image Processing [Lina]
magnitude
FFT
Coherent Image Processing [Lina]
magnitude phase
FFT
Coherent Image Processing [Lina]
magnitude phase
FFT
CWT
Coherent Wavelet Processing
feature magnitude phase
1 edge L coherent
“speckle” L incoherent> 1 edge
smooth S undefined
Coherent Segmentation
feature magnitude phase
1 edge L coherent
“speckle” L incoherent> 1 edge
smooth S undefined
Edge Geometry Extraction
r
• Extract 1-D edge geometry from piecewise smooth image
wedgelet
• Goal: Extract r and locally
Edge Geometry Extraction
r• CWT magnitude encodes angle
• CWT phase encodes offset r
Edge Geometry Extraction in 2-Doriginal estimated
Multiscale Edge Geometry Grammarfor Wavelet Transforms
• Inefficient -large number of significant WCs cluster around edge contours, no matter how smooth
Cartoon Processing
2-D Dyadic Partitions for Cartoons
• C2 smooth regions
• separated by edge discontinuities along C2 curves
2-D Dyadic Partitions for Cartoons
• Multiscale analysis
• Partition; not a basis/frame
• Zoom in by factor of 2 each scale
2-D Dyadic Partition = Quadtree
• Multiscale analysis
• Partition; not a basis/frame
• Zoom in by factor of 2 each scale
• Each parent node has 4 children at next finer scale
Wedgelet Representation [Donoho]
• Build a cartoon using wedgelets on dyadic squares
• Choose orientation (r,) from finite dictionary (toroidal sampling)
• Quad-tree structure
deeper in tree finer curve approximation
r
2-D Wedgelets
2-D Wedgelets
Wedgelet Representation
• Prune wedgelet quadtree to approximate local geometry (adaptive)
• Decorate leaves with (r,) parameters
Wedgelet Inference
• Find representation / prune tree to balance a fidelity vs. complexity trade-off
• For Comp(W) – need a model for the wedgelet representation(quadtree + (r,))
• Donoho: Comp(W) = #leaves
#Leaves Complexity Penalty
• Accounts for wedgelet partition size, but not wedgelet orientation
#leaves #leaves
“smooth”“simple”
“rough”“complex”
Multiscale Geometry Model (MGM)
Multiscale Geometry Model (MGM)
• Decorate each tree node with orientation (r,) and then model dependencies thru scale
• Insight: Smooth curve Geometric innovations small at fine scales
• Model: Favor small innovations over large innovations (statistically)
Multiscale Geometry Model (MGM)
• Wavelet-like geometry model: coarse-to-fine prediction
– model parent-to-child transitions of orientations
small innovations
large
MGM
• Wavelet-like geometry model: coarse-to-fine prediction
– model parent-to-child transitions of orientations
• Markov-1 statistical model– state = (r,) orientation of wedgelet – parent-to-child state transition matrix
MGM
• Markov-1 statistical model
• Joint wedgelet Markov probability model:
• Complexity = Shannon codelength = = number of bits to encode
MGM and Edge Smoothness
“smooth”“simple”
“rough”“complex”
MGM Inference
• Find representation / prune tree to balance the fidelity vs. complexity trade-off
• Efficient O(N) solution via dynamic programming
MGM Approximation
• Find representation / prune tree to balance the fidelity vs. complexity trade-off
• Optimal L2 error decay rate for cartoons
single scale multiscale
• Choosing wedgelets = rate-distortion optimization
Wedgelet Coding of Cartoon Images
Shannon code length
to encode
Joint Texture/Geometry Modeling
• Dictionary D = {wavelets} U {wedgelets}
• Representation tradeoff: texture vs. geometry
Zero Tree Compression
• Idea: Prune wavelet subtrees in smooth regions
– tree-structured thresholding
– spend bits on (large) “significant” wc’s
– spend no bits on (small) wc’s…
zerotree symbol Z = {wc and all decendants = 0}
Z
Zero Tree Compression
• Label pruned wavelet quadtree with 2 states
zero-tree - smooth region (prune)significant - edge/texture region (keep)
smooth
smooth
smooth
not smooth
S
S
S
SS
S
ZZ
Z
Z: all wc’s below=0
Wedgelet Trees for Geometry
• Label pruned wavelet quadtree with 3 states
zero-tree - smooth region (prune)geometry - edge region (prune)significant - texture region (keep)
• Optimize placement of Z, G, S by dyn. programming
smooth smooth
texture
geometry
S
S
S
SS
S
ZZ
G
G: wc’s below are wc’s of a wedgelet tree
Optimality of Wedgelets + Wavelets
• Theorem For C2 / C2 images, optimal asymptotic L2 error decay
and near-optimal rate-distortion
C2 contour (1-d)
C2 texture (2-d)
Practical Image Coder• Wedgelet-SFQ (WSFQ) coder
builds on SFQ coder [Xiong, Ramchandran, Orchard]
• At low bit rates, often significant improvement in visual quality over SFQ and JPEG-2k (much sharper edges)
• Bonus: WSFQ representation contains explicit geometry information
Improvement overJPEG-2000PSNR dB
bits per pixel
WSFQ
SFQ
Wet Paint Test Image
JPEG Compressed (DCT)PS
NR
= 2
6.3
2dB
@ 0
.01
03
bpp
JPEG2000 Compressed (Wavelets)PS
NR
= 2
9.7
7dB
@ 0
.01
03
bp
p
WSFQ CompressedPS
NR
= 3
0.1
9dB
@ 0
.01
02
bpp
WSFQ Contour Trees
ori
gin
al
JPEG
20
00
WS
FQS
FQ
Gain Over JPEG2000 – Cameraman
WSFQ
SFQ
40% MSE improvement
ExtensionsS
S
S
SS
S
SZ
W
Z D
W
zerotree
wedgeprint
S S
S S
ExtensionsS
B
S
BB
B
DZ
W
Z D
W
zerotree
wedgeprint
B B
B B
barprint
DCTprint
“Coifman’s Dream”
ExtensionsS
S
S
SS
S
SZ
W
Z D
W
zerotree
wedgeprint
S S
S S
ExtensionsS
B
S
BB
B
DZ
W
Z D
W
zerotree
wedgeprint
B B
B B
barprint
DCTprint
“Coifman’s Dream”
ExtensionsS
B
S
BV
B
DZ
W
Z D
W
zerotree
wedgeprint
B S
B B
Eeroprint
barprint
DCTprint
Bars / Ridges
16x16 image block
Multiple Wedgelet Coding
80 bits to jointly encode 5 wedgelets
“Barlet” Coding
22 bits to encode 1 barlet
(fat edgelet/beamlet)
Periodic Textures
DCTprint
5dB
DCT
wavelets
# terms
PS
NR
32x32 block = 1024 pixels
DCTprint
4x fewer coefficients
DCT
wavelets
# terms
PS
NR
32x32 block = 1024 pixels
Summary
• Wavelets and other bases are vocabularies• Image structure induces coefficient grammar• Statistical models
– extract higher-dimensional joint statistics via quadtrees, wedgelets, wedgeprints, …
• New vocabularies– curvelets, bandelets, complex wavelets, …– random projections [Candes, Romberg, Tao; Donoho]
• Grand challenge– rather than low-dimensional projections, work on the
manifold– potential for image processing, compression, ATR, …