besov bayes chomsky plato richard baraniuk rice university dsp.rice.edu joint work with hyeokho choi...

Besov

Bayes

Chomsky

Plato

Richard Baraniuk Rice Universitydsp.rice.edu

Joint work with Hyeokho ChoiJustin RombergMike Wakin

Multiscale Geometric Image Analysis

Low-Level Image Structures

• Smooth/texture regions

• Edge singularities along smooth curves (geometry)

geometry

texture

texture

Computational Harmonic Analysis

• Representation

Fourier sinusoids, Gabor functions, wavelets, curvelets, Laplacian pyramid

basis, framecoefficients

Computational Harmonic Analysis

• Representation

• Analysis study through structure of might extract features of interest

• Approximation uses just a few termsexploit sparsity of

basis, framecoefficients

Multiscale Image Analysis

• Analyze an image a multiple scales

• How? Zoom out and record information lost in wavelet coefficients

info 1 info 2 info 3

…

Wavelet-based Image Processing

• Standard 2-D tensor product wavelet transform

Transform-domain Modeling

• Transform-domain modeling and processing

transform

coefficient model

Transform-domain Modeling

• Transform-domain modeling and processing

transform vocabulary

coefficient model

Transform-domain Modeling

• Transform-domain modeling and processing

• Vocabulary + grammar capture image structure

• Challenging co-design problem

transform vocabulary

coefficient grammar model

Nonlinear Image Modeling

• Natural images do not form a linear space!

• Form of the “set of natural images”?

+ =

Set of Natural Images

Small: N x N sampled images comprise an

extremely small subset of RN2

Set of Natural Images

Small: N x N sampled images comprise an

extremely small subset of RN2

Set of Natural Images

Small: N x N sampled images comprise an

extremely small subset of RN2

Complicated: Manifold structure

RN2

Wedge Manifold

Wedge Manifold

3 Haar wavelets

Wedge Manifold

3 Haar wavelets

Stochastic Projections

• Projection

= “shadow”

Higher-D Stochastic Projections

• Model (partial) wavelet coefficient joint statistics

• Shadow more faithful to manifold

Manifold Projections …

Multiscale Grammars for

Wavelet Transforms

Wavelet Statistical Properties

Marginal Distribution

0

• manysmall Scoefficients

• smooth regions

Marginal Distribution

0

• a fewlarge Lcoefficients

• edgeregions

Wavelet Mixture Model

state: S or L

wc: Gaussian withS or L variance

Magnitude Correlations

• Persistenceof wc’s onquadtree

• S S

• L L

Wavelet Hidden Markov Tree

state: S or L

wc: Gaussian withS or L variance

A

Wavelet Statistical Models

• Independent wc’s w/ generalized Gaussian marginal – processing: thresholding (“coring”)– application: denoising

• Correlated wc magnitudes– zero tree image coder [Shapiro]

– estimation/quantization (EQ) model [Ramchandran, Orchard]

– JPEG2000 encoder– graphical models:

• Gaussian scale mixture [Wainwright, Simoncelli]

• hidden Markov process on quadtree (HMT)• processing: tree-structured algorithms

EM algorithm, Viterbi algorithm, …

– applications: denoising, compression, classification, …

Barbara

Barbara’s Books

Denoising Barbara’s Books

nois

y b

ooks

WT t

hre

shold

ing

DW

T H

MT

Denoising Barb’s Books

Wave

lets a

nd Subband C

oding

Vetterli

and K

ovace

vic

My

Life

as

a D

og

Pari

s H

ilton

Numeric

al Analys

is of W

avelet

Methods

Albert

Cohen

HMT Image Segmentation

HMT Image Segmentation

Zero Tree Compression

• Idea: Prune wavelet subtrees in smooth regions

– tree-structured thresholding

– spend bits on (large) “significant” wc’s

– spend no bits on (small) wc’s…

zerotree symbol Z = {wc and all decendants = 0}

Z

New Multiscale Vocabularies

• Geometrical info not explicit

• Modulations around singularities (geometry)

• Inefficient -large number of significant wc’s cluster around edge contours, no matter how smooth

Wavelet Modulations

• Wavelets are poor edge detectors• Severe modulation effects

Wavelet Modulations

• Wavelet wiggles…so its inner product with a singularity wiggles

L

S

L

S

scale

signal

Wavelet Modulation Effects

• Wavelet transform of edge

Wavelet Modulation Effects

• Wavelet transform of edge

• Seek amplitude/envelope

• To extract amplitude need coherent representation

1-D Complex Wavelets [Grossman, Morlet, Lina, Abry, Flandrin, Mallat, Bernard, Kingsbury, Selesnick, Fernandes, …]

• real waveleteven symmetry

imaginary waveletodd symmetry

• Hilbert transform pair(complex Gabor atom)

• 2x redundant tight frame• Alias-free; shift-invariant• Coherent wavelet representation (magnitude/phase)

2-D Complex Wavelets[Lina, Kingsbury, Selesnick, …]

• Even/odd real/imag symmetry• Almost Hilbert transform pair

(complex Gabor atom)• Almost shift invariant• Compute using 1-D CWT

-75 +75 +45 +15 -15 -45

real

imag

• 4x redundant tight frame• 6 directional subbands

aligned along 6 1-D manifold directions

• Magnitude/phase

Wavelet Image Processing

Coherent Wavelet Processing

real part

imaginarypart

+i

Coherent Wavelet Processing

|magnitude|

x exp(i phase)

Coherent Image Processing [Lina]

magnitude

FFT

Coherent Image Processing [Lina]

magnitude phase

FFT

Coherent Image Processing [Lina]

magnitude phase

FFT

CWT

Coherent Wavelet Processing

feature magnitude phase

1 edge L coherent

“speckle” L incoherent> 1 edge

smooth S undefined

Coherent Segmentation

feature magnitude phase

1 edge L coherent

“speckle” L incoherent> 1 edge

smooth S undefined

Edge Geometry Extraction

r

• Extract 1-D edge geometry from piecewise smooth image

wedgelet

• Goal: Extract r and locally

Edge Geometry Extraction

r• CWT magnitude encodes angle

• CWT phase encodes offset r

Edge Geometry Extraction in 2-Doriginal estimated

Multiscale Edge Geometry Grammarfor Wavelet Transforms

• Inefficient -large number of significant WCs cluster around edge contours, no matter how smooth

Cartoon Processing

2-D Dyadic Partitions for Cartoons

• C2 smooth regions

• separated by edge discontinuities along C2 curves

2-D Dyadic Partitions for Cartoons

• Multiscale analysis

• Partition; not a basis/frame

• Zoom in by factor of 2 each scale

2-D Dyadic Partition = Quadtree

• Multiscale analysis

• Partition; not a basis/frame

• Zoom in by factor of 2 each scale

• Each parent node has 4 children at next finer scale

Wedgelet Representation [Donoho]

• Build a cartoon using wedgelets on dyadic squares

• Choose orientation (r,) from finite dictionary (toroidal sampling)

• Quad-tree structure

deeper in tree finer curve approximation

r

2-D Wedgelets

2-D Wedgelets

Wedgelet Representation

• Prune wedgelet quadtree to approximate local geometry (adaptive)

• Decorate leaves with (r,) parameters

Wedgelet Inference

• Find representation / prune tree to balance a fidelity vs. complexity trade-off

• For Comp(W) – need a model for the wedgelet representation(quadtree + (r,))

• Donoho: Comp(W) = #leaves

#Leaves Complexity Penalty

• Accounts for wedgelet partition size, but not wedgelet orientation

#leaves #leaves

“smooth”“simple”

“rough”“complex”

Multiscale Geometry Model (MGM)

Multiscale Geometry Model (MGM)

• Decorate each tree node with orientation (r,) and then model dependencies thru scale

• Insight: Smooth curve Geometric innovations small at fine scales

• Model: Favor small innovations over large innovations (statistically)

Multiscale Geometry Model (MGM)

• Wavelet-like geometry model: coarse-to-fine prediction

– model parent-to-child transitions of orientations

small innovations

large

MGM

• Wavelet-like geometry model: coarse-to-fine prediction

– model parent-to-child transitions of orientations

• Markov-1 statistical model– state = (r,) orientation of wedgelet – parent-to-child state transition matrix

MGM

• Markov-1 statistical model

• Joint wedgelet Markov probability model:

• Complexity = Shannon codelength = = number of bits to encode

MGM and Edge Smoothness

“smooth”“simple”

“rough”“complex”

MGM Inference

• Find representation / prune tree to balance the fidelity vs. complexity trade-off

• Efficient O(N) solution via dynamic programming

MGM Approximation

• Find representation / prune tree to balance the fidelity vs. complexity trade-off

• Optimal L2 error decay rate for cartoons

single scale multiscale

• Choosing wedgelets = rate-distortion optimization

Wedgelet Coding of Cartoon Images

Shannon code length

to encode

Joint Texture/Geometry Modeling

• Dictionary D = {wavelets} U {wedgelets}

• Representation tradeoff: texture vs. geometry

Zero Tree Compression

• Idea: Prune wavelet subtrees in smooth regions

– tree-structured thresholding

– spend bits on (large) “significant” wc’s

– spend no bits on (small) wc’s…

zerotree symbol Z = {wc and all decendants = 0}

Z

Zero Tree Compression

• Label pruned wavelet quadtree with 2 states

zero-tree - smooth region (prune)significant - edge/texture region (keep)

smooth

smooth

smooth

not smooth

S

S

S

SS

S

ZZ

Z

Z: all wc’s below=0

Wedgelet Trees for Geometry

• Label pruned wavelet quadtree with 3 states

zero-tree - smooth region (prune)geometry - edge region (prune)significant - texture region (keep)

• Optimize placement of Z, G, S by dyn. programming

smooth smooth

texture

geometry

S

S

S

SS

S

ZZ

G

G: wc’s below are wc’s of a wedgelet tree

Optimality of Wedgelets + Wavelets

• Theorem For C2 / C2 images, optimal asymptotic L2 error decay

and near-optimal rate-distortion

C2 contour (1-d)

C2 texture (2-d)

Practical Image Coder• Wedgelet-SFQ (WSFQ) coder

builds on SFQ coder [Xiong, Ramchandran, Orchard]

• At low bit rates, often significant improvement in visual quality over SFQ and JPEG-2k (much sharper edges)

• Bonus: WSFQ representation contains explicit geometry information

Improvement overJPEG-2000PSNR dB

bits per pixel

WSFQ

SFQ

Wet Paint Test Image

JPEG Compressed (DCT)PS

NR

= 2

6.3

2dB

@ 0

.01

03

bpp

JPEG2000 Compressed (Wavelets)PS

NR

= 2

9.7

7dB

@ 0

.01

03

bp

p

WSFQ CompressedPS

NR

= 3

0.1

9dB

@ 0

.01

02

bpp

WSFQ Contour Trees

ori

gin

al

JPEG

20

00

WS

FQS

FQ

Gain Over JPEG2000 – Cameraman

WSFQ

SFQ

40% MSE improvement

ExtensionsS

S

S

SS

S

SZ

W

Z D

W

zerotree

wedgeprint

S S

S S

ExtensionsS

B

S

BB

B

DZ

W

Z D

W

zerotree

wedgeprint

B B

B B

barprint

DCTprint

“Coifman’s Dream”

ExtensionsS

S

S

SS

S

SZ

W

Z D

W

zerotree

wedgeprint

S S

S S

ExtensionsS

B

S

BB

B

DZ

W

Z D

W

zerotree

wedgeprint

B B

B B

barprint

DCTprint

“Coifman’s Dream”

ExtensionsS

B

S

BV

B

DZ

W

Z D

W

zerotree

wedgeprint

B S

B B

Eeroprint

barprint

DCTprint

Bars / Ridges

16x16 image block

Multiple Wedgelet Coding

80 bits to jointly encode 5 wedgelets

“Barlet” Coding

22 bits to encode 1 barlet

(fat edgelet/beamlet)

Periodic Textures

DCTprint

5dB

DCT

wavelets

# terms

PS

NR

32x32 block = 1024 pixels

DCTprint

4x fewer coefficients

DCT

wavelets

# terms

PS

NR

32x32 block = 1024 pixels

Summary

• Wavelets and other bases are vocabularies• Image structure induces coefficient grammar• Statistical models

– extract higher-dimensional joint statistics via quadtrees, wedgelets, wedgeprints, …

• New vocabularies– curvelets, bandelets, complex wavelets, …– random projections [Candes, Romberg, Tao; Donoho]

• Grand challenge– rather than low-dimensional projections, work on the

manifold– potential for image processing, compression, ATR, …