Richard G. Baraniuk Chinmay Hegde Sriram Nagaraj

Go With The FlowA New Manifold Modeling and Learning Framework for Image Ensembles

Aswin C. SankaranarayananRice University

Sensor Data Deluge

Concise Models

• Efficient processing / compression requires concise representation

• Sparsity of an individual image

pixels largewaveletcoefficients

(blue = 0)

Concise Models

• Efficient processing / compression requires concise representation

• Our interest in this talk: Collections of images

Concise Models

• Our interest in this talk: Collections of image

parameterized by q \in Q

– translations of an object q: x-offset and y-offset

– rotations of a 3D object: q pitch, roll, yaw

– wedgeletsq: orientation and offset

Concise Models

• Our interest in this talk: Collections of image

parameterized by q \in Q

– translations of an object q: x-offset and y-offset

– rotations of a 3D object: q pitch, roll, yaw

– wedgeletsq: orientation and offset

• Image articulation manifold

Image Articulation Manifold• N-pixel images:

• K-dimensional articulation space

• Thenis a K-dimensional manifoldin the ambient space

• Very concise model

articulation parameter space

Smooth IAMs• N-pixel images:

• Local isometry: image distance parameter space distance

• Linear tangent spacesare close approximationlocally

articulation parameter space

Smooth IAMs• N-pixel images:

• Local isometry: image distance parameter space distance

• Linear tangent spacesare close approximationlocally

articulation parameter space

Ex: Manifold Learning

LLEISOMAPLEHEDiff. Geo …

• K=1rotation

Ex: Manifold Learning

• K=2rotation and scale

Theory/Practice Disconnect: Smoothness

• Practical image manifolds are not smooth!

• If images have sharp edges, then manifold is everywhere non-differentiable [Donoho and Grimes]

Tangent approximations ?Isometry ?

Theory/Practice Disconnect: Smoothness

• Practical image manifolds are not smooth!

• If images have sharp edges, then manifold is everywhere non-differentiable [Donoho and Grimes]

Tangent approximations ?Isometry ?

Failure of Tangent Plane Approx.

• Ex: cross-fading when synthesizing / interpolating images that should lie on manifold

Input Image Input Image

Geodesic Linear path

Failure of Local Isometry

• Ex: translation manifold

all blue imagesare equidistantfrom the red image

• Local isometry

– satisfied only when sampling is dense

0 20 40 60 80 1000

0.5

1

1.5

2

2.5

3

3.5

Translation in [px]

Euc

lidea

n di

stan

ce

Tools for manifold processing

Geodesics, exponential maps, log-maps, Riemannian metrics, Karcher means, …

Smooth diff manifold

Algebraic manifolds Data manifolds

LLE, kNN graphs

Point cloud model

The concept ofTransport operators

))(()( xfIxIref

Beyond point cloud model for image manifolds

refIfI

Example

Example: Translation

• 2D Translation manifold

barring boundary related issues

• Set of all transport operators =

• Beyond a point cloud model– Action of the articulation is more accurate and meaningful

1I

11101 )( ),()( xxfxIxI

0I

2R

Optical Flow

• Generalizing this idea: Pixel correspondances

• Idea:OF between two images is a natural and accurate transport operator

(Figures from Ce Liu’s optical flow page)

OFfrom I1 to I2

I1 and I2

Optical Flow Transport

0

1 2

IAM

2I1

I0I

Articulations

• Consider a reference imageand a K-dimensional articulation

• Collect optical flows fromto all images reachable by aK-dimensional articulation

Optical Flow Transport

0

1 2

IAM

OFM at 0I

2I1

I0I

Articulations

• Consider a reference imageand a K-dimensional articulation

• Collect optical flows fromto all images reachable by aK-dimensional articulation

• Theorem: Collection of OFs is a smooth, K-dimensional manifold(even if IAM is not smooth)

OFM is Smooth (Rotation)

00 30 60 30 60

-80 -60 -40 -20 0 20 40 60 800

50

100

150

200

Articulation in [ ]

Inte

nsity

-80 -60 -40 -20 0 20 40 60 80-20

-10

0

10

20

Articulation in [ ]

Opt

ical

flow

vx i

n pi

xels

Articulation θ in [⁰]

Inte

nsi

ty I(θ

)O

p. flow

v(θ

)Pixel intensityat 3 points

Flow (nearly linear)

Main results

• Local model at each

• Each point on the OFM defines a transport operator– Each transport operator maps

to one of its neighbors

• For a large class of articulations, OFMs are smooth and locally isometric– Traditional manifold processing

techniques work on OFMs

0

1 2

IAM

OFM at 0I

2I1

I0I

Articulations

0I

0I

Linking it all together

0

1 2

IAM

OFM at 0I

2I1

I0I

Articulations

0e

1e 2e

Nonlinear dim. reduction

The non-differentiablity does not dissappear --- it is embedded in the mapping from OFM to the IAM.

However, this is a known map

))(()( such that },{ xfIxIIfI refref

The Story So Far…

0

1 2

IAM

OFM at 0I

2I1

I0I

Articulations

Tangent space at 0I

Articulations

0

1 2

2I1

I0I IAM

Input Image Input Image

Geodesic Linear pathIAM

OFM

OFM Synthesis

ISOMAP embedding error for OFM and IAM

2D rotations

Reference image

Manifold Learning

Manifold Learning

-6 -4 -2 0 2 4 6-5

-4

-3

-2

-1

0

1

2

3

4

5Two-dimensional Isomap embedding (with neighborhood graph).

Embedding of OFM

2D rotations

Reference image

Data196 images of two bears moving linearlyand independently

IAM OFM

TaskFind low-dimensional embedding

OFM Manifold Learning

IAM OFM

Data196 images of a cup moving on a plane

Task 1Find low-dimensional embedding

Task 2Parameter estimation for new images(tracing an “R”)

OFM ML + Parameter Estimation

• Point on the manifold such that the sum of geodesic distances to every other point is minimized

• Important concept in nonlinear data modeling, compression, shape analysis [Srivastava et al]

Karcher Mean

10 images from an IAM

ground truth KM OFM KM linear KM

Manifold Charting

• Goal: build a generative model for an entire IAM/OFM based on a small number of base images

• Ex: cube rotating about axis. All cube images can be representing using 4 reference images + OFMs

• Many applications– selection of target templates for classification– “next-view” selection for adaptive sensing applications

Greedy charting

Optimal charting

0 90 180 180 90

IAM

Summary

• IAMs a useful concise model for many image processing problems involving image collections and multiple sensors/viewpoints

• But practical IAMs are non-differentiable– IAM-based algorithms have not lived up to their promise

• Optical flow manifolds (OFMs)– smooth even when IAM is not– OFM ~ nonlinear tangent space– support accurate image synthesis, learning, charting, …

• Barely discussed here: OF enables the safe extension of differential geometry concepts– Log/Exp maps, Karcher mean, parallel transport, …

Open Questions

• Our treatment is specific to image manifolds under

brightness constancy

• What are the natural transport operators for other data manifolds?

dsp.rice.edu

Related Work

• Analytic transport operators– transport operator has group structure [Xiao and Rao 07]

[Culpepper and Olshausen 09] [Miller and Younes 01] [Tuzel et al 08]

– non-linear analytics [Dollar et al 06]

– spatio-temporal manifolds [Li and Chellappa 10]

– shape manifolds [Klassen et al 04]

• Analytic approach limited to a small class of standard image transformations (ex: affine transformations, Lie groups)

• In contrast, OFM approach works reliably with real-world image samples (point clouds) and broader class of transformations

Limitations

• Brightness constancy– Optical flow is no longer meaningful

• Occlusion– Undefined pixel flow in theory, arbitrary flow estimates in

practice– Heuristics to deal with it

• Changing backgrounds etc.– Transport operator assumption too strict– Sparse correspondences ?

Open Questions

• Theorem: random measurements stably embed aK-dim manifoldwhp [B, Wakin, FOCM ’08]

• Q: Is there an analogousresult for OFMs?

Image Articulation Manifold

• Linear tangent space at is K-dimensional

– provides a mechanism to transport along manifold

– problem: since manifold is non-differentiable, tangent approximation is poor

• Our goal: replace tangent spacewith new transport operator that respects the nonlinearity of the imaging process

Tangent space at 0I

Articulations

0

1 2

2I1

I0I IAM

OFM Implementation details

Reference Image

Pairwise distances and embedding

-5 0 5

-4

-2

0

2

4

100 200 300 400

100

200

300

400

0 2 4 6 8 100

0.1

0.2

0.3

0.4

0.5

Res

idua

l var

ianc

e

Isomap dimensionality

-5 0 5

-4

-2

0

2

4100 200 300 400

100

200

300

400

0 2 4 6 8 100

0.1

0.2

0.3

0.4

0.5

Res

idua

l var

ianc

e

Isomap dimensionality

Flow Embedding

Occlusion

• Detect occlusion using forward-backward flow reasoning

• Remove occluded pixel computations

• Heuristic --- formal occlusion handling is hard

Occluded

History of Optical Flow

• Dark ages (<1985)– special cases solved– LBC an under-determined set of linear equations

• Horn and Schunk (1985) – Regularization term: smoothness prior on the flow

• Brox et al (2005)– shows that linearization of brightness constancy (BC) is

a bad assumption– develops optimization framework to handle BC directly

• Brox et al (2010), Black et al (2010), Liu et al (2010)– practical systems with reliable code