Texture Optimization for Example-based Synthesis

Vivek Kwatra1,2

Irfan Essa1

Aaron Bobick1

Nipun Kwatra1

1GVU Center/College of Computing, Georgia Tech2Computer Science Dept., UNC Chapel Hill

Motivation

+ =

InputTexture

InputFlow Field

OutputSequence

Example-based Texture Synthesis

Example-based Texture Synthesis

1. Dynamic texturing (animation/video)

2. Greater control: external knobs, e.g., flow, shape, illumination

3. Explicit texture quality evaluation and refinement

Contributions I

• Global Optimization for texture synthesis– Global texture quality metric

– Iterative multi-scale optimization

Level 1

Level 2

Level 3

Level 4

Level 5

Level 6

Level 7

0 5 10 15 20 25 30 35 40 45 500

50

100

150

200

250

300

350

400

Number of Iterations

Contributions II

• Flow-guided texture animation– Flow incorporated into optimization

– Applications: fluid rendering, flow visualization

+ =

InputTexture

InputFlow Field

OutputSequence

Related Work (a selected sample)Lo

cal v

s. G

loba

l

Pixel vs. Patch

Non-parametricsampling

[Efros’99,Wei’00]Image Quilting

[Efros’01]

TextureOptimization

[Kwatra’05]

Histogram Matching[Heeger’95]

MRF Annealing[Paget’98]

Graphcut Textures[Kwatra’03]

Related Work (a selected sample)Lo

cal v

s. G

loba

l

Pixel vs. Patch

Non-parametricsampling

[Efros’99,Wei’00]Image Quilting

[Efros’01]

TextureOptimization

[Kwatra’05]

Histogram Matching[Heeger’95]

MRF Annealing[Paget’98]

Better Quality

Graphcut Textures[Kwatra’03]

Related Work (a selected sample)Lo

cal v

s. G

loba

l

Pixel vs. Patch

Non-parametricsampling

[Efros’99,Wei’00]Image Quilting

[Efros’01]

TextureOptimization

[Kwatra’05]

Histogram Matching[Heeger’95]

MRF Annealing[Paget’98]

Better Quality

Greater Controllability

Graphcut Textures[Kwatra’03]

Related Work

• Flow-guided Synthesis– Advected Textures [Neyret’03]

– Flow-based Video Editing [Bhat’04]

• Optimization– Image-based Priors [Fitzgibbon’03]

– Space-time Video Completion [Wexler’04]

• Flow consistency– Perceived motion similar to flow

• Texture similarity– Shape, size, orientation of texture

elements similar to source

Flow-guided Texture Animation

Target FlowFlowing Target

Source Texture

TextureSimilarity

FlowConsistency

Naive Approach

Target Flow

• Ignore Texture Similarity

• Warp each frame via flow field

X1 Xn

TargetFrames X0

Warp

Flowing Target

FlowConsistency

Source Texture

TextureSimilarity

Source Texture

TextureSimilarity

Naive Approach

Texture structure not maintained

Target FlowFlowing Target

FlowConsistency

Source Texture

TextureSimilarity

Warp

Instead of…

WarpXnX0 X1

TargetFrames

Warp

Instead of…

WarpXnX0 X1

TargetFrames

Warp + Correct

Our Approach

TargetFrames XX–

Warp

W

Correct

WarpedFrame

SynthesizedFrame

Link to original goals

“Correct” Operation

XTargetFrames X–

W

Correct

Warp

Link to original goals

“Correct” Operation

XTargetFrames X–

W

Correct

Flow ConsistencyWarp

Link to original goals

“Correct” Operation

XTargetFrames X–

W

Correct

Source

Flow Consistency

Texture Similarity

Warp

Energy = Flow Energy + Texture Energy

Optimization of Energy / Cost

XTargetFrames X–

W

Correct

Source

Flow Consistency

Texture Similarity

Warp

Flow Energy

XTargetFrames X–

Warp

W

Flow Energy

Texture Energy

• Tricky!– Pixel-by-pixel comparison of source and target

not possible

• Compare texture “elements”– Local pixel neighborhoods

• Want each target neighborhood to be similar to some source neighborhood

Texture Energy

X(target frame)

Z(source)

Energy for Single Neighborhood

p (pixel)Z(source)

X(target frame)

Energy for Single Neighborhood

Xp (neighborhood)Z(source)

X(target frame)

Energy for Single Neighborhood

Xp (neighborhood)Z(source)

X(target frame)

Energy for Single Neighborhood

(nearest neighbor) Zp

Xp (neighborhood)Z(source)

X(target frame)

Energy for Single Neighborhood

X

Z

(nearest neighbor) Zp

Xp (neighborhood)

Texture Energy(single neighborhood)

Texture Energy for Entire Image

X

Z

Texture Energy for Entire Image

X

Z

individual neighborhood energy

• Optimize Total Energy of target frame:

• Initialize: X W(target frame warped frame)

• Iteratively improve target frame

Optimization

Optimization: Intuition

Warped Frame

Source

Target Frame

Flow Force

Texture ForceZp

Xp

Wp

Optimization: Intuition

Source

Zp

Xp

Wp

Blend

Warped Frame

Target Frame

Overlapping Neighborhoods

Source

Wp

BlendXqZq

Xp

Warped Frame

Target Frame

Zp

We are not done yet!

Source

Xp

Wp

ZpNearest neighbor may change

Warped Frame

Target Frame

Iterative Algorithm

• Step 1 – Find Nearest Source Neighborhoods

• Step 2– Blend Source & Warped Neighborhoods

• Repeat until convergence

Iterative Algorithm

• Step 1 – Find Nearest Source Neighborhoods

• Minimize E( X ) w.r.t {Zp}

• Approximate nearest-neighbors– Hierarchical Clustering [Johnson’67] – Tree-Structured Vector Quantization [Wei’00]

Iterative Algorithm

• Step 2– Blend Source & Warped Neighborhoods

• Minimize E( X ) w.r.t X

• Set Solve Linear System

Robust Optimization

• Quadratic Texture Energy Et sensitive to outlier neighborhoods

• Non-quadratic Et more robust

– Et( X ) = p kXp – Zpkr, 0 < r < 2

Robust Optimization

• Quadratic Texture Energy Et sensitive to outlier neighborhoods

• Non-quadratic Et more robust

– Et( X ) = p kXp – Zpkr, 0 < r < 2

• Solve weighted quadratic at each iteration Et( X ) = p kXp – Zpkr-2kXp – Zpk2

= p p kXp – Zpk2

Minimize Texture Energy Alone

• Novel Static Texture Synthesis Algorithm

CorrectXX–

W

Texture Similarity

Flow Consistency

Source

• Random Initialization

• Multiple– Resolution Levels– Neighborhood Sizes

• Progressively refined output

Texture Energy Graph

Level 1

Level 2

Level 3

Level 4

Level 5

Level 6

Level 7

0 5 10 15 20 25 30 35 40 45 500

50

100

150

200

250

300

350

400

Number of Iterations

• Random Initialization

• Multiple– Resolution Levels– Neighborhood Sizes

• Progressively refined output

Texture Energy Graph

Results: Texture Synthesis

Results: Texture Synthesis

Input Wei-Levoy[Wei’00]

Image Quilting[Efros’01]

Graph-cuts[Kwatra’03]

Texture Optimization

[Kwatra’05]

Results: Texture Synthesis

Image Quilting[Efros’01]

Graph-cuts[Kwatra’03]

Texture Optimization[Kwatra’05]

Results: Flow-guided Synthesis

Results: Timing

• Static texture synthesis– 5-10 iterations per resolution level

• Flowing texture synthesis– 20-60 seconds per frame

Resolution One iteration Total time

64x64 2.5 sec 20 sec

128x128 10 sec 2 min

256x256 25 sec 8.5 min

Summary

• Global Optimization for controllable texture synthesis

• Explicit evaluation of texture quality

• Iterative progressive refinement of texture

• Flow-guided texture animation within optimization framework

+ =Input

TextureInput

Flow FieldOutput

Sequence

Level 1

Level 2

Level 3

Level 4

Level 5

Level 6

Level 7

0 5 10 15 20 25 30 35 40 45 500

50

100

150

200

250

300

350

400

Number of Iterations

Limitations and Future Work

• Optimization finds local minima– Blurry / misaligned texture elements

• Texture scale should be compatible with flow field

• Extensions– Controllable video textures

– Other control criteria like shape, illumination

Acknowledgements

• Eugene Zhang for flow design code

• Greg Turk, Gabriel Brostow, Delphine Nain, Ravi Ruddarraju, Gaurav Chanda, Stephanie Brubaker, Mitch Parry, Pei Yen

• Computational Perception Lab, Geometry Group @ Georgia Tech

• CMU Graphics Lab

Web page:

http://www.cc.gatech.edu/cpl/projects/textureoptimization/