hybrid texture synthesis

Hybrid Texture SynthesisHybrid Texture Synthesis

Andrew NealenAndrew NealenMarc AlexaMarc Alexa

Discrete Geometric Modeling Group Discrete Geometric Modeling Group (DGM)(DGM)

Technische Universität DarmstadtTechnische Universität Darmstadt

Eurographics Symposium on Rendering 2003

Outline

Introduction

2D Texture Synthesis

nxm Input Texture

NxM Output Texture

► The goal: Synthesize an output texture which is perceptually similar to the input texture. Also ensure that the result contains sufficient variation.

Introduction

Existing (and Impressive) Technology Pixel-Based:

Non-parametric Sampling [Efros and Leung 1999]

Tree-structured Vector Quantization [Wei and Levoy 2000]

Image Analogies [Hertzmann et al. 2001] Patch-Based:

Patch-Based Sampling [Guo et al. 2001] Image Quilting [Efros and Freeman 2001]

Introduction

Possible Drawbacks ? Loss of global structure Loss of scale Boundary mismatch Blurring

Wei/Levoy Algorithm:Can occur when using structured Textures with rich histograms.This is due to the L2 norm.

Introduction


Image Quilting Algorithm:boundary mismatch artifacts are noticeable

Introduction


Patch-Based Sampling Algorithm:Can occur when texture features are not well-aligned across patch boundaries

Hybrid Texture Synthesis Combine best aspects from other approaches

while avoiding (or improving over) known pitfalls: Patch-based algorithms

Are good at preserving global structure Can introduce artifacts along patch boundaries

Pixel-based algorithms Preserve local coherence (MRF model: local and

stationary) Possibly fail to preserve global structure This is especially problematic for textures with rich

histograms and many high frequency features, due to the smoothing nature of the distance metric (L2 norm)

Hybrid Texture Synthesis (HTS)

Hybrid, two-fold approach to texture synthesis Adaptive patch sampling [Soler et al. 2002]

Start with a uniform quadrilateral grid of patches (the output)

Recursively split a patch if the best fit taken from the input texture is not good enough (a user-defined tradeoff max)

Overlap re-synthesis (novel overlap repair strategy) Mark invalid pixels in the overlap region Validity map is defined by the error surface E(x)=(xsrc –

xdst)2 and a user-defined tolerance parameter max

Define an ordering for the invalid pixels and re-synthesize them using a per-pixel strategy

HybridSynthesize Algorithm1: HYBRIDSYNTHESIZE(T,P,ov,Δmax,δmax,R) : R2: for all patches pi ∈P do3: [Pi,Δi]← FINDBESTPATCH(T,Ri−1, pi,ov)4: if (Δi < Δmax or ISSINGLEPIXEL(pi)) then5: Si ← ERRORSURFACE(Pi,Ri−1)6: Pi ← MARKINVALIDPIXELS(Pi,Si,δmax)7: Mi ← BUILDTRAVERSALMAP(Pi)8: Ri,composite ← COMPOSE(Pi,Ri−1)9: Ri ← OVERLAPRESYNTHESIS(T,Ri,composite,Mi)10: else11: Ps ← SPLITPATCH(pi)12: ovs ←max(3, ov/2)13: Ri ← HYBRIDSYNTHESIZE(T,Ps,ovs,Δmax,δmax,Ri−1)

14: end if15: end for16: return R

FindBestPatch(Algo.)

1: FINDBESTPATCH(T,Ri−1, pi,ov) : [Pi,Δi]2: Ii, Ji ← BUILDIMAGEMASK(Ri−1, pi,ov)3: Ei ← ERRORIMAGE(T, Ii, Ji)4: Ei,trim ← TRIMINVALIDREGIONS(Ei, Ji, pi)5: [Pi,Δi]← BESTPATCH(Ei,trim,T)6: return [Pi,Δi]

In Search of Good PatchesSynthesizing a single patch

Synthesize Black Patch i

In Search of Good PatchesExtracting the image mask

Synthesize Black Patch i

Grow patch by overlap (toroidally)

In Search of Good PatchesExtracting the image mask

Image Mask (Ii)Synthesize Black Patch i Binary Support (Ji)

Extract image mask (Ii) and binary support (Ji), and circularly shift to upper left corner

In Search of Good PatchesComputing the error image

Image Mask (Ii)Input Texture (T) Binary Support (Ji)

* Compute error Ei(x0) between Ii and T for each circular shift x0=(x,y) of the input texture T

In Search of Good PatchesComputing the error image

Error Image (Ei)Input Texture (T)

x

xxxxxc

cicicii

i TIwJκ

E 20,,0 ))()(()(

1)(

C={R,G,B} (both with RGB color values in [0,1])

WR,G,B = {0.299,0.587,0.114}

X0: circular shift of T

)(xJκixi

(1)

In Search of Good PatchesSelecting a Patch

Error Image (E)Input Texture (T) Selected Patch (P)

)))((())((2)(1

)( 02

0,2

,0 XTJXTIIWκ

E ciccicic

ci

ix

xx (2)

Given that Ji(x) Ii(x) = Ii(x),and the cross correlation between two images (functions) f ◇ g is defined as ( f ◇ g )(x0) = Σx f (x)g(x+x0)The correlation f ◇ g between two functions can be computed in O(N logN)in fourier space as f ◇ g = F-1(F( f )∗F(g))(N :number of pixels in the input texture)In implementation it’s can be pre-compute the fourier transform for T and need only recompute the fourier transforms of Ii and Ji for each new patch pi.

In Search of Good PatchesSelecting a Patch

Error Image (E)Input Texture (T) Selected Patch (P)

Note: error image can be computed efficiently in the Fourier domain. Complexity: O(n log n) per patch.

Adaptive Patch SamplingPrinciple

Essentially: Hierarchical Pattern Mapping applied to the plane [Soler et al. 2002]

i > max Split PatchSynthesize White Patch i < max

Adaptive Patch SamplingTrade-off

Trade-off between preserving global structure and avoiding detail artifacts

max = 1.0max = 0.01 max = 0.04

Overlap Re-synthesisOverlap error and pixel invalidation

Patch and Image Mask Selected Patch (P)


Image Mask (I) Selected Patch (P)

c

cicicii PIwJS 2,, ))()(()()( xxxx


Image Mask (I) Selected Patch (P)Error Surface (S)

c

cicicii PIwJS 2,, ))()(()()( xxxx


Patch and Image Mask P with Invalid Pixels (blue)Error Surface (S)

If(Ji(x) !=0 and Si(x)> δmax )

as invalid and in need of per-pixel re-synthesis.else if( Ji(x) != 0 and Si(x) < δmax )are preserved and a stepwise alpha mask (feathering) is applied during Compositing.

(Trade-off: user-defined max is the pixel error tolerance. Setting to 1 results in pure feathering.)

Overlap Re-synthesisCompositing

Intermediate Result Patch with Invalid Pixels

P OVER Ri-1

Overlap Re-synthesisCompositing

Composited Result Patch with Invalid Pixels

Simple scanline re-synthesis: possibly insufficient causal neighborhood within valid pixel region

Solution: alternative ordering (Pixel Traversal Map)

Pixel Traversal-Map

1: BUILDTRAVERSALMAP(Pi) : Mi

2: level = 13: Dlevel ←binary image of size Pi initialized to 04: Dlevel ←set all valid pixels ∈ Pi to 15: Mi ←Dlevel

6: while (∃pixel ∈ Dlevel ∧ pixel = 0) do7: level = level+18: Dlevel ← DILATE(Dlevel−1,Disk)9: Mi ←Mi +(Dlevel ∧Dlevel−1) ∗level10: end while11: return Mi

Overlap Re-synthesisPixel Traversal Map (Ordering)

Composited Result Patch with Invalid PixelsPixel Traversal Map (M)

Pixel Traversal Map: repeated morphological dilation of the binary support for valid pixels with a euclidian disk of radius 1 (city-block distance transform).



Pixel Traversal Map: step 1



Pixel Traversal Map: step 2 ...



Pixel Traversal Map: ... step n



Art Restorer Analogy: stepwise restoration of the hole from the boundary of the existing image.

Overlap Re-synthesisPer-Pixel Re-synthesis

Pixel Traversal Map (M)

Synthesize red pixel from valid (or already re-synthesized) pixels

Intermediate Result


Image Mask (Ij)

Binary Support (Jj)

Extract image mask (Ij) and binary support (Jj) for best-pixel search in the input texture. Complexity: O(n log n) per pixel.

Intermediate Result


Intermediate Result Pixel Traversal Map (M)


Intermediate Result Shifted



Intermediate Result with Traversal Map



Overlap Re-synthesisStep 1



Overlap Re-synthesisResult


ResultsOverlap Repair Comparisons

No Repair HTSPBSIQ

ResultsSynthesis Comparisons

Input

Efros/Leung Wei/Levoy

IQ PBS HTS

ResultsSynthesis Comparisons

Input

PBS HTS

PBS HTS

Input

Results

presence of high frequency structure cause the error metric (L2 Norm)

Conclusions and Future Work

Improve Computational Complexity Pixel neighborhoods of patch and pixel

stage are not known a priori, so precomputation is not straightforward

We can solve this in the pixel stage by employing k-coherence search [Tong et al. 2002] [Ashikhmin 2001]

Improve Error Metric Still using the L2 norm due to its simplicity Develop a metric which takes feature

mismatch into account

hybrid texture synthesis

Documents

patchbased algorithmsare

texture features

patchbased sampling

pi markinvalidpixelspi

patches pi p do3

global structurecan

global structurethis

ri hybridsynthesizet