SOLID TEXTURE SYNTHE-SIS FROM 2D EXEMPLARSSIGGRAPH 2007Johannes Kopf, Chi-Wing Fu et al.

Abstract Present a novel method for synthesizing

solid textures from 2D texture exemplars Extend 2D texture optimization techniques

to synthesize 3D texture solids integrated with histogram matching effectively models the material in the inte-

rior of solid objects well-suited for synthesizing textures with a

large number of channels per texel

Introduction

Solid textures have several notable advan-tages over 2D textures Natural materials, such as wood and stone, may

be more realistically modeled using solid textures Solid textures obviate the need for finding a pa-

rameterization for the surface of the object to be textured

Possible to perform high-fidelity sub-surface scat-tering simulations, as well as break objects to pieces and cut through them

Tile-based On-demand evaluation Variety limited by number of tiles Distinctive features reveal tiling structure Cohen et al. 2003, Lefebvre et al 2003, Wei 2004

TilesCombined to form a texture

[Wei 2004][Cohen et al. 2003]

Patch-based Sequential Best results Little fine-scale variety Praun et al 2000, Liang et al 2001,

Efros and Freeman 2001, Kwatra et al 2003

Pixel-based Fine-scale variety Sequential Garber 1981, Popat & Picard 1993,

Efros & Leung 1999, Wei & Levoy 2000, Ashikhmin 2001, Hertzmann et al 2001, Tong et al 2002 …

Exemplar Synthesized

Parallel synthesis

Most neighborhood-matching synthesis al-gorithms cannot support parallel evaluation

Sequential - long chains of causal dependen-cies

Entire image must be synthesized at one time Cannot be mapped efficiently onto a parallel

architecture like a GPU or multi-core CPU

Order-independent synthe-sis Wei and Levoy 2003 Synthesize all pixels independently

Multi-scale pyramid Apply multiple passes of correction at each

pyramid levelread writeread writeread writeread writeread writeread write

Level 0 Level 1 Level 2Gen 0 Gen 1Gen 0 Gen 1Gen 0

(init)Gen 2 Gen 2

(result)

Our method

Extend previous approach using three novel ideas

Gaussian image stack Gaussian pyramids shifted at all locations of the exemplar im-

age

Coordinate up-sampling Initialize each pyramid level using coordinate inheritance

Correction sub-passes Split each neighborhood-matching pass into several sub-passes

Our method

More explicit, intuitive control Coordinate jitter

produces a tiling in the absence of jitter due to simple coordi-nate inheritance design

Set of continuous sliders that control the magnitude of random jitter

Also enables several forms of local control by adjusting spatial randomness

Synthesis magnification Use low-res texture to synthesize a map Use this map to efficiently sample a higher-res examplar

Basic scheme

Operate on coordinate instead of pixels

E[S[p]] = E[u] E : Exemplar image S : Synthesized image

For exemplar image E Create Gaussian image pyramid of total level L

S0, S1, …, (SL=S) For exemplar image E of size m x m

L = log2m

Basic scheme

Exemplar

Level 0 Level 1 Level 6

Level 0 Level 1 Level 6

Exemplarcoordinates

Basic scheme 3 fundamental steps

Up-sampling Jitter Correction

Upsampling Jitter Correction

ll-1

Up-sampling

Up-sample the coordinates of the parent pixels to its child

hl = 1 for pyramid hl = 2L-l for a stack

If jitter is disabled up-samplings create tiled image of exemplar E

,

10

,01

,10

,00

,mod)][2(:]2[ 1 mhpSpS lll

Up-sampling

(2,4) (3,4)

(2,5) (8,1)

(0,0)(1,0)

(0,1)(1,1)x2 +

(4,10)(5,10)

(4,11)(5,11)

(16,2)(17,2)

(16,3)(17,3)

(6,8) (7,8)

(6,9) (7,9)

(4,8) (5,8)

(4,9) (5,9)

l-1

Jitter Perturb the up-sampled coordinates at each level

Perturb coordinates using deterministic hash function

0 ≤ rl ≤ 1 : User-specified per-level randomness parameter

If the correction step is turned off, the effect of jitter at each level looks like a quad-tree of translated windows in the final image

Up-sampling / Jitter

Jitter Jitter

Correction Recreate neighborhoods similar to those in the exemplar

Generally perform two correction passes For each pixel p

Match 5x5 neighborhood vector NSl(p) with NEl(u) Consider only u in the exemplar given by 3x3 immediate neigh-

bors of p Pre-compute another candidate Cl 1…k for exemplar pixel u

Use only one more candidate in this paper (k = 2) For good spatial distribution each candidates are required to be sepa-

rated by at least 5% of the image size Penalize jumps to another candidate using parameter κ

Correction

Candidates { }Previous buffer (from jitter) OutputExemplar

?

Traditional Gaussian image pyramid

Synthesized features align with a coarser grid Coordinates in the synthesis pyramid are

snapped to the quantized positions of the exemplar pyramid

Gaussian image stack Allow synthesized coordinates u to have

fine resolution at all levels

Pyramid

Stack

Level 6Level 5Level 4Level 3Level 2Level 1Level 0

Gaussian image stack

Using Gaussian pyramid Using Gaussian Stack

Gaussian image stack Augment the exemplar image on all sides to

have size 2m×2m Additional samples come from

Actual larger texture Or tiling if toroidal Or mirrored copy of exemplar

Reassign hl = 2L-l

Update up-sampling step to account for parent-child relationship

mhpSpS lll mod)5.5.

][(:]2[ 1

Gaussian image stack If the exemplar is non-toroidal

Artifacts occur When the up-sampled coordinates of four sibling pix-

els span over ‘mod m’ Avoid getting boundary pixels (red in picture) as

candidate Cl

Gaussian image stack At the coarsest level (l = 0) correction step

has no meaning

Stack of level 0 is equal to the mean color of the exemplar image

Correction step on stack of level 2 or below tends to restrict alignment of features

So we disable correction step on l < 3

Level 0

Correction sub-passes Problem of traditional correction step

Pixels do not benefit from neighbors’ cor-rection May lead to slow convergence of pixel colors, or

even to cyclic behavior

Improve results by partitioning a correc-tion pass into a sequence of sub-passes Apply s2 sub-passes, each one processing

the pixels p such that p mod s = (i j)T, i, j ∈ {0…s-1}

Sub-passes

Nearly same amount of computation

Subpass 1Subpass 2Subpass 3Subpass 4Previous bufferPrevious buffer (from jitter)

Sub-passes Quality improves with more sub-passes

Not much beyond s2=9 Traditional sequential algorithm is similar to a large

number of sub-passes applied in scan-line order Yields worse results Gives fewer opportunities to fix earlier mistakes

On GPU, each sub-pass requires a SetRenderTarget() call, which incurs a small cost

Neighborhood error decreases with more corrections While texture gets to look less like the exemplar due to

disproportional bias cause by correction step

Sub-passes

Spatially deterministic computation For synthesizing a deterministic texture

window Wl

Need padded window Wl′⊃ Wl

Pixels needed for padding : 2cs2

c : number of correction step, s2 : number of sub-passes

PCA projection 5 x 5 neighborhood of 3 dimension vector

requires a lot of memory and time

Project 5 x 5 neighborhood into a lower-dimensional space Project NEl(u) into 6 dimension vector using

principal component analysis (PCA) matrix P6 Evaluate 6 dimensional distance by these

vectors

Quadrant packing Each correction sub-pass must write to a

set of nonadjacent pixels But GPU pixel shaders do not support effi-

cient branching on such fine granularity Reorganize pixels according to their ‘mod s’

location

Color caching / PCA Projection of colors /Channel quantization Fetching color information from exemplar image re-

quires two texture lookups Let S[p] store a tuple (u, E[u])

(u, E[u]) is 5 dimension(channel) vector Project this into 4 dimension vector using PCA to fit into 4 chan-

nels (RGBA) Store most of information into 8-bit/channel textures

(Cl1(u),Cl

2(u)) into RGBA texture Projected neighborhoods ŇE(u) into 2 RGB textures

Or into 4 RGBA textures of ((Cl1(u), ŇE(Cl

1(u)), Cl2(u), ŇE(Cl

2(u))) Byte sized coordinates limit exemplar size to 256 x 256

2D Hash function / GPU shader Define a hash function using 16 x 16 2D texture

Interaction of jitter across levels helps to hide im-perfections of the hash function

Hash function is only evaluated once per pixel dur-ing the jitter step

50 x 6 Matrix P6 takes up 75 constant vector4 registers 2D color of 5 x 5 pixels = 50 Requires shader model ps_3_0

Multi-scale randomness control rl set the jitter

amplitude at each level Set these param-

eters using a set of sliders

Coarse-scale jitter removes visible repetitive pat-terns at large scale

Spatial modulation Jitter is modulated by given randomness field

Over source exemplar By painting a randomness field RE over exemplar Create mipmap pyramid REl[u]

Over output By painting a randomness field RS over exemplar Create mipmap pyramid RSl[u]

Spatial modulation over ex-emplar

Spatial modulation over output

Feature drag-and-drop Locally overrides jitter to explicitly position texture fea-

tures Constrain the synthesized coordinates in a circular region of

the output Sl[p] := (uF + (p – pF)) mod m if ||p – pF|| < rF pF : circle center, rF : radius, uF : exemplar coordinate at pF Must apply this constraint across many synthesis levels

Actually store two radius Define inner radius ri , outer radius ro then interpolate across levels as

rF = ril/L + ro(L-l)/L Parameters are stored in the square cells associated with

coarse image IF at resolution level is 1 Can introduce feature variations by disabling fine scale

constraint

Near-regular textures

Some textures are near-regular

Approximately pe-riodic

Tiles may have ir-regular color

Near-regular textures Given a near-regular texture image E′,

resample it onto an exemplar E to be Regular

Can be achieved using the technique of [Liu et al 2004]

determines the two translation vectors repre-senting the underlying translational lattice and warps E′ into a “straightened” lattice

Subdivision of the unit square domain select an nx×ny grid of lattice tiles bounded by

a parallelogram and map it affinely to the unit square

Near-regular textures

Near-regular textures Maintain tiling periodicity by quantizing

each jitter coordinate Similarly for Jl,y(p) If local geometric distortion is desired dis-

able quantization at fine levels Constrain similarity set Cl(u) to the same

quantized lattice (ex. ux + i(m/nx) for inte-ger i) on levels for which hl ≥ (m/nx)

Near-regular textures

Synthesis magnification Obtain down-sampled version of exem-

plar EL with some higher-resolution ex-emplar EH

Use synthesized coordinates SL to create higher-resolution image

Can be used to overcome 256 x 256 ex-emplar size limit

Very simple algorithm which can be em-bedded into final surface pixel shader

Synthesis magnification

Given texture coordinates p

Access the 4 nearest texels Compute the exemplar coordinates that point p

would have if it was contained in the same parametric patch

Sample the high-resolution exemplar EH at those coordinates to obtain a color

Bi-linearly blend the 4 colors

Synthesis magnification

Comparison of resultsNVIDIA

GeForce 6800

DirectX 9.0

k = 2

s2 = 4

c = 2

64 x 64 or

128 x 128 Ex-emplar image

Synthesis speed GPU execution times

for a 64×64 exemplar Output is written di-

rectly into video mem-ory, avoid the over-head of texture upload

Synthesis magnifica-tion processes 100-200 Mpixels/sec can synthesize a

1600×1200 window by magnifying 320×240, all at 22 frames/sec

Preprocess / Representation compactness Preprocess

Gaussian stack, PCA for colors and neighborhoods, similarity sets Cl(u) Takes about a minute on 64 x 64 exemplar, 4-12 on 128 x 127 exemplar

Representation compactness For 64 x 64 exemplar

Total 214KB Minimum texture size for which the synthesis-based representation is more

compact than the final image is 270×270 For a 128×128 exemplar, it is 600×600 May try to reduce further by compressing the representation

Mipmapping To filter the texture, instead of comput-

ing mipmap pyramid, we may use inter-mediate-resolution synthesized image Subsequent jitter and correction will be ig-

nored but most of time they are adequate Must specify texture derivatives (ddx, ddy)

to avoid overly blurred mipmap levels at patch boundaries when using synthesis magnification

Toroidal synthesis Disable both the pyramid padding and the

quadrant packing then let NSl be evaluated toroidally Alternatively, use padded pyramid while using pe-

riodic jitter J`(p) = J(p mod n) n = size of synthesized image

Limitation It is actually not “infinite”

Numerical limits exist GPU supports 32-bit floats currently, so 2D integer lattice

of samples can be up to 224=16M samples on each axis

Drawback of neighborhood-based per-pixel syn-thesis

Summary Presented a parallel synthesis algorithm for creat-

ing infinite texture without the limited variety of tiles

Implemented as a sequence of pixel shading passes on a GPU

Can synthesize a 2562 window of deterministic tex-ture in 26 msec

Can pan the window at over 700 frames/sec Using synthesis magnification technique, can am-

plify content to fill a 1600×1200 screen in real time Various ways of user controls by jitter function

Future work Compress the Gaussian stack representation for im-

proved data amplification Support composite textures (i.e. texture-by-num-

bers) [Hertz-mann et al 2001; Zalesny et al 2005] Combine with geometry clipmaps for efficient terrain

synthesis and rendering [Losasso and Hoppe 2004] Incorporate synthesis of vector shapes such as poly-

gons, fonts, roads, and coastlines Automatically determine the best exemplar image

and random-ness parameters to visually approxi-mate a given texture