Texture Synthesis using the Image

Quilting AlgorithmFrom Image Quilting for Texture Synthesis and Transfer

by Efros & Freeman, SIGGRAPH 2001

Presenter: Vicente OrdonezDepartment of Computer Science

Stony Brook University State University of New York

Texture Synthesis

Given a sample texture image, generate an output texture of arbitrary size.

It Should Work on Different Types of Textures

Stochastic textures (no apparent structure)

It Should Work on Different Types of Textures

Repeated textures (Structured)

Other types of textures

Semi-structured textures. Repetition and stochastic processes involved.

The Taking Patch Samples from the Sample Image Approach

Xu et al. from MSR proposed the idea of taking random patch samples from the input image and then blending them together by blurring the edges.

Take the Patches so that they Agree on Some Overlapping Region

Idea:

Select the K patches from the input sample that match closely to the current overlapping region in the output texture.Pick one of those K patches randomly.

B1 B2

Neighboring blocksconstrained by overlap

Algorithm

Select a random patch from the input image and place it on the top left corner of the output texture.

Go through the output texture in raster scan order with steps of the size of the patch minus the dimension of the overlap region. Using the current overlap region in the output texture,

search for every location in the input image that minimizes the sum of square distances error between the overlap with top and left side of the last patch.

Choose randomly one of the patches identified in the previous step and paste it into the output texture.

How to avoid repeating calculations?

Idea: Don’t calculate the SSD for the overlapping regions for each possible positions one-bye-one but do it all in one pass.

This is the objective:

How to avoid repeating calculations?

Blend each new patch along a Minimum Error Boundary Cut

Calculate a cut between patches that minimizes the error along the overlap region.

B1 B2

Minimal errorboundary cut

Using Dynamic Programming to Calculate the Minimum Cost Path

Generate a minimum cost path table {Eij} using the overlap error area {eij} using Dynamic Programming as follows:

Traverse backwards the resulting table to find the minimum cost path.

),,min( 1,1,11,1,, jijijijiji EEEeE

Error Overlap Region and min-cut

Running the min-cut algorithm

How to get a better blending?

Alpha blending Pyramid blending Histogram matching Blending in the gradient domain Poisson Blending

User Controlled Parameters The size of the patches. This heavily affects the results. The

ideal size patch is the one that captures the minimum units of detail of the objects being repeated in the texture. We also have to be careful not to choose the size patch so large so that the stochastic nature of the results gets affected.

The size of the overlap. This mainly affects the cuts between adjacent patches. This parameter also depends on the patterns of the input texture. Having a larger overlap does not always results into a visual improvement in the generated texture.

The size of the error range when choosing patches. Making this range larger makes the selection converge into a random selection of patches, so it has to be sufficiently large to take advantage of the overlap constraints while leaving an stochastic process to affect the results.

Final Results. Generating some textures.

Generated in 12s

Final Results. Generating some textures.

Generated in 25s

Final Results. Generating some textures.

Generated in 11 s

Final results: Generating some textures.

Patch size = 50pxOverlap size = 8pxError range = (0 – 0.05)Execution time = 15sInput texture size = 148x139Output texture size = 428x428

Final Results. Generating some textures.

Patch size = 20pxOverlap size = 4pxError range = (0 – 0.05)Execution time = 26sInput texture size = 148x138Output texture size = 343x343

Conclusions

The results of the original paper have been successfully replicated.

The execution times are slow for a real time application but there are some hints about how to make it in real time using other research results.

Even when there are some recommendations for selecting the parameters, selecting the appropriate ones is an important step for the method and still largely depends on the image contents.

Despite the previous fact, the method performs well in a wide range of textures that were difficult to handle for previous methods.

Given the appropriate parameters the method seems to resemble well the visual appearance of the input texture.

Issues/Further Additions

Searching for a patch that agrees on an overlap region is computationally expensive, so it is necessary to search using heuristics or use another kind of efficient search (Tree structures).

The blending function still produces a visible seam. Maybe there’s room for improvement here. Pyramid Blending/ Gradient/Poisson Blending?

Other ways to choose the seam that allow more flexibility in the curvature of the seam.

References

Sample textures taken from the original Efros & Freeman paper and the project website athttp://graphics.cs.cmu.edu/people/efros/research/quilting.html