texture synthesis
DESCRIPTION
Texture Synthesis using Image QuiltingTRANSCRIPT
Texture Synthesis using the Image
Quilting AlgorithmFrom Image Quilting for Texture Synthesis and Transfer
by Efros & Freeman, SIGGRAPH 2001
Presenter: Vicente OrdonezDepartment of Computer Science
Stony Brook University State University of New York
Texture Synthesis
Given a sample texture image, generate an output texture of arbitrary size.
It Should Work on Different Types of Textures
Stochastic textures (no apparent structure)
It Should Work on Different Types of Textures
Repeated textures (Structured)
Other types of textures
Semi-structured textures. Repetition and stochastic processes involved.
The Taking Patch Samples from the Sample Image Approach
Xu et al. from MSR proposed the idea of taking random patch samples from the input image and then blending them together by blurring the edges.
Take the Patches so that they Agree on Some Overlapping Region
Idea:
Select the K patches from the input sample that match closely to the current overlapping region in the output texture.Pick one of those K patches randomly.
B1 B2
Neighboring blocksconstrained by overlap
Algorithm
Select a random patch from the input image and place it on the top left corner of the output texture.
Go through the output texture in raster scan order with steps of the size of the patch minus the dimension of the overlap region. Using the current overlap region in the output texture,
search for every location in the input image that minimizes the sum of square distances error between the overlap with top and left side of the last patch.
Choose randomly one of the patches identified in the previous step and paste it into the output texture.
How to avoid repeating calculations?
Idea: Don’t calculate the SSD for the overlapping regions for each possible positions one-bye-one but do it all in one pass.
This is the objective:
How to avoid repeating calculations?
Blend each new patch along a Minimum Error Boundary Cut
Calculate a cut between patches that minimizes the error along the overlap region.
B1 B2
Minimal errorboundary cut
Using Dynamic Programming to Calculate the Minimum Cost Path
Generate a minimum cost path table {Eij} using the overlap error area {eij} using Dynamic Programming as follows:
Traverse backwards the resulting table to find the minimum cost path.
),,min( 1,1,11,1,, jijijijiji EEEeE
Error Overlap Region and min-cut
Running the min-cut algorithm
How to get a better blending?
Alpha blending Pyramid blending Histogram matching Blending in the gradient domain Poisson Blending
User Controlled Parameters The size of the patches. This heavily affects the results. The
ideal size patch is the one that captures the minimum units of detail of the objects being repeated in the texture. We also have to be careful not to choose the size patch so large so that the stochastic nature of the results gets affected.
The size of the overlap. This mainly affects the cuts between adjacent patches. This parameter also depends on the patterns of the input texture. Having a larger overlap does not always results into a visual improvement in the generated texture.
The size of the error range when choosing patches. Making this range larger makes the selection converge into a random selection of patches, so it has to be sufficiently large to take advantage of the overlap constraints while leaving an stochastic process to affect the results.
Final Results. Generating some textures.
Generated in 12s
Final Results. Generating some textures.
Generated in 25s
Final Results. Generating some textures.
Generated in 11 s
Final results: Generating some textures.
Patch size = 50pxOverlap size = 8pxError range = (0 – 0.05)Execution time = 15sInput texture size = 148x139Output texture size = 428x428
Final Results. Generating some textures.
Patch size = 20pxOverlap size = 4pxError range = (0 – 0.05)Execution time = 26sInput texture size = 148x138Output texture size = 343x343
Conclusions
The results of the original paper have been successfully replicated.
The execution times are slow for a real time application but there are some hints about how to make it in real time using other research results.
Even when there are some recommendations for selecting the parameters, selecting the appropriate ones is an important step for the method and still largely depends on the image contents.
Despite the previous fact, the method performs well in a wide range of textures that were difficult to handle for previous methods.
Given the appropriate parameters the method seems to resemble well the visual appearance of the input texture.
Issues/Further Additions
Searching for a patch that agrees on an overlap region is computationally expensive, so it is necessary to search using heuristics or use another kind of efficient search (Tree structures).
The blending function still produces a visible seam. Maybe there’s room for improvement here. Pyramid Blending/ Gradient/Poisson Blending?
Other ways to choose the seam that allow more flexibility in the curvature of the seam.
References
Sample textures taken from the original Efros & Freeman paper and the project website athttp://graphics.cs.cmu.edu/people/efros/research/quilting.html