Fast Image Replacement Using Multi-Resolution Approach
Chih-Wei Fang and Jenn-Jier James Lien Robotics Lab
Department of Computer Science and Information Engineering
National Cheng Kung University
Robotics Lab.
Motivation
Photographs sometimes include unwanted objects. Lose data when the network transmits media (ex.
video)
System: Two modules
Texture Analysis (or Training) Module
Texture Synthesis Module
Texture Analysis Module
Input image for training (Size: W x H)
Is PeriodicYesNo
Non-periodic pattern
side-by-side
Periodic pattern
side-by-sideProcessing for Γ-shaped pattern
Principal Component Analysis (PCA) : Weight vector (W1…WN) for each Γ-shaped pattern
Vector Quantization (VQ) : Speeding up to search the best patches.
Mp > Mn
M = Mp or Mn
Processing for periodic pattern
There are Mp patches.
Processing for non-periodic pattern
There are Mn patches.
…………
……………………
…………
…………
1. Sampling
Processing for non-periodic pattern
A pixel-by-pixel shifting method is used to divide the input texture into M patches. M = (W–Wp+1)× (H–Hp+1)
W is the width of the input image (or texture) H is the height of the input image Wp is the width of the patch. Hp is the height of the patch.
Processing for periodic pattern
The pattern is divided into M patches, where M=W×H. Periodic patterns have continuous veins between two equivalent patterns
when positioned side by side. When the patch reaches the boundary of the input image, in order to form a
complete patch, additional pixels are supplied from the opposite border.
Processing for the Γ-shaped pattern
Taking the whole pieces of the patch as training data may lead to an overestimation of the underlying structure of the patch and will certainly increase the length of the training time.
In addition, searching the matching patches by considering their whole contents generally produce unsatisfactory results since the whole contents tend to be quite different from the initial random values and it may cause the rim effect to become distinct.
ω pixels
Each patch has K elements. During training, PCA is used to transform the original K×M matrix to an N×K matrix, where M>>K>N. And there are N eigenvectors. This can reduce most of the time complexity operations, and hence dramatically increases the performance.
Recombining the appearance of the features while maintaining the coherence of the characteristic content
2. Principal Component Analysis (PCA)
KMKK
M
M
PPP
PPP
PPP
21
22221
11211
ΝKΝΝ
K
K
ΕΕΕ
ΕΕΕ
ΕΕΕ
Ψ
21
22221
11211
PCA
After the PCA process and a sort based on the eigenvalues with corresponding eigenvectors: The first several eigenvectors control the global geometrical structure. The middle eigenvectors control the local features. The last few eigenvectors are controlled by some noises.
These noises cause the photograph to appear truer, but have no influence on the geometric structure.
Therefore a good matching structure need only compare the first several eigenvectors.
N
K
ΝKΝΝ
K
K
W
W
W
P
P
P
ΕΕΕ
ΕΕΕ
ΕΕΕ
2
12
1
21
22221
11211
3. Vector Quantization (VQ)
Searching for the best matching pattern from the eigenspace Ψ during the synthesis process is a computationally expensive task.
In order to speed up the synthesis process, the training data is initially projected onto the eigenspace Ψ in order to retrieve the weight vectors.
The data is then separated into C clusters by means of VQ. Using this approach, the computational time can be reduced from O(M) to O( ). M
3. Similarity measure (based on SSD): Using the weight vector to find the closest cluster and then find the best matching patch from this cluster in the input image. Then this matching patch is called to fill the region of the search window.
Texture Synthesis Module
1. Initialization: Each pixel is assigned a random value in the WoxHo-pixel output image area.
Shifting this search window over this image in the scan-line order from the top-left order to the bottom-right order.
………
2. Γ-shaped pattern projectiong: Each WpxHp-pixel search window (or patch) is taken the Γ-shaped pattern and then projected onto the eigenspace Ψ to obtain one N-dimensional weight vector.
……… ………
4. Result: Find the best patches to fill in the WoxHo-pixel output image.
Texture Synthesis Module : Demo
Image Replacement
Down-sampling with Texture Analysis
Up-sampling with Texture SynthesisI0
I1
I2
I3
I4 β4
β3
β2
β1
β0
C0C3C4 C2 C1
1. Preprocessing Using Multi-resolution Approach
The input image I0 and the inverse matte β0 to do l times down-sampling ↓ , then to get each level of input image Ii and inverse matte βi, i=0~l
Bi denotes the background region and the foreground region Fi is then utilized to search the background Bi
liβIΒ iii ~0, liβIF iii ~0,
2. Training Process Based on Background Region
If the same Γ-shaped pattern of the search window (or patch) is applied, the fragment between two different textures will be mapped by only one texture rather than being mapped by the mixing of two different textures. Hence, the search window is modified.
ω
K pixels…
3. Completion Process
The reconstructed image Ci First, the system reconstructs the level i (i=l) image Ii
The replaced part is reconstructed (or synthesized), called Ii’ Only the reconstructed part of Ii’ is preserved at position Fi
The other parts are replaced by the original background Bi
The system applies the up-sampling ↑ technique from the reconstructed image Ci at level i and proceeds to only replace
the level i-1 image Ii-1 at position Fi-1
iiiii βIβIC
1111 )( iiiii βCβII
Image Replacement : Demo
Experimental ResultsTable 1: Average training time in Section 3 for input images of various sizes. Units of time: milliseconds (ms).
Size of input image (pixels)
64x64 96x96 128x128
PCA 250 437 875
Projecting to eigenspace 359 703 1656
VQ 203 391 953
Total time (ms) 812 1531 3484
Table 2: Average synthesizing time in Section 4 for output images of various sizes.
Size of output image (pixels)
200x200 300x300 400x400 600x600
Synthesis time (ms) 11 37 59 140
Table 3: The replacement information of each level in Figure 9 and Figure 12 for multi-resolution approach. “~0” means close to 0 ms.
Level Width (pixels) Height (pixels)Number of whole
patchesSynthesis time (ms)
0 392 364 114034 63
1 196 182 24272 16
2 96 91 4024 ~0
3 49 46 110 ~0
4 25 23 0 ~0
Table 4: The time in Figure 9 and Figure 12 for various processes. The total time of the training and synthesis processes include the processes of the multi-resolution approach, PCA, projection (the patches are projected onto eigenspace Ψ to obtain the weight vectors), VQ, and synthesis.
Method Time (ms)
Data type Gray value RGB space
Training process
Multi-resolution 32 31
PCA 891 7719
Projecting to eigenspace 1828 4906
VQ 421 375
Synthesis process
Synthesis 78 108
Total time 3250 13139
Conclusions
We developed a system including two modules the texture analysis and synthesis modules. This system is able to be applied
to the two different purposes: the synthesis of a large image, and the replacement of local removed region.
According to the training non-periodic or periodic pattern we use different sampling methods to obtain different amount of patches in
order to reduce the emergences of the seams of the output synthesized image.
Reduce most of the time complexity operations the analysis module can reduce dimensions of the training data and cluster
these data, so the synthesis module can synthesize a large output image very fast and keep geometrical structures and veins continuous.
Conclusions
Without needing to assign initial random values The same process can also be used to replace the removed regions. Here,
the multi-resolution approach is applied to the image replacement without needing to assign initial random values or approximate values.
Multi-resolution go through the image replacement process The down-sampling step is used for the analysis process as compiling the
training data, and the up-sampling step is used for the reconstructing process as assigning initial values.
Handle the large removed region So this approach enables the system to handle the large removed region and
obtain more realistic image (or textures) quickly.
Experimental ResultsTexture Synthesis (1)
Experimental ResultsTexture Synthesis (2)
Experimental ResultsTexture Synthesis (3)
Experimental ResultsTexture Synthesis (4)
Experimental ResultsTexture Synthesis (5)
Experimental ResultsImage Replacement
(1)
Experimental ResultsImage Replacement
(2)