texture segmentation and synthesis

of 14

TEXTURE SEGMENTATION AND SYNTHESIS Wayne Weiyi Chen

1.1. INTRODUCTION Texture analysis lies in the very fundamental and important part of pattern recognition and computer vision. It has a broad range of application such as map segmentation and so on.

In this problem, we will investigate two techniques, one being texture segmentation and the other being texture synthesis. The purpose of segmentation is that to distinguish several parts with certain patterns within one single image; while synthesis is to propagate a given texture “seed” into a new image with larger size.

Figure 1. Texture Segmentation

Figure 2. Texture Synthesis

of 14

1.2. METHODOLOGY

1.2.1. Texture Segmentation The process of texture segmentation can be divided into three parts: parallel filtering, energy computation, and segmentation. Now we will briefly introduce these three steps separately.

Parallel Filtering Firstly we will let the input image pass through a group of parallel filters. These filters are combined with 5 different features, being shown as following:

Table 1. Laws Filters

Name Kernel

L5 Level [1 4 6 4 1]

E5 Edge [-1 -2 0 2 1]

S5 Spot [-1 0 2 0 -1]

W5 Wave [-1 2 0 -2 1]

R5 Ripple [1 -4 6 -4 1]

The 25 5*5 square parallel filters is composed by the tensor product of these 5 kernels. They play as different frequency pass filters. For example, L5*L5 is being the low pass filter and its output is the de-noising result which tell us the brightness of the image.

Energy Computation After the parallel filtering, now we have 25 output images which are called 25 different features. Before doing segmentation we need to compute the bulk energy of each feature. The energy is computed using formula as below:

(17)

where the window size can be different. As we shall see from result part, typically a window size of 11~15 can deliver a satisfactory output.

The procedure of energy computation is we slide the window from the upper-left corner to the down-right corner and within each window we summarize the square of the pixel value to obtain the energy value of that pixel.

( ) ( )( ) ( )

2

, ,, ,i i

x y W x yE x y M x y

¢ ¢Î

¢ ¢ = å

of 14

Segmentation: K-means Algorithm The segmentation can be done by using K-means algorithm. The basic idea of K-means algorithm is to repeat dividing image points into k groups and re-calculating the centroids of each group until the change between steps converges into a small tolerance. It can be elaborated as following:

• Initializing data set as choosing arbitrary k points as the initial centroid.

• Force data set into k clusters using nearest neighbors rule by calculating Euclid distance.

• Find a new centroid in each cluster.

• Repeat last two steps until the change of clusters is in the range of a small tolerance.

The code of the k-means algorithm is provided and we only need to adjust the form of output data from energy computation to fit the given program. As we shall see in the result part, we can reduce the converging time and performance by choosing better initial centroids.

1.2.2. Texture Synthesis In this part, we will implement both naïve version of block-based texture synthesis and Efros-Freeman algorithm.

Naïve Method The naïve method is randomly pick blocks from the seed image and simply put them one by one into output image. We shall see this procedure will introduce discontinuous boundaries and does not deliver satisfactory results.

Efros-Freeman Algorithm To improve the result from the last method, we introduce Efros-Freeman Algorithm. The mainly improvement and therefore key points of this algorithm is we choose neighbor blocks with boundary having similarity and connect them by certain method so that the “jump” effect in the boundary being reduced to a negligible level. Here we provide the pseudo-code for an explanation of our implementation.

Algorithm Efros-Freeman (seed, BlockSize, StripeWidth, error)

1st Block Choosing

Location = randomly assign a number, being the upper-left corner of that block

Block1 = crop the image from Location to Location + BlockSize*BlockSize

Exhaustive Search the Next Block

for row in seed:

for column in seed:

of 14

SSD[row][column] = summation for the squared error inside the stripe area

If SSD[row][column] < error

Count += 1

RowCandidate[Count] = row; ColumnCandidate[Count] = column

Candidate = randomly assign a number between 0 and Count

NextBlockRow = RowCandidate[Candidate]

NextBlockColumn = ColumnCandidate[Candidate]

BlockNext = crop the seed according to NextBlockRow and NextBlockColumn

Minimum Cut

Calculate the cumulative minimum error from top to bottom inside the stripe area

for row in [NextBlockRow, NextBlockRow+BlockSize]

for column in [NextBlockRow, NextBlockRow+StripeWidth]

e[row][column] = BlockNext[row][column] – BlockLast[row][column]

Error[row][column] = e[row][column] +

min(Error[row-1][column]),Error[row-1][column-1]),Error[row-1][column+1])

Find the start of the path from the last row

path[row] = min(Error in the last row)

Search back the path from the bottom to top inside the stripe area

for row in [NextBlockRow+BlockSize-1, NextBlockRow]

for column in [NextBlockRow+StripeWidth, NextBlockRow]

path[row] = the index of min( Error[ row][ path[row]], Error[ row][ path[row]-1]

Error[ row][ path[row]+1] )

Cut the block according to PathStripe1 (row direction) and pathStripe2(column direction)

for row in NextBlock

for column in row

If row< PathStripe1 and column < PathStripe2 is not true

of 14

Copy NextBlock[row][column] to output image

The above procedure should be repeated until the final output image is filled with blocks. As we can be seen in the pseudo-code, when we are looking for the minimum cut path, we firstly compute the cumulative minimum error from top to bottom inside the stripe area, then search back from bottom to top to find a path that have minimum error. After the cut path is found, we left top and left side of the path in the output image and copy the block for the rest part.

1.3. RESULTS

1.3.1. Texture Segmentation First we show some of the results from parallel filtering:

Figure 3. First 10 Outputs from Parallel Filtering

As we can see from the result, the 1st output being the brightness of the image as we mentioned above and with result being more close to the right bottom we have the higher frequency component of the original image.

The next step is energy computation. Here we only show some of the result from 11*11 window size. We will discuss the influence introducing by window size to the final result in the discussion part.

of 14

Figure 4. Some of the Results from Energy Computation

Figure 5. A Direct Result from Energy Outputs in Fig.12

of 14

As we can see in the Fig.12, after energy computation and normalization, the five parts of the image has not been distinguished very well (especially the on the right side), which means we shall not expect K-means algorithm performing on these features can deliver a satisfactory result. We show the result directly from these energy features in the Fig.13 and show final outputs with preprocessed features in the Fig.14 in the below. We will have a detailed discussion on this preprocessing procedure in the discussion part.

Figure 6. Final Result of Segmentation with Window Size

a) 11*11 b) 13*13

a) 15*15 b) 19*19

As we can clearly see from Fig.14, the results have been significantly improved from result in Fig.13 by using the pre-processing. We will also discuss the influence of window size in the

of 14

discussion part. Notice not all of the features are used in the final result, we only pick part of them for the input of K-means algorithm, we will also have a discussion on this in the next part.

1.3.2. Texture Synthesis Firstly we show the naïve method in the Fig.15.

Figure 7. Naive Method

It is obvious that strong artifacts occur on every boundary of each blocks.

In the next page we show synthesis results from Efros-Freeman algorithm in Fig.16. As we can clearly see, the output image has been significantly improved comparing to the former cases due to the block choosing and minimum cut.

As we can see in the Fig.16, different size of blocks will have slightly different results. We will discuss this difference in the discussion part.

of 14

Figure 8. Efros-Freeman Algorithm with Block Size

Left) 24*24 | right) 60*60

of 14

1.4. DISCUSSION

1.4.1. Preprocessing Before Utilizing K-means Algorithm We briefly mentioned above that from the direct result of energy computation the K-means algorithm will not deliver satisfactory result due to the reason that there are no distinguishable features in all of the five parts in the result. As a result, to improve the performance of segmentation, we need to enhance the contrast of five different parts.

Therefore, we pre-process the result using histogram equalization, as some examples showing in the following Fig.17.

Figure 9. Preprocessing the Energy Computation Result with Histogram Equalization

We can see from above that after histogram equalization, the contrast between different parts becomes more significantly.

of 14

1.4.2. Feature Choosing We also mention in the last part that we only pick certain features as the input of K-means algorithm. This is because two reasons: the first is to compute all of the 25 features is time-consuming and computationally expensive; the second is after energy computation it is actually can be seen that only part of features can provide useful information, that is, have obviously distinguishable 5 parts standing out, even after histogram equalization.

Therefore, to remove the less important features is not only for the purpose of reducing computation intensity but also improving the final output result. We finally choose {0,1,2,3,4,5,6,7,10,11,12,16,20} as our final input features for K-means algorithm.

Another important points of feature choosing is that we add two extra features beside the above 13 features, one being the row index and the other being the column index of the image. The reason we doing this is in this specific case we have 5 textures in the five clusters in the different locations. Two images contains of index information provide more weighting on closer pixels (because the closer two pixels are, the closer pixel value they have in these two maps) so that it would help k-means algorithm to segment textures and group closer points together to obtain a better result.

To demonstrate our method by choosing and adding extra features can improve the performance, in Fig.18 we show the corresponding results: the left being using all 25 features and the right being only using our proposed 15 features.

Figure 10. All 25 Features vs. 13 Features + Extra 2 Features

(Energy Computation Window Size 15*15 )

1.4.3. Energy Computation Window Size Impact on the Segmentation As we can see from Fig.14, with different window size when we do energy computation we can have different results: smaller size window generally put more emphasize on detailed texture which is not important in this case so they do not deliver as good result as the larger ones; however, larger size window introduce a boarder boundary (more obvious on the perimeter of the central circle).

of 14

As a result, there is a trade-off between the choice of different window size, we will prefer larger size in the cases like this problem where we have bulked texture while we would like to choose smaller one if the texture is interleaved within the image.

1.4.4. Initialization of K-means Algorithm We also discover that the initialization of centroid heavily influence the final result as showing below. If we simply divide the data into 5 parts with equal length and set the centroid as the start point of each segment, then the converge result will have some bias.

However, if we manually choose 5 center points of each texture, we will obtain a much better result from K-means algorithm. So, K-means algorithm will deliver a more satisfactory performance with reliable prior knowledge.

1.4.5. Block Size and Stripe Width Impact on the Synthesis We show two different sizes of blocks in Fig 16, the left side being 24*24 and right side being 60*60. Smaller size blocks will combine blocks more disorderly with more artifacts on the boundary (in all of the three images there are obvious discontinues boundaries); Larger size will reduce these artifacts (there are much fewer discontinues edges in all of the three images), however it suffers from a disadvantage that introduce more periodic repeating of blocks (especially obvious in the apple case) which let human eyes easier to realize this is an artificial result.

of 14

The reason of this result is because by using larger block size, we choose a more large block which means inside the block we do not to worry about the discontinuous problem and we will encounter less such discontinue boundary in the whole output image. However at the same time, we have less choice on different blocks so we have to more rapidly repeat the same pattern.

So there is also a trade-off when we are choosing block size.

In Fig.19 we show different stripe width. It is obviously that the 18 one almost has perfect linked boundaries. However, the over-repeated patterns in the same image can be observed. On the other hand, in the 6 one, the apples are arranged more randomly but there are more artifacts on the boundaries. This is because with larger stripe width, we will have more data to compute the SSDs so that we will finally pick a more similar one when using larger width, but again the choice range becomes smaller so a rapid repeating would not be a surprise.

As a result, we may compromise some artifacts within boundaries with a more natural look of the image, as we are showing in the second image in the Fig.19 with stripe width 12.

Figure 11. Stripe Width: 6, 12 and 18, Block Size 48*48

Last but not least, we want to point out that the choice of block size is an ad hoc problem because we are dealing with very different texture in various cases and it is reasonable to think that some specific block size and stripe width will perform better at each particular kind of texture. Therefore, we here show a group of result besides the apple case (block size 48 and stripe width 12) above in Fig.20 that we think they are better compared to using other parameters of size by experiment. We consider both less boundary artifacts and pattern diversity as good criteria when choosing parameters.

of 14

Figure 12.

Brick Texture Synthesis with Block Size 60 and Stripe Width 20

Weave Texture Synthesis with Block Size 48 and Stripe Width 18

texture segmentation and synthesis

Documents