spie proceedings [spie is&t/spie electronic imaging - burlingame, california, usa (sunday 3...

A modified Hierarchical graph cut based video segmentation approach for high frame rate video

Xuezhang Hua , Sumit Chakravartyb , Qi Shec and Boyu Wang d a ,c,dNanjing University of Post and Telecommunications, Nanjing, China;

b New York Institute of Technology, Nanjing, China

ABSTRACT

Video object segmentation entails selecting and extracting objects of interest from a video sequence. Video Segmentation of Objects (VSO) is a critical task which has many applications, such as video edit, video decomposition and object recognition. The core of VSO system consists of two major problems of computer vision, namely object segmentation and object tracking. These two difficulties need to be solved in tandem in an efficient manner to handle variations in shape deformation, appearance alteration and background clutter. Along with segmentation efficiency computational expense is also a critical parameter for algorithm development. Most existing methods utilize advanced tracking algorithms such as mean shift and particle filter, applied together with object segmentation schemes like Level sets or graph methods. As video is a spatiotemporal data, it gives an extensive opportunity to focus on the regions of high spatiotemporal variation. We propose a new algorithm to concentrate on the high variations of the video data and use modified hierarchical processing to capture the spatiotemporal variation. The novelty of the research presented here is to utilize a fast object tracking algorithm conjoined with graph cut based segmentation in a hierarchical framework. This involves modifying both the object tracking algorithm and the graph cut segmentation algorithm to work in an optimized method in a local spatial region while also ensuring all relevant motion has been accounted for. Using an initial estimate of object and a hierarchical pyramid framework the proposed algorithm tracks and segments the object of interest in subsequent frames. Due to the modified hierarchal framework we can perform local processing of the video thereby enabling the proposed algorithm to target specific regions of the video where high spatiotemporal variations occur. Experiments performed with high frame rate video data shows the viability of the proposed approach. Keywords: Graph Cut, Image Analysis, Video Image Processing, Hierarchical Grap Guts, Hierarchical Video Segmentation, Multiresolution

1. INTRODUCTION In computer vision bilayer Video Segmentation (bilayer VS) is frequently desired task which has many applications. They include video edit, video composition, object recognition. Generally, a bilayer VS system has two primary segments. This includes object tracking and video segmentation. There are numerous algorithms to solve object tracking such as mean shift algorithm [1], particle filter [2], classifier based design like random forest and multiclassification. There are also a great deal of works on object segmentation, such as level set methods [6], graph cut [7] and grab cut [8].

Image Processing: Machine Vision Applications VI, edited by Philip R. Bingham, Edmund Y. Lam, Proc. of SPIE-IS&T Electronic Imaging, SPIE Vol. 8661, 86610V · © 2013 SPIE-IS&T

CCC code: 0277-786X/13/$18 · doi: 10.1117/12.2008522

SPIE-IS&T/ Vol. 8661 86610V-1

Downloaded From: http://proceedings.spiedigitallibrary.org/ on 03/14/2013 Terms of Use: http://spiedl.org/terms

Figure 1.video based flare detection and control: an overview

In this paper we propose the use of video based monitoring system to monitor and quantify the amount of flare generated from an industrial flare using efficient multiresolution graph cut based segmentation techniques. We use digital videos d of the flaring from chimneys to obtain quantification for fire. A Flare serves as a safety device, as it burns off the gases vented from drilling or refinery process units. In petroleum refineries, it is a common practice to flare up the exhaust gases before releasing them to the atmosphere in order to reduce the environment pollution. If the flare stack flame is either out or is performing abnormally, then a) hazardous gases can be vented accidentally into the environment resulting in equipment damage, environmental pollution and possible human fatality b) an abnormal flare condition could indicate suboptimal performance of the refinery. Consequently, the continuous verification and control of the presence of the flame is critical for safety and performance and is a highly significant requirement. Figure 1 shows an overview picture for video based flare detection operation. In this proposal we address the monitoring aspect of the problem using advanced image analysis technique called multiresolution graph cuts. A successful monitoring can generate alarms in case of critical process conditions. This would allow operators to take preventive actions before any catastrophic environmental hazard occurs and thus impacts not only performance but also safety of a plant. The current norm in industry is to either rely on manual monitoring (which being a very mundane task can cause poor performance or error) or use heat monitoring of flares (which can be misled based on surrounding environmental conditions and furthermore provide no information about the amount of un-burnt or residual gases). The merit of this proposal is to use a camera sensor to obtain live video images which are fed to and analyzed by a PC with special flare and smoke detecting and quantifying algorithm developed for this purpose.

2. TECHNICAL DESCRIPTION In this section we describe the technical details of the three approaches to video segmentation of object as undertaken in this paper. For graph cut based segmentation approach we have to provide an initial segmentation estimate as will be discussed in section 2.2. In section 2.1 we detail the preprocessing required for graph cut estimation for our application. Next we give a brief insight to graph cut segmentation and show its basic implementation for video segmentation application. In section 2.3 we discuss our first approach for efficient segmentation of video. Subsequently in section 2.4 we propose the new multiresolution based graph cut segmentation as our second approach. An optional hierarchical step

Video Monitoring

Standard filtering

Parameter extraction (e.g. colour, smoke)

flare

Pilot

flare stack

Controller

Set Point

SPIE-IS&T/ Vol. 8661 86610V-2


23-55N-frameo005

which can be used when coarse resolution based segmentation is not sufficiently accurate is also elaborated. In the next section we provide results for our experiments and draw conclusions.

2.1 Preprocessing for Graph Cut Implementation Graph cut based image segmentation requires aprori knowledge of pixel class distribution. So initial segmentation of the video frame is done. The accuracy of initial segmentation is not required to be as significant as that of the graph cut. Various procedures can be used as the preprocessing step. Such techniques include algorithms like k-means. For our analysis we use color based flare detection approach. For flame detection, the EO data is converted to HSI (Hue, Saturation and Intensity) space. The motivation behind it is the fact that red color which defines the significant component of flame has low hue values and typically moderate to high saturation value. The intensity parameter on the other hand is typically moderate. Figure 2 shows results of typical HSI space conversion. Figure 3 shows the results of flare flame extraction as a pseudo colored fire mask on top of the original flare image of figure 2(a).

hue sat int

a

b c d Figure 2. (a) shows a typical flare image. (b,c,d) shows the result of color space conversion to HSI space on the temporally processed

flare image(a). As seen from the image Hue as well as intensity provide good separation parameters

SPIE-IS&T/ Vol. 8661 86610V-3


Figure 3. Flare fire extraction

2.2 GraphCut based Image Segmentation To segment a video using graph cuts, we can perform individual graph cut segmentation on each video frame. An image can be represented as a graph wherein we are interested in partitioning the graph into a number of separate connected components. A partitioning of a graph is commonly represented as a graph cut. Graph cut technique has frequently been applied to image segmentation since been introduced by Boykov and Jolly [12]. It is well suited for binary labeling problem. Assume that an image is represented as a graph G = (V,E) with a set of vertices (nodes) V representing pixels or image regions and a set of edges E connecting the nodes. Each edge is associated with a nonnegative weight. The binary labeling problem is to assign each node i with a unique label xi, that is, xi {0 (background) , 1 (foreground)} such that X = { xi } minimizes the following energy function: E (X) = λ R(X) + B (X) where λ is a positive parameter for adjusting the relative weighting between R and B. R(X) is the data energy determining the energy to assign label xi to the mode i and B(X) is the smoothing energy term denoting the cost of assigning the labels xi and xj to adjacent pixels i and j. λ is the weighting factor with balances the weightage between the above two terms [7]. Given the initial foreground and background segmentation of the image, we can estimate the two energy terms provided they satisfy the non negative criteria. The maximum flow principle is used to optimize the Graph Cut. It states that to maximize flow means to route as much flow as possible from the source to the sink. The minimum cut principle is to minimize c(S; T),(figure 4) i.e. to find an s-t cut with minimal cost [5]. The max-flow min-cut theorem states: The maximum value of an s-t flow is equal to the minimum weight of an s-t cut [6]. Graph cut in essence tries to segment an image by constructing a graph such that the minimal cut of this graph will cut all the edges connecting the pixels of different objects with each other [11]. This approach of video segmentation does not consider any temporal correlation and is significantly computational expensive. Our next approach delves to identify and exploit temporal correlation amongst video frames.

SPIE-IS&T/ Vol. 8661 86610V-4


(a) Image with seeds.

1J

i;.;,ck.,,ivutkitarrninal

/I' . M

P T qCD CD

(d) Segmentation results.

Objeci. S Objectterminal terminal

(b) Graph. (c) Cut.

Figure 4: Graph Cuts as applied to image processing

2.3 Video GraphCut with temporal correlation Graph-based image processing methods typically operate on pixel adjacency graphs, i.e., graphs whose vertex set V is the set of image elements, and whose edge set E is given by an adjacency relation on the image elements. Commonly, E is defined as all pairs of vertices v and w such that d(v,w) <,where d(v,w) is the Euclidean distance between the points associated with the vertices v and w and is a specified constant. This is called the Euclidean adjacency relation. In 2D images, with pixels sampled in a regular Cartesian grid, d=1 gives a 4-connected graph. The edge weights in a pixel adjacency graph are typically chosen to reflect the image content in some way. The weights may be based on, e.g., local differences in intensity, or other features, between adjacent image elements. Performing individual graph cut on images is highly computationally intensive as it does not exploit any temporal correlation present in the video frame sequence. This becomes especially relevant when the video used has a high frame rate. In order to exploit the inter-frame correlation we can perform a temporal change detection process. The temporal change detection process instead of individual frames considers a sequence of image frames at a time. It then finds frames amongst the set which have significant deviation. Two approaches have been used for detection of frames having significant temporal deviations. In method 1 we perform inter-frame based detection wherein we check the amount of change in successive frames. We can use inter-frame change index as a threshold factor--first, we get the absolute value of the subtraction between the matrixs of the 1st and 2nd frames, if the value is bigger than that threshold, we will put the 2nd frame into the list, which we are going to perform graph cut on. And then we compare the absolute value of the subtraction of the matrixs of 2nd and the 3rd frame with the threshold. If the absolute value of the subtraction between the matrixs of the 1st and 2nd frames is not bigger than the threshold, we will compare he absolute value of the subtraction between the matrixs of the 1st and 2nd frames. We do the same processing to the whole sequence and get the highly variant frames. We will explain this method with the following algorithm flow in Figure5. The second method is to compare the amount of temporal deviation from statistical average of the ensemble. Figure 6 shows the flow diagram of the two procedures for frame detection.

SPIE-IS&T/ Vol. 8661 86610V-5


N Y N N Y Y

Figure 5 algorithm flow of the 2nd method

Stack of temporal frames

Inter frame temporal processing

Set of high temporal variation frames and their frame no

1

2

34

2

3

begin

i=2, templet=frame(i-1)

s>threshold

templet=frame(i)

Put frame (i) into highly variant frame list

i> frame No

i=i+1

i=i+1

end

s=|frame(i)-templet|

i>frame No

SPIE-IS&T/ Vol. 8661 86610V-6


(a)

(b)

Figure 6(a) Inter frame temporal processing (b) ensemble mean based temporal processing

2.4 Video GraphCut with temporal correlation and multiresolution analysis We propose next an entirely novel process of video object segmentation using graph cuts. This algorithm performs graph cuts on the temporal based selected frames as discussed in section 2.4. At first, we can generate the masks of the fire frames after converting the original frames into HSI frames and setting an threshold in HSI. The masks are shown in Figure 7. Second, we calculate the difference of every two continuous frames and get the sum of the differences, which can show the variation in that frame sequence. Actually, the high intensity (bright parts) in the sum image denotes that lots of variation happens there while low intensity (dark parts ) means less variation. Third, instead of performing full resolution spatial graph cut on the selected frames, we first change the resolution of the original according to the sum of the differences. At last, we perform graph cut on it.

Figure 7 mask sequence(shown partly)

We get the extent of variance of different regions by calculating the sum of pixels intensity in every region of interest. We propose to use a lower resolution for highly temporally variant regions. The argument behind this choice is that high temporal variant parts correspond to large change of object for which high precision of image mask in not necessary as

Stack of temporal frames

Inter frame temporal processing

Set of high temporal variation frames and their frame no

12

34

2

3

Average gray scale variation of the stack

SPIE-IS&T/ Vol. 8661 86610V-7


they will not have long enough temporal longevity. While on the other hand low temporal variant regions correspond to sufficient longevity of the temporal mask and hence the mask can be carried over to successive frames and hence we can afford for high resolution graph cut processing without incurring too high computational cost. Figure 8 shows multiresolution image processing wherein we see we can process images at a variety of resolutions. Figure 9 shows the work flow process of the proposed multiresolution graph cut video object segmentation. The Figure 9 shows a temporally varying significant frame is first obtained from the image stack. Depending upon the amount of regions' variation the resolution size is selected in step 2. In the next step graph cuts is performed on the image of adjusted resolution. This process reduces the computation complexity of performing graph cut on the frame to get the segmentation result as will be discussed in the next section when we present the experiment results.

Figure 8 Multiresolution image

Figure 9 Multiresolution based Graph Cut clustering wherein different regions are given different resolutions

2.5 Optional hierarchical representation Optional hierarchical representation can be used in case low resolution representation is does not provide a sufficiently accurate segmentation. In such a case technique similar to reference 15 can be used for multiple resolution graph cut analysis. We can use a test criteria based on the difference between graph cut results and initial segmentation to decide if such hierarchical representation is required.

3. EXPERIMENTATION

3.1 Segment the frame sequence sample basically Firstly, we choose 15 continuous frames and process this frame sequence with a basic method, which means that we just perform graph cut on every frame of this sequence without any preprocessing. Some of the segmentation results can be shown in Figure 10. Generally speaking, the segmentation is satisfying as long as graph cut can work. However, for the whole sequence, the success ratio of the segmentation algorithm graph cut is 61% (we perform graph cut on each of 15

Stack of temporal frames of video

SPIE-IS&T/ Vol. 8661 86610V-8


frames for 10 times and get 91 successful segmentations). We use k-means as initial segmentation in graph cut algorithm. The result of this initial segmentation is random and important for the 2nd segmentation of graph cut on the frame. As k-means algorithm is based on random initialization, the initial segmentation results vary to some extent every time we run the operation; thereby we get different result of the graph cut segmentation as well. Besides, for the segmentations, the mean time of performing graph cut on each frame is 4.5 seconds. So, it is unreasonable and unrealistic to segment every frame in a video since a video commonly includes thousands of frames. In order to decide the success of segmentation we perform logical set operation with the results of manual segmentation and count to number of common pixels and the number of dissimilar pixels. The ratio of the common pixels with respect to the total count of common and dissimilar pixels gives us a quantitative measure of the segmentation performance.

Figure 10 segmentation result

3.2 Graph Cut on highly variant frames got with inter-frame change index According to the first method to get the highly variant frame, we make the threshold equal to 0.4*width*length for the our frame sequence Then, we choose the 5th ,9th and 12th frames to segment. The result is shown in Figure 10. The total success ratio of the segmentation on these three images is 90% and the mean time of the successful segmentation is 3.7seconds, both of which are improved apparently compared with those in the basic method. The improvement of the success ratio of segmentation and runtime is mainly because the backgrounds in the frames we choose are relatively more different with the foregrounds in them, which shows that our method works when it comes to pick up those different frames with obvious features.

3.3 Graph Cut on the most variant frame with changed Resolution Using our 2nd method to choose the highly variant frame, we can get the most variant frame in the sequence. And, we can also get the sum of the differences between every two continuous frames to show the temporal variance in the sequence. The sum of differences is shown in Figure 11. Before we perform graph cut on it, we change the resolution of the image according to Figure 11. At last, we perform graph cut on the frame with changed resolution and get that the segmentation success ratio on the preprocessed frame is 100% and the mean time of segmentation is only 3.7 seconds. However, the segmentation success ration of graph cut on this frame with original resolution is only 70% and the mean time of successful segmentation is 8.2 seconds, which shows that k-means works stably and well after the changing resolution processing This is primarily because with lower resolution, the variability in the data is reduced and soothe randomness of the k-means initialization is also curtailed. In other words, our method of lowering the resolution of the regions of high variance is effective in reducing noise and lowering the difficulty of performing graph cut on the frame of interest. Besides, the segmentation results of both of these two images are almost the same as shown in Figure12.

SPIE-IS&T/ Vol. 8661 86610V-9


Figure 11 sum of the differences between Figure 12 segmentation result every two continuous frames in a sequence

At last, we choose another frame sequence whose background and foreground are obviously more different, which means that it is easy for us to separate the background and foreground with graph cut. At this time, we also change the resolution of the most variant frame according to the sum of the differences of masks. The result shows that both of the segmentation success ratios are 100% and it takes the same time to do the segmentation. However, the segmentation of the frame with adjusted resolution is still a little better than that of the frame with original resolution, which can be shown in Figure 13. Table 1shows the details regarding performance of our methods and processing.

a b Figure 13 a) segmentation of the frame with changed resolution b) segmentation of the frame with the original resolution

SPIE-IS&T/ Vol. 8661 86610V-10


Table 1: Table showing Comparative performance of the different methods

Method No of processed

frames

Mean time of successful segmentation(s)

Success ratio of graph cut

Segmenting all the frame s without preprocessing

15 4.5 61%

Segmenting the highly variant frame after preprocessing in 1nd way

3 3.7 90%

Segmenting the most variant frame after preprocessing in 2nd way(30

times)

1 3.7 100%

Segmenting the most variant frame with original resolution (30 times)

1 8.2 70%

4. CONCLUSION Performing individual graph cut on images involves lots of computation intensive since it does not exploit any temporal correlation present in the video frame sequence. However, in our paper, we get the temporal correlation in the video frame sequence successfully with two methods. With the first proposed method, we get the highly variant frames and reduce the runtime of graph cut apparently. Besides, the mean time of successful segmentation is reduced and the segmentation success ratio has also been improved obviously. With the second method, we get the most variant frame in the frame sequence. Afterwards, we change the resolution according to different regions' temporal variance. As is shown in the result, we improve the success ratio of segmentation by 30% and reduce the mean time of segmentation by more than 50%(details shown in table 1). All of these have shown that, our methods to get the temporal correlation are effective and function very well to improve efficiency of running graph cut on the frames in the video.

REFERENCES

1. A. Yilmaz, O. Javed, and M. Shah, “Object tracking: A survey,” ACM Comput. Surv., vol. 38, no. 4, p. 13, 2006. 2. D. Comaniciu and P. Meer, “Mean shift analysis and applications,” in The Proceedings of IEEE International Conference on Computer Vision, vol. 2, 1999, pp. 1197 –1203. 3. K. Nummiaro, E. Koller-Meier, and L. J. V. Gool, “An adaptive color based particle filter,” Image Vision Comput., vol. 21, no. 1, pp. 99–110, 2003. 4. H. Grabner and H. Bischof, “On-line boosting and vision,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, 2006, pp. 260 – 267. 5. A. Saffari, C. Leistner, J. Santner, M. Godec, and H. Bischof, “On-line random forests,” in IEEE International Conference on Computer Vision Workshops, 2009, pp. 1393 –1400. 6. A. Reza Mansouri and J. Konrad, “Motion segmentation with level sets,” in IEEE International Conference on Image Processing, 1999, pp. 126–130. 7. Y. Boykov and M. pierre Jolly, “Interactive graph cuts for optimal boundary and region segmentation of objects in n-d images,” in IEEE International Conference on Computer Vision, 2001, pp. 105–112. 8. C. Rother, V. Kolmogorov, and A. Blake, “Grab cut: interactive foreground extraction using iterated graph cuts,” ACM Transactions on Graphics, vol. 23, pp. 309–314, 2004. 9. Richard Szeliski, “Computer Vision:Algorithms and Applications” ,Springer. Filip Malmberg, “Graph-based Methods for Interactive image Segmentation”, ACTA Univeritatis Uppasala,2011 10. Jan erik Solem, “Programming Computer vision with Python”, O’Riley. 11.Wikipedia, 2012,” http://en.wikipedia.org” 12. M. Sonka, V. Hlavac, R. Boyle, “Image Processing, Analysis, and Machine Vision”,Thomson.

SPIE-IS&T/ Vol. 8661 86610V-11


spie proceedings [spie is&t/spie electronic imaging - burlingame, california, usa (sunday 3...

Documents