F.L. Wang et al. (Eds.): CMSP 2012, CCIS 346, pp. 8–15, 2012. © Springer-Verlag Berlin Heidelberg 2012

A Multi-features Based Particle Filtering Algorithm for Robust and Efficient Object Tracking

Shuang Ye1,3, Yanguo Zhao1, Feng Zheng1, and Zhan Song1,2

1 Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China

2 The Chinese University of Hong Kong, Hong Kong, China 3 Wuhan University of Technology, Wuhan, China

{shuang.ye,yg.zhao,feng.zheng,zhan.song}@siat.ac.cn

Abstract. This works presents a novel approach for robust and efficient object tracking. To make the feature representation more robust, color and the local binary pattern features are fused via a proposed scheme. The partial filter is used for the feature tracking. To improve its efficiency, a mean shift based method is introduced to decrease the required partials so as to decrease the computation cost. With the robust multi-features description and boosted partial filter algorithm, satisfied tracking results can be obtained via the experiments with different datasets, and showed distinct improvements in both tracking robustness and efficiency.

Keywords: object tracking, multi-features, mean-shift, particle filter.

1 Introduction

Object tracking has been an important research issue in computer vision domain. Current tracking algorithms can be generally divided into two categories according to Ref. 1: the certainty method and the random method. The former tracks the target by looking for the optimal matching target, such as the mean-shift (MS) algorithm [2] .The later tracks the target via state estimation, such as the particle filter (PF) algorithm [3]

Various image features have been used for the target representation and tracking. However, the tracking via sole feature is usually lack of robustness subject the complicate background and scenarios. Much of recent methods have focused on the fusion of multiple image features. [4-12] In Ref. 4, a PF-based method combined with multiple image cues is proposed for the object tracking, but its efficiency still needs improvement for real-time application. In Ref. 7, a geometric particle filter algorithm is presented based on the affine group with optimal importance functions. In Ref. 12, a new approach combined with MS of regional color distribution and the PF algorithms is introduced for the efficient object tracking.

In this paper, a multi-features based approach for the efficient and robust object tracking is presented. The color and local binary pattern (LBP) features are adopted

A Multi-features Based Particle Filtering Algorithm 9

and fused for the robust feature representation. And then, the traditional partial filter method is improved by the introduction of mean shift method to boost its efficiency. The combined tracking algorithm utilizes advantages of the two algorithms and shows distinct improvements in both robustness and efficiency via extensive experiments and comparisons.

The paper is organized as follows. Section 2 presents the fusing scheme of color and LBP features. Section 3 introduces how the mean shift method is used to improve the efficiency of the traditional partial filter method. Experiments are given in Section 4 and the conclusion is offered in Section 5.

2 The Fusion of Color and LBP Features

Let {xi i=1,…,n} be the normalized pixel location in the target region centered at 0, n is the pixel number, b(xi) associates the pixel xi to the histogram bin. By diving the feature space into m subspace, the target model and the probability of the feature u=1,…,m in the target area can be expressed as:

1,...,{ }u u mq = q =

( ) ( )[ ]2n

i i

i=1uq = C k x δ b x - u (1)

where k (.) is the kernel profile, δ is the Kronecker delta function. C is norm-aliz

ed coefficient defined as 2=1 (|| || )

niiC 1 k x/= .

Similarly, let {xi , i=1,…,nh} denote the pixel positions in the candidate region centered at y in the current frame, the target candidate model p(y) can be defined as:

( ) { ( )}u u=1....mp y = p y ( )2(|| ( ) / ||( ) )hn

ii=1

u h ip y C k δ b x - uy - x h=

(2)

Where pu(y) is the feature u=1,…,m in the target candidate area, Ch is a normalized

coefficient defined as 2=1 (||( )/ || )hn

iihC 1 k y-x h/= , h is the bandwidth defines the

scales of the target candidate. The image is firstly transformed from the RGB space to HSV space. The b(xi) is

represented as C(xi) in HSV space, and the histograms of HSV channels are divided into 8×8×4 bins according to Eqn.(2), then we obtain the color model.

To ensure the model to be adaptive to illumination changes, the feature of texture is represented via the well-known LBP operator, which is defined as follows:

-1

=1( ) ( )2

P i

P,R p ciyLBP = s g - g

(3)

where R is the distance between the central pixel xi and its neighbors, P is the number of neighboring pixels, gc is the intensity of centeral point y, gp is the intensity of P which distributed equally on the circle with radius of R. s(x) is a two-valued function, which is equal to 1 if x>0. Since s(.) is only related to the relative pixel intensity, LBP is robust to illumination changes.

10 S. Ye et al.

By varying P and R, we have LBP operators under different quantization of the angular space and spatial resolution and multi-resolution analysis can be accomplished by using multiple LBPP, R operators. In this work, as shown in Fig. 1, we choose the LBP8, 1. b(xi) is represented as LBP (xi) in LBP, we obtain the LBP model according to Eqn. (2), the LBP histograms in target area are divided into 27=128 bins. The LBP image is obtained by LBP8, 1 as shown in Fig.2.

35

79

91

57 33

124

24

4477

0

0

0

63 1 1

1

1

1

1

thresholding 2(01011111) = 95

Fig. 1. Illustration of LBP8, 1

Fig. 2. The LBP image processed with LBP8, 1.

The tracking procedure is performed via two stages. Fuse the color and LBP features to get the weights in the PF and MS algorithm, and integration of MS and PF to prevent degeneracy problem and reduce the required number of particles. With respect to the two stages, how to fuse the two image features are depicted as follows.

Treated as the observation information, the Bhattacharyya distance between target

and candidate region histograms

i

cd and i

ld can be calculated by ( )d = 1 - ρ y with

the Bhattacharyya coefficient [ ]( ) ( ) ( )m

i=1 u uρ y ρ p y ,q = p y q≡ . Given the

variance of Gaussian distribution σ, define 2 2( ) /exp ( ) /2 2i i

c cw d πσ σ−= as the

weights extracted from color information, and 2 2( ) /exp ( ) /2 2i i

l lw d πσ σ−= as the

weights extracted from LBP. And denote the likelihood measurement (Z | X )i

k kp as

the weights of particles in PF. In the PF algorithm, we fuse the color and LBP information to get the particle weights by multiplying the weights:

2

2 2

2

( ) ( ){ } /

2exp 2

+i i i

c l

i il cd d

w w wσ

πσ= −= (5)

A Multi-features Based Particle Filtering Algorithm 11

Further iterative optimization by MS is necessary to prevent degeneracy problem and reduce the required number of particles, we compute the weights of pixels in target area λi by additive fusion in MS stage.

i i

i c l= α +(1 - α)λ λ λ (6)

Where i

cλ is the weights extracted from color information, i

lλ is the weights extracted

from LBP according to following equation, λi is obtained by Taylor expansion around

the values0

( )u

p y . For more information about MS, please refer to Ref. 2.

0[ ( ) ] / ( )

i u u

m

i=1

i = δ b x - u q p yλ

(7)

α is a balancing variable, which can be set to 0.5 empirically. The feature fusion procedure can be depicted as Fig. 3.

Fig. 3. Fusion procedure of color and LBP features

3 Tracking Algorithm Based on the Multi-features

The main idea of the PF is an infinite approximation of the pdf(posterior probability density function)of the system state with a set of weighted particles that sampled from the importance density function q (.).The major limit of PF is the limited capability of the weighted particles which describe the pdf when the state space is not densely sampled. To overcome this problem, a large number of particles is required thus increasing the computational load. For more information about PF, please refer to Ref. 3. MS algorithm can find the most similar region to the target in the new frame, but if the center of the object shifts more than the kernel size in two consecutive frames or there is an occlusion, the tracking is likely failed. How to overcome the defects of MS and PF and inherit their advantages is a key problem.

Based on the fused image features, a combined tracking algorithm is also introduced in this work. Firstly Np particles are sampled from q (.) by PF algorithm. Secondly, Nm particles near the possible position that calibrated by MS are randomly sampled. They effectively utilize the observation information of the target, keep the diversity of particles thus reduce the required number of particles, in addition, the Np

12 S. Ye et al.

particles restrain the defects that MS is easy to fall into local optimum. Thirdly, both the Np particles and Nm particles are updated by PF and we obtain the state output of the Np particles and Nm particles. And then, likelihood in the state output of Nm and Np

particles are calculated. If the likelihood in the state output of Nm particles is larger than the Np particles, remove Nm particles in the Np particles and add the Nm optimized particles by MS method and we obtain new Np particles. Otherwise, remain the primary Np particles. Finally, the final position of target with new Np particles can be computed. Details about the tracking procedure can be depicted as follows.

1) Initialization a) Extract the color and LBP template at frame k=0, and calculate the LBP and

color histograms of the target model; b) Set Np as the particles number that sampled from q(.) in PF stage, Nm as the

particles number that sampled in MS stage, Nthr as the resample threshold;

c) Sample the initial particles S={ X i

0, i=1…. Np } from the prior probability

density function. Particles weights are set as 1/ Np, let k=1. 2) Sampling

In PF stage: Load the next frame, transfer particles of last time by target State

transition model X = (X ) +k k -1 k -1

f u.

For i = 1,…, Np, sample the particles

{ X i

k, i=1…Np} at time k from q(.)= (X | X )i i

k k -1p , then the particles set is

updated to be Sp= {0:

X i

k} =

0: 1{X X | , , }, i i

k k pi 1 N− = ….

In MS stage: Search the target possible position by MS and calibrate it by back

projection to a new position y1=2 2

1 1

/( ( ) / ) ( ( ) / )n n

i i i i i

i i

h h

g y - x h x g y - x hλ λ= =

according to Ref. 2, the weights λi is calculated by additive fusion according to Eqn. (6). Randomly sample Nm particles near the new position as the PF stage

do , the particles set is Sm= {0:

X i

k} =

0: 1{X X | , , }, i i

k k mi 1 N− = … .

3) Weight update While a measurement Zk is available, extract color and LBP features, calculate

observation likelihood function i

cw and i

lw . i

kw

is obtained by multiplying the

weights extracted in color and LBP according to Eqn. (5). The weights of Nm

particles Sm and Np particles Sp are updated to:

(Z | X )i i i

k k -1 k kw w p∝ i∈ Nm or Np (8)

After normalization, the weights of particle set Sm and Sp can be calculated as:

1

/N

i i

k k

i

mi

kw w w=

= ，

1

/N

i i

k k

i

pi

kw w w=

= (9)

4) State output State estimation of two particle set Sp and Sm can be calculated as:

A Multi-features Based Particle Filtering Algorithm 13

1

X XN

ii

k k

i

p

p w=

≈ , 1

X XN

ii

k k

i

m

m w=

≈

(10)

The Bhattacharyya likelihood between target candidate position X p , Xm and the

target position can be calculated. If the likelihood in Xm is larger than X p

,

randomly remove the low-ranking Nm particles in the Np particles and add the Nm particles to the Np particles. Otherwise remain the primary Np particles.

5) Resample Compute an estimate of the effective number of particles as Neff

2

1

1 / )N

i

k

i

p

effN w=

= （ (11)

If Neff <Nthr, copy the particles with large weight and remove the particles with small weight.

Let i

kw =1/ Np, i=1... Np, we can obtain the new particles set { X j*

k, i

kw ,

j=1….Np } as the initial particles at time k+1, and then go back to step 2).

4 Experiment Results

The experiments are conducted on three video clips, face.avi, hand.avi, and soccer.avi. The former two videos are captured via a normal USB webcam at 25fps with the resolution of 240×320 pixels. The last one is downloaded from public database with the resolution of 480×360 pixels. The algorithms are implemented with OpenCv2.0 and vs. 2008 on a desktop with 2G RAM and Core2 2.4GHz CPU. The MS algorithm from Ref. 2, and the PF algorithm from Ref. 3 are used for the comparison.

To compare the efficiency, the traditional PF method can achieve a running speed of 75fps on the face and hand datasets. In comparison, the proposed method is more efficient with a speed of 125fps. Since the particle number of 100 is required by PF, only 70 particles are needed in the proposed method.

Fig. 4(a) shows the tracking results under illumination changing. The 1st row shows the results by our method, the 2nd and 3rd rows are the results by MS and PF algorithms respectively. As the result shows, when illumination suddenly changed from bright to dim, the proposed algorithm can still locate the eyes precisely. But the MS and PF algorithms are seriously affected. It strong robustness to illumination change mainly comes from the use of LBP feature in the proposed algorithm.

Fig. 4(b) evaluates the performance of different algorithms with occlusions. The 1st row shows the result by the proposed algorithm, and the 2nd and 3rd rows show the results by PF and MS algorithms respectively. When a partial of the hand is occluded, the PF and our method still work but the MS algorithm failed. When occlusion is seriously, both PF and MS algorithms loss the tracking target.

The dataset of soccer.avi is used to evaluate the performance of different algorithms under the target with continuous and huge appearance change. The

14 S. Ye et al.

tracking results on the frame 23, 96, 120, 132, 165, 235, 278 are given for comparison. As time going on, the degradation of particles is seriously by PF method. As shown in Fig. 5, the 1st row shows the tracking result by the proposed method in different frames, and the results by MS and PF algorithms are displayed in the 2nd and 3rd rows. When it goes to 835th frame, there is serious deviation by the method of PF. However, the proposed tracking method still works well, since it can effectively utilize the observation information of the target in current frame. And thus, it can keep the diversity of particles and treat well with the degeneracy problem.

(a) (b)

Fig. 4. Tracking results by the proposed algorithm (1st row), MS (2nd row) and PF (3rd row) methods. (a) Tracking of eyes on the dataset of face.avi; (b) Tracking of hand with conclusion on the dataset of hand.avi.

Fig. 5. Experiment with the dataset of soccer.avi, the tracking results by the proposed, MS, and PF are illustrated in 1st, 2nd, and 3rd row respectively

5 Conclusions

This study presents a fused approach for the efficient and robust object tracking. To represent the target more robust, both color and LBP features are adopted. To improve the tracking efficiency, the mean shift method is introduced to boost the performance of traditional particle filter algorithm. MS preserves the diversity of particles and reduce the required number of particles, the particle filter restrains the defects local

A Multi-features Based Particle Filtering Algorithm 15

optimum. Experiments with several datasets with illumination change, occlusion and appearance change demonstrate the improvements by the proposed algorithm in both robustness and efficiency.

Acknowledgments. This work was supported in part by the National Natural Science Foundation of China (NSFC, grant no. 61002040), NSFC-GuangDong (grant no. 10171782619-2000007), and the Introduced Innovative R&D Team of Guangdong Province-Robot and Intelligent Information Technology R&D Team.

References

1. Yilmaz, A., Javed, O., Shah, M.: Object tracking: A survey. ACM Journal of Computing Surveys 38, 13–57 (2006)

2. Comaniciu, D., Ramesh, V., Meer, P.: Kernel-based object tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence 25, 564–577 (2003)

3. Isard, M., Blake, A.: Condensation—Conditional Density Propagation for Visual Tracking. International Journal on Computer Vision 25, 5–28 (1998)

4. Brasnett, P., Mihaylova, L., Bull, D., Canagarajah, N.: Particle Filtering with Multiple Cues for Object Tracking in Video Sequences. Image and Vision Computing 25, 1217–1227 (2007)

5. Hui, T., Yiqin, C., Tingzhi, S.: Face Tracking using Multiple Facial Features based on Particle Filter. In: 2nd International Asia Conference on Informatics in Control, Automation and Robotics, vol. 3, pp. 72–75. Elsevier, Wuhan (2010)

6. Tang, S.L., Kadim, Z., Liang, K.M., Lim, M.K.: Hybrid Blob and Particle Filter Tracking Approach for Robust Object Tracking. In: International Conference on Computational Science, vol. 1, pp. 2549–2557. Elsevier, Wuhan (2010)

7. Kwon, J., Lee, K.M., Park, F.C.: Visual tracking via geometric particle filtering on the affine group with optimal importance functions. In: International Conference on Computer Vision and Pattern Recognition, pp. 991–998. IEEE Press, Miami (2009)

8. Kwon, J., Lee, K.M.: Visual tracking decomposition. In: International Conference on Computer Vision and Pattern Recognition, pp. 1269–1276. IEEE Press, San Francisco (2010)

9. Comaniciu, D., Ramesh, V., Meer, P.: Real-time tracking of non-rigid objects using mean-shift. In: International Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 142–149. IEEE Press, Hilton Head Island (2000)

10. Fang, J., Yang, J., Liu, H.: Efficient and robust fragments-based multiple kernels tracking. International Journal of Electronics and Communications 65, 915–923 (2011)

11. Leichter, I., Lindenbaum, M., Rivlin, E.: A probabilistic framework for combining tracking algorithms. In: International Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 445–451. IEEE Press, Washington (2004)

12. Deguchi, K., Kawanaka, O., Okatani, T.: Object tracking by the mean-shift of regional color distribution combined with the particle-filter algorithms. In: International Conference on Pattern Recognition, vol. 3, pp. 506–509. IEEE Press, Cambridge (2004)