2008 International Conference on Computer Science and Software Engineering
Multiple Non-Rigid Objects Tracking by Modified Kernel Particle Filter
CHENG Xiao-ping， CHEN Jia-shu
Department of Computer Science and Information Technology, SouthWest University,
Chongqing, China 400715
xpcheng@swu.edu.cn,
swucjs@swu.edu.cn
Abstract
This paper proposed a new method for multi-target
tracking in video sequences by combining two trackers,
sum-of-squared differences (SSD) and kernel particle
filter (KPF). In our work, the idea of Object Likelihood
Value of pixel is proposed. Instead of using direct
propagation resample result from the previous sample
set, a weighted SSD displacement is used for
reinitializing and resample before next KPF iteration.
Experiment results on soccer athletes tracking show
that the combination of SSD with KPF tracker forms a
simple and powerful multiple non-rigid object tracking
system.
1. Introduction
Object tracking is an important task in many
computer vision applications including surveillance,
gesture recognition, smart rooms, vehicle tracking,
augmented reality, video compression, and medical
imaging, etc. The tracking of real-world objects is a
challenging problem due to the presence of noise,
occlusion, clutter and dynamic changes in the scene
other than the motion of objects of interest. A variety
of tracking algorithms have been proposed and to
overcome these difficulties. The particle filters, also
known as condensation or sequential Monte Carlo,
have proved to be powerful tools for object tracking
[7]. The current density of the state is represented by a
set of random samples with associated weights and the
new density is computed based on these samples and
weights. Kernel Particle Filter (KPF) was proposed
because the standard particle filter often fails to
produce a particle set that captures the “irregular”
motion under some dynamic system conditions [2]. In
KPF, a kernel density estimate (KDE) [10] is used to
approximate the posterior PDF. In particular, KPF
estimates the gradient of the kernel density and moves
particles toward the modes of the posterior, leading to
978-0-7695-3336-0/08 $25.00 © 2008 IEEE
DOI 10.1109/CSSE.2008.1173
a more effective allocation of particles. The gradient
estimation and particle allocation is implemented by
the mean shift algorithm [1], [11]. [2] Which improved
sampling efficiency and the visual tracking robustness.
Though Kernel Particle Filter performs well on
sequences with relatively small object displacement, its
performance is not guaranteed when the objects move
fast as well as when they undergo partial occlusion,
especially during long and continuous full occlusion.
The motion pattern of objects may changes
dramatically in the situation listed above. Here, we
attempt to improve the performance of KPF tracker
when the object undergoes large displacements (when
the object regions do not overlap between the
consecutive frames) and in the event of partial /full
occlusion. The problem of large displacements is
tackled by cascading an SSD (sum-of-squared
differences) tracker with the KPF tracker.
The outline of this article is as follows. In Section2
we briefly give the key steps of the Kernel Particle
Filter and in Section 3 we describe our modification of
SSD algorithm. In section 4, the SSD-KPF tracking
system we proposed is discussed in detail and in
Section 5 we present some experimental results.
Finally, in Section 6, we summarize our research.
2. Kernel Particle Filter
2.1 Estimation of Posterior PDF and the
gradient of PDF
Kernel density estimate (KDE) [10] is used to form
a continuous estimate of the posterior in order to
facilitate gradient estimation. The target is
characterized through the KDE. Given a particle set at
n N
time t : st ={ st }n =1 and associated weights {πt
n N
}
n=1 ,
the KDE approximates the posterior PDF with kernel
K can be formulated as
903
 N ( n) approximated density modes, but the weight will
pˆٛ( Χt |Ζ1:t ) = ∑ K h ( Χt − st )π tn (1) contain a factor that offsets this effect.
n =1
Where K h is the kernel scaled by the width h ;
3 Modified SSD tracker
⎛ ( n) ⎞
( n) 1 ⎜ Χt − st ⎟
K h ( Χt − st ) = K (2)
h nx ⎜⎝ h ⎟
⎠
SSD tracker localizes the object in a given search
window of the successive frame based on minimum
The kernel and width h are chosen so as to minimize distance criterion between the target and candidate
the Mean Integrated Square Error (MISE) between the ˆ
posterior PDF and the corresponding kernel estimate. object images. Given the state estimate Χ t −1 at
Given the approximation of the posterior PDF, the time t − 1 , the SSD-based displacement estimate is
steepest direction toward the mode of the posterior ssd
could be searched, which can be found employing the D = arg m in ∑ [ I ( i+ Χ
ˆٛ ˆٛ
t − 1 + j ) − I t − 1 ( Χ t − 1 + j )]
2
t j∈ R
t ( 7)
mean shift procedure [11]. In this procedure, each i∈ S

particle is moved to its sample mean determined by Where I t and I t −1 are the two consecutive intensity
N (l ) l ( n ) (l )
∑ st π t g ( st − st ) images, S is the search window, and R is the ellipse
( n ) l =1 modeled the object.
st′ = (3)
N l ( n ) (l )
∑ π t g ( st − st )
l =1 3.1 Object Likelihood Factor and Weighted
After several iterations, the vector s t′ ( n ) − s t( n ) Displacement Estimation
would be in the gradient direction of (1) [11].
In the initialize step, there must be some method
2.2 Particle Reweighting (either manually or automatically) to segment object to
tracking from the background. Since the object is
(roughly) segmented in this step, it is relatively easy to
When particles change their positions, the new obtain color distribution of the object, which generally
particles do not follow the posterior distribution significantly different from that of the background. We
anymore. This is circumvented by re-weighting the can employ this characteristic to determine whether a
particles. Denote the particle set after the i th mean pixel belong to the tracking object. We define Object
shift procedure at time t as {st(,⋅i) } . After each mean Likelihood Factor (OLF) of a pixel in tracking scene at
time t as
shift procedure, the weight is recomputed as the
posterior density evaluated at the new particle
m ax { f i , o , ε } (7)
L ( i ) = lo g
positions augmented with a particle density balancing
t m ax { f i , b ,ε }
factor [2, 3]
(n) Where f i ,o is the probability that a pixel belong to the
p ( st ,i | Ζ1:t )
π t(,ni ) = (4) object, while f i ,b is the probability that a pixel belong
(n)
qt ,i ( st ,i )
to background. f i ,(⋅) is r-g-b combinational PDF， ε
Where the denominator is the new proposal density is a non-zero small positive number. It is evidently
that captures the non-uniformity of the new particle
set. if fi ,o > fi ,b ， Lt (i ) is positive ， else it is negative.
N Our view is that during computation of SSD
(l )
qt ,i ( Χt )= ∑ K h ( Χt − st ,i ) (5) displacement, the contribution of pixels in matching
l =1 template should not be taken as equal. The pixel more
Thus the posterior density can be given by likely on the object should assign greater weights.
(n) (n) N (n) (l ) Thus we have the following Weighted Displacement
p( st |Ζ1:t )∝ p( Ζt |st ) ∑ Kh ( st −st −1)πtn−1 (6)
l =1
Estimation
When mean shift is performed on st , the next
generation of particles will concentrate more on the

904
 ssd ˆٛ + j ) − I ˆٛ
D = arg min ∑ Wt [ It ( D +Χ t −1 t −1 ( Χ t −1 + j )]
t
D∈S j∈R
In which
⎧ L (i ) Lt (i )>θ
Wt (i ) =⎨ t (10)
⎩ 0 else
θ is the threshold of the likelihood. In our system,
θ =0.8. With the information of Lt (i ) , we need not
compute every pixel of template. In the case of partial
occlusion, the occluded particles with zero OLF value
would be discarded and only those particles really
contribute to tracking process left.
3.2 Occlusion Order Determination
To determine the occlusion order is important to
interpret the interaction among objects during
occlusion. The Dtssd originated from SSD represents
pixel-wise intensity conservation of a search blob.
Figure 1 shows two objects overlapped each other.
Object A is above on B. We can obtain more
appearance information of A than that of B obviously.
In other words, object A conserved more intensity than
B. It is easy to assume that Dtssd − A > Dtssd − B is true in
generic situation. Inversely, we can follow occlusion
order from the size relation of the objects during
occlusion. To improve the confidence of the induced
ssd
result, we can set a threshold value DT . If
Dtssd − A ≥ Dtssd − B + DTssd (11)
is true, we believe object A is above on B.
Figure1. A scenario of two targets overlapped.
4 SSD Modified KPF Tracker
The idea discussed above implemented as follows:
1 Initialized searching window according to object size
2 2 Compute combinational PDF of the object and
(9) (9)
background,
3 Compute pixel OLF value.
4 Compute a weighted SSD displacement Dt
ssd
by (11)
5 Update particle set st′,0
( ⋅)
= st(,0⋅) + D(10)
ssd
t
6 Do Mean-Shift KPF tracking procedure using those
particles with Li(t)> threshold,
7 Go step2 till the last frame processed.
5 Experiments
The proposed algorithm was applied to many
sequences and found it work very robustly despite the
difficulties of suddenly change of velocity, background
clutter, and temporary occlusion.
The first experiment shows our approach is robust to
resolve the motion correspondence even under the
background contained similar targets when tracking a
single object (see Fig. 2). The tracker can still keep
tracking the soccer player on the right after walking
through a crowd of soccer players with similar clothing
color.
Figure2. Tracking result of a soccer player passing-by a group of
soccer players. Frames shown 8, 10, 19, 21
The second example (in Fig. 3) shows the result of
multi-target tracking. There are several players going
through the scene in different directions, which result
many instances of their passing-by each other. In order
to test whether our algorithm can track objects
undergoes large displacement, we set a low FPS rate at
3fps. Our algorithm performs well as expected.
905

Fig .3 Tracking result of multiple soccer players. Frames shown 9, 11,
13, 15, 18, 20
The third example (in Fig. 4) is a segment includes 2
players interacting from a long sequence. Player 1 is
occluded by player 2 in several frames. We can see our
tracker can still distinguish the one from the other,
ssd − player1 ssd − player 2 ssd
since the condition Dt ≥D
t
+D
T
is
always true during occlusion. In frame 12, it can still
track the player 1 after long and continuous almost full
occlusion.
Figure 4. Tracking result of two soccer players. The player in blue
ellipses was occluded by the other player in several continuous
frames. But he can still be captured after he split from the group.
Frames shown 1, 4, 12
6 Conclusions
Multiple non-rigid objects tracking such as athletes
tracking is quite a challenge for computer vision. We
believe single type tracker always have its own
limitation. In this paper, we have proposed an efficient
visual tracker by coupling SSD and KPF, which have
complementary properties. The key idea is to improve
KPF particle set update by a weighted SSD algorithm,
which moves the particles in KPF tracker to the most
dominant modes quickly, since SSD give direct object
pixel motion information between frames. While KPF
trackers are used here for finer adjustments of object
location,
906
With help from SSD, the combined tracker can
address situations when objects undergo long and
continuous occlusion. It is expected the successfully
“fusion” of different type of tracking method will
further improve the performance of tracking system.
[1] D.Comaniciu,V. Ramesh, and P. Meer. Kernel-based
object tracking. In IEEE Trans. Pattern Anal. Machine
Intell.25 (5) (2003) 564-577.
[2] C.Chang and R. Ansari. Kernel particle filter for visual
tracking. In IEEE Signal processing Lett., 12(3):242-
245, March 2005.
[3] C.Chang and R. Ansari. Multiple objects tracking with
kernel particle filter. In IEEE CVPR 20-25 June 2005
Page(s):566-573 vol.1.
[4] Babu, RV and Perez P. Robust tracking with motion
estimation and local Kernel-based color modeling.
Image and Vision Computing 25 (8): 1205-1216 AUG 1
2007.
[5] T. B. Moeslund and A. Hilton. A survey of advances in
vision-based human motion capture and analysis.
Comput. Vis. Image Understanding, vol.104, no.2-3, pp.
90-127, 2006.
[6] K. Okuma, and A. Taleghani. A boosted particle filter:
Multitarget Detection and Tracking. In ECCV, 2004.
[7] Pérez, P., Hue. C, Vermaak, J., Gangnet, M.: Color-
Based Probabilistic Tracking. In ECCV,2002.
[8] M. Isard and A. Blake. Condensation——conditional
density propagation for visual tracking. Int. J. Comput.
Vision, vol.29, no. 1, pp. 5-28, 1998.
[9] S. Arulampalam, S. Maskell, N. Gordon, and T. Clapp.
A tutorial on particle filters for on-line non-linear/non-
Gaussian Bayesian tracking. In IEEE Trans. Signal
Process., Vol. 50, no. 2, pp. 174-189, 2002.
[10] B. W. Silverman. Density Estimation for Statistics and
Data Analysis. London, U .K.: Chapman & Hall, 1986.
[11] Y. Cheng. Mean shift, mode seeking, and clustering. In
IEEE Trans. Pattern Anal. Mach. Intell., vol. 17, no. 8,
pp. 790-799, Aug. 1995.
[12] D. Comaniciu, V. Ramesh, and P. Meer. Real-time
tracking of non-rigid objects using mean shift. In proc.
IEEE Conf. on Computer Vision and Pattern
Recognition, Hilton Head, SC, volumeⅡ, June 2000, pp.
142-149.
[13] B. Han, D. Comaniciu, Y. Zhu, and L. Davis.
Incremental density approximation and kernel-based
Bayesian filtering for object tracking. In IEEE CVPR,
Jun. 2004.
[14] C. Musso, N. Oudjane, and F. LeGland. Improving
regularized particle filters. In Sequential Monte Carlo
Methods in practice, A. Doucet, J. F. G. de Freitas, and
N. J. Gordan, Eds. New York: Springer-Verlag, 2001.
[15] D. Comaniciu and P. Meer. Mean shift: A robust
approach toward feature space analysis. In IEEE Trans.
Pattern Anal. Machine Intell., vol. 24, no. 5, pp. 603-619,
2002.