ARTICLE IN PRESS

Pattern Recognition 43 (2010) 2003–2016

Contents lists available at ScienceDirect

Pattern Recognition
journal homepage: www.elsevier.com/locate/pr

Multi-focus image fusion using PCNN

Zhaobin Wang a,b, Yide Ma a,, Jason Gu b
a
School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China
b
Department of Electrical and Computer Engineering, Dalhousie University, Halifax, Nova Scotia, Canada B3J 2X4

a r t i c l e in fo abstract

Article history: This paper proposes a new method for multi-focus image fusion based on dual-channel pulse coupled
Received 13 May 2009 neural networks (dual-channel PCNN). Compared with previous methods, our method does not
Received in revised form decompose the input source images and need not employ more PCNNs or other algorithms such as
24 November 2009
DWT. This method employs the dual-channel PCNN to implement multi-focus image fusion. Two
Accepted 17 January 2010
parallel source images are directly input into PCNN. Meanwhile focus measure is carried out for source
images. According to results of focus measure, weighted coefficients are automatically adjusted. The
Keywords: rule of auto-adjusting depends on the specific transformation. Input images are combined in the dual-
PCNN channel PCNN. Four group experiments are designed to testify the performance of the proposed
Image fusion
method. Several existing methods are compared with our method. Experimental results show our
Focus measure
presented method outperforms existing methods, in both visual effect and objective evaluation criteria.
Finally, some practical applications are given further.
& 2010 Elsevier Ltd. All rights reserved.

1. Introduction
Optics of lenses with a high degree of magnification suffers
from the problem of a limited depth of field. The larger the focal
length and magnification of the lens, the smaller the depth of field
becomes. As a result, fewer objects in the image are in focus.
However, sometimes people like to get the image in which more
objects are in focus. This is to say that people want to see
more and clearer objects in one image. The perfect state is that
the whole image is clear or in focus. This directly leads to the
appearance of multi-focus image fusion technology.
Multi-focus image fusion is the process in which different
images with different focus settings are fused to produce a new
image with extended depth of field. Its purpose is to attempt to
increase the apparent depth of field through the fusion of object
within several different fields of focus. Hence it plays important
roles in many different fields such as biomedical imaging and
computer vision. Multi-focus image fusion is one of the main
research fields of image fusion.
Nowadays, there are two common schemes to be used in the
field of multi-focus image fusion. The one is to use multi-
resolution approaches which usually employ the discrete wavelet
transform (DWT) [15,22] or various pyramid algorithms such as
contrast pyramid [25], FSD pyramid [1], gradient pyramid [3],
morphological pyramid [24], etc. However, this scheme is
complicated and time-consuming to implement. The other is
 Corresponding author.
E-mail addresses: zhaobin_wang@hotmail.com (Z. Wang),
ydma@lzu.edu.cn (Y. Ma).
based on the selection of image blocks from source images [14].
The idea of this scheme is to select the better image blocks from
source images to construct the fused image. Clearly, the latter is
simpler than the former. However, the latter has some disadvan-
tages. For instance, the effect of this scheme depends on focus
measurement to a great extent. Some errors in the fused image
are obtained by this scheme. Therefore, the third scheme is
employed in the paper.
PCNN is a biologically inspired neural network based on the
work by Eckhorn et al. [4]. Pioneering work in the implementation
of these algorithms was done by Johnson and his colleagues
[11,10,9,21]. It has been proven that PCNN is widely applicable in
the field of image processing [18,19] such as image segmentation,
image enhancement, pattern recognition, etc. PCNN also plays an
important role in image fusion. In fact, Broussard et al. [2] applied
PCNN into image fusion for object detection as early as 1999. In
the same year Johnson and Padgett [9] pointed out that there was
the great potential for PCNN in the field of image and data fusion.
Many multi-focus image fusion algorithms based on PCNN
[7,8,13,20,2,16] have been published in different journals or
proceedings so far. Although different authors have different
schemes, most of them employ the same characteristic of PCNN,
namely, the mechanism of synchronous pulse bursts.
For multi-focus image fusion, different from other methods,
PCNN has its own advantages. Firstly, PCNN model derives from
the research on cat visual cortex. Its mode of information
processing is much closer to the mode of human visual
processing. And then PCNN has the flexible structure which can
be changed according to different tasks. Additionally, the existing
PCNN methods also show PCNN has the higher performance.
However, the problem is that the existing PCNN methods are still

0031-3203/$ - see front matter & 2010 Elsevier Ltd. All rights reserved.
doi:10.1016/j.patcog.2010.01.011
 ARTICLE IN PRESS
2004 Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016
complicated and time-consuming. After careful analysis, we find:
(1) almost all the methods with PCNN employ more PCNNs
[7,8,13,20] or combine with other algorithms such as DWT [2,16];
(2) almost authors adopt the simplified PCNN model and all
models have one stimulus. Generally an intricate algorithm
usually costs much time to compute and operate intermediate
variables so as to make the whole system inefficient. From (1) we
know that their schemes are very complicated, it is why these
existing methods are not efficient. From (2) we believe that only
having one stimulus is the root cause of complication and
inefficiency of PCNN methods. In other words, to some extent,
the standard PCNN structurally limits its application in image
fusion. In order to make PCNN more suitable for image fusion, we
improve the standard PCNN and propose the dual-channel PCNN,
which can solve the problem of complication and inefficiency of
PCNN methods very well.
Compared with previous methods, this method with the dual-
channel PCNN does not decompose the input source images and
need not employ more PCNNs or combine with other algorithms
such as DWT. So its scheme of image fusion is very simple and
experimental results also show the proposed method is feasible
and efficient.
The rest of the paper is organized as follows. In Section 2, the
standard PCNN is briefly reviewed, and then the dual-channel
PCNN is introduced in details. Section 3 concretely describes our
proposed multi-focus image fusion algorithm based on the dual-
channel PCNN. It mainly includes focus measurement introduc-
tion, principle and implementation of algorithm, and other
relative contents. Section 4 gives experimental results and
performance evaluation, and then introduces some practical
applications. Conclusions are summarized in the end.
2. PCNN model
Because the dual-channel PCNN is proposed based on the
PCNN model, firstly PCNN model is briefly introduced. After
analyzing its standard model, we will improve the model
according to practical demands of multi-focus image fusion.
2.1. Standard PCNN model
In the standard PCNN model, the PCNN neuron consists of
three parts: the dendritic tree, the linking modulation, and the
pulse generator, as shown in Fig. 1.
The role of the dendritic tree is to receive the inputs from two
kinds of receptive fields. Depending on the type of the receptive
field, it is subdivided into two channels (the linking and the
surrounding
neurons
Si j
exp( −α F )
VF
W Ti j
Fi j
Feeding
Linking
Li j
M
VL
exp(−α L )
Dendritic tree
Fig. 1. The structure of PCNN.
feeding). The linking receives local stimulus from the output of
surrounding neurons, while the feeding, besides local stimulus,
still receives external stimulus.
In the following expressions, the indexes i and j refer to the
pixel location in the image, k and l refer to the dislocation in a
symmetric neighborhood around one pixel, and n denotes the
current iteration (discrete time step). Here n varies from 1 to
N (N is the total number of iterations)
X
Fij ½n ¼ eaF Fij ½n1 þ VF wijkl Yij ½n1 þSij ð1Þ
k;l
X
Lij ½n ¼ eaL Lij ½n1 þVL mijkl Yij ½n1 ð2Þ
k;l
Uij ½n ¼ Fij ½nð1 þ bLij ½nÞ ð3Þ
(
1; Uij ½n 4 Tij ½n
Yij ½n ¼ ð4Þ
0; otherwise
Tij ½n ¼ eaT Tij ½n1 þ VT Yij ½n ð5Þ
The dendritic tree is given by Eqs. (1)–(2). The two main
components F and L are called feeding and linking, respectively.
wijkl and mijkl are the synaptic weight coefficients and S is the
external stimulus. VF and VL are normalizing constants. aF and aL
are the time constants; generally, aF o aL. The linking modulation
is given in Eq. (3), where Uij[n] is the internal state of the neuron. b
is the linking parameter and the pulse generator determines the
firing events in the model in Eq. (4). Yij[n] depends on the internal
state and threshold. The dynamic threshold of the neuron is
Eq. (5), where VT and aT are normalized constant and time
constant, respectively. Here is a brief review of the standard PCNN.
The detailed description of the implementation of the standard
PCNN model on digital computers can be found in literature [9].
More details about PCNN will be found in the literatures [18,19].
2.2. Dual-channel PCNN model
Analysis of the PCNN exposes a defect preventing one PCNN
from fusing multi-focus images. In this case, we modify the
standard model and propose a new improved model which
overcomes some limits of the standard model in multi-focus
image fusion.
The dual-channel neuron model (see Fig. 2) consists of three
parts: the dentritic tree, information fusion pool and the pulse
generator. The function of dentritic tree is to receive the stimulus
including external inputs and surrounding neuron stimuli;
information fusion pool is the place where all data are fused;
the pulse generator is to generate the output pulse.
Threshold
exp( −α T ) VT
1 Ui j Yi j
β
Linking modulation Pulse generator
 ARTICLE IN PRESS
Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016 2005

surrounding
neurons SijA β ijA
1 Threshold
A
H ij exp( −α T ) VT
W σ Ti j

Ui j Yi j

M
H ijB
1

S B
ij β B
ij

Dendritic tree Information fusion pool Pulse generator

Fig. 2. The structure of dual-channel PCNN.

In the improved model there are two input channels. So both (5) If all neurons have fired, go to the next step, else go back to
stimuli can be input into the model at the same time. The step (4).
following expressions describe its mathematical model: (6) O is the output of the dual-channel PCNN.
X
HijA ½n ¼ SAij þ wijkl Ykl ½n1 ð6Þ
Note, ‘‘a neuron fires’’ means a PCNN’s neuron generates a pulse.
k;l
Here we would like to explain the principle of data fusion
X
HijB ½n ¼ SBij þ mijkl Ykl ½n1Þ ð7Þ using the dual-channel PCNN. Eqs. (6)–(8) mathematically show
k;l the way of data fusion. For one neuron, the final output value is
A B not completely determined by weighting coefficients (bA and bB).
Uij ½n ¼ ð1þ bij HijA ½nÞð1 þ bij HijB ½nÞ þ s ð8Þ
The output from surrounding neurons also plays a role via the
( mechanism of synchronous pulse bursts and this output is
Uij ½nSurij ½n; Uij ½n 4 Tij ½n1
Yij ½n ¼ ð9Þ unexpected and even random for one neuron. The mechanism
0; otherwise of synchronous pulse bursts, implemented by time-domain
( iterative processing, can make neurons with similar state
eaT Tij ½n1; Yij ½n ¼ 0 generate synchronous pulses; hence, the way of PCNN fusing
Tij ½n ¼ ð10Þ
VT ; otherwise data is not linear but nonlinear.
For the dual-channel PCNN model, it inherits good features
Compared with the standard PCNN, the dual-channel PCNN
from the standard PCNN. For example, the dual-channel model
has few parameters. In the dual-channel model, instead of feed
remains the mechanism of synchronous pulse bursts. The
channel (F) and link channel (L), HA and HB stand for two
exponential attenuation characteristic of the threshold is also
symmetrical input channels. bA and bB are the weighting coeffi-
kept. It is believed that this characteristic is coincident with
cients of two symmetrical channels, s the level factor to adjust the
human visual characteristic. These remaining features are
average level of internal activity. When s =  1, UZ0. Parameters
propitious to image processing.
(U, T, Y, VT, wijkl, mijkl, and aT) have the same meanings as these in
the standard model. Sur denotes the input from surrounding
P
neurons. Generally, kijkl ¼ wijkl ¼ mijkl ; Surij ¼ k;l kijkl Ykl ½n1. 3. Image fusion algorithm
Now we describe the data fusion process of the dual-channel
PCNN. Firstly, two channels of neuron receive external stimuli and In the section, multi-focus image fusion scheme used in the
output of surrounding neurons. Furthermore, the data from these paper is introduced. Image sharpness measure is so important for
channels are weighted and mixed in the information fusion pool multi-focus image fusion that some common evaluation methods
according to the weighting coefficients. Finally, the mixed data of image sharpness are given in order to make the paper
are released by neuron as its output with the attenuation of the understandable and logical.
threshold. The implementation of the dual-channel PCNN in our
experiments is described as follows: 3.1. Image focus measure

(1) Initialize parameters and matrices. U=O = Y= 0, T= 1. The
Initialization of W and M are different from those of other
matrices. Here K= W=M. Its values are determined manually.
(2) If SA =SB, then O = SA or SB; and go to step (6)
(3) Normalize external stimuli to lie within [0, 1].
(4) Sur= YK;
HA = SA +Surij;
HB = SB + Sur;
U= (1 + bA  HA) (1+ bB  HB)+ s;
If Uij 4Tij then Yij = Uij  Surij, else Yij = 0;
If S Aij ¼ S Bij or bAij ¼ bBij , then Oij ¼ S Aij or S Bij ; else Oij = Yij;
If Yij = 0 then Tij =exp(  aT)  Tij, else Tij = VT;
For multi-focus image fusion, we usually select one or more
focus measure methods to estimate image sharpness. So, it is
crucial to choose a good focus measure method.
The focus measure defined in literatures [12,17] is a maximum
for the best focused image and it generally decreases as the
defocus increases. That is, the focused image should produce
maximum focus measures, while the defocused should produce
minimum focus measures.
Recently, there are many focus measure techniques. Here are
some focus measure methods: variance, energy of image gradient
(EOG), energy of Laplacian of the image (EOL), sum-modified-
Laplacian (SML), and spatial frequency (SF). Huang and Jing [7,8]
 ARTICLE IN PRESS
2006 Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016

assessed these methods according to some objective standards.
Experiment results show that SML and EOL can provide better
performance than other focus measures. However, SML spends
more implementation time under the same conditions. Therefore,
EOL is used in the paper to measure image focus. Because
evaluation of different focus measure methods is not our topic of
this paper, more details can be found in the Refs. [7,8]. For EOL, its
corresponding mathematic expressions are also given as follows.
Let f(i, j) denote the gray level intensity of pixel (i, j).
XX
EOL ¼ ½f ði1; j1Þ4f ði1; jÞf ði1; j þ 1Þ
i j
4f ði; j1Þ þ 20f ði; jÞ4f ði; j þ 1Þf ði þ1; j1Þ
4f ði þ 1; jÞf ði þ1; j þ1Þ2
This method carries out focus measure by analyzing high
spatial frequencies associated with image border sharpness, and it
is implemented through Laplacian operator.
weighting coefficients [5]. This method is explained in details in
the following paragraphs.
Suppose source images IA(i, j) and IB(i, j) turn into measured
images MA(i, j) and MB(i, j) after focus measure. The difference
between MA(i, j) and MB(i, j) is defined by: D(i, j)=MA(i, j)–MB(i, j).
Generally speaking, if D(i, j)40, it indicates the pixel value at
location (i, j) in image A should be chosen. Otherwise, its counterpart
from B is selected. However, this measure alone in practice is not
sufficient to pick out the better focused image on a pixel-by-pixel
basis, since using single pixel information to make decision is
vulnerable to wide fluctuations caused by outer environment such
as various noises. So, it is necessary to maintain robustness of the
algorithm through more information from neighboring pixels.
ð11Þ In order to make full use of surrounding information, it is
necessary to require summing the D(i, j) s over a (r+ 1)  (r + 1)
region surrounding each decision-point
r=2
X r=2
X
Dði; jÞ ¼ Dði þ m; j þ nÞ ð12Þ
m ¼ r=2 n ¼ r=2

Hence their weighting coefficients are

3.2. Principle of multi-focus mage fusion
1
bAij ¼ ð13Þ
Now we explain how to fuse multi-focus image using the dual- 1þ eZDði;jÞ
channel PCNN. Suppose IAand IB are two multi-focus images with the and
same size. IA(i, j) and IB(i, j) denote the pixels of IAand IB in the same
position, respectively. The goal of multi-focus image fusion is to get 1
bBij ¼ ð14Þ
clear one from IA(i, j) and IB(i, j). Actually, from Section 2.2, we know 1þ eZDði;jÞ
the dual-channel PCNN can carry on data fusion. Because the data where Z is a constant. Constant Z has an important influence on
from two channels are weighted according to the weighting weighting coefficients bA and bB (shown in Fig. 3). Larger Z
coefficients, as long as clear pixel get large weighting coefficient markedly increases the difference between bA and bB while
while blurred pixel get small one, the dual-channel PCNN can reducing Z will shrink this difference. Typically, when Z = 0, bA = bB.
achieve the purpose of multi-focus image fusion. Hence, adjusting Z will change the trend of bA and bB and their
Therefore, the core of this method is how to make sure that the difference. Because weighting coefficients play a crucial role in
change of the weighting coefficients depends on the clarity of the multi-focus image fusion, constant Z is usually set via several
input stimuli. Here a new method is employed to implement the experiments to meet practical demands.
transformation from the importance of input stimuli to the

3.3. Implementation of fusion algorithm

1

According to the above statements, we introduce the implemen-

0.8 tation of our proposed multi-focus image fusion algorithm based on
η = 0.05 βA the dual-channel PCNN in this section. The dual-channel PCNN used
0.6 in our experiments is a single layer two-dimensional array of
β ij

laterally linked neurons and all neurons are identical. The number of
0.4
η = 0.01
neurons in the network is equal to the number of pixels in the input
βB

0.2
Table 1
Parameter setting in the experiment.
0
-300 -200 -100 0 100 200 300
Parameters s aT VT r Z
D (i, j )
Their values 1 0.12 1000 14 0.01
Fig. 3. The relation between constant Z and weighting coefficients.

Source IA Focus MA Coefficient βA

image A Measure transformation

IA SA
Dual-channel Fused
PCNN image
IB SB

Source Focus Coefficient

image B IB Measure MB transformation βB

Fig. 4. Procedure of multi-image fusion in the paper.

 ARTICLE IN PRESS
Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016 2007

Table 2 image. In terms of position there exists a one-to-one correspondence

Parameters of Huang’s method. between the image pixels (IA(i, j) and IB(i, j)) and the neuron
(N(i, j)). In other words, the external stimuli of N(i, j) are IA(i, j) and
Parameters aL aT VL VT r
IB(i, j).
Their values 1.0 5.0 0.2 20.0 13.0 Now we describe the implementation of algorithm. The
procedure of our proposed multi-focus image fusion is shown in

Fig. 5. Test images which consist of four groups: LETTER, BADGE, TEAPOT, and PHOTO. Each group include two source images (labeled by A and B). (a) LETTER A, (b) LETTER
B, (c) BADGE A, (d) BADGE B, (e) TEAPOT A, (f) TEAPOT B, (g) PHOTO A, and (h) PHOTO B.
 ARTICLE IN PRESS
2008 Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016
Fig. 4. Note that our algorithm is introduced on the assumption
that input multi-focus images have been registered.
The proposed multi-focus image fusion method consists of the
following steps:
(1) Carry out focus measure with Eq. (11) for two multi-focus images
(IA, IB). Denote measured images by MA and MB, respectively.
(2) Compute the weighting coefficients (bA and bB) via MA and MB
according to Eqs. (12)–(14).
(3) Input IA and IB, taken as two stimuli, into the dual-channel
PCNN, and then start PCNN.
(4) Fuse the multi-focus images via PCNN.
(5) Obtain the fused image after finishing the process of PCNN.
4. Simulations and results
This section consists of three parts: parameter setting,
performance evaluation, and practical applications. Different
parameters of algorithm can produce different effects. So this
section begins with parameter setting. In order to testify the
capability of the proposed method, performance evaluation is
done in succession with visual effect and objective evaluation
criteria. Finally, we illustrate some applications in practical work.
Fig. 6. Test results (LETTER). (a) Our method, (b) CP method, (c) Huang’s method, (d) FSD method, (e) GP method, (f) MP method, (g) SIDWT method, and (h) reference
image.
4.1. Parameter setting
In the dual-channel PCNN, parameter settings are as follows:
large K will make computer simulation cost much time, so
synaptic weight matrices K =[1, 0.5, 1; 0.5, 0, 0.5; 1, 0.5, 1]; level
factor, time constant and normalized offset parameter see Table 1.
In addition, parameters in Eqs. (12)–(14) are also in Table 1. Note
that these parameters are manually set by experiments.
For comparison, several existing methods are used: contrast
pyramid (CP), FSD pyramid (FSD), gradient pyramid (GP),
morphological pyramid (MP), SIDWT with Harr (SIDWT), and an
existing PCNN method (Huang’s method). In order to make
comparison reliable and repeatable, we use the image fusion
toolbox for MATLAB developed by Rockinger [23]. The toolbox
includes all the above methods except Huang’s method and can be
downloaded on the web (http://www.metapix.de/toolbox.htm).
Their parameters in toolbox are set by: pyramid level= 4, selection
rules: highpass= select max, lowpass= average. Note, with this
toolbox, our experiment can be reproduced exactly according to
parameters provided above. If you want to obtain more details,
please read related references [6]. For the scheme and PCNN
model of Huang’s method, Huang and Jing [7,8] give a detailed
explanation in their paper. Its iteration time is 300 and other
parameters see Table 2.
 ARTICLE IN PRESS
Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016 2009

4.2. Performance evaluation 4.2.1. Objective evaluation

To evaluate objectively those methods mentioned above, we
To evaluate the performance of our proposed fusion method, choose two methods: the root mean squared error (RMSE) and the
extensive experiments with multi-focus image fusion and structural similarity (SSIM) index. Here R and F denote reference
different sensor image fusion have been performed. Here, we image and fused image respectively. RMSE is used to evaluate the
give four groups (LETTER, BADGE, TEAPOT, and PHOTO) experi- performance of the focus measures. RMSE is defined as
ments. All test images used in the experiment are shown in Fig. 5. vffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
u
u 1 X M X N
Note, images in LETTER, BADGE and TEAPOT are synthetic while RMSE ¼ t ½Fði; jÞRði; jÞ2 : ð15Þ
images in PHOTO are real with synthetic blur. The content of M  Ni¼1j¼1
LETTER is simple; there is more information in BADGE and
Usually, less RMSE signifies better performance for the
TEAPOT; PHOTO is the real picture acquired by camera. Hence, the
algorithm of fusion. The results of RMSE assessment are shown
content of image becomes more and more complicated from
in Figs. 7, 10, 13 and 16.
LETTER to PHOTO.
The structural similarity (SSIM) index is proposed by Wang
Here we give an explanation for figures: images in Figs. 6, 9, 12
et al [26]. Because it is a better approach to image quality
and 15 are experimental results obtained by several methods. In
measure, we use it to assess the performance of different
each figure, there are nine images: one reference image and
methods, too. The mathematical expression of SSIM index is
eight images fused by eight corresponding algorithms. Figs. 7, 8,
10, 11, 13, 14, 16 and 17 show the performance of different ð2mR mF C1 Þð2sRF þ C2 Þ
SSIM ¼ ; ð16Þ
algorithms. ðm2R þ m2F þ C1 Þðs2R þ s2F þ C2 Þ

2.5

1.5

0.5

0
Our Huang's
CP FSD GP MP SIDWT
method method
RMSE 0.0216 0.5914 1.6782 2.8283 2.8288 0.6319 1.2887

Fig. 7. Objective evaluation of LETTER (RMSE).

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0
Huang's
Our method CP FSD GP MP SIDWT
method
SSIM index 0.9999 0.9999 0.9308 0.8291 0.829 0.9372 0.9863

Fig. 8. Objective evaluation of LETTER (SSIM index).

 ARTICLE IN PRESS
2010 Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016

where C1 =(K1L)2, C2 = (K2L)2. Where L is the dynamic range of the
pixel values (255 for 8-bit grayscale images); K1, K2 51 are small
constants. More descriptions are found in reference. Note that
SSIM index MATLAB program is downloaded on the website
(http://www.ece.uwaterloo.ca/  z70wang/research/ssim/). The
values of SSIM index in the paper are computed with default
parameters. SSIM index describes the similarity of two inputs.
Larger value shows both inputs more similar. Experimental
results of SSIM index are shown in Figs. 8, 11, 14 and 17.
As for objective evaluation, our proposed method is not inferior
to most methods from Figs. 7, 8, 10, 11, 13 and 14. According to the
evaluation rule of RMSE and SSIM index, the better method of multi-
focus image fusion should have smaller RMSE and larger SSIM index.
In fact, experimental results demonstrate that our method has the

Fig. 9. Test results (BADGE). (a) Our method, (b) CP method, (c) Huang’s method, (d) FSD method, (e) GP method, (f) MP method, (g) SIDWT method, and (h) reference
image.

0
Huang's
Our method CP FSD GP MP SIDWT
method
RMSE 0.5792 1.5996 2.3938 4.6158 4.5491 6.4707 2.7038

Fig. 10. Objective evaluation of BADGE (RMSE).

 ARTICLE IN PRESS
Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016 2011

smallest RMSE (in Figs. 7, 10, 13 and 16) and the largest SSIM index
(in Figs. 8, 11 and 17) among several algorithms. However, Fig. 14
shows SIDWT have the largest SSIM index. At the same time, the
suggested method, CP, SIDWT, and Huang’s methods have similar
performances in Figs 13, 14, 16 and 17. Hence it is essential to carry
on subjective evaluation in this case.
4.2.2. Subjective evaluation
As we know, performance evaluation usually includes two
sides: subjective (or qualitative) evaluation and objective (or
quantitative) evaluation. For multi-focus image fusion subjective
evaluation means the evaluation of visual effect. Here we use
visual evaluation to evaluate every method.

0.99

0.98

0.97

0.96

0.95

0.94
Huang’s
Our method CP FSD GP MP SIDWT
method
SSIM index 0.9996 0.9981 0.9934 0.9818 0.9821 0.9625 0.9953

Fig. 11. Objective evaluation of BADGE (SSIM index).

Fig. 12. Test results (TEAPOT). (a) Our method, (b) CP method, (c) Huang’s method, (d) FSD method, (e) GP method, (f) MP method, (g) SIDWT method, and (h) reference
image.
 ARTICLE IN PRESS
2012 Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016

From Figs. 6, 9, 12, and 15, it is clear that these two images
fused by FSD method and GP method have lower lightness than
the others. It shows FSD method and GP method make the fused
images have low contrast. This is not what we want. MP method
has a good contrast but the process of edges in images is bad,
typically such as Figs. 6f, 9f, and 12f.
Although the remaining four methods (the suggested method,
CP, SIDWT, and Huang’s methods) have the similar visual effect,
the difference among them still exists. In fact, our method has the
better performance. Here we use one method to make this
performance easily seen. We take LETTER (in Fig. 6) and TEAPOT
for example; our schemes of magnifying the difference among
these four methods are as follow:
For LETTER:
(3) Obtain error images. E0= XOR(R0,Bs); E1= XOR(R1,Bs); E2=
XOR(R2,Bs);E3= XOR(R3,Bs). Here XOR indicates the exclusive
OR operation. The results see Fig. 18.
For TEAPOT:
(1) Let R0, R1, R2, and R3 denote the fused image of the suggested
method, CP, SIDWT, and Huang’s methods, respectively.
(2) Make the difference between Rs and R0, R1, R2, R3, respectively.
Namely, D0=jRs R0j; D1=jRsR1j; D2=jRs R2j; D3=
jRs R3j. Here, Rs denotes reference image, and || indicates
computing the absolute value of the difference.
(3) Obtain error images. E0 =D0 40; E1= D140; E2 =D2 40;
E3= D340. The results see Fig. 19.

(1) Let R0, R1, R2, and R3 denote the fused image of the suggested
method, CP, SIDWT, and Huang’s methods, respectively.
Rs denotes reference image.
(2) Get the binary images. B0= R040; B1 =R140; B2 = R24 0;B3
= R340; Bs =Rs 40.
For Figs. 18 and 19, all the images are binary images and light
point means error. The error image of our method in Fig. 18 is
completely dark, which shows the fused image using our method
is more similar to reference image than others. Obviously, CP
method and Huang’s method have very bad results in Fig. 19.

0
Huang's
Our method CP FSD GP MP SIDWT
method
RMSE 3.5905 4.5103 3.6549 5.1081 5.12 6.0499 3.7865

Fig. 13. Objective evaluation of TEAPOT (RMSE).

0.99

0.98

0.97

0.96

0.95

0.94

0.93

0.92
Huang's
Our method CP FSD GP MP SIDWT
method
SSIM index 0.9839 0.9884 0.9886 0.9608 0.9619 0.9456 0.9903

Fig. 14. Objective evaluation of TEAPOT (SSIM index).

 ARTICLE IN PRESS
Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016 2013

Comparing Fig. 19(1) with Fig. 19(4), you will find that the big
teapot have the same light while two little teapots in Fig. 19(1)
are clearly darker than the ones in Fig. 19(4). It is indicated that
the proposed method is better than SIDWT method. Hence, our
method is better than others.
By the way, we would like to compare our method with
Huang’s method from the references [7,8]; after all, they have
similar performance in both objective and subjective standards.
Compared with Huang’s method, our method at least has
two advantages. One is that our method has less complexity

Fig. 15. Test results (PHOTO). (a) Our method, (b) CP method, (c) Huang’s method, (d) FSD method, (e) GP method, (f) MP method, (g) SIDWT method, and (h) reference
image.

0
Huang's
Our method CP FSD GP MP SIDWT
method
RMSE 2.3865 2.3951 2.402 6.0212 6.0169 4.0758 2.4105

Fig. 16. Objective evaluation of PHOTO (RMSE).

 ARTICLE IN PRESS
2014 Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016

0.99

0.98

0.97

0.96

0.95

0.94

0.93
Huang's
Our method CP FSD GP MP SIDWT
method
SSIM index 0.9954 0.9934 0.9905 0.9876 0.9877 0.9583 0.9936

Fig. 17. Objective evaluation of PHOTO (SSIM index).

Fig. 18. Error test of LETTER. (a) Our method, (b) CP method, (c) Huang’s method, and (d) SIDWT.

Fig. 19. Error test of TEAPOT. (a) Our method, (b) CP method, (c) Huang’s method, and (d) SIDWT.

because it does not need to divide the input images into many Table 3
image blocks and it also need not compare and choose the best Comparison of time cost between two methods (time unit: second).
image block. Obviously, the other is that our method has less time
Methods LETTER BADGE TEAPOT PHOTO
cost (shown in Table 3), for it has fewer operates than the latter.
Additionally, a large amount of time, spent in the process of Our method 0.3340 0.4131 0.6866 0.6469
determining the adjustable parameter b, greatly decreases the Huang’s method 5.5749 10.1060 18.5210 20.6628
latter’s efficiency. Here, what we need to explain is that data in
Table 3 are obtained in such experimental platform that the
computer is IBM graphics workstation (Intellistation Z Pro) and all
codes have been run in Matlab 7.0. are microscope images required by Motic digital microscope
with focus change (see Figs. 20 and 21). The latter two groups
are a body of photographs (see Figs. 22 and 23) which are
4.3. Practical application obtained from the internet (http://www.imgfsr.com/ifsr_ifs.html;
http://www2.mta.ac.il/  tal/Fusion/). Image A and image B are
Here we take four groups of images as example to demonstrate source images with different focuses. Image C is the fused image
our method’s applications in practice. The first two groups obtained by our proposed method.
 ARTICLE IN PRESS
Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016 2015

Fig. 20. Microscope images (1). (a) image A, (b) image B, and (c) image C.

Fig. 21. Microscope images (2). (a) image A, (b) image B, and (c) image C.

Fig. 22. Example of photography (1). (a) image A, (b) image B, and (c) image C.

Fig. 23. Example of photography (2). (a) image A, (b) image B, and (c) image C.

5. Conclusion
This paper presents a novel multi-focus image fusion algorithm
based on the dual-channel PCNN. This method improves the
standard PCNN model and simplifies the process of image fusion
using PCNN in comparison with previous methods. Previous
methods usually employ more PCNNs or combination with other
algorithms such as DWT, while the proposed method in this paper
just uses one dual-channel PCNN to implement multi-focus image
fusion. Experimental results show that our presented method
excels the existing methods in both visual effect and objective
evaluation criteria. In practical applications, it is proved that our
method is feasible. Because the method is simple and easy to
implement, it is suitable to work in real-time system platform, too.
Acknowledgements
We thank the associate editor and the reviewers for their
helpful and constructive suggestions. The authors also thank Ying
 ARTICLE IN PRESS
2016 Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016

Zhu for her support and help. This paper is jointly supported by
National Natural Science Foundation of China (No.60872109),
Program for New Century Excellent Talents in University (NCET-
06-0900), China Scholarship, and the Fundamental Research
Funds for the Central Universities of Lanzhou University in China
(lzujbky-2009-129).
References
[1] C.H. Anderson, A filter-subtract-decimate hierarchical pyramid signal
analyzing and synthesizing technique. US Patent 718104. 1987.
[2] R.P. Broussard, S.K. Rogers, M.E. Oxley, G.L. Tarr, Physiologically motivated
image fusion for object detection using a pulse coupled neural network, IEEE
Transaction Neural Networks 10 (1999) 554–563.
[3] P.J. Burt, A gradient pyramid basis for pattern-selective image fusion, Society
for Information Displays (SID) International Symposium Digest of Technical
Papers 23 (1992) 467–470.
[4] R. Eckhorn, H.J. Reitboeck, M. Arndt, P.W. Dicke, Feature linking via
synchronization among distributed assemblies: Simulation of results from
cat cortex, Neural Computation 2 (1990) 293–307.
[5] H.A. Eltoukhy, S. Kavusi, A computationally efficient algorithm for multi-focus
image reconstruction, in: Proceedings of SPIE-The International Society for
Optical Engineering, Santa Clara. 2003, pp.332–341.
[6] R.C. Gonzalez, P. Wintz, Digital image processing, Perason Education, NJ,
1978.
[7] W. Huang, Z.L. Jing, Multi-focus image fusion using pulse coupled neural
network, Pattern Recognition Letter 28 (2007) 1123–1132.
[8] W. Huang, Z.L. Jing, Evaluation of focus measures in multi-focus image fusion,
Pattern Recognition Letter 28 (2007) 493–500.
[9] J.L. Johnson, M.L. Padgett, PCNN models and applications, IEEE Transaction
Neural Networks 10 (1999) 480–498.
[10] J.L. Johnson, H.S. Ranganath, G. Kuntimad, H.J. Caulfield, Pulse coupled neural
networks, Neural Networks and Pattern Recognition (1998) 1–56.
[11] J.L. Johnson, D. Ritter, Observation of periodic waves in a pulse-coupled
neural network, Optical Letter 18 (1993) 1253–1255.
[12] E. Krotkov, Focusing, International Journal of Computer Vision 1 (1987)
223.
[13] M. Li, W. Cai, Z. Tan, A region-based multi-sensor image fusion scheme using
pulse-coupled neural network, Pattern Recognition Letter 27 (2006)
1948–1956.
[14] S. Li, J.T. Kwok, Y. Wang, Multifocus image fusion using artificial neural
networks, Pattern Recognition Letter 23 (2002) 985–997.
[15] H. Li, B.S. Manjunath, S.K. Mitra, Multisensor image fusion using the
wavelet transform, Graphical Models and Image Processing 57 (1995)
235–245.
[16] W.Li, X.F. Zhu, A new image fusion algorithm based on wavelet packet
analysis and PCNN, in: Proceedings of the Fourth International Conference on
Machine Learning and Cybernetics. Guangzhou. 2005, pp.5297–5301.
[17] G. Ligthart, F. Groen, A comparison of different autofocus algorithms, in:
Proceedings of International Conference on Pattern Recognition, Munich.
1982, pp. 597–600.
[18] T. Lindblad, J.M. Kinser, Image processing using pulse-coupled neural
networks (second edition), Springer press, New York, 2005.
[19] Y.D. Ma, L. Li, Y.F. Wang, R.L. Dai, Principle of Pulse Coupled Neural Network
and Its Applications, Science Press, Beijing, 2006.
[20] Q. Miao, B. Wang, A novel adaptive multi-focus image fusion algorithm based
on PCNN and sharpness, in: Proceedings of SPIE-the International Society for
Optical Engineering, Orlando, 2005, pp. 704–712.
[21] H.S. Ranganath, G. Kuntimad, J.L. Johnson, Pulse coupled neural networks for
image processing, in: Proceedings of Southeast conference on ‘Visualize the
Future’. Raleigh. 1995, pp. 37–43.
[22] O. Rockinger, Image sequence fusion using a shift-invariant wavelet trans-
form. In: Proceedings of the IEEE International Conference on Image
Processing. Santa Barbara. 1997, pp. 288–291.
[23] O. Rockinger, 1999. Image fusion toolbox for Matlab. Technical report,
Metapix. /http://www.metapix.de/toolbox.htmS.
[24] A. Toet, A morphological pyramidal image decomposition, Pattern Recogni-
tion Letter 9 (1989) 255–261.
[25] A. Toet, L.J. van Ruyven, J.M. Valeton, Merging thermal and visual images by a
contrast pyramid, Optical Engineering 28 (1989) 789–792.
[26] Z. Wang, A.C. Bovik, H.R. Sheikh, E.P. Simoncelli, Image quality assessment:
from error visibility to structural similarity, IEEE Transactions Image
Processing 13 (2004) 600–612.

About the Author—ZHAOBIN WANG received the B.A. degree in electronic information science and technology from Yantai University, Yantai, China, in 2004. He is
currently pursuing the Ph.D. degree in radio physics at Lanzhou University, Lanzhou, China. Since 2004, he has been working on image processing techniques, most in the
biological images. In particular, he has developed a special interest in biomedical image segmentation and measurement, pattern recognition and artificial neural network
especially PCNN.

About the Author—YIDE MA received his B.S. degree and M.S. degree in Electronic Engineering from University of Electronic Science and Technology of China, Chengdou
City, China. But his Ph.D. degree was received from Lanzhou University. He is now a professor in School of Information Science and Engineering, Lanzhou University. His
research focuses on image processing and embed-system’s application. He has authored and coauthored more than 100 publications in journals, books and international
conference proceedings.

About the Author—JASON GU received a B. Sc. degree in 1992 from the Department of Electrical Engineering and Information Science (Special Class for the Young Gifted
1987–1990), University of Science and Technology of China, M. Sc. degree in 1995 from Biomedical & Instrumentation Engineering, Jiaotong University (Shanghai), China
and Ph.D. degree from the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, Alberta, Canada. Since September 2000, he has been with
the Department of Electrical and Computer Engineering at Dalhousie University and is presently an associate professor of Electrical Engineering. His research focuses on
Robotics, Biomedical Engineering, Intelligent Systems, etc.