442 IEEE TRANSACTIONS ON INFORMATION TECHNOLOGY IN BIOMEDICINE, VOL. 13, NO.

4, JULY 2009
A Pattern Similarity Scheme for Medical
Image Retrieval
Dimitris K. Iakovidis, Member, IEEE, Nikos Pelekis, Evangelos E. Kotsifakos, Ioannis Kopanakis,
Haralampos Karanikas, and Yannis Theodoridis, Member, IEEE
AbstractIn this paper, we propose a novel scheme for efcient
content-based medical image retrieval, formalized according to the
PAtterns for Next generation DAtabase systems (PANDA) frame-
work for pattern representation and management. The proposed
scheme involves block-based low-level feature extraction from im-
ages followed by the clustering of the feature space to form higher-
level, semantically meaningful patterns. The clustering of the fea-
ture space is realized by an expectationmaximization algorithm
that uses an iterative approach to automatically determine the
number of clusters. Then, the 2-component property of PANDA
is exploited: the similarity between two clusters is estimated as a
function of the similarity of both their structures and the measure
components. Experiments were performed on a large set of refer-
ence radiographic images, using different kinds of features to en-
code the low-level image content. Through this experimentation, it
is shown that the proposed scheme can be efciently and effectively
applied for medical image retrieval from large databases, provid-
ing unsupervised semantic interpretation of the results, which can
be further extended by knowledge representation methodologies.
Index TermsContent-based image retrieval (CBIR), feature
extraction, patterns, pattern similarity, semantics.
I. INTRODUCTION
O
NE of the primary tools used by physicians is the compar-
ison of previous and current medical images associated
with pathologic conditions. As the amount of pictorial informa-
tion stored in both local and public medical databases is growing,
efcient image indexing and retrieval becomes a necessity.
During the last decade, the advances in information technol-
ogy allowed the development of content-based image retrieval
(CBIR) systems, capable of retrieving images based on their
similarity with one or more query images. Indicative examples
of such systems are QBIC [1], SIMPLicity [2], and FIRE [3].
It is interesting that more than 50 CBIR systems are surveyed
in [4].
Manuscript received April 30, 2007; revised March 12, 2008. Current version
published July 6, 2009. This work was supported in part by the Operational Pro-
gramme Information Society of the Greek Ministry of Development, General
Secretariat for Research and Technology, and in part by the European Union.
D. K. Iakovidis is with the University of Athens, GR15784 Panepistimiopolis,
Ilisia, Greece (e-mail: dimitris.iakovidis@ieee.org).
N. Pelekis, E. Kotsifakos, and Y. Theodoridis are with the Department
of Informatics, University of Piraeus, 185 34 Piraeus, Greece (e-mail:
npelekis@unipi.gr; ek@unipi.gr; ytheod@unipi.gr).
I. Kopanakis is with Technological Educational Institute of Crete, 71 004
Crete, Greece (e-mail: i.kopanakis@emark.teicrete.gr).
H. Karanikas is with the University of Manchester, Manchester M13 9PL,
U.K. (e-mail: bkaranikas@phs.uoa.gr).
Color versions of one or more of the gures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identier 10.1109/TITB.2008.923144
The benets emanating fromthe application of content-based
approaches to medical image retrieval range from clinical de-
cision support to medical education and research [5]. These
benets have motivated researchers either to apply general-
purpose CBIR systems to medical images [3] or to develop
dedicated ones explicitly oriented to specic medical domains.
Specialized CBIR systems have been developed to support the
retrieval of various kinds of medical images, including high-
resolution computed tomographic (HRCT) images [6], breast
cancer biopsy slides [7], positron emission tomographic (PET)
functional images [8], ultrasound images [9], pathology im-
ages [10], and radiographic images [11].
Common ground for most of the systems cited earlier is that
image retrieval is based on similarity measures estimated di-
rectly from low-level image features. This approach is likely
to result in the retrieval of images with signicant perceived
differences from the query image, since low-level features usu-
ally lack semantic interpretation. This has motivated researchers
to focus on the utilization of higher-level semantic represen-
tations of image contents for content-based medical image re-
trieval. Recent approaches include semantic mapping via hybrid
Bayesian networks [12], semantic error-correcting output codes
(SECC) based on individual classiers combination [13], and a
framework that uses machine learning and statistical similarity
matching techniques with relevance feedback [14]. However,
these approaches involve supervised methodologies that require
prior knowledge about the dataset and introduce constraints to
the semantics required for the image retrieval task.
A state-of-the-art CBIR approach has been presented in [15].
It utilizes a continuous and probabilistic image representation
scheme that involves Gaussian mixture modeling (GMM) along
with information-theoretic image matching via the Kullback
Leibler (KL) measure. The results reported in [15] show that
this approach is very effective for radiographic image retrieval;
however, its efciency for large image retrieval tasks still re-
mains a challenge.
In this paper, we propose an unsupervised approach for ef-
cient content-based medical image retrieval that utilizes sim-
ilarity measures, dened over higher-level patterns that are as-
sociated with clusters of low-level image feature spaces. The
term pattern is considered in the context of a state-of-the-art
framework called PAtterns for Next generation DAtabase sys-
tems (PANDA) developed for the representation and the man-
agement of data mining results, and it describes a compact,
rich-in-semantics result of a data mining process [16].
The proposed approach combines the advantages of
the clustering-based CBIR methodologies [17][19] with a
1089-7771/$25.00 2009 IEEE
IAKOVIDIS et al.: PATTERN SIMILARITY SCHEME FOR MEDICAL IMAGE RETRIEVAL 443
semantically rich representation of medical images. Moreover,
unlike related CBIR approaches that exploit multidimensional
indexing techniques, such as R-trees [1], [20], iconic index
trees [21], and meshes of trees [22], the efciency of the pro-
posed approach is hardly affected by increasing the dimension-
ality of the low-level feature representation.
The major contributions of this paper are the following.
1) We dene a novel representation of medical images treated
as rich-in-semantics complex patterns. Each complex pat-
tern comprises a set of simple patterns representing clus-
ters of image regions associated with anatomic specimens
in an unsupervised way. The pattern representation of clus-
ters involves both structural descriptors and quality mea-
sures.
2) We propose a novel scheme for the assessment of the
similarity between complex patterns (i.e., medical images)
for CBIR purposes.
3) We conduct a comprehensive set of experiments over a
publicly available set of radiographic images, in order
to thoroughly evaluate our approach and demonstrate its
effectiveness and efciency in comparison to state-of-the-
art techniques.
The rest of this paper is structured as follows. Section II
outlines the PANDA framework, which provides necessary
background information to the reader. The proposed pattern
similarity scheme for medical image retrieval is presented in
Section III. The results obtained from the experimental evalu-
ation of the proposed scheme are apposed in Section IV. The
conclusions along with the future perspectives are summarized
in Section V.
II. PANDA FRAMEWORK
The efcient management of patterns extracted from medical
image databases is of vital importance due to the extremely large
storage requirements as well as the complexity of such kind of
raw data. Taking advantage of the PANDA framework [16], we
adopt the idea of a pattern-base (PB) keeping information about
extracted patterns in a compact and unied way. A PB consists
of three basic layers: the pattern type, the pattern, and the class.
Apattern type is a description of the pattern structure. Apattern
is an instance of the corresponding pattern type and class is a
collection of semantically related patterns of the same pattern
type [16].
Formally, a pattern type PTis dened as a pair PT=SS, MS),
where SS denes the pattern space by describing the structure
schema of the pattern type, while the measure schema MS quan-
ties the quality of the source data representation achieved by
patterns of this pattern type. As an example, consider a pattern
type representing Euclidean-distance, spherical-like clusters in
a D-dimensional space. The structure of such a pattern type may
be modeled by specifying the cluster center (a D-dimensional
vector) and a radius (a real value). The measure for a cluster
might be, for instance, its support, that is, the fraction of the
data points represented by the cluster. As such
EuclideanCluster=
_
SS : (center : [Real]
D
1
, radius : Real)
MS : (sup p : Real)
_
.
A pattern-type PT is called complex if its structure schema
SS includes another pattern type, otherwise PT is called simple.
Thus, a EuclideanCluster is a simple pattern type, whereas a
clustering extracted, e.g., by a partitioning clustering algorithm
is considered a complex pattern type since it can be modeled as
a set of clusters with no measure component
PartitioningCluster =
_
SS : EuclideanCluster
MS :
_
.
In this notation, if PT is a pattern type, then p = s, m) is
an instance of PT, where s, m are the corresponding structure
and measure values of the pattern. With respect to the previous
example, a possible instance of a 3-D EuclideanCluster could
be
Cluster1 =
_
s : (center : [0.1, 0.3, 0.45], radius : 0.77)
m : (sup p : 0.15)
_
.
According to PANDA, the distance dis between two simple
patterns p
1
, p
2
of the same type is computed by combining
the distance between both the structure s and the measure m
components with a gathering function f
gath
[16]
dis(p
1
, p
2
) = f
gath
(dis
struct
(p
1
.s, p
2
.s),
dis
meas
(p
1
.m, p
2
.m)) (1)
where p
i
.s and p
i
.m denote the structure and the measure, re-
spectively, of the pattern p
i
. The dot in this notation denotes
that the variable on the right is a member of the pattern instance
on the left, according to the notation used in object-oriented
modeling.
On the other hand, the distance between two complex patterns
is dened as the aggregate distance between their constituent
patterns, according to a coupling that associates constituent pat-
terns (this is a recursive denition since a complex pattern could
be composed of other complex patterns, and so on).
The denition of p.s and p.m components of a pattern p ex-
tracted from medical images and the selection of appropriate
dis
struct
, dis
meas
, f
gath
functions are challenges adopted in this
paper, and will be discussed in depth in the following section.
III. MEDICAL IMAGE RETRIEVAL USING PATTERNS
The proposed content-based medical image retrieval scheme
is outlined in Fig. 1. It involves four steps: 1) low-level fea-
ture extraction from each of the registered and query images;
2) clustering of the extracted feature vectors per image; 3) pat-
tern instantiation of the resulted clusters; and 4) computation
of pattern similarities. The registration of a new image into
the database involves steps 1)3), whereas step 4) is processed
during the retrieval task.
A. Low-Level Image Feature Extraction
Each of the images registered in the database, as well as the
query image are raster scanned with a sliding window of user-
dened size, sampling image blocks at a given sampling step.
The sampling step may allowconsecutive blocks to overlap. For
each block, a set of N features f
i
, i = 1, . . ., N, is calculated to
form a single feature vector F. The number of feature vectors
444 IEEE TRANSACTIONS ON INFORMATION TECHNOLOGY IN BIOMEDICINE, VOL. 13, NO. 4, JULY 2009
Fig. 1. Outline of the proposed content-based image retrieval methodology.
The black arrows indicate the data ow for image retrieval, whereas the grey
arrows indicate the data ow for the registration of a new image.
produced for each image depends on the size, the dimensions
of the sliding window, and the sampling step. Typically, the
sampling parameters and the features characterizing the low-
level image content are selected based on the details associated
with the image collection and the retrieval task [23]. Color,
texture, and shape are the three major classes of image features
commonly used in CBIR [1], [4]. Considering an image as a set
of block samples, the features used with the proposed pattern
similarity scheme should describe properly the local content
of the image. The appropriateness of different local descriptors
depending on the kind of medical images is discussed in [5]
and [14].
In the case of radiographic medical image retrieval, local grey
level intensity and texture features have proved to discriminate
best the depicted specimens. Such features include raw pixel
values used along with an image distortion similarity model,
local feature histograms, and local relational features [23], [24].
Recently, in [15] it was shown that highest retrieval precision can
be achieved by combining intensity and texture contrast along
with the corresponding spatial coordinates. However, the intro-
duction of spatial information into the feature vectors makes
them dependent on the patients position. Although patients are
usually positioned in a standard way during the acquisition of a
radiograph, there are still many cases in which this is not practi-
cally feasible. For example, this is the case with the acquisition
of radiographs of critically ill patients using portable radio-
graphic devices [25] and with the acquisition of radiographs of
upper or lower extremities [26].
In this paper, we adopt a standard, multiscale statistical ap-
proach for the representation of the radiographic image regions
that preserves local features, and does not depend on spatial
coordinates. It is based on the 2-D discrete wavelet transform
(2D-DWT), an efcient, yet effective transformation that has
proved useful in a variety of medical image processing and anal-
ysis applications, including the CBIR [5], [27][30]. It enables
coding of image texture into detail (higher frequency) coef-
cients, whereas image intensity information can be extracted
from its approximation (lower frequency) coefcients [27]. A
compact representation of the distributions of the approxima-
tion and the detail coefcients can be obtained by rst-order
statistical approximation.
However, it should be noted that this paper focuses on the
utility of the proposed pattern similarity scheme rather than on
the selection of an optimal feature set for a particular image
retrieval task.
B. Clustering
The low-level feature vectors are clustered using mixture
models that model the data by a number of Gaussian distri-
butions. A cluster corresponds to a set of distributions, one for
each dimension of the dataset. Each distribution is described in
terms of mean and standard deviation. A probabilistic approach
to assigning feature vectors to clusters is used.
For 1-D datasets, a mixture is a set of c Gaussian probabil-
ity distributions, representing c clusters. The parameters of a
mixture model are determined by the expectation maximization
(EM) algorithm [31]. With c Gaussians, the probability density
function of a variable X is
f(X[) =
c

i=1
pp
i
1
_
(2)
d
[
i
[
e

1
2
(X
i
)
T

1
i
(X
i
)
(2)
where pp
i
> 0,

c
i=1
pp
i
= 1, and d is the dimension of the
feature vector. The set of model parameters pp
i
,
i
,

i
, i =
1, . . ., c, consists of the prior probabilities pp
i
of the Gaussian i,
the mean vector
i
, and the covariance matrix

i
for the Gaus-
sian i, respectively. The EM algorithm is used to estimate the
maximumlikelihood L of given a set of features x
1
, . . . , x
N

L( [ X) = log
N

j=1
f(x
j
[ ). (3)
The model parameters are initialized with randomvalues. The
algorithm starts by calculating the probabilities that a vector
should belong to each distribution. These probabilities are used
to compute a newestimate for the parameters. The whole process
is repeated until the parameters converge to a constant or almost-
constant estimate. The algorithm results in a set of distributions,
a vector of pairs of means and standard deviations , each of
which corresponds to a feature, and outputs the size of the cluster
(the number of vectors that belong to the cluster). The vector
of means of the distributions for every feature represents the
centroid of the cluster.
The EM algorithm exhibits many advantages over other clus-
tering algorithms that make it appealing for use with the CBIR
methodology described in this paper. Combining EM with the
v-fold cross-validation algorithm [32], the number of clusters
in the output of the algorithm can automatically be determined.
The v-fold cross-validation technique works by partitioning the
data into v equally sized segments. Starting with one cluster,
EM is performed v times holding out one segment at a time for
test purposes and the likelihood is averaged over all the results.
Next, EM is performed over two clusters, and if the likelihood
IAKOVIDIS et al.: PATTERN SIMILARITY SCHEME FOR MEDICAL IMAGE RETRIEVAL 445
increases, the number of clusters is set to two and the process is
repeated until the estimated likelihood begins to decrease [33].
Furthermore, the EM algorithm is more general than, e.g.,
K-means [34], as it can nd clusters of different sizes and ellip-
soidal shapes. Most importantly, the distributions representing
the clusters at the output of the EM algorithm can be easily uti-
lized for pattern instantiation by the PANDA framework, which
is discussed next.
C. Pattern Instantiation
The clusters resulting from the EM algorithm are considered
as patterns extracted from the image database, and are rep-
resented and handled according to the PANDA formalization
presented in Section II. Hence, given a clustered image com-
prising of M simple patterns P
i
, i = 1, . . ., M, and with respect
to the output of the EM algorithm, a Specimen
i
is instantiated
for each pattern P
i
representing a physical anatomic specimen
in a medical image
Specimen
i
=
_
SS : (D : [ : [Real], : [Real]]
N
1
)
MS : (pp : [Real], SV : [Real])
_
. (4)
More specically, the structure schema SS of a specimen is
represented by the pair (, ) of the distribution D
j
for each
of the N features (j = 1, . . ., N) in pattern P
i
, respectively.
Correspondingly, the measure schema MS of a specimen is
represented by two values, the prior probability (pp) and the
scatter value (SV) of P
i
. Formally, the prior probability pp
is dened as the fraction of the feature vectors of the image
that belong to pattern P
i
. Intuitively, pp is equivalent with the
support measure widely used in data mining models. In this
case, it provides an indication of the size of the specimen. On
the other hand, SV is a measure of the cohesiveness of the data
items in a cluster with respect to the centroid of the cluster,
and it is a commonly used intrinsic measure of the quality of
a cluster [35]. Formally, the scatter value SV of a specimen is
dened as
SV =

kP
i
(x
k
c
P
i
)
2
(5)
where x
k
are the feature vectors that belong to pattern P
i
and
c
P i
is the corresponding centroid, which is also a vector having
the same dimensionality as x
k
, and its value in each dimension
is computed as the average from the corresponding features
values belonging to pattern P
i
. A low scatter value indicates
good scatter quality, but it should be noted that this is a relative
measure of quality, since it depends on the number of items in
the cluster.
In this context, a medical image MI is considered as a complex
pattern
MI =
_
SS : Specimen
MS :
_
(6)
consisting of a set of simple patterns (i.e., specimens), which
follow the denition in (4).
D. Computation of Pattern Similarities
Let us recall that the objective of a CBIRsystemis the estima-
tion and ranking of the similarity between query and registered
images. Aiming at the estimation of the similarity between two
medical images [dened as complex patterns in (6), we rst
have to dene the distance over the structures and the measures
of two simple patterns P
1
and P
2
. Since complex patterns are
decomposed into a number of simple patterns, in comparing
two medical images, MI
1
and MI
2
, we need a way to associate
component patterns of MI
1
to component patterns of MI
2
. To
this end, the coupling type constrains the way component pat-
terns can be associated (i.e., matched). Next, we rst propose
an effective way to measure the distance between two simple
patterns, and then we present (see 11), which is our choice for
coupling them.
The distance between the measures of two patterns is pro-
posed to be dened as the absolute difference of the scatter
values, each one weighted by the corresponding prior proba-
bility of the patterns, normalized by the sum of the two scatter
values. Formally
dis
mean
(P
1
, P
2
) =
[P
1
.pp P
1
.SV P
2
.pp P
2
.SV[
P
1
.SV + P
2
.SV
. (7)
Intuitively, (7) quanties the interpattern divergence between
the cohesiveness of two clusters. It should be noted that this
denition overrides the inefciency of the relativeness of the
scatter value with respect to the number of items in the cluster,
as each scatter value is weighted by the fraction of the feature
vectors of the image that belong to pattern P
i
.
Regarding the structural similarity between P
1
and P
2
, we
search for a measure that evaluates the closeness of two sets
of distributions, as P
1
and P
2
are. Further decomposing the
problem, we should rst dene a method of computing the
similarity between two distributions D
1
and D
2
. To achieve this,
we use the standardized difference d between two distributions,
as dened by Cohen [36]. Cohens d is dened as the absolute
difference between the means of the distributions, divided by
the root- mean square of the two standard deviations
d(D
1
, D
2
) =
_
[D
1
.D
2
.[
_
D
1
.
2
D
2
.
2
2
, if D
1
. ,= 0 or D
2
. ,= 0
[D
1
. D
2
.[ , otherwise
.
(8)
Cohens distance d is a nonnegative real number interpreting
the overlap between two distributions. If d is zero, the distribu-
tions are identical. Low d indicates quite similar distributions
whereas high d indicates quite dissimilar distributions. If both
standard deviations are zero, the absolute difference between
the means is used as the distance between the distributions.
Cohens distance is the vehicle to automate and materialize
the intuitive overlap between two distributions. Having this, we
dene that the structural distance between two sets of distri-
butions (i.e., two patterns P
1
and P
2
) should be the result of
an aggregate function g
aggr
(9), which interrelates the different
446 IEEE TRANSACTIONS ON INFORMATION TECHNOLOGY IN BIOMEDICINE, VOL. 13, NO. 4, JULY 2009
distance scores achieved by each pair of distributions
dis
struct
(P
1
, P
2
) = g
aggr
_
d(D
1
j
, D
2
j
)

_
j = 1, 2, . . . , N
(9)
where d is Cohens distance and is a normalization factor
of the domain of values of the g
aggr
function (g
aggr
: [0, 1]
[0, 1]), which intuitively corresponds to the Cohens d score
over which two distributions are considered totally dissimilar
(i.e., they do not overlap). In this connection, g
aggr
function
can be any mapping that initially performs a feature selection
process and subsequently applies the aggregation function upon
the selected features. Examples of such functions include: 1)
the minimum function g
min
(i.e., selection of the most similar
distributions); 2) the average function g
avg
(i.e., selection of
the average among the distances computed for each pair of the
N features); and 3) the average of the k nearest distributions
function g
avg kND
(i.e., selection of k N most similar pairs
of distributions). In the last case, the k parameter may not be
given explicitly, yet it can be dened implicitly by relaxing the
parameter. Formally
g
min
=
N
min
j=1
d(D
1
j
, D
2
j
), g
avg
=
1
N
N

j=1
d(D
1
j
, D
2
j
), g
avg kND
=
1
k
k

j=1
kND(d(D
1
j
, D
2
j
))
where function kND returns the k most similar distributions.
To this point, we have dened dis
meas
and dis
struct
[(7) and
(9), respectively] between two patterns (i.e., clusters on fea-
tures extracted by two medical images following the proposed
CBIRmethodology). In the sequel, we aggregate these distances
by using a wise weighted sum function. Formally, the distance
dis(P
1
, P
2
) between two patterns P
1
and P
2
is dened as
dis(P
1
, P
2
) = dis
struct
(P
1
, P
2
)
+ (1 dis
struct
(P
1
, P
2
)) dis
mean
(P
1
, P
2
)
2
.
(10)
The intuition behind our choice is that the more similar
the structures are, the more the measure distance should con-
tribute to the total distance score. This implies that if structures
are totally different, the distance should be 1, irrespective of
the measure. This choice further implies that we give emphasis
on the structural similarity. This is additionally strengthened by
multiplying the factor 1 dis
struct
(i.e., the similarity between
the structures of the patterns) with a smaller value than the actual
measure distance dis
meas
. Recall that dis
meas
takes values in the
domain [0, 1], so by taking its square we denote the relaxing of
the dis
meas
contribution.
Having dened the distance between simple patterns, to com-
pare two medical images MI
1
and MI
2
(i.e., two complex pat-
terns), we adopt the coupling methodology between the different
patterns of each image as follows
dis(MI
1
, MI
2
) =
1
M K

M

i=1
K

j=1
dis(P
MI
1
i
, P
MI
2
j
) (11)
where M and K are the respective numbers of constituent simple
patterns of each image with respect to the output of the EM
algorithm. Though various coupling types can be applied in
the context of the PANDA framework [16], we adopt the all-
by-all matching expressed by (11) so as to avoid bias toward
specic patterns. The nal outcome is the average of all possible
matchings.
IV. RESULTS
A number of experiments were performed with radiographic
images fromthe image retrieval in medical applications (IRMA)
dataset [37], which is often used as a reference for medical image
retrieval tasks. It currently contains 10 000 arbitrarily selected
anonymous radiographic images taken randomly from patients
of different ages, genders, and pathologies during medical rou-
tine. The images are categorized into 116 classes according
to the IRMA code [38]. This code comprises four elds: 1) the
imaging modality; 2) direction of the imaging device and the pa-
tient; 3) the anatomic body part that is examined; and 4) the sys-
tem under investigation. The particular dataset comprises only
plain X-ray images of various directions (such as anteroposte-
rior and mediolateral), anatomic body parts (such as cranium,
spine, arm, elbow, and chest) and systems under investigation
(such as musculoskeletal, gastrointestinal, and uropoietic). The
IRMAcode information of each image is provided as the ground
truth along with the dataset. Other patient data and pathology
information are unavailable. All radiographic images are in 8-bit
greyscale format and have been downscaled to t into a 256
256-pixel bounding box maintaining the original aspect ratio.
Fromthe available dataset, a subset of 90%of the images was
registered in the database, whereas a nonoverlapping subset of
10% of the images was used for querying the pattern-base.
Each image was sampled in blocks using overlapping sliding
windows. The details of the feature extraction method used
(Section III) include a 3-level biorthogonal spline wavelet de-
composition of each sampled block and the estimation of the
rst two wavelet moments from each band. This process results
in a 20-D feature vector per block.
The determination of the sampling parameters was based on
preliminary experiments seeking the maximum average dis-
tance (11) between complex patterns MI of the different cat-
egories comprising the registered dataset. The sampling pa-
rameters tested before each CBIR experiment include sliding
windows of 32 32, 64 64 and 128 128 pixels. In all
cases, the maximum average distance was obtained with win-
dows of 64 64 pixels. Variation of the overlap (0%, 25%,
50%, and 75%) between the sampled blocks did not affect
this result. Increasing the overlap provides better localization
of the patterns but produces many more sampled blocks, affect-
ing the efciency of both the feature extraction and the pattern
instantiation tasks. Thus, a 50% overlap, i.e. a 32 pixel step, was
used as a compromise between localization and efciency.
IAKOVIDIS et al.: PATTERN SIMILARITY SCHEME FOR MEDICAL IMAGE RETRIEVAL 447
Fig. 2. (a) Original radiographic images, (b) clustering output, and (c) 3-D
visual representation of the feature spaces.
In the following, we present qualitative results of the pattern
instantiation realized via clustering, and measure the perfor-
mance of the proposed scheme, in terms of effectiveness and in
terms of efciency.
A. Pattern Instantiation via Clustering
The feature vectors extracted from each image were clus-
tered using an implementation of the EM algorithm available in
the Waikato environment for knowledge analysis (WEKA) data
mining tool [33] using the 10-fold cross-validation algorithm to
determine the number of clusters. Each cluster was represented
by a pattern Specimen
i
, i = 1, . . ., M (see (4)), and each image
was represented by a complex pattern MI [see (6)]. Fig. 2(a)
illustrates three radiographic images from breast, abdomen, and
hand categories (from left to right). The respective clusterings
obtained are illustrated in Fig. 2(b). The different grey levels
in Fig. 2(b) indicate the different specimen patterns found in
the images. Fig. 2(c) illustrates projections of the 20-D feature
vectors to a 3-D space constructed according to the centroid-
preserving projection technique [39]. It can be observed that
the clustering produced is quite meaningful in terms of seman-
tics, i.e., the breast and the perceived differences in its structure
are clearly depicted, the region of the abdomen is well dened
and separated from the upper part of the body, and the palm
is differentiated from the ngers. However, for the ngers, the
algorithm assigned two specimens instead of one, but this can
be attributed to the large size of the sampled blocks as compared
with the gap between the ngers.
B. Effectiveness
The patterns from the registered radiographic images were
used to build a pattern base (see Fig. 1). In order to quantita-
tively assess the effectiveness of the proposed pattern similarity
scheme, we evaluate its capability to retrieve images by adopt-
ing the popular recall and precision measures, where recall is
Fig. 3. Average precision versus recall using g
avg kND,
g
avg
, and g
min
ag-
gregation functions for (a) all, (b) chest, and (c) cranium, categories.
dened as the ratio of the relevant images retrieved over the total
relevant images in the database, and precision is dened as the
ratio of the relevant images retrieved over the total number of
images retrieved, relevant or not. To enable comparisons with
other medical image retrieval methodologies using a standard
single-gure measure, the area under the interpolated precision-
recall curve (AUC) is estimated [40].
The proposed scheme was tested using the three alternative
aggregation functions g
aggr
discussed in Section III-D. The re-
sults, in terms of average precision versus recall estimated for
all 116 categories, are illustrated in Fig. 3(a). Indicatively, in
Fig. 3(b) and (c), we present the precision versus recall charts for
two independent categories of chest and cranium radiographs. It
is evident that best retrievals are achieved by using the average
of the k nearest distributions function
avg kND
.
Fig. 3(a) shows that, for a recall of 90%, the average precision
achieved using
avg kND
is almost 45%, and the corresponding
AUC estimated is 74%. It is worth noting that these results
could only marginally improve upon a denser sampling scheme.
Compared with a simple method that uses global grey-level his-
tograms as features and histogramintersection as an appropriate
dissimilarity measure [41], the average precision for 90% re-
call is approximately 10%, and the corresponding AUC reaches
only 17%. The AUC obtained with the proposed scheme us-
ing local grey-level histogram information reaches 34%. The
corresponding precision versus recall curves are illustrated in
Fig. 4.
The precision reported in [15] for 90% recall seems to be
comparable with the one achieved with the proposed approach;
however, the dataset from which that precision is estimated is
signicantly smaller comprising only 1500 radiographs from
17 categories. In order to derive comparable estimates between
448 IEEE TRANSACTIONS ON INFORMATION TECHNOLOGY IN BIOMEDICINE, VOL. 13, NO. 4, JULY 2009
Fig. 4. Comparative precision versus recall chart.
Fig. 5. (a) Query requesting nine chest images similar to the upper-left image
(1,1): All retrieved images belong to the same category; (b) A query requesting
nine abdomen-gastrointestinal system images similar to the upper-left image
(1,1): all retrieved images belong to the same category, except (1,4) and (2,5),
which belong to the abdomen-uropoietic system. (Notation (i, j) indicates the
positioning of an image at the ith row, jth column in the gure.).
the two CBIR approaches, a retrieval experiment was run with
the proposed scheme on a subset of the available data generated
according to the guidelines provided in [15]. The AUCestimated
for the proposed approach on this subset reached 78%, whereas
the AUC estimated from [15] is approximately 66%.
Two example retrievals using g
avg kND
are illustrated in
Fig. 5. The rst image of each sequence is the query image,
and the rest are the nine retrieved images requested. Fig. 5(a)
shows that all the retrieved images belong to the same category.
Fig. 5(b) shows that two of the retrieved images belong to a
different class than that of the query image. However, the main
difference between the two categories is hardly perceivable and
located in the region of pelvis (lower part of the image at the
center). Similar observations are valid for queries performed
using radiographic images from other categories.
Fig. 6. Speedup factor between the conventional and the proposed approach
as a function of the number of blocks per image.
C. Efciency
In this section, we measure the efciency of the proposed
medical image similarity scheme that involves pattern com-
parisons, in comparison with the performance of the conven-
tional scheme that involves exhaustive comparisons of the fea-
ture vectors. A vector comparison in the conventional approach
is considered equivalent to a pattern comparison in the proposed
scheme. The experiments were performed on a workstation with
Intel Pentium M1.6 processor having 1 GB RAM and 60 GB
hard disk.
We have chosen the sequential, exhaustive scan as the yard-
stick for our method, as other common methods such as R-trees
are sensitive to the high dimensionality of the feature vectors,
which is usual in CBIRapplications (e.g., a dimensionality of 64
in [42] and at least 2 N=40 in our case, where Nis the number
of features in a pattern). The performance of these approaches
degrades rapidly as dimensionality increases. For instance, it
has been shown that even for a dimensionality of as low as 5,
the R

-tree behavior in similarity search is problematic [42].

The main reason is that, with the growth of the dimensionality,
the overlap in the internal nodes of the tree increases and, as
such, its discrimination ability decreases.
The speedup factor between the conventional and the pro-
posed approach as a function of the number of blocks per image
is illustrated in Fig. 6. It can be observed that the advantage
of the proposed approach increases with the number of blocks
per image (e.g., by increasing the sampling step), and for a few
hundreds of blocks per image, it requires almost three orders of
magnitude fewer comparisons than the conventional approach.
On the other hand, in [15], a speedup of two orders of magni-
tude compared to the conventional approach is reported. More-
over, in the same research, it is noted that the GMM-KL frame-
work is not yet capable of coping with large image retrieval tasks
that extend more than 6000 images due to the computational load
involved with the KL measure. We further estimated the aver-
age processing time (CPU plus I/O time) for the comparison
of a pair of images. For the aforesaid experimental setting, the
proposed pattern similarity scheme requires always less than
0.1 ms. The average time required for the mixture model pa-
rameters to converge to a constant or almost-constant estimate is
0.22 0.04 s.
IAKOVIDIS et al.: PATTERN SIMILARITY SCHEME FOR MEDICAL IMAGE RETRIEVAL 449
V. CONCLUSION
We presented a novel scheme for efcient content-based med-
ical image retrieval. This scheme utilizes rich-in-semantics pat-
tern representations of medical images, dened in the context
of PANDA, a framework for representing and handling data
mining results. The theoretical contributions of this paper are
validated by comprehensive experimentation on the IRMA ref-
erence collection of radiographic images. The results advocate
both its efciency and effectiveness in comparison with state of
the art.
Future perspectives of this paper include: 1) systematic eval-
uation of the proposed scheme for the retrieval of various kinds
of medical images, such as endoscopic [29] and ultrasound im-
ages [43] according to their pathology; 2) the enhancement of the
retrieval performance by using image indexing techniques based
on specialized data structures; and 3) the integration of the pro-
posed scheme with ontology-based information extraction and
data mining techniques for the retrieval of medical images us-
ing heterogeneous data sources. By storing the semantically rich
patterns along with low-level features in a unied way, accord-
ing to the PANDA framework, will enable the extension of the
CBIRmethodologies with knowledge representation techniques
for semantic processing and analysis.
ACKNOWLEDGMENT
We would like to gratefully acknowledge the contribution
of the anonymous reviewers for their valuable comments and
suggestions, which allowed us to substantially improve the pre-
sentation quality of our paper. The collection of images used in
this paper is courtesy of Dr. T. M. Lehmann, Image Retrieval
in Medical Application (IRMA) Group, Department of Medical
Informatics, RWTHAachen, Germany (http://irma-project.org).
REFERENCES
[1] C. Faloutsos, R. Barber, M. Flickner, J. Hafner, W. Niblack, D. Petkovic,
and W. Equitz, Efcient and effective querying by image content, J.
Intell. Inf. Syst., vol. 3, pp. 231262, 1994.
[2] J. Z. Wang, J. Li, and G. Wiederhold, SIMPLIcity: Semantics-sensitive
integrated matching for picture libraries, IEEE Trans. Pattern Anal.
Mach. Intell., vol. 23, no. 9, pp. 947963, Sep. 2001.
[3] T. Deselaers, D. Keysers, and H. Ney, FIREexible image retrieval
engine: ImageCLEF 2004 evaluation, in Lecture Notes in Computer Sci-
ence, vol. 3491, 2004, pp. 688698.
[4] R. Veltcamp and M. Tanase, Content-based image retrieval systems: A
Survey, Dept. Comput. Sci., Utrecht Univ., Tech. Rep. UU-CS-2000-34,
2000.
[5] H. M uller, N. Michoux, D. Bandon, and A. Geissbuhler, A review of
content-based image retrieval systems in medicineclinical benets and
future directions, Int. J. Med. Inf., vol. 73, pp. 123, 2004.
[6] C. R. Shyu, C. E. Brodley, A. C. Kak, A. Kosaka, A. M. Aisen, and
L. S. Broderick, ASSERT: A physician-in-the-loop content-based im-
age retrieval system for HRCT image databases, Comput. Vis. Imag.
Understand., vol. 75, pp. 111132, 1999.
[7] F. Schnorrenberg, C. S. Pattichis, C. N. Schizas, and K. Kyriacou,
Content-based retrieval of breast cancer biopsy slides, Technol. Health
Care, vol. 8, pp. 291297, 2000.
[8] W. Cai, D. D. Feng, and R. Fulton, Content based retrieval of dynamic
PET functional images, IEEE Trans. Inf. Technol. Biomed., vol. 4, no. 2,
pp. 152158, Jun. 2000.
[9] D.-M. Kwak, B.-S. Kim, O.-K. Yoon, C.-H. Park, J.-U. Won, and
K.-H. Park, Content-based ultrasound image retrieval using a coarse
to ne approach, Ann. NY Acad. Sci., vol. 980, pp. 212224, 2002.
[10] L. Zheng, A. W. Wetzel, J. Gilbertson, and M. J. Becich, Design and
analysis of a content-based pathology image retrieval system, IEEE
Trans. Inf. Technol. Biomed., vol. 7, no. 4, pp. 249255, Dec. 2003.
[11] I. El-Naqa, Y. Yang, N. P. Galatsanos, R. M. Nishikawa, and M. N.
Wernick, Asimilarity learning approach to content-based image retrieval:
Application to digital mammography, IEEE Trans. Med. Imag., vol. 23,
no. 10, pp. 12331244, Oct. 2004.
[12] C.-Y. Lin, J.-X. Yin, X. Gao, J.-Y. Chen, and P. Qin, Asemantic modelling
approach for medical image semantic retrieval using hybrid Bayesian
networks, in Proc. 6th Int. Conf. Intell. Syst. Des. Appl. (ISDA), 2006,
pp. 482487.
[13] J. Yaoa, I, S. Antani, R. Longb, G. Thoma, and Z. Zhanga, Automatic
medical image annotation and retrieval using SECC, in Proc. 19th Int.
Symp. Comput.-Based Med. Syst. (CBMS), Salt Lake City, UT, Jun. 2006.
[14] M. M. Rahman, P. Bhattacharya, and B. C. Desai, A framework for
medical image retrieval using machine learning and statistical similarity
matching techniques with relevance feedback, IEEE Trans. Inf. Technol.
Biomed., vol. 11, no. 1, pp. 5867, Jan. 2007.
[15] H. Greenspan and A. T. Pinhas, Medical image categorization and re-
trieval for PACSusing the GMM-KLframework, IEEETrans. Inf. Techol.
Biomed., vol. 11, no. 2, pp. 190202, Mar. 2007.
[16] I. Bartolini, P. Ciaccia, I. Ntoutsi, M. Patella, and Y. Theodoridis, A
unied and exible framework for comparing simple and complex pat-
terns, in Proc. 8th Eur. Conf. Principles Pract. Knowl. Discov. Database
(PKDD), 2004, pp. 496499.
[17] R. O. Stehling, A. X. Falcao, and M. A. Nascimento, An adaptive and
efcient clustering-based approach for content-based image retrieval in
image databases, in Proc. Int. Symp. Database Eng. Appl., pp. 356
365.
[18] C. Carson, S. Belongie, H. Greenspan, and J. Malik, Blobworld: Image
segmentation using expectation-maximization and its application to im-
age querying, IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 8,
pp. 10261038, Aug. 2002.
[19] C. Yixin, J. Z. Wang, and R. Krovetz, CLUE: Cluster-based retrieval of
images by unsupervised learning, IEEE Trans. Image Process., vol. 14,
no. 8, pp. 11871201, Aug. 2005.
[20] E. G. M. Petrakis and C. Faloutsos, Similarity searching in medical image
databases, IEEE Trans. Knowl. Data Eng., vol. 9, no. 3, pp. 435447,
May/Jun. 1997.
[21] J. K. Wu and A. D. Narasimhalu, Identifying faces using multiple re-
trievals, IEEE Multimedia, vol. 1, no. 2, pp. 2038, Aug. 1994.
[22] W.-M. Jeng and J.-H. Hsiao, An efcient content based image retrieval
system using the mesh-of-trees architecture, J. Inf. Sci. Eng., vol. 21,
pp. 797808, 2005.
[23] T. Deselaers, D. Keysers, and H. Ney, Features for image retrievala
quantitative comparison, in Proc. 26th DAGM Symp., Lect. Notes Com-
put. Sci., 2004, pp. 228236.
[24] L. Setia, A. Teynor, A. Halawani, and H. Burkhardt, Image classication
using cluster-cooccurrence matrices of local relational features, in Proc.
8th ACM Int. Workshop Mult. Inf. Retrieval, Santa Barbara, CA, 2006,
pp. 173182.
[25] F. S. Bongard and D. Y. Sue, Current Critical Care: Diagnosis and Treat-
ment, 2nd ed. New York: McGraw-Hill/Appleton and Lange, 2002.
[26] S. B. Dowd and B. G. Wilson, Encyclopedia of Radiographic Positioning.
vol. 2, Philadelphia, PA: Saunders, 1995.
[27] S. Mallat and A. Wavelet, Tour of Signal Processing, 2nd ed.
Washington, DC: Academy, 1999.
[28] J. Z. Wang, G. Wiederhold, O. Firschein, and S. X. Wei, Content-based
image indexing and searching using Daubechies wavelets, Int. J. Digital
Libraries, vol. 1, no. 4, pp. 311328, 1998.
[29] S. A. Karkanis, D. K. Iakovidis, D. E. Maroulis, D. A. Karras, and
M. Tzivras, Computer aided tumor detection in endoscopic video us-
ing color wavelet features, IEEE Trans. Inf. Technol. Biomed., vol. 7,
no. 3, pp. 141152, Sep. 2003.
[30] J. Z. Wang, Wavelets and imaging informatics: Areviewof the literature,
J. Biomed. Inf., vol. 34, pp. 129141, 2001.
[31] A. Dempster, N. Laird, and D. Rubin, Maximum likelihood from incom-
plete data via the EM algorithm, J. Roy. Stat. Soc., Ser. B, vol. 39, no. 1,
pp. 138, 1977.
[32] M. Stone, Cross-validation choice and assessment of statistical predic-
tions, J. Roy. Stat. Soc., vol. B-36, pp. 111147, 1974.
[33] I. H. Witten and E. Frank, Data Mining: Practical Machine Learning
Tools and Techniques, 2nd ed. San Francisco, CA: Morgan Kaufmann,
2005.
[34] J. A. Hartigan, Clustering Algorithms. New York: Wiley, 1975.
450 IEEE TRANSACTIONS ON INFORMATION TECHNOLOGY IN BIOMEDICINE, VOL. 13, NO. 4, JULY 2009
[35] D. Littau, Using low-memory approximations to cluster very large data
sets, in Proc. 3rd SIAM Int. Conf. Data Min. (SDM), 2003, pp. 341345.
[36] J. Cohen, Statistical Power Analysis for the Behavioral Sciences, 2nd ed.
Hillsdale, NJ: Erlbaum, 1988.
[37] T. M. Lehmann, M. O. G uld, C. Thies, B. Plodowski, D. Keysers,
B. Ott, and H. Schubert, IRMAContent-based image retrieval in med-
ical applications, in Proc. 14th World Congr. Med. Inf. (Medinfo), IOS,
Amsterdam, The Netherlands, vol. 2, pp. 842848, 2004.
[38] T. M. Lehmann, H. Schubert, D. Keysers, M. Kohnen, and B. B. Wein,
The IRMA code for unique classication of medical images, in Proc
SPIE 2003, vol. 5033, pp. 109117.
[39] I. Kopanakis and B. Theodoulidis, Visual data mining modelling tech-
niques for the visualization of mining outcomes, J. Vis. Lang. Comput.,
Spec. Issue Vis. Data Min., vol. 14, no. 6, pp. 543589, 2003.
[40] J. Davis and M. Goadrich, The relationship between precision-recall and
ROC curves, in Proc. 23rd Int. Conf. Mach. Learn. (ICML), Pittsburgh,
PA, 2006, pp. 233240.
[41] M. J. Swain and D. H. Ballard, Color indexing, Int. J. Comput. Vis.,
vol. 7, no. 1, pp. 1132, Nov. 1991.
[42] R. Weber, H.-J. Schek, and S. Blott, A quantitative analysis and perfor-
mance study for similarity-search methods in high-dimensional spaces,
in Proc. Very Large Data Bases Conf. (VLDB), 1998, pp. 194205.
[43] D. E. Maroulis, M. Savelonas, D. K. Iakovidis, S. A. Karkanis, and N.
Dimitropoulos, Variable background active contour model for computer-
aided delineation of nodules in thyroid ultrasound images, IEEE Trans.
Inf. Technol. Biomed., vol. 11, no. 5, pp. 537543, Sep. 2007.
Dimitris K. Iakovidis (M05) was born in Athens,
Greece, in 1973. He received the B.Sc. degree in
physics from the University of Athens, Athens, and
the M.Sc. degree (with honors) in cybernetics and the
Ph.D. degree in informatics from the Department of
Informatics and Telecommunications, University of
Athens, in April 2001 and 2004, respectively.
For over ten years, he has been involved in the
eld of medical informatics and has been collaborat-
ing with many European hospitals and health centers
as an expert on biomedical systems. Currently, among
other academic positions, he holds a Senior Researcher position at the University
of Athens. He has been actively involved in more than 10 European and National
R&D projects. He is the author or coauthor of more than 60 research papers
and book chapters. His current research interests include image processing and
analysis, data mining, and pattern recognition with applications on biomedical
systems and bioinformatics.
Dr. Iakovidis is a Reviewer of ten international journals, including the
IEEE TRANSACTIONS ON IMAGE PROCESSING and the IEEE TRANSACTIONS
ON BIOMEDICAL ENGINEERING.
Nikos Pelekis was born in 1975. He received the
B.Sc. degree from the Department of Computer Sci-
ence, University of Crete, Crete, Greece, in 1998, and
the M.Sc. degree in information sciences engineering
and the Ph.D. degree in moving object databases in
2002 from the Department of Computation, Univer-
sity of Manchester Institute of Science and Technol-
ogy (UMIST), Manchester, U.K., in 1999 and 2002,
respectively.
Since March 2004, he has been a Postdoctoral
Researcher in the Information Systems Laboratory,
Department of Informatics, University of Piraeus, Piraeus, Greece. His current
research interests include spatiotemporal databases, management of location-
based services, data mining, and geographical information systems.
Evangelos E. Kotsifakos was born in Athens,
Greece, in 1978. He received the Bachelors and
Masters degrees in information systems fromthe De-
partment of Informatics, Athens University of Eco-
nomics and Business, Athens, in 2001 and 2003, re-
spectively. He is currently working toward the Ph.D.
degree in the Department of Informatics, University
of Piraeus (UniPi), Praeus, Greece.
He has professional experience in software en-
gineering and information technology. His current
research interests include pattern management, data
mining, and scientic databases.
Ioannis Kopanakis received the Diploma in com-
puter science from the University of Crete, Crete,
Greece, in 1998, and the M.Sc. degree in informa-
tion technology and the Ph.D. degree in computation
from the University of Manchester Institute of Sci-
ence and Technology (UMIST), Manchester, U.K., in
1999 and 2003, respectively.
He is currently an Assistant Professor and Head
of the Department of Commerce and Marketing at
the Technological Institution of Crete, Crete. He is
the Scientic Director of the e-Business Intelligence
Laboratory at the Center for Technological Research, Crete. His current research
interests include data mining, visual data mining, and business intelligence.
Haralampos Karanikas was born in 1972. He re-
ceived the B.Sc. degree in physics from Aristotle
University of Thessalonica, Thessalonica, Greece, in
1996, and the M.Sc. degree in computation from the
University of Manchester Institute of Science and
Technology (UMIST), Manchester, U.K., in 1998.
Currently, he is working toward the Ph.D. degree at
the University of Manchester, Manchester.
He is actively involved in text mining in collabora-
tion with leading European Universities, research in-
stitutes, and industrial partners (i.e., PARMENIDES
IST200139023 European Communities). His current research interests include
the eld of temporal text mining, which, in turn, comprises document ware-
house, automatic ontology construction, data mining, and business intelligence.
Yannis Theodoridis (S93M95) was born in 1967.
He received the Diploma and Ph.D. degrees in elec-
trical and computer engineering from the National
Technical University of Athens, Athens, Greece, in
1990 and 1996, respectively.
He is currently an Assistant Professor in
the Department of Informatics, University of Pi-
raeus (UniPi), Piraeus, Greece. Currently, he is a
Scientist in charge for UniPi in the EC-funded
GeoPKDD project (20052008) on geographic
privacy-aware knowledge discovery and delivery, and
is also involved in several national-level projects. He is the author or coauthor
of three monographs and over 50 articles in scientic journals (including the
Algorithmica, ACM Multimedia, and the IEEE TRANSACTIONS ON KNOWLEDGE
AND DATA ENGINEERING) and conferences (including the ACM SIGMOD, the
PODS, the VLDB, and the ICDE) with over 400 citations in his work. His
current research interests include spatial and spatiotemporal databases, geo-
graphical information management, knowledge discovery, and data mining.
Dr. Theodoridis is a member of the Association for Computing Machinery.
He serves on the ProgramCommittee for several major conferences in databases
and data mining, and is on the Editorial Board for the International Journal on
Data Warehousing and Mining.