b i o s y s t e m s e n g i n e e r i n g x x x ( 2 0 1 5 ) 1 e1 7

Available online at www.sciencedirect.com

ScienceDirect

journal homepage: www.elsevier.com/locate/issn/15375110

Research Paper
Special Issue: Robotic Agriculture

Detection of tomatoes using spectral-spatial

methods in remotely sensed RGB images captured
by UAV

J. Senthilnath a,b, Akanksha Dokania c, Manasa Kandukuri c,

Ramesh K.N. d, Gautham Anand e, S.N. Omkar a,*
a
Department of Aerospace Engineering, Indian Institute of Science, Bangalore, India
b
Geospatial Sciences Center of Excellence, South Dakota State University, Brookings, USA
c
Department of Electronics and Electrical Engineering, IIT Guwahati, Guwahati, India
d
Department of Electronics and Communication Engineering, UVCE, Bangalore, India
e
Department of Electronics, National College Basavanagudi, Bangalore, India

article info
The spectral-spatial classification of high spatial resolution RGB images obtained from
Article history: unmanned aerial vehicles (UAVs) for detection of tomatoes in the image is presented.
Published online xxx Bayesian information criterion (BIC) was used to determine the optimal number of clusters
for the image. Spectral clustering was carried out using K-means, expectation max-
imisation (EM) and self-organising map (SOM) algorithms to categorise the pixels into two
Keywords: groups i.e. tomatoes and non-tomatoes. Due to resemblance in spectral intensities, some of
Unmanned aerial vehicle the non-tomato pixels were grouped into the tomato group and in order to remove them,
Spectral clustering spatial segmentation was performed on the image. Spatial segmentation was carried out
Spatial segmentation using morphological operations and by setting thresholds for geometrical properties. The
number of pixels grouped in the tomato cluster is different for each clustering method. EM
doesn't pick up the land patches as tomato pixels. As a result, the size of the tomatoes
picked up is different than K-means and SOM. Since threshold values chosen for carrying
out spatial segmentation are shape and size dependent, different threshold values are
applied to different methods of clustering. A synthetic image of 12
12 pixels with
different labels is created to illustrate the effect of each method used for spatial segmen-
tation on the clustered image. Two representative UAV images captured at different
heights from the ground were used to demonstrate the performance of the proposed
method. Results and comparison of performance parameters of different spectral-spatial
classification methods were presented. It is observed that EM performed better than K-
means and SOM.
© 2015 IAgrE. Published by Elsevier Ltd. All rights reserved.

* Corresponding author. Tel.: þ91 080 229 32873; fax: þ91 080 236 00134.
E-mail address: omkar@aero.iisc.ernet.in (S.N. Omkar).
http://dx.doi.org/10.1016/j.biosystemseng.2015.12.003
1537-5110/© 2015 IAgrE. Published by Elsevier Ltd. All rights reserved.

Please cite this article in press as: Senthilnath, J., et al., Detection of tomatoes using spectral-spatial methods in remotely sensed RGB
images captured by UAV, Biosystems Engineering (2015), http://dx.doi.org/10.1016/j.biosystemseng.2015.12.003
2 b i o s y s t e m s e n g i n e e r i n g x x x ( 2 0 1 5 ) 1 e1 7
Nomenclature
UAV unmanned aerial vehicle
LAI, NDVI leaf area index, normalised difference
vegetative index
SRS, URS, LARS satellite remote sensing, UAV remote
sensing, low altitude remote sensing
VTOL vertical take-off and landing
ESC, SBC electronic speed controllers, single board
computer
BIC bayesian information criterion
EM expectation maximisation
SOM self-organising map
J objective function of K-means
P colour thresholding methods. Regunathan and Lee (2005)
mk, k, gnk mean, covariance of the data, posterior
probability
N, P(x), pk multivariate gaussian distribution, mixture of
gaussian distributions, mixing proportion of
the gaussian distributions
L(q; x) log likelihood function
dj, wij, ht discriminant function, weight vector, learning
rate
f, s, srot image, structuring element, rotated structuring
element
S(A), B skeletonisation of object A, structuring element
SI, P, A shape index, perimeter, area of the object
ROC receiver operating characteristics
TP, FP, FN true positive, false positive, false negative
1. Introduction
With the increasing global population, increasing agricultural
production is the key to meet global food security goals. The
imperative is to produce more food with less resources. The
projected growth rate of total world consumption of agricul-
tural products which include food, fodder and fibre is 1.1% per
annum. From the year 2005e2007 to 2050 (Alexandratos &
Bruinsma, 2012) the world level (not for individual countries
or regions) consumption of agricultural products is estimated
to be equal to its production, this means global production in
2050 should be more than 60% than that of 2005e2007. The
total world food production is proposed to increase from 2217
million tonnes in 2005e2006 to 3291 million tonnes in 2050
(Alexandratos & Bruinsma, 2012). With land availability and
water resources becoming scarcer, increases in production in
such proportions will not be easy.
Estimating yield is a critical input in crop management to
increase production and productivity. Several researchers
have analysed yield estimates at both macro and micro levels
(the level of individual plants) and crop yield assessments at a
macro level have been simulated and empirically evaluated
considering factors such as soil, atmosphere and plant geno-
type. Satellite remote sensing (SRS) data (Launay & Guerif,
2005; Shi & Xingguo, 2011; Yuping et al., 2008) has been used
to increase the predictability and accuracy of crop models
(Lobell, 2013). Doraiswamy, Moulin, Cook, and Stern (2003)
successfully demonstrated a method for integrating SRS
Please cite this article in press as: Senthilnath, J., et al., Detection of tomatoes using spectral-spatial methods in remotely sensed RGB
images captured by UAV, Biosystems Engineering (2015), http://dx.doi.org/10.1016/j.biosystemseng.2015.12.003
data and a crop simulation model for monitoring crop growth
and yields of spring wheat at the country and state level. Mo
et al. (2005) used SRS to measure the normalised difference
vegetative index (NDVI) and derive the leaf area index (LAI) of
crops to aid the development of crop growth models. Johnson
(2014) used SRS data for corn and soybean yield forecasting
models across the US Corn Belt at the country level.
At a micro level, plant level yield estimation can be carried


out. Stajnko & Cmelik (2005) used visible spectrum image
analysis to model harvested apple fruit yield under orchard
conditions. Stajnko, Lakota, & Hoc
evar (2004) analysed ther-
mal camera acquired images of apple trees to detect apple
fruits. Zhou, Damerow, Sun, and Blanke (2012) used visible
spectrum images to detect both green and red apples using
implemented three different classification techniques
namely Bayesian, neural network and Fischer's discrimina-
tion to differentiate fruit pixels from the background pixels.
Hannan, Burks, and Duke (2009) proposed an algorithm based
on adaptive segmentation and shape analysis for orange fruit
detection. Patel, Jain, and Joshi (2011) developed an algorithm
for fruit detection based on multiple features in which
different weights were assigned to different features such as
intensity, colour, orientation and edge of an image. Seng and
Mirisaee (2009) proposed a recognition approach which com-
bined colour-based, shape-based and size-based methods and
further used the nearest neighbour classification to classify
the fruit pixels.
UAVs provide an interesting option for remote sensing for
agricultural applications. UAV remote sensing (URS) features
high spatial resolution, low temporal resolution and the abil-
ity to operate with cloud cover. Herwitz et al. (2004) used an
UAV to acquire multispectral imagery of a coffee plantation.
The chosen study area had coffee trees that were pruned a
priori. Spectral index was used to determine the ripeness of
cherry beans. Kooistra et al. (2014) used a hyperspectral
camera mounted on a UAV to monitor LAI as an indicator of
biomass in a potato field. Hunt et al. (2010) used near infrared
reflectivity (NIR) and green and blue photographs to measure
LAI and correlate the output with vegetation index.
It can be seen that SRS is widely used to drive crop models at
a macro level. At a micro level, individual plants have been
analysed. While UAVs are used for agricultural operations such
as monitoring and measuring site specific bio-physical pa-
rameters, they are not extensively used for yield estimation.
UAV images allow detection of the tree crowns, observa-
tion of individual plants, fruits and patches. This motivates
the application of URS for intermediary levels between tradi-
tional macro and micro level crop modelling techniques to
analyse and study actual fruits in yield estimation.
In this paper two representative images acquired by URS
were analysed to detect tomatoes. Successful detection of
tomatoes demonstrates the potential of using UAV images for
actual yield estimates in future. The UAV images of an open
field tomato crop were analysed. Spectral clustering tech-
niques followed by spatial segmentation were used to extract
tomatoes. This work can be broadly divided into two stages: 1)
Spectral clustering: The optimum number of clusters is deter-
mined using Bayesian information criteria (BIC) and three
different clustering algorithms, namely K-means, expectation
 b i o s y s t e m s e n g i n e e r i n g x x x ( 2 0 1 5 ) 1 e1 7
maximisation (EM) and self-organising map (SOM), applied on
the image to generate the clusters and 2) Spatial segmentation:
Morphological operations with threshold for size and shape
based properties such as area and shape intensity are applied
to separate the tomatoes from other objects which were
falsely picked up in the spectral clustering process.
Section 2 describes the UAV system used in this study and
the data acquisition procedure. Section 3 describes the
spectral-spatial methods applied on UAV images for tomato
detection. In Section 4, a synthetic image is used to illustrate
the proposed methods. Sections 5 and 6 details the perfor-
mance measures and results and a discussion on the results
respectively. The conclusions are discussed in Section 7.
2. Unmanned aerial vehicle for remote
sensing
In this section the type of UAVs suitable for remote sensing
and more specifically for agriculture applications are dis-
cussed. UAVs typically fly at low altitudes to acquire remote
sensing data also known as low altitude remote sensing
(LARS) (Saberioon et al., 2014). For LARS, most UAVs are fixed-
wing or rotary-wing aircraft with low payload and short
endurance capabilities. Payload size and weight are critical
factors for UAVs in agricultural applications. The most
important payload component in URS is the imaging sensor.
The imaging sensor for URS therefore needs to be of small size
and low mass.
Co rcoles, Ortega, Herna  ndez, and Moreno (2013) used a
vertical take-off and landing (VTOL) micro-drone quad-rotor
aircraft to determine LAI and canopy cover mapping of onion
crop. Berni, Zarco-Tejada, Sua  rez, and Fereres (2009) used
helicopter based UAV with hyper-spectral and thermal im-
aging for vegetation monitoring. Xiang and Tian (2011) used a
14 kg helicopter with multispectral cameras and autonomous
capabilities to collect field image data with GPS navigation.
Sugiura, Noguchi, and Ishii (2005) used remote sensing for
vegetation monitoring using helicopter UAV. From the litera-
ture it can be observed that LARS for agriculture is carried out
using cameras mounted on small low payload UAVs. These
requirements drive the system characteristics of the UAV
used in this work.
2.1. System description
In this section, the characteristics and preferred configuration
of URS for agriculture application is discussed. Further, the
Table 1 e UAV specifications.
Parameters
Class of UAV:
Gross Weight:
Material:
Motors:
Length:
Endurance:
Altitude:
Battery:
Please cite this article in press as: Senthilnath, J., et al., Detection of tomatoes using spectral-spatial methods in remotely sensed RGB
images captured by UAV, Biosystems Engineering (2015), http://dx.doi.org/10.1016/j.biosystemseng.2015.12.003
3
process of building the UAV for data acquisition has been
detailed.
The UAV used in this study is built using off-the-shelf
components. It is a quad-copter which belongs to a class of
multi-rotor aircraft. The UAV was an electrically powered,
VTOL, with low endurance and low cost. The main advantage
of VTOL UAVs are their hovering capabilities which makes
them stable platforms during image acquisition. It is piloted
remotely from the ground station by a trained pilot. The take-
off weight was 2.1 kg including a payload of 0.1 kg. The UAV
and camera specifications are listed in Tables 1 and 2
respectively.
Figures 1 and 2 show an image of the system and a block
diagram, respectively. The URS comprised the following sub-
systems, namely, a UAV, an imaging sensor and ground
control.
2.1.1. UAV sub-system
This sub-system consisted of all the components and devices
required for the UAV to be airborne. The UAV sub-system has
four brushless DC motors mounted on a quad-copter frame
operated using four electronic speed controllers (ESC) that
were powered by Lithium-Polymer batteries. The quad-copter
was stabilised using an on-board flight stabilisation system.
The flight stabilisation system was an open-source multi-
rotor platform called ardupilot (APM2.5, 3d Robotics Inc, Ber-
keley, California, USA). It consisted of MEMS based gyroscope,
accelerometer, magnetometer, barometer, GPS module and
AT-mega 2560 AVR microcontroller. As the pilot navigated
using the remote control, the control commands were trans-
mitted over the air interface to the on-board flight stabilisa-
tion system. The flight stabilisation system varied the speed of
the motors to achieve the desired altitude. The pitch-roll
camera stabilisation maintained the nadir view which was
independent of the orientation of UAV.
2.1.2. Imaging sensor sub-system
The imaging system is realised using a Raspberry Pi® (Model
1 B, Raspberry Pi Foundation, Caldecote, Cambridgeshire,
United Kingdom). The Raspberry Pi® is a credit card sized
single board computer (SBC) that has a camera board as a
standard add on. The SBC runs a Linux operation system.
Aerial video was acquired using the camera module of the
Raspberry Pi® computer. The Raspberry Pi® camera module
specification is shown in Table 2. The camera has a fixed focal
length of 3.6 mm and a fixed horizontal and vertical field of
view. The UAV was flown up to an altitude of 50 m. The images
chosen for analysis were captured at altitudes below 20 m.
UAV specifications
Micro Aerial Vehicle (MAV), Rotor aircraft, Quadcopter
2.1 kg
Fibre glass, aluminium
Four (4) BLDC motors, 1000 Kv, 210 W each
550 mm
20 min
500 m
Lithium polymer 3S 5000 mAh
4 b i o s y s t e m s e n g i n e e r i n g x x x ( 2 0 1 5 ) 1 e1 7

UAV. The images were acquired with nadir orientation which

Table 2 e Camera specifications.
provides a top view of the tomato fruits. An oblique orienta-
Parameter specification Value tion may occlude the visibility of the fruits because of plant
Camera make Camera module of Raspberry PI stems and leaves. Although nadir orientation also causes oc-
Video HD resolution 1080 p, 30 Hz clusion, occlusion was predominantly due to the stalks of the
Sensor resolution 2592
1944, 5 mega pixels fruits, which our proposed method addresses.
Sensor image area per pixel 1.4 mm
1.4 mm
The video was reviewed offline to extract frames (images)
Focal length 3.60 mm
that have a good aerial views of the tomato crop. Regions of
Horizontal field of view 53.50
Vertical field of view 41.41 interest (RoIs) were selected within the extracted frames that
depicted the tomatoes well. It must be noted that the size of
the images chosen for analysis vary based on the size of the
The camera used the default calibration of average of sensors. RoI.
The camera is pre-tuned to an 'average' of the sensors as a Tomato crops shed their leaves during the terminal stages
default setting and the same default setting is used in image of crop development. During this stage, the fruits are visible in
acquisition. aerial images. Images acquired during the terminal stage were
analysed for yield estimation. In earlier stages of the crop,
2.1.3. Ground control sub-system where the plants were leafy and the fruits were concealed
This system consists of wireless devices required to control below, image analysis was not possible.
the UAV flight path and camera. A nine channel remote con- The experimental site was at Mudimadagu village (Latitude
trol is used to manually control the UAV. It is required to 13.56N, Longitude 78.36E) in the Rayalpad subdivision of Sri-
operate the SBC on field. To achieve this objective, a battery nivaspur Taluk, Kolar district, in the south-east of peninsular
operated display and keyboard are required. A palm size India.
display module was connected to the SBC through a com-
posite video port. The wireless keyboard was used to interface
with the SBC. 3. Methodology

2.2. Data acquisition Spectral-spatial classification of tomatoes and non-tomatoes to

The SBC along with the camera module was used for image
acquisition. As the UAV flew over the farm, the SBC captured
video for 10 s and wrote the data into a file. This is repeated for
the entire flight duration of 5 min. This functionality was
accomplished using a shell script program running on the
SBC. The spatial resolution of the extracted frames vary based
on the altitude at which the image was acquired. Since the
altitude of the UAV changed as it flew, the spatial resolution,
or the depth of the image, varied based on the altitude of the
UAV images was applied. Spectral-spatial methods have been
successfully applied to satellite remote sensing data
(Senthilnath et al., 2013; Tarabalka, Benediktsson, & Chanussot,
2009). To achieve better classification of spectral-spatial
methods majority voting on the pixel wise classification is
performed using an adaptive neighbourhood. Here BIC was
used to predict the optimal number of clusters for UAV remote
sensing images and carried out the spectral clustering using K-
means, EM and SOM. For spatial segmentation morphological
operations were used with geometrical attributes.

Fig. 1 e Unmanned aerial vehicle (UAV) system.

Please cite this article in press as: Senthilnath, J., et al., Detection of tomatoes using spectral-spatial methods in remotely sensed RGB
images captured by UAV, Biosystems Engineering (2015), http://dx.doi.org/10.1016/j.biosystemseng.2015.12.003
 b i o s y s t e m s e n g i n e e r i n g x x x ( 2 0 1 5 ) 1 e1 7 5

X K 
N X 2
 
J¼ xn  mk  (3)
n¼1 k¼1

where xn is the nth pixel and m k is the kth cluster centre. The
BIC-K-means steps are

Step 1: Find the number of clusters K to be generated using

Fig. 2 e UAV system block diagram.
3.1. Spectral clustering
Three unsupervised spectral clustering methods, namely, K-
means, EM and SOM are used for grouping pixels into to-
matoes and non-tomatoes.
The RGB value of any pixel in an image depends on the
reflectance properties of the object it represents. Pixels can be
classified into different groups depending on their spectral
intensities. The aim was to classify the image into two groups:
tomatoes and non-tomatoes. However, due to resemblance in
the spectral intensities of some non-tomato pixels were
classified with the tomato pixels. To overcome this problem, a
spatial classification was applied as discussed in Section 3.2.
A key process for efficient clustering of RGB images is the
aggregation and assignment to information class because
many subclasses may be included in a single information class.
For example, the information class ‘tomatoes’ may include
several intensity subclasses such as ripe and non-ripe to-
matoes. To determine the number of clusters for such varying
intensity distribution, a BIC (Schwarz, 1978) was adopted.
  1
BICzln L b
q  *kj *logðnÞ
2 PðxÞ ¼
    
q ¼ p xb
where L b q; M
Equation (2) represents the maximised likelihood measure
of model M for dataset x and b q are the parameter values that
maximise the likelihood. kj represents the number of free
parameters to be estimated for a specific number of clusters
and n represents the number of attributes for the given data-
set. For i number of clusters of m dimensional data, the
number of free variables will be i * (1 þ m þ (m*(m þ 1)/2))
where 1 is for responsibility, m is for the mean and the third
term is for the degrees of freedom of a covariance matrix.
With an increase in the number of clusters, BIC at first tends to
increase, reaches a maximum value and then decreases. The
value at which it attains the maximum was considered to be
the optimal number of clusters.
BIC.
Step 2: Randomly choose K pixel values from the image as
cluster centres.
Step 3: Calculate the Euclidean distance of every pixel from
each cluster centre.
Step 4: Assign the pixel to the nearest cluster centre i.e. the
cluster centre for which Euclidean distance is minimum.
Step 5: Recalculate the cluster centres for each cluster by
computing the average of the pixel values allocated to that
specific cluster.
Step 6: Repeat the process until there is no further change
in the distribution of the pixel intensities.
Step 7: Select the cluster which is dominated with to-
matoes for spatial segmentation, further apply spatial
methods to remove misclassified tomato regions as dis-
cussed in section 3.2.
3.1.2. Expectation maximisation (EM)
EM is an iterative method for determining maximum likeli-
hood estimates in which it is assumed that data points are
drawn from a mixture of distributions (Li, Zhang, & Jiang,
2005). In this study, the statistical model considered is a
mixture of Gaussian models where mean and covariance are
the parameters to be estimated. The BIC-EM algorithm steps
are (Li et al., 2005):
Step 1: Consider a mixture of Gaussian distributions given
by
(1) X
K
pk Nðxjmk ; Sk Þ (4)
i¼1
(2) where K is the total number of Gaussian distributions in the
mixture model, pk2[0,1] is the mixing proportion of the
P
Gaussian model k satisfying the property Kk¼1 pk ¼ 1, mk is the
mean of the Gaussian model k, Sk is the covariance of the
Gaussian model k and N(xjmk,Sk) is the multivariate Gaussian
distribution.
Step 2: Find out the number of distributions K in the
mixture by using BIC.
Step 3: Randomly initialise mean values, covariance
matrices and mixing coefficients.
Step 4: Using these values, evaluate the posterior proba-
bility gnk(responsibility) of kth cluster towards the pixel xn
given by

pk Nðxn jmk ; Sk Þ
3.1.1. K-means gnk ¼ P    (5)
K 
K-means aims at minimisation of the Euclidean distance j¼1 pj N xn mj ; Sj

value (Selim & Ismail, 1984). The objective function J is given

by Step 5: Assign every pixel to the cluster with greatest pos-
terior probability.

Please cite this article in press as: Senthilnath, J., et al., Detection of tomatoes using spectral-spatial methods in remotely sensed RGB
images captured by UAV, Biosystems Engineering (2015), http://dx.doi.org/10.1016/j.biosystemseng.2015.12.003
6 b i o s y s t e m s e n g i n e e r i n g x x x ( 2 0 1 5 ) 1 e1 7

Step 6: Update the parameters of each distribution using basis for competition. The discriminant function for a
Eqs. (6)e(8) neuron is given by

PN
gnk xn X
D
 2
mnew
k ¼ n¼1
(6) dj ¼ xi  wij (11)
Nk
i¼1

PN   T
Xnew gnk xn  mnew xn  mnew where x is the input vector of D dimension and wij is the
¼ n¼1 k k
(7) weight vector connecting the ith component of the input
k Nk
vector to the jth neuron. Hence, the winning neuron is the
XN neuron that most closely matches the input i.e. the neuron for
Nk
pnew
k ¼ ; where; Nk ¼ gnk (8) which the discriminant function is minimum.
N n¼1
When one neuron is activated, its closest neighbours tend
where mnew
k ; Sk ; pk
new new
are the updated parameters after each to get excited more than those farther away. There is a topo-
iteration, Nk is the total sum of the responsibilities of a logical neighbourhood that decays with distance (Kohonen,
Gaussian distribution and N is the total number of pixels in the 1990). This neighbourhood set Nc around the winning
image. neuron where c is defined.

Step 7: Calculate the log likelihood function given by Step 4: Update the winning neuron and the neurons within
the topological neighbourhood using

X
n
LðQ; XÞ ¼ logðpxi jQÞ (9)  
i¼1 Dwij ¼ hðtÞ xi  wij (12)

where the observation X ¼ {xiji ¼ 1,…,n} are independently
drawn from the distribution p(x) parameterised by Q.
The optimised likelihood is given by
e
LðQ; XÞ ¼ LðQ; XÞ þ gPðX; yjQÞ
where the regulariser P is a functional of the distribution of the
complete data given by Q and the positive value g is the reg-
ularisation parameter that controls the compromise between
the degree of regularisation of the solution and the likelihood
function.
Repeat the steps 4e6 till the log likelihood value converges.
Step 8: Select the cluster which is dominated with to-
matoes for spatial segmentation, further apply spatial
methods to remove misclassified tomato regions as dis-
cussed in Section 3.2.
3.1.3. Self-organising maps (SOM)
SOM is an unsupervised algorithm which belongs to the class
of competitive learning (Kohonen, 1990). In competitive
learning, output neurons compete with each other to get
activated.
SOM transforms an incoming signal pattern into one or two
dimensional discrete map in a topological fashion. The neu-
rons become selectively tuned to various input patterns or
classes of input patterns during the course of competitive
learning. The BIC-SOM algorithm steps are:
Step 1: Find out the number of neurons K in the mixture
using BIC.
Step 2: Randomly initialise the connection weights of the
neurons.
Step 3: For each input pattern, compute the value of the
discriminant function for each neuron. This provides the
where Dwij indicates the change in the weight and h(t) is the
time dependent learning rate which decreases with time. In
this study the basic Kohonen network was implemented. The
above steps were repeated until there was no further change
(10) in the topography.
Step 5: Select cluster which is dominated with tomatoes for
spatial segmentation, further apply spatial methods to
remove misclassified tomato regions as discussed in Sec-
tion 3.2.
3.2. Spatial segmentation
As discussed in the last step of K-means, EM and SOM algo-
rithm, here the spatial features of the clustered image were
used to remove misclassified tomato regions.
The steps used in spatial segmentation and the rationale
for the chosen sequences of steps were as follows:
3.2.1. Closing operation
Some of the single tomatoes appear bisected in the binary
segmented image because of the overhead stalks. While
counting, these may be counted as two instead of one. In order
to join these, close operation is applied on the image with an
appropriate structuring element.
The closing of an image f by a structuring element s is
dilation followed by erosion (Haralick, Sternberg, & Zhuang,
1987). The structuring element is a small matrix of pixels,
each with a value of zero or one. The dimensions of the matrix
state the size of the structuring element and the pattern of
ones and zeros specifies its shape.
f $s ¼ ðf 4srot ÞQsrot (13)
where srot means that the dilation and erosion should be
performed with a rotated structuring element. In case of

Please cite this article in press as: Senthilnath, J., et al., Detection of tomatoes using spectral-spatial methods in remotely sensed RGB
images captured by UAV, Biosystems Engineering (2015), http://dx.doi.org/10.1016/j.biosystemseng.2015.12.003
 b i o s y s t e m s e n g i n e e r i n g x x x ( 2 0 1 5 ) 1 e1 7 7

symmetrical structuring elements, the rotation does not make
a difference. 4 represents the dilation of f with srot and Q
represents erosion. A disk structuring element of radius 2 is
used in this study.
3.2.2. Area thresholding and skeletonisation
The noise was predominantly due to patches of soil, stalks
and pebbles. It is not possible to eliminate all noise in one step
due to different geometric characteristics of the noises. The
soil patches, being the largest, were remove first by setting a
threshold for area values. The threshold of area values for a
given image depends on the resolution of the image. The
resolution of the image is a function of the altitude of the
image from the ground. The objects in the image were first
labelled and then the area for each object was calculated. Area
of an object was calculated by finding the total number of
pixels depicting that object. However, problems arose in
selecting a threshold when the tomatoes in the image
overlapped.
In some cases, the area values of tomatoes that overlapped
and soil patches were similar. We overcome this difficulty by
taking into account the skeleton representation of each object
calculated using morphological means. The skeleton of an
object provides a simple and compact representation of a
shape that preserves the topological and the original size of
the object. The skeletonisation S(A) (Sagar, 2013) of an object A
is carried out using the following equations:
Sk ðAÞ ¼ ðAQkBÞ  ðAQkBÞoB
K
SðAÞ ¼ ∪ Sk ðAÞ
k¼0
where B is a structuring element of appropriate size and
shape. At first, k successive erosions of object A were carried
out with the structuring element B. The eroded image was
then opened using B and then it is subtracted from the eroded
image to obtain the skeleton points as shown in Eq. (14). The
operations were done iteratively for different sizes of B. The
union of all such skeleton points gave the skeleton represen-
tation of the object. In this study, instead of taking the union
as shown in Eq. (15), only Eq. (14) was implemented for a
suitable k and B to find out the number of skeleton points for
each object. Due to overlapping, the connected tomatoes had
a complex shape and they need a greater number of pixels to
represent their skeleton. Therefore, along with an area
threshold, a threshold for skeleton points can be put to retain
the tomatoes and eliminate the land patches.
3.2.3. Removal of stalks
Tomatoes are nearly circular in shape while the stalks are
relatively long, thin and have branches. This distinction in the
geometrical features of the stalks can be used to categorise
them into non-tomato group. The geometric parameter shape
index (SI) (Senthilnath et al., 2013) given by
P
SI ¼ pffiffiffiffi (16)
4* A
where P represents the perimeter of the object (i.e. the number
of pixels on the boundary of the object) and A represents the
area of the object. For a given area, the perimeter value of a
stalk is greater than that of a tomato, which makes the ratio of
perimeter to area, a higher value for the stalks. Thus, a
threshold value was set for SI to remove stalks. The morpho-
(14) logical operation open was applied on the image to detach the
stalks attached with tomatoes.
(15)
3.2.4. Removal of elliptical noise
The unwanted objects remaining in the image were mostly
pebbles which are commonly elliptical in shape. The best way
to distinguish an ellipse from a circle is by finding the ratio of

Fig. 3 e Illustrative example (a) Clustered image (b) Structuring element disk of radius 1 (c) Image after applying closing
operation (d) Effect of area thresholding (e) Effect of shape intensity thresholding (f) Removing the elliptical noise by
calculating the ratio of major to minor axis.

Please cite this article in press as: Senthilnath, J., et al., Detection of tomatoes using spectral-spatial methods in remotely sensed RGB
images captured by UAV, Biosystems Engineering (2015), http://dx.doi.org/10.1016/j.biosystemseng.2015.12.003
8 b i o s y s t e m s e n g i n e e r i n g x x x ( 2 0 1 5 ) 1 e1 7

the length of the major axis to the minor axis. Circular objects shown in Fig. 3(e). The major to minor axis ratio calculated for
have the ratio almost one while elliptical objects have ratio each object is
greater than one. The labels having ratio greater than the
threshold are categorised into non-tomato group while the
others are retained in the tomato group.
The result obtained, after implementing all the noise Label 1: 1.9861 Label 5: 2.2638
Label 6: 1.4639 Label 7: 1.0000
removal methods, was an image consisting tomatoes present
in the image.
Label 5 represented a pebble in the image. It had the
highest major to minor axis ratio and was eliminated by
setting a threshold of 2 for the ratio. The final result obtained
4. Illustrative example
after eliminating all the misclassified tomato regions is shown
in Fig. 3(f).
A synthetic image of size 12
12 pixels was assumed to
illustrate the effect of each step in the proposed algorithm.
Figure 3(a) shows the clustered image with tomatoes and
misclassified tomato regions. The challenge here is to retain 5. Performance measures
tomatoes by removing misclassified tomato regions, i.e. to
obtain labels 1, 6 and 7. The performance measures using ROC parameters were ana-
Firstly, close operation is applied on the image with a disk lysed in this study by comparing the spectral-spatial classi-
structuring element of radius 1 as shown in Fig. 3(b). Closing fiers. We have the reference data which shows the location of
fills the gaps in the regions while retaining the initial region tomato pixels in the image. It is collected using field survey.
size, as shown in Fig. 3(c). The processed binary image is overlaid on the reference data
The number of pixels (area) of each label was specified as: and the tomatoes and non-tomatoes pixels are identified
respectively. The number of true tomatoes and false tomatoes
were counted separately to evaluate the performance
measures.
Label 1: 13 Label 2: 02
ROC parameters calculated for UAV images to extract to-
Label 3: 11 Label 4: 08
Label 5: 08 Label 6: 03
matoes are defined in terms of true positive (TP), false positive
Label 7: 04 Label 8: 01 (FP) and false negative (FN) (Fawcett, 2006; Senthilnath,
Shivesh, Omkar, Diwakar, & Mani, 2012). A TP means that
the extracted object is a tomato and the database also in-
From Fig. 3(c), it can be observed that number of skeleton dicates the object to be a tomato. If the extracted object is not a
points for each label was: tomato but the database indicates it to be a tomato, then it is
counted as a FN. For an FP the extracted object is a tomato but
the database indicates it not to be a tomato.
Based on these ROC parameters, performance measures:
Label 1: 4 Label 2: 0
Recall, Precision and F-Measure are being evaluated.
Label 3: 2 Label 4: 2
Label 5: 2 Label 6: 1
Label 7: 0 Label 8: 0 i. Recall:

A threshold value is put for area in between 9 and 3 along

TP
with a threshold of 3 for number of skeleton points. So, all the R¼ (17)
TP þ FN
labels having area greater than 9 and less than 3 and skeleton
points less than 3 are assigned label 0. Although the area of
label 1 falls outside the threshold range, it is able to retain its
label due to the threshold used for number of skeleton points ii. Precision:
as shown in Fig. 3(d). The labels 2, 3 and 8 are no longer pre-
sent in the image as shown in Fig. 3(d). Next, the SI value for
each label was calculated using the perimeter value of the
TP
respective label. The SI value for each label was: P¼ (18)
TP þ FP

Label 1: 1.0212 Label 4: 0.9786

Label 5: 0.8536 Label 6: 0.4928 Label 7: 0.5000 iii. F-Measure:

Labels 1 and 4 had larger shape intensity values than other

labels. A threshold value less than 0.9, along with skeleton 2PR
F¼ (19)
points threshold of 3 removes label 4 from the image, as PþR

Please cite this article in press as: Senthilnath, J., et al., Detection of tomatoes using spectral-spatial methods in remotely sensed RGB
images captured by UAV, Biosystems Engineering (2015), http://dx.doi.org/10.1016/j.biosystemseng.2015.12.003
 b i o s y s t e m s e n g i n e e r i n g x x x ( 2 0 1 5 ) 1 e1 7
For a good spectral-spatial classifier, all the above mea-
sures will be close to one.
6. Results and discussion
In this section, the experimental results obtained using UAV
images for tomato detection are presented. In the images and
the spatial resolution of images acquired in this study are
dynamic since the altitude of UAV varied. The nearer the UAV
was to the tomato field, better features it picked up in the
image. The two images examined were acquired at different
altitudes and hence different spatial resolution were studied.
The detection of tomatoes was carried out using spectral-
spatial classification.
6.1. UAV image 1
The size of the first UAV image was 293
415 pixels as shown
in Fig. 4. The optimal number of clusters to be chosen for
clustering depends on the data set. The BIC curve reached a
maximum at 3 as shown in Fig. 5. Hence all the spectral
methods generated 3 clusters.
The spectral methods, namely, K-means, EM and SOM,
were applied to group tomato and non-tomato regions in the
image. The resultant images are as shown in Figs. 6(a), 7(a) and
8(a) respectively. The spectrally classified image contained
soil patches, stalks and pebbles among other objects that were
misclassified as belonging to the tomato region.
To overcome this problem, spatial classification was
applied on spectrally classified image as discussed in Section
3.2. For K-means clustered image, an area threshold value of
25 was used where all the labels having area <25 were grouped
into non-tomato pixels. To remove the stalks an SI threshold
value of 1.4125 was used along with a threshold value of 26 for
the number of skeleton points. Therefore, all the labels having
SI value >1.4125 were classified into non-tomato pixels.
However, due to the threshold value for number of skeleton
points, certain overlapped tomatoes, despite having SI values
>1.4125 were retained in tomato group. For eliminating the
elliptical shaped noise, a value of 2.3033 was used as the
Fig. 4 e UAV image 1.
Please cite this article in press as: Senthilnath, J., et al., Detection of tomatoes using spectral-spatial methods in remotely sensed RGB
images captured by UAV, Biosystems Engineering (2015), http://dx.doi.org/10.1016/j.biosystemseng.2015.12.003
9
maximum threshold point for the ratio of major to minor axis
along with a threshold value of 26 for the skeleton points.
Likewise, for EM clustered image an area threshold value of
16 and SI threshold value of 1.424 were used along with a
threshold of 8 for the number of skeleton points. For the ratio
of the major axis length to minor axis, a threshold value of
2.0655 along with a threshold value of 25 for skeleton points
was used. For SOM clustered image, an area threshold value of
18 and SI threshold value between 2.5077 and 1.4402 was used.
So, all those labels having an SI value in the threshold range
were categorised as non-tomato pixels. For eliminating the
elliptical shaped noise, a value of 2.4079 was used as the
threshold value for major to minor axis ratio along with a
threshold value of 18 for skeleton points.
The spatial classification leads to improve the tomato
detection as shown in the Figs. 6(b), 7(b) and 8(b) for K-means,
EM and SOM respectively. Figures 6(c), 7(c) and 8(c) show the
spectral-spatial classified tomato region overlaid on the orig-
inal image.
The results of the Figs. 6(c), 7(c) and 8(c) are represented in
Table 3 which compares the ROC parameters of all the
methods, namely, K-means, EM and SOM. The performance
measures based on ROC parameters were calculated for each
method, as shown in Table 4. For this was lower for SOM. The
advantage of SOM is that it had higher precision value than the
other two methods as it selects less false positives. To analyse
the results, the graph between the precision and recall was
plotted as shown in Fig. 9. The desirable outcome was the point
where both the measures i.e., recall and precision were one.
The point closest to the coordinate (1, 1) was selected as the
best result. The results of SOM and EM were equally good for
both precision and recall compared to the K-means process.
In all these algorithms used for tomato detection, the
number of clusters chosen for clustering played an important
role. Since, there are two classes of objects, i.e. tomato and
non-tomato region, the performance measures were analysed
by taking exact partitioning into 2 clusters. This was
compared with the clusters generated using BIC, which is 3
clusters, for this image. The ROC parameters and the perfor-
mance measures using two clusters are shown in Tables 5 and
Fig. 5 e BIC curve of UAV image 1.
10 b i o s y s t e m s e n g i n e e r i n g x x x ( 2 0 1 5 ) 1 e1 7

Fig. 6 e (a) Spectral clustering using K-means on image 1 (b) Spatial classification on the clustered image (c) Spectral and
spatial classified tomato region overlaid on the original image.

6 respectively. While comparing the performance measures, it
is noticed that choosing BIC generated 3 cluster centres gave a
better recall as well as precision values than 2 clusters for K-
means and SOM. For EM, the precision value was better for 2
clusters than for 3 clusters generated by BIC. However, the
number of false positives picked was high for the case of 2
clusters as compared to 3 clusters generated by BIC. Hence,
clustering using 3 clusters was a better solution than 2 clus-
ters. Figure 9 shows the precision verses recall values using
BIC and without BIC for K-means, EM and SOM. It was
concluded that using the number of clusters generated by BIC
gave a better result compared with 2 clusters.
In EM method regularisation factor plays a key role in
classification of the pixels as tomatoes. In order to decide the
regularisation factor which gives the best result, the change in
the ROC parameters and performance measures with respect
to different regularisation factors was examined. The com-
plete analysis is shown in Table 7. Since, F-Measure takes into
account both recall and precision, it can be taken as the prime
factor for deciding the best regularisation value. From Table 7,
it can be seen that F-Measure reached maximum value of
0.8911 for 0.0005. Hence, the obtained result using 0.0005 as
the regularisation value was used for comparing the perfor-
mance of EM method with the other two methods.
SOM considers learning rate as an important parameter
for deciding the classification of pixels. The optimal value
was observed by varying the learning rate and then
comparing the performance measures obtained for each
case. The complete analysis is shown in Table 8, from this
table it can be observed that the value 0.99 gives the better
result.
The algorithms were tested on a computer with Core-i3
processor, 4 GB RAM with Matlab (R2013a, The MathWorks,
Inc., Natick, Massachusetts, USA), and the execution time for
UAV Image 1 was recorded. The time of execution for the UAV
Image 1 for K-means spatial method was 874.41 s, the EM-
spatial method took 3453.818 s and SOM-spatial method
took 2886.445 s.
6.2. UAV image 2
The second UAV image is of size 1080
1920 pixels, as shown
in Fig. 10. The number of clusters at which the BIC curve
reaches its maxima was analysed. From Fig. 11, it can be
observed that 5 is the optimal number of clusters to be applied
on this image for all the clustering methods.
The clustering of the image using K-means, EM and SOM is
as shown in Figs. 12(a), 13(a) and 14(a) respectively. In the
clustered image, misclassification can be observed where
non-tomato pixels are misclassified as tomato pixels. In order
to remove the misclassified pixels, spatial classification is
performed on these images.

Please cite this article in press as: Senthilnath, J., et al., Detection of tomatoes using spectral-spatial methods in remotely sensed RGB
images captured by UAV, Biosystems Engineering (2015), http://dx.doi.org/10.1016/j.biosystemseng.2015.12.003
 b i o s y s t e m s e n g i n e e r i n g x x x ( 2 0 1 5 ) 1 e1 7 11

Fig. 7 e (a) Spectral clustering using EM on image 1 (b) Spatial classification on the clustered image (c) Spectral and spatial
classified tomato region overlaid on the original image.

In case of EM, since no soil patches were classified into the
tomato group, and “salt and pepper” noise was removed first.
To achieve the same, a threshold value of 62 was used for
areas where all the labels having area <62 were grouped into
non-tomato pixels. All those labels having SI value in the
range of 1.9486 to 0.9219 were retained as tomato pixels. To
remove the pebbles, a threshold value of 2.6084 was set for the
ratio of major to minor axis. For K-means and SOM, soil
patches were also misclassified into the tomato group. In case
of K-means, soil patches have areas >1475 and the “salt and
pepper noise” have areas <140. Therefore, an area threshold
value in the range of 1475 and 140 along with 50 as the
threshold value for the number of skeleton points was used to
retain the tomatoes. An SI threshold value was set at 1.6137
and for the ratio of major axis to minor axis, 1.8030 was cho-
sen as the threshold value and 5 as the threshold for number
of skeleton points. Similarly for SOM, the area threshold value
chosen was in the range of 2070 and 130 along with 25 as the
threshold for the number of skeleton points. The threshold
values for SI and ratio of major to minor axis were set as 1.7768
and 1.9396 respectively. The resultant images were as shown
in Figs 12(b), 13(b) and 14(b) for K-means, EM and SOM
respectively. Figures 12(c), 13(c) and 14(c) are the spectral-
spatial classified tomato region overlaid on the original image.
Due to the similar spectral intensities, few tomato pixels
and non-tomato pixels were misclassified with each other.
The number and the type of pixels grouped in the tomato
cluster were different in each method. For example, for EM,
the tomato cluster did not pick up the soil patches and did not
capture the tomato pixels fully. The size of the tomatoes
picked up in EM tomato cluster was smaller compared to other
two methods. In the case of SOM, the soil patches, stalks and
pebbles were captured in larger proportion than other two
methods. Since the threshold values were shape and size
dependent, different threshold values were applied to
different methods of clustering.
Threshold values were data dependent due to the
different resolutions used in this study. Threshold values of
these above mentioned parameters were chosen empirically.
A sensitivity analysis for the spatial method applied on K-
means spectrally clustered image was carried out. Table 9
shows how the results varied with the change in threshold
values for spatial method. As the threshold values were
increased for all these three geometrical properties, indi-
vidually or together, (i.e. the labels having values greater
than these thresholds are removed) the number of true
positives as well as false positives increased. Hence, the
threshold values for spatial method were chosen for the
empirically set value so as to give a balance between the
precision and recall. This was done by calculating F-Measure
in our study. The thresholds, which give the better F-Mea-
sure value, were set.

Please cite this article in press as: Senthilnath, J., et al., Detection of tomatoes using spectral-spatial methods in remotely sensed RGB
images captured by UAV, Biosystems Engineering (2015), http://dx.doi.org/10.1016/j.biosystemseng.2015.12.003
12 b i o s y s t e m s e n g i n e e r i n g x x x ( 2 0 1 5 ) 1 e1 7

Fig. 8 e (a) Spectral clustering using SOM on image 1 (b) Spatial classification on the clustered image (c) Spectral and spatial
classified tomato region overlaid on the original image.

The ROC parameters and the performance measures for was greater than the difference in the recall values, which
the second UAV image are shown in Tables 10 and 11 makes EM a better method than SOM. The same inference can
respectively. From Table 10, it can be seen that along with be observed from Fig. 15 where the EM (precision and recall)
the largest number of true positives, SOM gave the largest
number of false positives. Because of this the precision value
for SOM was the least among the three methods. For EM, the
number of true positives was less than SOM, but the number
of false positives was also less when compared to SOM or K-
means. The difference in the precision values of SOM and EM

Table 3 e ROC parameters comparison for UAV Image 1

with BIC.
ROC parameter K-means EM SOM
TP 45 45 44
FP 06 05 03
FN 06 06 07

Table 4 e Evaluating performance measures based on

ROC parameters in Table 3.
Performance measure K-means EM SOM
Recall 0.8824 0.8824 0.8627
Precision 0.8824 0.9000 0.9362
F-Measure 0.8824 0.8911 0.8979
Fig. 9 e Precision versus recall curve for UAV image 1.

Please cite this article in press as: Senthilnath, J., et al., Detection of tomatoes using spectral-spatial methods in remotely sensed RGB
images captured by UAV, Biosystems Engineering (2015), http://dx.doi.org/10.1016/j.biosystemseng.2015.12.003
 b i o s y s t e m s e n g i n e e r i n g x x x ( 2 0 1 5 ) 1 e1 7 13

clusters derived using BIC, which is 5 clusters for this image.

Table 5 e ROC parameters comparison for UAV image 1
The ROC parameters and performance measures obtained
without BIC.
using 2 clusters are shown in Tables 12 and 13 respectively.
ROC parameter K-means EM SOM
From Table 12 it can be observed that true positives for all the
TP 38 38 43 three methods in the case of 2 clusters decreased drastically
FP 19 01 05 compared to the true positives shown in Table 10, for the case
FN 13 13 08
of 5 clusters. K-means is affected the most with 2 clusters as
there is a substantial decrease in the number of true positives
and increase in the number of false positives. It can also be
Table 6 e Evaluating Performance measures based on concluded that SOM and K-means give a better result using
ROC parameters in Table 5. BIC. However, for EM the results for both the number of
Performance measure K-means EM SOM clusters were equally good but using 5 clusters gave a better
Recall 0.7451 0.7451 0.8431 quantity of true positives while using 2 clusters gives less
Precision 0.6667 0.9744 0.8958 number of false positives. The complete comparison can be
F-Measure 0.7037 0.8445 0.8687 seen in Fig. 15.

Table 7 e ROC parameters and performance measures for different regularisation factor used in EM for UAV Image 1.
Regularisation factor True positives False positives False negatives Recall Precision F-Measure
0 47 8 04 0.9216 0.8545 0.8868
0.0005 45 5 06 0.8824 0.9000 0.8911
0.0008 18 0 33 0.3529 1.0000 0.5217
0.0010 17 0 34 0.3333 1.0000 0.4000
0.0030 39 1 12 0.7647 0.9750 0.8571
0.0050 41 4 10 0.8039 0.9111 0.8541
0.0080 42 2 09 0.8235 0.9545 0.8842
0.0100 35 2 16 0.6863 0.9459 0.7955
0.0200 09 3 42 0.1765 0.7500 0.2858

Bold indicates the value used for further processing.

Table 8 e ROC parameters and performance measures for different learning parameter used in SOM for UAV Image 1.
Learning rate True positives False positives False negatives Recall Precision F-Measure
0.10 39 4 12 0.7647 0.9070 0.8298
0.30 39 7 12 0.7647 0.8478 0.8041
0.50 42 5 09 0.8235 0.8936 0.8571
0.70 42 5 09 0.8235 0.8936 0.8571
0.90 42 4 09 0.8235 0.9130 0.8659
0.99 44 3 07 0.8627 0.9362 0.8979

Bold indicates the value used for further processing.

values are nearer to (1, 1). K-means performance measure

values were lower than the performance measures of EM.
To comprehend the significance of BIC, the analysis was
carried out again using exact partitioning of 2 clusters and

Fig. 10 e UAV image 2. Fig. 11 e BIC curve of UAV image 2.

Please cite this article in press as: Senthilnath, J., et al., Detection of tomatoes using spectral-spatial methods in remotely sensed RGB
images captured by UAV, Biosystems Engineering (2015), http://dx.doi.org/10.1016/j.biosystemseng.2015.12.003
14 b i o s y s t e m s e n g i n e e r i n g x x x ( 2 0 1 5 ) 1 e1 7

Fig. 12 e (a) Spectral clustering using K-means on image 2 (b) Spatial classification on the clustered image (c) Spectral and
spatial classified tomato region overlaid on the original image.

Fig. 13 e (a) Spectral clustering using EM on image 2 (b) Spatial classification on the clustered image (c) Spectral and spatial
classified tomato region overlaid on the original image.

Please cite this article in press as: Senthilnath, J., et al., Detection of tomatoes using spectral-spatial methods in remotely sensed RGB
images captured by UAV, Biosystems Engineering (2015), http://dx.doi.org/10.1016/j.biosystemseng.2015.12.003
 b i o s y s t e m s e n g i n e e r i n g x x x ( 2 0 1 5 ) 1 e1 7 15

Fig. 14 e (a) Spectral clustering using SOM on image 2 (b) Spatial classification on the clustered image (c) Spectral and spatial
classified tomato region overlaid on the original image.

Table 9 e Sensitive analysis for UAV image 2 with BIC e K-means.

Area threshold Shape intensity Major axis/Minor axis Recall Precision F-Measure
140e1475 1.6137 1.8030 0.6400 0.8067 0.7137
150e2000 1.6137 1.8030 0.5733 0.7748 0.6590
150e2000 2.0000 1.8030 0.6000 0.7692 0.6741
150e2000 2.0000 2.1000 0.6133 0.7077 0.6571
140e1475 2.0000 1.8030 0.6533 0.7716 0.7075
140e1475 2.0000 2.1000 0.6600 0.7279 0.6923
140e1475 1.6137 2.1000 0.6533 0.7259 0.6877

Bold indicates the value used for further processing.

To find the optimal regularisation factor and learning rate

Table 10 e ROC parameters comparison for UAV Image 2 in EM and SOM algorithm respectively, similar analyses were
with BIC.
carried out to obtain optimal value. It is observed that the
ROC parameter K-means EM SOM regularisation factor set to 0.003 and learning rate 0.99 gave
TP 96 109 114 the best result.
FP 23 20 36 In order to improve the overall performance, all the three
FN 54 41 52 clustering methods can be considered at a time, such that if a
pixel is classified as a tomato pixel in two, or more than two,
output images it will be considered as a tomato pixel. Other-
wise, it will be declared as a non-tomato pixel. The output of
this combined algorithm is shown in Fig. 16.
Table 11 e Evaluating performance measures based on
ROC parameters in Table 10.
Performance measure K-means EM SOM
7. Conclusion
Recall 0.6400 0.7267 0.6867
Precision 0.8067 0.8450 0.7600 Tomatoes grow in clusters and are generally partially hidden
F-Measure 0.7137 0.7814 0.7215
by their leaves and stalks. It becomes challenging to identify

Please cite this article in press as: Senthilnath, J., et al., Detection of tomatoes using spectral-spatial methods in remotely sensed RGB
images captured by UAV, Biosystems Engineering (2015), http://dx.doi.org/10.1016/j.biosystemseng.2015.12.003
16 b i o s y s t e m s e n g i n e e r i n g x x x ( 2 0 1 5 ) 1 e1 7

Fig. 16 e Output of the combined algorithm.

acquisition (rotation, translation and more importantly

Fig. 15 e Precision versus recall curve for UAV image 2. scaling effect) by maintaining constant resolution. The
thresholds for tomato area and geometric properties are
empirically determined. Adaptive methods can be used to
automatically determine thresholds. This work of detecting
tomatoes can be extended to counting of tomatoes to esti-
Table 12 e ROC parameters comparison for UAV Image 2 mate the yield.
without BIC.
ROC parameter K-means EM SOM
TP 89 91 103 Acknowledgement
FP 34 08 38
FN 61 59 47
We thank the managing editor, guest editors and reviewers for
their thorough reading and suggesting many important as-
pects. All the suggestions were very useful in revising our
tomatoes manually and estimate the yield due to poor visual
study.
interpretation (as in Fig. 10). Hence, it is necessary to automate
the tomato detection process thus to validate the process we
experimented with two UAV images of different resolutions. references
The task of tomato detection using UAV images was suc-
cessfully carried out by the procedure of pixel based spectral
information for clustering and shape information for spatial Alexandratos, N., & Bruinsma, J. (2012). World agriculture towards
segmentation. Applying BIC with unsupervised clustering 2030/2050: The 2012 revision. ESA Work. Pap, 3.
techniques on the images gave better results than the exact Berni, J., Zarco-Tejada, P. J., Sua  rez, L., & Fereres, E. (2009).
partitioning (2 clusters). The performance is measured using Thermal and narrowband multispectral remote sensing for
vegetation monitoring from an unmanned aerial vehicle.
ROC parameters: precision, recall and F-Measure. A compar-
Geoscience and Remote Sensing, IEEE Transactions on, 47(3),
ative study of the outputs for three different clustering 722e738.
methods was carried out and EM proved to be better than K-  rcoles, J. I., Ortega, J. F., Herna
Co  ndez, D., & Moreno, M. A. (2013).
means and SOM. Estimation of leaf area index in onion (Allium cepa L.) using an
This work demonstrates a novel method to detect to- unmanned aerial vehicle. Biosystems Engineering, 115(1), 31e42.
matoes using UAV images. Tomato detection was success- Doraiswamy, P. C., Moulin, S., Cook, P. W., & Stern, A. (2003). Crop
fully carried out on two representative images. The results yield assessment from remote sensing. Photogrammetric
Engineering & Remote Sensing, 69(6), 665e674.
are promising; however as an extension to this work is
Fawcett, T. (2006). ROC Graphs: Notes and practical considerations for
planned to include how UAV remote sensing can be used for researchers. Technical Report HPL-2003-4, HP Labs.
larger areas cultivated with tomatoes. To achieve this image Hannan, M. W., Burks, T. F., & Duke, M. B. (2009). A machine
stitching is required to overcome the challenges of image vision algorithm combining adaptive segmentation and shape
analysis for orange fruit detection. Agricultural Engineering
International: the CIGR E Journal, XI, 1e7.
Haralick, R. M., Sternberg, S. R., & Zhuang, X. (1987). Image
Table 13 e Evaluating performance measures based on analysis using mathematical morphology. Pattern Analysis and
ROC parameters in Table 12. Machine Intelligence, IEEE Transactions on, 4, 532e550.
Performance measure K-means EM SOM Herwitz, S. R., Johnson, L. F., Dunagan, S. E., Higgins, R. G.,
Sullivan, D. V., Zheng, J., et al. (2004). Imaging from an
Recall 0.5933 0.6066 0.6867
unmanned aerial vehicle: agricultural surveillance and
Precision 0.7236 0.9191 0.7305
decision support. Computers and Electronics in Agriculture, 44(1),
F-Measure 0.6520 0.7308 0.7079
49e61.

Please cite this article in press as: Senthilnath, J., et al., Detection of tomatoes using spectral-spatial methods in remotely sensed RGB
images captured by UAV, Biosystems Engineering (2015), http://dx.doi.org/10.1016/j.biosystemseng.2015.12.003
 b i o s y s t e m s e n g i n e e r i n g x x x ( 2 0 1 5 ) 1 e1 7
Hunt, E. R., Hively, W. D., Fujikawa, S. J., Linden, D. S.,
Daughtry, C. S., & McCarty, G. W. (2010). Acquisition of NIR-
green-blue digital photographs from unmanned aircraft for
crop monitoring. Remote Sensing, 2(1), 290e305.
Johnson, D. M. (2014). An assessment of pre-and within-season
remotely sensed variables for forecasting corn and soybean
yields in the United States. Remote Sensing of Environment, 141,
116e128.
Kohonen, T. (1990). The self-organizing map. Proceedings of the
IEEE, 78(9), 1464e1480.
Kooistra, L., Suomalainen, J., Franke, J., Bartholomeus, H.,
Mücher, S., & Becker, R. (2014, May). Monitoring agricultural
crops using a light-weight hyperspectral mapping system for
unmanned aerial vehicles. In EGU General Assembly Conference
Abstracts (Vol. 16, p. 2790).
Launay, M., & Guerif, M. (2005). Assimilating remote sensing data
into a crop model to improve predictive performance for
spatial applications. Agriculture, Ecosystems & Environment,
111(1), 321e339.
Li, H., Zhang, K., & Jiang, T. (2005). The regularized EM algorithm.
In Proceedings of the 20th National Conference on Artificial
Intelligence (pp. 807e812).
Lobell, D. B. (2013). The use of satellite data for crop yield gap
analysis. Field Crops Research, 143, 56e64.
Mo, X., Liu, S., Lin, Z., Xu, Y., Xiang, Y., & McVicar, T. R. (2005).
Prediction of crop yield, water consumption and water use
efficiency with a SVAT-crop growth model using remotely
sensed data on the North China Plain. Ecological Modelling,
183(2), 301e322.
Patel, H. N., Jain, R. K., & Joshi, M. V. (2011). Fruit detection using
improved multiple features based algorithm. International
Journal of Computer Applications, 13(2), 1e5.
Regunathan, M., & Lee, W. S. (2005, July). Citrus fruit identification
and size determination using machine vision and ultrasonic
sensors. In ASAE Annual International Meeting.
Saberioon, M. M., Amin, M. S. M., Anuar, A. R., Gholizadeh, A.,
Wayayok, A., & Khairunniza-Bejo, S. (2014). Assessment of rice
leaf chlorophyll content using visible bands at different
growth stages at both the leaf and canopy scale. International
Journal of Applied Earth Observation and Geoinformation, 32,
35e45.
Sagar, B. S. D. (2013). Mathematical morphology in geomorphology and
GISci. CRC Press.
Schwarz, G. (1978). Estimating the dimension of a model. The
Annals of Statistics, 6(2), 461e464.
Please cite this article in press as: Senthilnath, J., et al., Detection of tomatoes using spectral-spatial methods in remotely sensed RGB
images captured by UAV, Biosystems Engineering (2015), http://dx.doi.org/10.1016/j.biosystemseng.2015.12.003
17
Selim, S. Z., & Ismail, M. A. (1984). K-means-type algorithms: a
generalized convergence theorem and characterization of
local optimality. Pattern Analysis and Machine Intelligence, IEEE
Transactions on, 1, 81e87.
Seng, W. C., & Mirisaee, S. H. (2009). (2009, August). A new method
for fruits recognition system. In , ICEEI'09. International
Conference on: Vol. 1. Electrical engineering and informatics (pp.
130e134). IEEE. August.
Senthilnath, J., Omkar, S. N., Diwakar, P. G., Mani, V., Nitin, K., &
Shreyas, P. B. (2013). Crop stage classification of hyperspectral
data using unsupervised techniques. IEEE Journal of Selected Topics
in Applied Earth Observations and Remote Sensing, 6(2), 861e866.
Senthilnath, J., Shivesh, B., Omkar, S. N., Diwakar, P. G., &
Mani, V. (2012). An approach to multi-temporal MODIS image
analysis using image classification and segmentation.
Advances in Space Research, 50(9), 1274e1287.
Shi, H., & Xingguo, M. (2011). Interpreting spatial heterogeneity of
crop yield with a process model and remote sensing. Ecological
Modelling, 222(14), 2530e2541.


Stajnko, D., & Cmelik, Z. (2005). Modelling of apple fruit growth by
application of image analysis. Agriculturae Conspectus
Scientificus (ACS), 70(2), 59e64.
Stajnko, D., Lakota, M., & Hoc
evar, M. (2004). Estimation of
number and diameter of apple fruits in an orchard during the
growing season by thermal imaging. Computers and Electronics
in Agriculture, 42(1), 31e42.
Sugiura, R., Noguchi, N., & Ishii, K. (2005). Remote-sensing
technology for vegetation monitoring using an unmanned
helicopter. Biosystems Engineering, 90(4), 369e379.
Tarabalka, Y., Benediktsson, J. A., & Chanussot, J. (2009).
Spectralespatial classification of hyperspectral imagery based
on partitional clustering techniques. Geoscience and Remote
Sensing, IEEE Transactions on, 47(8), 2973e2987.
Xiang, H., & Tian, L. (2011). Development of a low-cost
agricultural remote sensing system based on an autonomous
unmanned aerial vehicle (UAV). Biosystems Engineering, 108(2),
174e190.
Yuping, M., Shili, W., Li, Z., Yingyu, H., Liwei, Z., Yanbo, H., et al.
(2008). Monitoring winter wheat growth in North China by
combining a crop model and remote sensing data. International
Journal of Applied Earth Observation and Geoinformation, 10(4),
426e437.
Zhou, R., Damerow, L., Sun, Y., & Blanke, M. M. (2012). Using
colour features of cv. ‘Gala’ apple fruits in an orchard in image
processing to predict yield. Precision Agriculture, 13(5), 568e580.