Literature Review: Food Recognition Methods
Suvarna Pansambal, Yamini Tawde, Chetali Surti, Chhaya Patil, Dhara Patel
Department of Computer Engineering,
University of Mumbai, Atharva College of Engineering,
Malad West, Mumbai, India
Abstractpresently a days sharing food
related photographs via web-based
networking media has turned into a pattern
and individuals are searching for the intrigued
food dishes and the eateries. So distinguishing
the food things, arranging them and breaking
down have been the subject of top to bottom
reviews for different applications identified
with food recognition, dietary habits and
dietary appraisal. This paper gives a wide
investigation of food recognition methods. It
likewise concentrates on the different feature
extraction strategies and in addition the
characterization systems. This paper
additionally gives a brief review of the
datasets accessible for the food.
Index TermsFeature extraction, food
recognition, multiple food
recognition,classification.
I. INTRODUCTION
Food related photographs have turned out to
be mainstream, because of informal
organizations, food suggestion and dietary
appraisal frameworks. Interpersonal interaction
destinations are these days overwhelmed with
Food related photographs. For example, new
pattern is sharing eating out encounters on
informal organizations. Truth be told,
individuals are progressively intrigued by finding
and sharing new cooking styles, and knowing
more about various parts of the food they
expend. Many takes a shot at food recognition
have been proposed as of late in view of various
visual representations [1] [2] [3] the vast
majority of them are constrained to a couple
food classes in controlled settings. Precise food
recognition from just visual data is still a
troublesome assignment. Rather than articles,
food things are deformable and with high intra-
class inconstancy, e.g. distinctive cooking styles
and seasonings will prompt to various
appearances of a similar food. In addition,
diverse foods share numerous fixings and
regularly contrasts between some food classes
are hard to distinguish. Additionally the
distinction in appearance and presentation of
same dish at different eateries add to the
multifaceted nature of perceiving the dish. So
the food recognition consist of detection of
food in the image and background
subtraction[10][11]. After that food features are
extracted, followed by classification[11] is
done.In this paper the various food recognition
methods are reviewd in section 2. After that the
section 3 gives the brief review of publically
available food datasets followed by futurework
and conclusion. II. LITERATURE SURVEY M. M.
Zhang et al.[10] utilized attribute-based
classification, they grouped plates of food to
the right cooking by the nation, utilizing the
ingredients as attributes for a plate of food.
Initially distinguished the ingredients, gave
every ingredient a likelihood name, and after
that utilized pairwise nearby components
among the ingredients to decide the food
classification, by computing the separation,
introduction, and different properties between
every combine of ingredients. To distinguishing
cooking utilizing ingredient, attribute-based
classification is utilized. In the first place layer
incorporates utilization of Earth Movers
Distance as the low-level element. EMD ends up
being hazardous for dishes where just a single
ingredient is available. Ground truth for test
tests at middle level is the pictures range
proportion for the particular ingredient
classifier, as separated from the pictures
attribute vector. In the last layer, anticipating
the cooking classification from the attributes,
this attribute vectors is used to prepare the last
classifier, with the food class ID as ground truth.
Thus the last order of the cooking styles is
separate from the crude pictures, as it uses the
middle of the road abnormal state attributes
layer to anticipate the outcomes. Y. Maruyama
et al.[14] comprises of work in view of
FoodLog framework, where the client takes
photographs of the foods and transfers them to
the framework, and the framework performs
picture handling to distinguish food pictures
and decide the food balance. The foodlog
framework permits the client to redress the
aftereffects of the framework., They proposed a
technique to make utilization of the recti-
fications made by the client by Naive Bayes
which is one of the Bayesian systems. It
additionally researches how to enhance the
precision by utilizing client criticisms. Initially,
the precision of the execution amongst SVM
and Naive Bayes is analyzed.Then, making
utilization of the clients adjustments as input,
the Bayesian system is redesigned to enhance
the execution. Ciocca et al [20] proposed a new
dataset for the evaluation of food recognition
algorithms that can be used in dietary
monitoring applications. Each image depicts a
real canteen tray with dishes and foods
arranged in different ways containing multiple
instances of food classes. The dataset contains
1,027 canteen trays for a total of 3,616 food
instances belonging to 73 food classes. They
designed an automatic tray analysis technique
that takes a tray image as input, finds the
regions of interest, and predicts for each region
the corresponding food class. First step is tray
analysis which takes the tray image as input.
The output is a list of regions of interest i.e. list
of recognized food which are processed by the
food class predictor. To predicate the food class
first global approach extracts the visual features
from the whole region of interest , second local
approach extracts the visual features from local
patches of the region of interest. Final approach
combines 2 the posterior probabilities
calculated by the global and local classifiers
with the sum and product operators. Second is
tray segmentation module used to detect the
regions of interest. It is composed of four main
steps. First the input RGB image is resized to an
height of 320 pixels which undergoes two
separate processing pipeline: a saturation-
based one, and a color texture one. The
segmentation detects the regions having similar
visual characteristics. The segmented image is
then processed in order to remove non relevant
regions .The final segmented image contains
with high probability the food regions and few
non relevant ones. In overall, the proposed
segmentation method outperforms the JSEG
one with an FMeasure of 0.724 against
0.323and shows that the proposed
segmentation strategy is able to effectively
locate the food regions. These images are then
fed to a feature extractor where several visual
descriptors are computed. They used two
classifiers as predictor: the k-Nearest Neighbour
(k-NN) and Support Vector Machines. The
metric for the evaluation of the error in the tray
analysis is the tray accuracy. This is defined as
the percentage of trays correctly analysed. A
tray is correctly analysed when all the foods
contained are correctly recognized. C. Rother et
al [13] develops the graph cut approach in three
regards. Initially, built up an all the more
effective, iterative variant of the advancement.
Second, the force of the iterative calculation is
utilized to disentangle significantly the client
connection required for a given nature of result.
Third, a hearty calculation for outskirt
tangling has been created to gauge all the
while the alpha-matte around a protest limit
and the shades of closer view pixels. Diagram
slice is utilized to accomplish strong division
even in disguise, when forefront and foundation
shading circulations are not very much isolated.
Grapph cut technique is flopped as far as client
connections, can happen in three cases: (i)
regions of low difference at the move from
closer view to foundation (ii) disguise, in which
the genuine frontal area and foundation
appropriations cover somewhat in shading
space (iii) foundation material inside the client
rectangle happens not to be enough spoken to
out of sight region. Graph cut is effective by
where the bouncing rectangle alone is an
adequate client association to empower closer
view extraction to be finished naturally by
GrabCut. Bettadapura et al [19] proposed a
technique that leverages the sensor data (i.e.
location) captured at the time photos are taken,
with additional information about the
restaurant available online, combined with
computer vision techniques to recognize the
food being consumed. To train the classifier
images from restaurants online menu databases
were used. For feature extraction from the
training and test data, a Harris-Laplace point
detector is used and for feature descriptor six
descriptors, two 2 color-based descriptors and 4
SIFT-based are used. Color-based descriptors
used are Color Moment Invariants and Hue
Histogram. SIFT based descriptors used are
CSIFT, Opponent SIFT , RGB-SIFT ,SIFT. For
image classification SMO-MKL multi-class SVM
classification framework is used. They evaluate
the performance of system in unconstrained,
real-world settings with food images taken in 10
restaurants across 5 different types of food
(American, Indian, Italian, Mexican and Thai).
First using the geo-location information, the
menu for each restaurant was automatically
retrieved. Next, they perform interest point
detection, feature extraction, codebook
building for Bag of Words representation,
kernel pre-computation and finally classification
using SMO-MKL .And the results are
summarized as the individual confusion
matrices. Using this approach they achieved
good classification accuracy with American,
India and Italian cuisines. However, for the
Mexican and Thai cuisines, the accuracy is
limited due low degree of visible variability
between food types belonging to the same
cuisines. The approach is based on the
hypothesis that knowing the location (through
geo-tags) helps us in reducing down the number
of food categories which in turn increases
recognition rates. To test this location
information was dropped and SMO-MKL
classifier was trained for all training images .
The setup provided accuracy of 15.67% for 600
images. But the overall average accuracy across
the 5 cuisines was 63.33%. Thus it was seen that
the average performance increased by 47.66%
when location prior was included. It was
concluded that knowing the location of eating
activities helps in food recognition, and that it is
better to build several smaller
restaurant/cuisine specific classifiers rather
than one all-category food classifier. Bejibom et
al[18] concentrates on restaurant scenario and
introduce a technique that addresses this
problem specifically called Menu-Match. In
their work, database contains atomic items as
they are served at specific restaurants. Thus a
meal can be directly classified as belonging to a
particular restaurant for example, the
cheeseburger at Joes at Solo Grill in Toronto,
and then accurate nutritional statistics can be
read from the database. They propose an
application in which location information
available on mobile devices (e.g., GPS) limits the
search for a particular image to a small set of
nearby restaurants, which greatly simplifies
recognition and offers accurate mapping of
images to nutritional information. The image
recognition framework is based on the bag of
visual words approach In the first step, five
types of base features are extracted from the
images: color , histogram of oriented gradients
(HOG) ,scale-invariant feature transforms (SIFT)
, local binary patterns (LBP) , and filter
responses from the MR8 filter bank . These base
features are encoded with locality constrained
linear encoding (LLC) , using a dictionary with
1024 words learned via k-means clustering. The
encoded base features are then pooled using
max-pooling in a rotation-invariant pooling
scheme. The implementation includes Semi-
Automated Food Item Identification using
oneversus-rest linear Support Vector
Machine(SVM) and Fully Automated Estimation
Of Food Statistics based on regression. The
results show that fusing several features that is
the joint feature representation significantly
outperforms the performance of any single
feature type. The rotationally invariant pooling
method increased the mean average precision
for the joint feature from 38.3% to 51.2%
compared to the traditional spatial pyramid
pooling. They collected a dataset of actual meal
images from three local restaurants. Which
contains accurate nutritional information and
realistic food images called Menu-Match
dataset. It contains a total of 646 images, with
1386 tagged food items across 41 categories.
Also calorie counts for all food items thus
containing nutritional metadata 3 . The
advantage of this approach it is not needed to
identify every item, considering the meal as a
whole entity, ingredients and preparation
details are encoded into the database, volume
estimation is no longer needed. The technique
has some limitations such when dish do not
come in discrete serving sizes, such as salad
bars. To provide results for home-cooked food,
it requires to build a custom menu .In case of
Takeout food the location information where
the food is consumed is not relevant, this
requires a user to specify the restaurant
manually. Martin et al [17] proposed a
committee based recognition system that
chooses the optimal features out of the existing
plethora of available ones (e.g., color, texture,
etc.). Each committee member is an Extreme
Learning Machine trained to categorize food
plates on the basis of a single feature type.
Single member classifications are then
considered by a structural Support Vector
Machine to produce the final ranking of
expected matches. This is accomplished by
filtering out the irrelevant features/classifiers,
thus considering only the relevant ones. The
SCORE approach uses as many different
features as possible but exploits only a subset
of those to obtain optimal ranking performance.
To demonstrate the benefits of the proposed
SCORE approach extensive evaluations on three
benchmark datasets has been done. Yanai et
al[16] examined the effectiveness of deep
convolutional neural network (DCNN) for food
photo recognition task. In the DCNN approach,
an input data of DCNN is a resized image, and
the output is a class-label probability. That is,
DCNN comprises all the object recognition
phases such as local feature extraction, feature
coding, and learning. The first technique is by
means of a pre-trained DCNN with a large scale
dataset such as the ILSVRC dataset as a feature
vector extractor for a small-scale data. The
second technique is fine-tuning of the pre-
trained DCNN. Fine-tuning is tuning the
parameters pre-trained with a large-scale data
using an additional small-scale data. With fine-
tuning, the DCNN originally for a large-scale
data is altered and adapted to other tasks. In
this case, a DCNN is used as both a feature
extractor and a classifier. In the experiments,
they have achieved the best classification
accuracy, 78.77% and 67.57%, for the UEC-
FOOD100/256 dataset, which proved that the
fine-tuning of the DCNN pre-trained with a large
number of food-related categories (DCNN-
FOOD) can boost the classification performance
the most greatly. In addition, they applied the
food classifier employing the best combination
of the DCNN techniques to Twitter photo data.
He et al[15] paper presents an ingredient-based
food recognition method. DietCam, which
specifically addresses the variation of food
appearances. DietCam consists two major
components, ingredient detection and food
classification. Food ingredients are identified
through a combination of a deformable part-
based model and a texture verification model,
and apply a multi-view multi-kernel SVM to
classify the food ingredients. Their first
innovation is the improvement of the current
part-based object recognition model toward
textures oriented and location-flexible to detect
food ingredients. They modified the detector in
three ways for the purpose of food ingredient
detection. The ingredient detector tries to find
Fig. 1. EXAMPLE OF THE UNIMIB2016 FOOD
DATASET[22] food ingredients on a single scale
in order to retain relative ingredient scales.
After that, the scale invariance is achieved in a
multi-scale support vector machine (SVM)
during food classification. Their second
contribution is development of a multi-view
multi-kernel SVM to classify various
combinations of food ingredients under
occlusion. In the experiment, DietCam presents
reliability and outperformance in recognition of
food with complex ingredients on a database of
55 food types with 15262 food images. The
recognition precision of DietCam was around
90% for general food items, and 85% for food
items in DCs 5 and 6. III. DATASETS FOR FOOD
RECOGNITION A. UNIMIB2015 DATASET The
dataset contains 2,000 images of 15 classes of
foods placed on trays. The images were
acquired in a real canteen location, and are
paired with the corresponding leftover images
acquired after the meals. It is the first dataset
explicitly designed for both food recognition
and leftover estimation. UNIMIB2015 dataset is
the only dataset presented to date specifically
designed for canteen environments. The
annotation is done using polygonal areas and
the dataset is publicly available[20] [22] . B. THE
UNIMIB2016 FOOD DATASET The dataset
contains 1,027 tray images, 73 food categories,
and a total of 3,616 food instances and has
been collected in a real canteen environment.
Each image depicts different foods on a tray,
and some foods (e.g. fruit, bread and dessert)
are placed on the placemats rather than on
plates. The acquisition of the images has been
performed in a semi-controlled environment so
the images present visual distortions as well as
illumination changes due to shadows making
this dataset challenging requiring both the
segmentation of the trays for food localization,
and a robust way to deal with multiple foods
containing dishes. The UNIMIB2016 dataset is
characterized by images that contain multiple
foods with accurate segmentation. The dataset
is annotated using improved version of our
Image Annotation Tool that will help research
work on methods for food segmentation, as
well as food quantity estimation [20][22]. 4 C.
FOOD 50 AND FOOD 85 DATASET Food 50
contains 50 food categories (mostly Japanese
food) and a total of 5000 food images, gathered
from the Web. It contains single food image .
Using MKL-based feature fusion, recognition
accuracy of 61.34% was obtained. This dataset
was enlarged to 85 food categories containing
8500 food images called Food 85 dataset.The
annotation of both these datasets is done using
labels. These two datasets are proprietary [23].
D. TADA DATASET The TADA dataset contains
images of real foods (256 images) as well as
food replica (50 images) and are acquired in lab
controlled settings. The images can have
multiple food present making the dataset more
challenging as it will require the segmentation
of each food in the image. It is a proprietary
dataset [4]. E. UEC FOOD 256 The dataset UEC
FOOD 256 contains 256-kind food photos both
single and multi food type acquired in an
unconstrained environment . It contains 31,397
food images and each has a bounding box
indicating the location of the food item in the
photo. For the recognition, SVM classifiers with
color histogram and SURF features are used,
achieving a classification rate of 74.4% for the
top 5 category candidates when the ground-
truth bounding boxes are given. Most of the
food categories in this dataset are popular
foods in Japan and other countries. The
annotation is done using Bounding Box and the
dataset is publicly available [5]. F. UNICT889
DATASET UNICT889 is the dataset with single
food type but has most food categories. It
contains 889 classes on a total of 3,583 images
acquired in an unconstrained environment . The
purpose of the dataset was near duplicate food
retrieval. Different features are tested and the
best results for near duplicate retrieval was
achieved by color Bag-of-Textons with a mean
average precision of 67.5%. The annotation is
done using labels and the dataset is publicly
available [6]. G. DIABETES DATASET
Anthimopoulos et al. presented a dataset of
4,868 single food images organized into 11
classes acquired in an unconstrained
environment .The dataset can be used for food
recognition system based on the Bag-of-
Features model. The system is intended to help
diabetic patients in controlling their
carbohydrates daily consumption. Different
visual features and classification strategies were
tested and the best combination gave a
classification accuracy of slightly less that 78%
using a 10,000 words dictionary. The annotation
is done using labels and the dataset is publicly
available[7] . H. FOOD 101 Food -101 is the
largest dataset currently available acquired in
an unconstrained environment. It contains
101,000 single food images divided into 101
food categories. Random forest are used to
mine discriminant super pixel grouped parts in
the food images. These parts are then classified
with SVM achieving an average accuracy of
50.76% on the 101 classes. The annotation is
done using labels and the dataset is publicly
available [8] . I. PITTSBURGH FAST FOOD IMAGE
DATASET Pittsburgh fast-food image data sets
contain 61 categories of food items. These 61
categories are a subset of the 101 categories.
There are 3 instances of each food item, each
taken on different days. Classification accuracy
on this 61 categories are -Bag of SIFT features-
9.2%, Color histogram- 11.3% [9]. IV.
CONCLUSION The paper has given the vast
review of the recent developments in food
recognition methods. Researchers have worked
on many different problems like dietary
monitoring, food photos are taken, with
additional information about the restaurant,
restaurant Menu-Matching etc.it is observerd
that the quality of the photo does affect the
recognition of food. It seems that, keeping in
mind todays scenario of eating habits and
searching for good food by using the internet is
a trend and many researchers are working on
food recognition domain. recognizing a single
food item is easy as compare to recognising the
multiple food items in a dish. Results are good
for single item recognition as compare to
multiple item recognition. The photos taken
with the location information helps the user to
find the place quickly. V. FUTURE WORK
Researchers have worked on so many
approaches for the food recognition. . But
practically they are very far. Some further
research work is outlined as follows. 1]
Researchers an work on to catogorise the food
countrywise. 2] To get the high quality images
and make the food recognition easy try to keep
the uniformity in picture taking devices. It will
help to make a centralized system for all the
restaurant in a perticular continent.
REFERENCES [1] Z. Zong, D. T. Nguyen, P.
Ogunbona, and W. Li, On the combination of
local texture and global structure for food
classification, in International Symposium on
Multimedia, 2010, pp. 204211. [2] F. Kong and J.
Tan, Dietcam: Regular shape food recognition
with a camera phone, in International
Conference on Body Sensor Networks, 2011, pp.
127132. [3] S. Yang, M. Chen, D. Pomerleau,
and R. Sukthankar, Food recognition using
statistics of pairwise local features, in
International Conference on Computer Vision
and Pattern Recognition, 2010, pp. 22492256.
[4] Wang, Xin, Devinder Kumar, Nicolas Thome,
Matthieu Cord, and Frederic Precioso, Recipe
recognition with large multimodal food
dataset, In Multimedia & Expo Workshops
(ICMEW), 2015 IEEE International Conference
on, pp. 1-6. IEEE, 2015. 5 [5] Hassannejad,
Hamid, Guido Matrella, Paolo Ciampolini, Ilaria
De Munari, Monica Mordonini, and Stefano
Cagnoni, Food Image Recognition Using Very
Deep Convolutional Networks. Proceedings of
the 2nd International Workshop on Multimedia
Assisted Dietary Management. ACM, 2016. [6]
Farinella, Giovanni Maria, Dario Allegra, and
Filippo Stanco, A benchmark dataset to study
the representation of food images, European
Conference on Computer Vision. Springer
International Publishing, 2014. [7]
Anthimopoulos, Marios M., Lauro Gianola, Luca
Scarnato, Peter Diem, and Stavroula G.
Mougiakakou, A food recognition system for
diabetic patients based on an optimized bag-of-
features model. IEEE journal of biomedical and
health informatics 18.4 (2014): 1261-1271. [8]
Bossard, Lukas, Matthieu Guillaumin, and Luc
Van Gool, Food- 101mining discriminative
components with random forests. European
Conference on Computer Vision. Springer
International Publishing, 2014. [9] Chen,
Mei,Dhingra, K., Wu, W., Yang, L., Sukthankar,
R., & Yang, J.l. PFID: Pittsburgh fast-food image
dataset. 2009 16th IEEE International P. Napoletano, and R. Schettini, Food
Conference on Image Processing (ICIP). IEEE, recognition: a new dataset, experiments and
2009 [10] Swati, Shirke, and Suvarna results, IEEE journal of biomedical and health
Pansambal, Enhancement of IRIS recognition informatics (2016). [21]
using Gabor over FFBPANN, Computing for http://www.ivl.disco.unimib.it/activities/food-
Sustainable Global Development (INDIACom), recognition/ [22] Senyuva, Hamide Z., and John
2015 2nd International Conference on. IEEE, Gilbert, Immunoaffinity column cleanup
2015. [11] Shirke Suvarna, Pawar S, Shah K techniques in food analysis: A review, Journal
(2014), Literature review: model free human of Chromatography B 878.2 (2010): 115-132
gait recognition, In: 2014 fourth international
conference on communication systems and
network technologies (CSNT). IEEE, pp 891895
[12] Zhang, Mabel Mengzi, Identifying the
cuisine of a plate of food, Univ. of California at
San Diego, La Jolla, CA, USA, Tech. Rep. CSE 190
(2011). [13] C. Rother, V. Kolmogorov, and A.
Blake, GrabCut: Interactive foreground
extraction using iterated graph cuts, ACM
Transactions on Graphics, vol. 23, no. 3, pp.
309314, 2004. [14] Y. Maruyama, G. C. de Silva,
T. Yamasaki, and K. Aizawa, Personalization of
food image analysis, in International
Conference on Virtual Systems and Multimedia,
2010, pp. 7578. [15] He, Hongsheng, Fanyu
Kong, and Jindong Tan, DietCam: Multiview
Food Recognition Using a MultiKernel SVM.
IEEE journal of biomedical and health
informatics 20.3 (2016): 848-855. [16] Yanai,
Keiji, and Yoshiyuki Kawano, Food image
recognition using deep convolutional network
with pre-training and fine-tuning, Multimedia
& Expo Workshops (ICMEW), 2015 IEEE
International Conference on. IEEE, 2015. [17]
Martinel, Niki,Claudio Piciarelli, Christian
Micheloni,and Gian Luca Foresti, A Structured
Committee for Food Recognition, In
Proceedings of the IEEE International
Conference on Computer Vision Workshops, pp.
92-100. 2015. [18] Oscar Beijbom, Neel Joshi,
Dan Morris, Scott Sapona, Siddharth Khullar,
Menu-Match: Restaurant-Specific Food Logging
from Images, 2015 IEEE Winter Conference on
Applications of Computer Vision. [19] Vinay
Bettadapura, Edison Thomaz, Aman Parnami,
Gregory D. Abowd, Leveraging Context to
Support Automated Food Recognition in
Restaurants, 2015 IEEE Winter Conference on
Applications of Computer Vision. [20] Ciocca, G.,