Food Quality and Preference 26 (2012) 267–277

Contents lists available at SciVerse ScienceDirect

Food Quality and Preference

journal homepage: www.elsevier.com/locate/foodqual

Rapid descriptive sensory methods – Comparison of Free Multiple Sorting,

Partial Napping, Napping, Flash Profiling and conventional profiling
Christian Dehlholm a, Per B. Brockhoff b, Lene Meinert c, Margit D. Aaslyng c, Wender L.P. Bredie a,⇑
a
Department of Food Science, Faculty of Life Sciences, University of Copenhagen, Rolighedsvej 30, DK-1958 Frederiksberg C, Denmark
b
Informatics and Mathematical Modelling, Technical University of Denmark, Richard Petersens Plads, Build. 321, DK-2800 Kongens Lyngby, Denmark
c
Danish Meat Research Institute, Maglegårdsvej 2, DK-4000 Roskilde, Denmark

a r t i c l e i n f o a b s t r a c t

Article history: Two new rapid descriptive sensory evaluation methods are introduced to the field of food sensory eval-
Received 24 August 2011 uation. The first method, free multiple sorting, allows subjects to perform ad libitum free sortings, until
Received in revised form 22 February 2012 they feel that no more relevant dissimilarities among products remain. The second method is a modal
Accepted 23 February 2012
restriction of Napping to specific sensory modalities, directing sensation and still allowing a holistic
Available online 3 March 2012
approach to products. The new methods are compared to Flash Profiling, Napping and conventional
descriptive sensory profiling. Evaluations are performed by several panels of expert assessors originating
Keywords:
from two distinct research environments. Evaluations are performed on the same nine pâté products and
Sensory analysis
Descriptive profiling
within the same period of time. Results are analysed configurationally (graphically) as well as with RV
Napping coefficients, semantically and practically. Parametric bootstrapped confidence ellipses are applied for
Free sorting the graphical validation and comparisons. This allows similar comparisons and is applicable to single-
Flash profile block evaluation designs such as Napping. The partial Napping allows repetitions on multiple sensory
Meat quality modalities, e.g. appearance, taste and mouthfeel, and shows the average of these repetitions to be signif-
icantly more closely related to the conventional profile than other methods. Semantic comparison shows
large differences, with closest relations found between the two conventional profiles. This suggests that
semantic results from an assessor in an evaluation type with no training sessions are dependent on the
assessors’ personal semantic skills. Comparisons of the methods’ practical differences highlight the time
advantage of the rapid approaches and their individual differences in the number of attributes generated.
Crown Copyright Ó 2012 Published by Elsevier Ltd. All rights reserved.

1. Introduction sions by Rosenberg et al. (1968). In this sorting procedure, the

Since the introduction of the flavour profile method (Cairncross
& Sjôstrom, 1950) a number of different sensory profiling methods
have been developed. Some methods focus on the training of sub-
jects on a sensory vocabulary, whereas other methods let subject
more freely decide how to indicate differences between samples.
Beside conventional descriptive profiling techniques such as
Quantitative Descriptive Analysis and the Spectrum method (Ci-
ville & Lyon, 1996; Stone, Sidel, Oliver, Woolsey, & Singleto,
1974), faster alternatives have gained some popularity for instance
Projective Mapping (Risvik, McEwan, Colwill, Rogers, & Lyon, 1994)
and its subsequent variants e.g. Napping (Pagès, 2003, 2005), the
Flash Profile (FP) (Dairou & Sieffermann, 2002) and Sorting (Rosen-
berg, Nelson, & Vivekana, 1968).
Sorting was introduced to the field of food sensory evaluation in
a cheese-based study by Lawless, Sheng, and Knoops (1995) with
references to earlier free sorting of non-food personality impres-
⇑ Corresponding author. Tel.: +45 35 33 32 42; fax: +45 35 33 35 09.
E-mail address: wb@life.ku.dk (W.L.P. Bredie).
assessors were asked to sort samples once into groups in a way
that made sense to the individual assessor only. However, a sorting
procedure known as the Free Sorting (Steinberg, 1967) had previ-
ously been introduced, with an important difference in the meth-
odological approach. While Steinberg asked subjects to perform
multiple sortings, provided it made sense to the subject, Rosenberg
et al. only asked for one sorting per subject. When performing mul-
tiple sortings, the assessor was allowed to make additional sortings
of the same sample set, as long as it made sense to the individual.
After the evaluation, data analysis was performed on the merged
data. Despite including only two sortings per subject from the mul-
tiple sorting in the analysis, Rosenberg later concluded that multi-
ple sorting would often be superior when compared with single
sorting (Rosenberg & Kim, 1975). In the present study, the Free
Multiple Sorting (FMS) approach including all individual sortings
is introduced to the field of food science, representing the branch
of sorting methodologies. The free single sorting evaluation has
been evaluated several times with various results (Faye et al.,
2004; Lelievre, Chollet, Abdi, & Valentin, 2008), whereas the free
multiple sorting as applied here has not.

0950-3293/$ - see front matter Crown Copyright Ó 2012 Published by Elsevier Ltd. All rights reserved.
http://dx.doi.org/10.1016/j.foodqual.2012.02.012
268 C. Dehlholm et al. / Food Quality and Preference 26 (2012) 267–277
When Free-choice profiling was introduced by Williams and
Langron (1984), development and training of the ideosyncratic
vocabulary was the method’s cardinal point. The use or not of
the free-choice vocabulary made up a substantial methodological
difference compared to, how the ‘conventional descriptive profile’
(CP) was and still is, applied in laboratories. Flash Profiling was
introduced later by Dairou and Sieffermann (2002) as a rapid ap-
proach based on Free-choice profiling (Williams & Langron,
1984), leaving out the training sessions and implementing rank
ordering of samples. Instead, assessors received an introduction
to the samples and were told to generate their own vocabulary
free-of-choice, based on their own sensory perception and to cover
the sensory variations in the samples. After generating relevant
attributes, they were allowed to see other assessors’ vocabularies
and to add or substitute attributes in their own list.
Dairou and Sieffermann (2002) compared a single sorting with a
conventional descriptive profile (CP) and found configurational
relations between the methods and, to a lesser extent, between
vocabularies but suggested a study with more similar products.
This was done on two product sets by Delarue and Sieffermann
(2004), who drew similar conclusions. Later, Blancher et al.
(2007) compared a CP with the FP and FMS and concluded that
the FP was configurationally closer to the CP than FMS. However,
the sorting was a single sorting.
Projective Mapping was introduced to the field of food sensory
evaluation by Risvik et al. (1994). In Projective Mapping, assessors
were introduced to the method, but had no further training. They
were supplied with a A4 paper sheet and the sample set and were
instructed to place samples perceived as similar close to each other
and samples perceived to be more different further apart. Risvik
et al., (1994) introduced Projective Mapping, recording the sample
space using A4 paper sheets with unstructured line scales with
trained assessors, but performed it later on A3 paper sheets with
structured line scales on consumers, coupling the evaluation to a
CP on trained assessors (Risvik, McEwan, & Rodbotten, 1997). King,
Cliff, and Hall (1998) experimented with Projective Mapping using
60 cm  60 cm paper sheets (also used later by Kennedy and Hey-
mann (2009) and Nestrud and Lawless (2010)) on untrained asses-
sors and using both unstructured and structured hedonic line
scales.
Similar to Projective Mapping, Pagès later introduced Napping.
The basic ideas of Napping and Projective Mapping were similar
as they were both having similar assessor instructions and that
the raw data were based on Euclidean product distances. Although,
important methodological differences were found in the Napping
concept, e.g. the framework of data collection had to be rectangu-
lar, data were not to be scaled and data analysis had to be done
using Multiple Factor Analysis (Escofier & Pagès, 1994; Pagès, Ass-
elin, Morlat, & Robichet, 1987). The framework consisted of a
60 cm  40 cm paper sheet, which is approximately A2 size.
Although stated otherwise in other publications, Napping and Pro-
jective Mapping are not the same, and Napping should be seen as a
special restricted and defined case of Projective Mapping. Napping
was normally coupled with the Ultra Flash Profiling (UFP) (Pagès,
2003) to collect subjects’ semantic responses, so called in order
to draw conceptual parallels to the free choice and rapid features
of vocabulary used in the FP. Perrin et al. (2008) showed that the
UFP is a good descriptive supplement to Napping, though it pro-
vides less descriptive information than that derived from a CP. As
Napping is in continuous development, one of the latest variations
is the Sorted Napping (Pagès, Cadoret, & Lê, 2010) in which Nap-
ping is combined with a sorting procedure instead of the UFP. This
study will examine the original Napping attached to the UFP. Pagès
mentioned in his original paper that it might be a good idea to per-
form a Napping restricted to one sensory modality. Even though
this Napping on modalities has been implemented in some sensory
laboratories, its performance has not previously been documented.
Hence, we introduced this approach as the ‘Partial Napping’ (PN) as
opposite to the non-restricted ‘Global Napping’ (GN). Here, we per-
form PNs on appearance, taste and mouthfeel and a GN.
The aim of this paper is to compare different rapid descriptive
methods with each other and with a consensus profiling method
based on the Quantitative Descriptive Analysis (Stone et al.,
1974). Different features are compared: their configurational out-
put, their semantic output, their reliability and their practical dif-
ferences. Furthermore, applicability is studied among several
sensory panels. A new approach that applies parametric bootstrap-
ping allows ellipses of confidence to be displayed for single block
evaluation designs, e.g. Napping data.
2. Materials and methods
2.1. Samples
Nine different types of commercially available liver pâté were
bought in local Danish supermarkets. Products and brands were se-
lected in cooperation with one manufacturer so that major varie-
ties within the Danish product range were covered. Samples
were different variations of liver pâté produced by three different
manufacturers and were all from a similar production date. Pack-
aging sizes varied between 200 and 500 g. Table 1 shows the prod-
uct information available on the package. The pâtés span the
overall branded differences in texture (coarsely chopped or not),
fat content (5–25 g/100 g) and bacon content (with or without).
The product numbers will be used throughout for identification
of the pâtés.
For sensory evaluation, samples were prepared by cutting the
surfaces and crust so that each sample would be of similar size
(5 cm  3 cm  1 cm). The samples were stored at 2 °C and were
served temperate to 15 °C in transparent PET containers, blind-la-
belled and with three-digit codes.
2.2. Methodology
2.2.1. Experiment overview
This study compared a number of different descriptive sensory
evaluation methods both within and between sensory panels. The
experimental set-up consisted of two different professional sen-
sory panels, each of which performed a CP, FMS, a GN followed
by a UFP and three PNs followed by UFPs. Furthermore, Panel A
performed an FP. Panel B did not consist of enough assessors to
perform an independent FP. Fig. 1 shows the overall experimental
set-up. All evaluations were performed on the same products and
within the same period of time. Evaluations took place in a stand-
ardised sensory environment (ISO 8589, 2007) and followed good
sensory practice (Lawless & Heymann, 2010). Randomised com-
plete block designs were assigned to the CPs and the FP, while ran-
domised complete designs were assigned to the FMSs and the
nappings.
2.2.2. Panels
Assessors from two professional sensory panels participated in
this study. Panel A consisted of expert assessors, while Panel B con-
sisted of specialised expert assessors (ISO 5492, 2008). Panel A was
a panel attached to a university environment (University of Copen-
hagen), and assessors were mainly university students. It was a
broad panel in the sense that the assessors were used to perform-
ing sensory analysis on a diverse product range. Panel B was a pa-
nel at the Danish Meat Research Institute, which supports the meat
industry. For this reason, their daily work on sensory analysis was
focused on meat products, and they possessed prior specific
 C. Dehlholm et al. / Food Quality and Preference 26 (2012) 267–277 269

Table 1 Assessors performing one type of methodology could not auto-

Product information. matically perform other types of methodologies. However, the pa-
Product Brand information Pork Nutritional information – nel pools of employed expert assessors were limited. For this
liver protein/carbohydrate/fat/NaCl reason, the assessors performing the CP (nine for Panel A and eight
content (approx. g per 100 g) for Panel B) were allowed to perform the FMS as a part of their CP
(%)
training. Similarly, assessors performing the restricted PNs had in a
P1 Brand 1, liver pâté, 30 10/10/15/2 previous session performed the more general GN.
coarsely chopped,
less fat
The fact that the panel pools were limited was the reason that
P2 Brand 1, liver pâté, 30 10/10/15/2 Panel B did not perform an FP.
added bacon, less fat
P3 Brand 2, liver pâté, 35 10/5/20/1.5 2.2.3. Conventional Descriptive Profile
coarsely chopped
P4 Brand 2, liver pâté, 35 10/5/20/-
The CP applied in this study was a consensus descriptive sen-
added bacon sory profile. It consisted of a period of training before the collection
P5 Brand 3, liver pâté, 35 10/15/5/- of repeated measurements. In the analysis, the individual panels
less fat proceeded as normal independent panels and also developed their
P6 Brand 4, liver pâté, 35 10/5/20/1.5
vocabularies independently from one another.
coarsely chopped
P7 Brand 4, liver pâté, 30 10/5/25/1.5 In each panel, the assessors generated a number of attributes
coarsely chopped, which they agreed would cover the sensory variations in the nine
added bacon samples. The panel was subsequently trained to score the intensi-
P8 Brand 4, liver pâté, 35 10/5/15/1 ties of the attributes on 15 cm unstructured line scales anchored
coarsely chopped,
less fat
from ‘absent’ to ‘extreme’. In their training sessions, Panel A used
P9 Brand 5, liver pâté 35 10/5/25/1.5 references that covered all developed attributes, while Panel B
did not use references as they were familiar with the product type.
The vocabulary development and training for Panel A lasted 6 h
knowledge of the products used in this study. The research insti- and was divided into three sessions, while for Panel B it lasted
tute was not located in a university environment and the assessors 4½ h and was divided into two sessions. In the following sessions,
were associated with the labour market. On average, they were the products were evaluated in sensory booths using the balanced
older than those in Panel A. sample presentation design. Each product was evaluated four
Nine Panel A assessors performed the CP. Eight Panel B asses- times by each assessor. Data were collected using the Fizz software
sors performed the CP, but one assessor was only able to carry from Biosystemes. The entire profiling period, including training,
out two out of four repetitions, so his data were not included in was performed over 5 days for Panel A and 4 days for Panel B.
the analysis. Nine Panel A assessors performed the FP. Each Nap-
ping procedure, followed by the UFP, was performed by nine Panel 2.2.4. Flash profiling
A assessors and nine Panel B assessors. FMS was performed by 13 The FP was performed according to Dairou and Sieffermann
Panel A assessors and 10 Panel B assessors. (2002).

Fig. 1. Overview of experimental set-up. Expert assessors from Panels A and B performed the listed methodologies on the same products within the same period of time.
Subpanel sizes are shown for individual methodologies. Subpanels with the same subscript notation performed the same methodology; assessors performing the
conventional profile had performed a free multiple sorting as a part of their training and assessors performing the partial nappings had performed a global napping as a
previous session. Comparisons of methodologies were done both within and between panels.
270 C. Dehlholm et al. / Food Quality and Preference 26 (2012) 267–277
Assessors received an introduction to the evaluation. After a
short sample presentation, assessors generated their own free-
choice vocabularies covering the sensory variations in the nine
samples. Assessors were allowed to see other assessors’ generated
vocabularies to add or substitute attributes in their own list. The
vocabulary development lasted 1 h and was divided into two
sessions.
For each attribute, samples were ranked according to their
intensity on an ordinal scale anchored from ‘lower’ to ‘higher’.
The assessors were allowed to apply the same rank to two or more
samples if no difference was perceived. Each product was evalu-
ated three times by each assessor in sensory booths, and data were
collected using the Fizz software from Biosystemes. The entire pro-
filing period, including vocabulary development, lasted 4 h and
was divided into 2 days. Only Panel A performed the FP.
2.2.5. Napping
Napping was carried out according to Pagès’ (2003) original
idea but was separated into two different approaches, GN and
PN. Assessors were introduced to the method, but had no further
training. They were provided with a 60 cm  40 cm paper sheet
and were instructed to place samples perceived as similar close
to each other and samples perceived to be more different further
apart. While the assessors gradually tasted the samples, they took
notes on their sensory impressions. When all the samples had been
placed on the paper, the assessors substituted them with an X and
the samples’ codes. The UFP consisted of the noted sample charac-
teristics and impressions written next to each X.
When performing a GN, there were no restrictions in the asses-
sor’s separation process, and the assessor was free to give any form
of semantic expression. In PN, the word ‘partial’ referred to a
restriction in sensory modalities. In this study, assessors perform-
ing Nappings started evaluating samples with a GN where they had
no prior experience of evaluating the products. In subsequent ses-
sions, they continued to evaluate samples with three PNs, one re-
stricted to separating samples on appearance, one on taste and
the last one on mouthfeel. All four Nappings were completed both
by Panel A and Panel B in 2 h during 1 day.
In the following sections of this paper, the GN is treated as one
methodology, and the three PNs as one methodology, which then
consists of three different kinds of measurements on the same
subjects.
Each sample’s data consisted of the X’s coordinates on the paper
sheet plus the characteristics that count as attributes.
2.2.6. Free Multiple Sorting
The FMS is based on the original idea of multiple sorting by
Steinberg (1967) adapted to food samples.
The assessors were asked to sort the samples in front of them
into groups in a way that made sense to the individual assessor
only. After one sorting of all samples, they were pooled again
and the assessor was allowed to make additional sortings of the
same sample set. In each sorting, they were allowed to generate
between two and eight groups containing one to eight samples.
They were instructed to continue in this way until they felt that
all sorting possibilities had been covered. They did not need to
be able to explain why they sorted as they did, provided it made
sense to the assessors themselves. If they had a clear idea of why
they sorted as they did, they were encouraged to write down the
reason. For each sorting task, data collected for a sample consisted
of the description of the group that the sample belonged to. There
was no preliminary training, and the FMS was carried out both by
Panel A and Panel B during a 1 h session.
2.3. Data analysis
To maintain a similar and comparable approach, results ob-
tained from the different methodologies were analysed in similar
ways. The CP raw data set consisted of a matrix with one variable
per attribute, each variable being numeric and between 0 and 15.
Panel A had developed a vocabulary consisting of 29 attributes
and Panel B one consisting of 20 attributes. CP data sets were ana-
lysed with Principal Component Analysis (PCA) using averaged
data. It was chosen to work with unscaled data in this study.
Other data sets were analysed with Multiple Factor Analysis
(MFA). MFA combines PCA and Multiple Correspondence Analysis
(MCA) according to the nature of the variables (Escofier & Pagès,
1994) and allows grouping of variables.
The FP data consisted of a number of variables, one for each
attribute belonging to a single assessor. The total number of attri-
butes was 189 with each assessor having between 16 and 20 attri-
butes. As the evaluation was performed as rank ordering, each
variable consisted of the rank number, 1–9, given to that sample
(same ranks of samples were allowed). The FP data set was aver-
aged over individual repetitions and grouped by assessor for the
MFA.
The FMS data set consisted of one variable for each sorting per-
formed. In this way, the total number of variables was 104 for Pa-
nel A and 96 for Panel B. The average of sortings per assessor were
8 for Panel A and 10 for Panel B, ranging between 3 and 12 per
assessor for Panel A and 6 and 14 per assessor for Panel B. Each var-
iable were categorical and consisted of the name of the group that
each sample belonged to. The variables were grouped for the MFA
with all sortings performed per assessor as constituting one group.
The Napping data consisted of two variables per subject, one
being the X coordinate and one being the Y coordinate of each sam-
ple’s placement on the evaluation sheet. These two numeric vari-
ables were grouped for the MFA. Data were not scaled. All
Napping evaluations were performed with nine assessors resulting
in a data set of 18 variables. These variables were in each data set
followed by a contingency table with the UFP results. In the contin-
gency tables, each variable name were a sample attribute given by
any subject and the value for each sample was the frequency of
that attribute applied for each sample by all assessors. This re-
sulted in contingency tables for GNs with Panel A applying 115
attributes and Panel B applying 49 attributes. For PNs on appear-
ance, taste and mouthfeel, Panel A applied 71, 71 and 51 attributed
and Panel B applied 40, 65 and 43 attributes, respectively.
Comparisons of overall sample configurations were made using
the RV coefficient (Robert & Escoufier, 1976) to see how closely re-
lated they were according to use of different MFA model dimen-
sions. The significance of the RV coefficient was tested using the
standardised RV coefficient according to Josse, Pagès, and Husson
(2008).
To graphically display uncertainty within individual methodo-
logical approaches and to compare differences in uncertainty be-
tween methods, 95% confidence ellipses were applied around the
sample mean points in a multidimensional space. The construction
of the confidence ellipses followed the parametric bootstrap proce-
dure as outlined by Dehlholm, Brockhoff, and Bredie (2012), which
bootstraps groups of coordinates based on the MFA groups of vari-
ables. This procedure makes it possible to construct confidence
ellipses for one-repetition methodologies like Napping. To apply
the same approach for all methodological approaches, results with
more than one repetition were averaged over repetitions.
The sensory attributes used in the different methods were ana-
lysed for their significance of discriminating the samples. For CP re-
sults, significant attributes were selected at p < 0.05 using the
following analysis of variance model:
 C. Dehlholm et al. / Food Quality and Preference 26 (2012) 267–277
Attributei ¼ aðProducti Þ þ bðAssessori Þ þ dðProducti ; Assessori Þ
þ ei
The ANOVA model used to select significant attributes for the FP
results was:
Attributei ¼ aðProducti Þ þ bðRepetitioni Þ þ ei
Only those attributes reported by a minimum of three assessors
were included for interpretation.
For the FMS, GN and PS results, the reported descriptions of
products were based on frequency analysis. Beforehand, lemmati-
sation was performed in obvious cases. For FMS, characterisations
were reported, if the term did positively describe the product
(p < 0.25) and was used by at least two assessors. For GN and PN,
attributes were reported, if the term did positively describe the
product (p < 0.1) and was applied by at least two assessors. After
independent semantic examinations per methodology, results
were separated and listed in tables of terms describing product
appearance, taste or mouthfeel. Before reporting, all semantic re-
sults were translated from Danish to English.
All analyses were performed in R version 2.12.1 (Ihaka & Gen-
tleman, 1996; R Development Core Team, 2010) applying function-
ality from the add-on packages FactoMineR (Husson, Josse, Le, &
Mazet, 2010; Lê, Josse, & Husson, 2008) and SensoMineR (Husson
& Lê, 2010; Lê & Husson, 2008).
3. Results
3.1. Configuration comparisons
The distributions of samples according to the applied sensory
methodology are compared in Fig. 2 for both panels. All configura-
tions are obtained by individual models and are shown with differ-
ent colours for each methodological approach and panel. Fig. 2
gives the overall picture that products are separated in similar
ways, since all results for an individual product are placed in the
same area. Products 4 and 9 show a low variability among results,
while Products 1 and 2 seem to show the largest variability. For in-
stance, Product 2 shows contradictory results obtained for the GN
by Panel A and Panel B, for the FMS by Panel A and for the CP by
Panel B.
Fig. 2. Comparative MFA on individual configurations of a conventional descriptive
profile (CP), the flash profile (FP), the free multiple sorting (FMS), the global napping
(GN) and the partial nappings’ (PN) mean, showing Product 1 (P1) to Product 9 (P9).
Suffix-A: Panel A; suffix-B: Panel B.
271
The relation between the methodologies can be evaluated by
comparing the RV coefficients. Fig. 3 shows the RV coefficients ob-
tained from comparisons of the individually obtained MFA config-
urations of the applied methods. RV coefficients were calculated
for all possible combinations of methodologies and dimensionality,
but only dimensions that are typically compared are illustrated in
Fig. 3. For Panel A, the values of all demonstrated comparisons
show corresponding levels, and they are all significantly close to
the CP. For two model dimensions, the RV values lie between
0.86 for the PN and 0.74 for the FMS. Panel B shows larger differ-
ences especially for the GN with an RV value of 0.48 for two model
dimensions and is not configurationally similar to the CP. FMS is
more similar, with RV coefficients between 0.60 and 0.75 according
to choice of dimensions, while PN for two model dimensions is
0.74. Panel A and Panel B show the same rank in the methodolo-
gies’ relation to the CP in two-dimensional model comparisons.
In that way, PN is the most highly related, followed by FMS and
then GN. However, comparing higher model dimensions increases
similarity significantly for some methods, more than would be ex-
pected as an artefact. Especially higher model dimensions of the FP
of Panel A and the FMS of Panel B could obtain information similar
to the CP.
When comparing RV coefficients between panels, the CP results
show highest similarity (0.94) as the two panels performed an
overall product configuration that was approx. 95% similar. The
two PN configurations show approx. 80% similarity (0.82), and
FMS configurations show around 70% similarity (0.71) but rising
when including a higher number of dimensions. The GN results
show lower configurational similarities (0.48) between the panels.
All possible combinations of dimensions, e.g. comparisons of an
uneven number of dimensions between methods, have been calcu-
lated and examined (not shown here). These show RV coefficients
to be similar to the levels presented in Fig. 3. Other combinations
are not higher or lower than the corresponding variations for an-
other method comparison. Method comparisons showing a high
variation within the RV coefficient include the FP and some FMSs.
Individual model configurations for each panel and applied
methodology are shown in Fig. 4. The resample-based 95% graph-
ical confidence intervals are superimposed as ellipses for all sam-
ples and show a graphical overview of uncertainty within each
methodology. The area covered by confidence ellipses is observed
to be larger for the lower plots in the figure and smaller for the
upper plots. GN showed the largest ellipses, while the CPs, the FP
and the PNs showed smaller ellipses. FMS is observed to have
smaller ellipses than the ones of GN. In methods showing larger
ellipses, the GN and the FMS, one repetition of the evaluation
was performed, while the CP, the FP and the PNs included several
repetitions. Although, large differences existed between the CP, FP
and PNs in the time spend by the assessors to evaluate the samples
(Table 5). The plots shown in Fig. 4 have been arranged according
to the amount of time spent on evaluation from most time (top) to
least time (bottom). When comparing the two panels in Fig. 4 sim-
ilar structures of uncertainty were observed.
3.2. Semantic comparisons
The results of attribute analysis for all methodological ap-
proaches are shown for each product in the following tables (Ta-
bles 2–4) according to appearance, taste and mouthfeel
attributes, respectively. The sensory descriptions given by the sen-
sory panels were also compared to the sensory information given
on the product labels. The number of attributes varied greatly be-
tween the different methodologies, ranging from 20 terms in the
CP by Panel B to 189 terms in the FP by Panel A (Table 5). The three
PNs produced on average 171 terms by each panel. That is an aver-
age of 57 terms for each sensory modality.
272 C. Dehlholm et al. / Food Quality and Preference 26 (2012) 267–277
Fig. 3. RV coefficients for method configuration comparisons. Left: Comparisons between CP and other applied methods for Panel A. Middle: Comparisons between CP and
other applied methods for Panel B. Right: Comparisons between panels within the same applied method. The abscissa shows the number of model dimensions chosen for
comparison (2, 3 or 4 from each MFA result, respectively). The ordinate shows the RV coefficient. For each point, the level of significance for overall configuration similarity is
shown (–p > 0.05; ⁄p < 0.05; ⁄⁄p < 0.01; ⁄⁄⁄p < 0.001).
Regarding appearance attributes (Table 2), the red/reddish col-
our for Product 2 and the light grey colour for Product 9 were usu-
ally mentioned throughout all methods applied. When looking at
characteristics for specific methodologies, it is worth mentioning
that in PN both panels described Product 2 as soft, Product 3 as
hard/compact (as did the FP) and Product 7 as coarse/crumbly
(as did the FP). Furthermore, both panels in the CP describe Product
7 as red/pink and Products 6, 7 and 8 as coarse/lumpy/structural.
This matches the products’ label information (Table 1), which
states that Products 1, 6, 7 and 8 are coarsely chopped, and also
from the same brand. Panel B found in both FMS, GN and PN that
Products 3 and 9 have a light colour. For both products, the panel
did not find these differences during the CP, which may indicate
that new differences might not come into play when using a stan-
dard training procedure.
Regarding taste attributes (Table 3), bacon taste for Product 2,
peppery/spicy taste for Product 5 and bacon/smoked/salami taste
for Product 7 were mentioned throughout all applied methods. In
addition, the CP described Product 1 as spicy (as did the FP) and
Product 2 as smoked and salty (as did the FP). In FMS, Product 7
was also described as having pork/stale taste and Product 8 as hav-
ing liver taste. To match the product label information (Table 1),
the added bacon in Products 2 and 7 was found by all applied
methodologies. Nothing was found for Product 4, even though
the label text indicated added bacon.
Regarding mouthfeel attributes (Table 4), the terms coarse/
lumpy/incoherent were generally mentioned for Product 7. Fur-
thermore, the CP was used by both panels to describe Product
3 as burning and lumpy, Product 4 as lumpy, Product 5 as burn-
ing and hard/structural, Product 6 as lumpy/coarse (as did the
FMS) and Product 7 as burning and gritty/grainy (as did the
FP). Applying PN, the panels agreed that Product 2 had a fast
meltdown/greasy mouthfeel. In FMS, GN and PN, only Panel B
indicated that Product 3 was compact. When comparing with
the product label information (Table 1), there was a general rela-
tion between results and Product 7 being labelled as coarsely
chopped, while only some of the applied methods showed Prod-
ucts 3, 6 and 8 to be coarsely chopped. No methods showed
Product 1 as being coarsely chopped, indicating that it is not
generally perceived that way.
3.3. Overall comparisons
Except for the CP, all applied methodologies have one thing in
common. They can be performed relatively rapidly. Similarities
and differences between working with the different methods are
collected in Table 5, separated into a ‘training and introduction’
part, a ‘measuring samples’ part and a ‘results’ part. The overall
time needed to perform a rapid descriptive profile is up to 3 h,
while the CP is performed within a minimum of 9 h over several
days and is not rapid. Even though the actual sample measure-
ments in the FP are not considered to be rapid, the overall approach
is regarded as rapid, and, according to the experimenterś choice of
repetitions, the method has the option of being performed within
half a day.
4. Discussions
4.1. Configuration comparisons
When comparing different methodologies, the CP is chosen as a
reference because of its assumed sound and valid results and its
widespread use in academia and industry. The fact that GN is
shown to have the fewest similarities to the CP should not be inter-
preted in any way hedonically. The configurational differences
shown between the GN and the CP are a product of applied meth-
odologies following different concepts. To choose the right rapid
approach for any given task is not a matter of choosing the one
most closely related to a CP, but rather choosing the one best sui-
ted to answer the problem in question.
It is chosen to compare all performed methods in the same MFA
plot (Fig. 2). This is done to examine the relative configuration dif-
ferences. In a normal work situation, the analyst would only exam-
ine one data set’s plot with regard to the product configuration.
When applying MFA on multiple results, a more optimistic picture
of different configurations’ closeness is obtained, as the groups of
results are rotated. In that case, one should remember to look at
the relative differences. Hence, the figure shows a good single plot
for comparing the differences between methods rather than the
exact placement of the individual sample.
 C. Dehlholm et al. / Food Quality and Preference 26 (2012) 267–277 273

Fig. 4. Individual configurations for all performed methodologies showing each sample’s 95% confidence interval. The higher the plot, the more time is spent on sample
evaluation. CP: conventional descriptive profile, FP: flash profile, FMS: free multiple sorting, GN: global napping, PN: partial nappings’ mean.

Although the RV coefficient is a good single measure for overall
similarity when comparing multidimensional results, it does not
alone provide more than an overall picture. For instance, if a sam-
ple in one applied method is placed as an outlier, e.g. Panel B’s PN
measurement of Product 1 in Fig. 2, it will have an effect on the
overall comparison and result in a lower RV coefficient (Fig. 3),
even though all other samples are perceived similar to other meth-
odological approaches. Also, the more rapid approach as GN with
more variation in results and between panels might result in less
similarity and larger differences in RV coefficients.
274 C. Dehlholm et al. / Food Quality and Preference 26 (2012) 267–277

Table 2
Appearance attributes significantly applied for each methodology and product.

Method Panel Product Product 2 Product 3 Product Product 5 Product 6 Product 7 Product 8 Product 9
1 4
Conventional A Moist Moist, red Grey Coarse, lumps of Coarse, lumps of fat, Coarse, lumps Grey
profilea fat, moist, red red of fat, moist
B Reddish Brown Brown, lumpy, Brown, lumpy, pink, Lumpy,
structural structural structural
Flash profileb A Red Compact Compact Coarse Coarse, grainy Coarse Compact,
firm
Free multiple A Reddish Brown Light
sortingc B Reddish Light Grey brown Light
brown brown,
lumpy
Global nappingd A Light Dry Smooth
B Reddish Light Grey,
brown light
Partial napping on A Soft Hard Jelly Crumbly, gritty, Firm,
appearanced incoherent, lumpy light
B Half- Reddish Compact, Compact, Coarse Light
darkened brown, soft light dark
a
Attribute selection based on ANOVA model: Attributei = a(Producti) + b(Assessori) + d(Producti, Assessori) + ei showing terms positively describing (p < 0.05) products.
b
Attribute selection based on ANOVA model: Attributei = a(Producti) + b(Repetitioni) + ei showing terms positively describing (p < 0.05) products and applied by at least
three assessors.
c
Attribute selection based on chi-square statistics showing terms positively describing (p < 0.25) products and applied by at least two assessors.
d
Attribute selection based on chi-square statistics showing terms positively describing (p < 0.1) products and applied by at least two assessors.

The product range examined in this study consists of liver pâtés
from different manufactures and with variations in their main
ingredients. The sampling is by the authors thought to cover both
larger and smaller product differences. Results, configurationally
and semantically, of cause represents the sample set. To generalise
results on studies with other sample sets, one must assess the nat-
ure of the differences among products.
4.2. Semantic comparisons
Only the CPs involved vocabulary training. As each panel was
encouraged to develop the optimal vocabulary for the panel con-
sensus, each panel’s vocabulary was not completely similar. It is
difficult to compare all acquired vocabularies in a completely fair
way. As vocabularies are generated through different concepts,
the concept itself defines the output. The CP and FP are clearly
reductionistic in their vocabulary generating phases, as it is often
attempted to break down the product into easier recognisable con-
cepts with common understandings. The actual measurement is al-
ways on a reference term, and whatever the form of reference, the
reference will have to be thought of or defined in some way. Hence,
this leads to a more reductionistic approach than those of methods
being more holistic. This is the case, even when profiling more
complex terms. In free-choice vocabulary examples like the FP,
vocabularies are potentially idiosyncratic and tailor-made for the
individual assessor. Nevertheless, the attributes must be definable.
In FMS and Napping evaluations, the assessor instructions allow
samples to be separated without being able to express the differ-

Table 3
Taste attributes significantly applied for each methodology and product.

Method Panel Product Product 2 Product 3 Product 4 Product 5 Product 6 Product 7 Product 8 Product 9
1
Conventional A Onion, Bacon, salty, Peppery, Spicy, sweet creamy Peppery, Rancid Bacon, salty, Rancid Liver
profilea spicy smoked sweet spicy smoked
creamy
B Liver, Salty, smoked Salty, sourish Liver, spicy Smoked bacon,
spicy bacon, sweet sweet
Flash profileb A Onion, Bacon, onion, Liver, sweet Herbs (rosemary, Liver, Bacon, smoked,
spicy salty, smoked creamy thyme), liver, spicy, peppery, spicy
sweet spicy
Free multiple A Soap, Bacon, chicken Creamy, Liver, sourish (different) Strong, Nutty Bacon, pork, Liver Liver
sortingc spicy liver liver peppery (coconut) smoked (canned
food)
B Bacon Sourish, spicy, sweet Liver, spicy Bacon, smoked, Liver Abnormal
stale (fishy) (meat smell
taste)
Global A Bouillon Bacon, butcher Peppery Intense,
nappingd salami,
smoked
B Liver pâté Spicy
Partial A Bacon Mild Sweet Spicy Salami
napping B Spicy Bacon, mealy Onion Peppery Bacon, smoked, Cardboard,
on tasted spicy old
a
Attribute selection based on ANOVA model: Attributei = a(Producti) + b(Assessori) + d(Producti, Assessori) + ei showing terms positively describing (p < 0.05) products.
b
Attribute selection based on ANOVA model: Attributei = a(Producti) + b(Repetitioni) + ei showing terms positively describing (p < 0.05) products and applied by at least
three assessors.
c
Attribute selection based on chi-square statistics showing terms positively describing (p < 0.25) products and applied by at least two assessors.
d
Attribute selection based on chi-square statistics showing terms positively describing (p < 0.1) products and applied by at least two assessors.
 C. Dehlholm et al. / Food Quality and Preference 26 (2012) 267–277 275

Table 4
Mouthfeel attributes significantly applied for each methodology and product.

Method Panel Product Product 2 Product 3 Product 4 Product 5 Product 6 Product 7 Product 8 Product 9
1
Conventional A Burning aftertaste, Coherent, hand, Burning aftertaste, Lumpy Burning aftertaste, Coherent,
profilea coherent, hard, lumpy, sticky gritty, hard gritty, lumpy hardness,
lumpy, sticky sticky
B Soft Soft Burning, lumpy Lumpy Burning, structural Lumpy, soft, Burning, gritty, Gritty,
structural lumpy, soft, lumpy, soft
structural
Flash profileb A Compact, greasy Coarse Grainy, lumpy Coarse, Compact,
lumpy firm
Free multiple A Fine, Dry, lumpy Coarse Coarse Coarse Fine, hard
sortingc smooth
B Compact Greasy Compact, gritty Coarse, lumpy, old
Global nappingd A Mealy
B Loose Compact Compact Coarse Fine, firm
Partial napping A Fast Greasy, smooth, Incoherent Coherent,
on mouthfeeld meltdown sticky dry, tough
B Soft Greasy Compact Dry, loose Gritty Lumpy Coarse
a
Attribute selection based on ANOVA model: Attributei = a(Producti) + b(Assessori) + d(Producti, Assessori) + ei showing terms positively describing (p < 0.05) products.
b
Attribute selection based on ANOVA model: Attributei = a(Producti) + b(Repetitioni) + ei showing terms positively describing (p < 0.05) products and applied by at least
three assessors.
c
Attribute selection based on chi-square statistics showing terms positively describing (p < 0.25) products and applied by at least two assessors.
d
Attribute selection based on chi-square statistics showing terms positively describing (p < 0.1) products and applied by at least two assessors.

ence in words. In this way, the approach to the sample can be said
to be more holistic. This is thought to be the case for both GN and
PN. Where PN is restricted to one sensory modality, GN would be
the most holistic approach. In many cases, the assessors were able
to explain, after the evaluation, why samples had been separated
as they were. This indicated that, even though the assessor was al-
lowed to be multidimensional in the response, it was often chosen
to separate samples using ‘unidimensional’ terms. So, in GN, the
assessor is given the possibility to respond in a holistic way, but
may choose not to. And on the other hand, unidimensional idiosyn-
cratic terms might have been given as an explanation for a
perceived difference that was not fully understandable.
Regarding the GN evaluation, the attached UFP makes it possi-
ble to use unlimited semantic terms. As with the FMS and the FP,
an excessively large vocabulary can be an interpretational draw-
back. As for GN, acquired terms are fewer and more diverse. In this
paper, it was chosen to compare method semantics by calculating a
significant relationship to samples based on frequencies for
methods with only one repetition. As a result of the sparse and di-
verse GN vocabulary, this frequency-based approach resulted in
few significant descriptive attributes. Another approach to looking
at Napping data and vocabularies could be to calculate the terms
significantly describing model dimensions. But then it is only
possible to describe an overall model configuration and not the

Table 5
Comparison and differences in applied sensory methodologies.

Conventional profile Flash profile Free multiple sorting Global napping + ultra Partial nappingb + ultra
flash profiling flash profiling
Training and introduction
Training sessions Yes No No No No
Vocabulary introduction Yes Yes No No No
Vocabulary style Consensus Individual Individual Individual Individual
Tasting samples before evaluation Yes Yes No No No
Time used on traininga / 5–6 h 1h 10 min 10 min 10 min
introduction
Measuring samples
Fixed / predetermined vocabulary Yes Yes No No No
Measurements 15 cm unstructured Ranks Contingencies Euclidean distances Euclidean distances
VAS
Repetitions 4 3 1 1 1/1/1
Time used on measurementsc 4h 120 min 50 min 30 min 70 min
Overall time 9–10 h 3h 60 min 40 min 80 min
Holistic sample approach No No Yes Yes Yes
Rapid measurements No No Yes Yes Yes
Results
Individual assessor data sets (k- No Yes Yes Yes Yes
sets)
Output (No. of attributes) Panel A: 29Panel B: 20 Panel A: Panel A: 104Panel B: Panel A: 115Panel B: Panel A: 71/71/51Panel B: 40/65/
189 96 49 43
Statistics applied PCA MFA MFA MFA MFA
Rapid configurational results Yes Yes Yes Yes Yes
Rapid descriptive results Yes No No Yes Yes
a
For the conventional profile, sessions were spread over several days.
b
The partial nappings (PN) in the experiment consisted of one PN at appearance, one PN at taste and one PN at mouthfeel.
c
For the conventional profile and flash profile, sessions were spread over several days.
276 C. Dehlholm et al. / Food Quality and Preference 26 (2012) 267–277
individual samples. We should consider whether applied basic sta-
tistics have the properties to justify rapid methodologies at all.
Normally, when applying GN and UFP, interpretation would prob-
ably be less conservative than here and more interpretational
based.
In this study, it was chosen to apply all methodologies to expert
panelists in order to maintain a consistent comparative strategy.
Some methods may be applicable to naive subjects, consumers
or also experts. Several studies have shown that the sensory pro-
files are very similar when comparing trained and untrained asses-
sors (Husson & Pagès, 2003) or experts and consumers (Worch, Lê,
& Punter, 2010). If the methods had been performed on naive
assessors, it might have been necessary to recruit more subjects
for each evaluation in order to maintain the same separation of
samples.
4.3. Overall comparisons
The obvious difference between methods is found in the time it
takes to complete the evaluation. Except for the CP, all approaches
can potentially be performed in less than half a day, morning or
afternoon. It is important to take that into consideration in any
application where the focus is on the use of resources. Provided
the results are valid, it is optimal to minimise resources. Table 5
shows the time spend on measurements to be 30, 50, 70,
120 min and 4 h for the GN, FMS, PNs, FP and CP, respectively.
The increasing amount of time, spend on sample measurements,
seems to comply with the increasing size of the confidence ellipses
shown in Fig. 4. Notice, that spending 50–70 min on measurements
rather than 30 min seems to lower sample confidence close to
methods, where 2–4 h are spend on measurements. It seems
unnecessary to spend 4 h on measurements to show significant dif-
ferences among the selected products. Nevertheless, the choice of
method will have to rely more on the project aim.
An important difference can be seen in the way vocabularies are
applied to samples. Both the CP and FP apply a perceptually ‘reduc-
tionistic’ approach, meaning that they have to break down a whole
sensory perception into individual attributes. In the CP training,
assessors have to adapt the common vocabulary. As a group, or
machine, they are calibrated to perform in the same way. A more
individual approach takes place in the FP, where both the calibra-
tion part and the training are left out (as opposed to Free-choice
profiling). With the CP and FP, a predetermined amount of measur-
ing points is applied to the sample set. Hence, the focus during
measuring is placed on the chosen semantics as well as on the
sample. This is different for FMS and Napping methods, which
try to follow another concept with the entire focus kept on the
sample. Here, the approaches are individual and based on the dif-
ferences the subjects experience between samples. It is not until
after a separation has taken place that subjects are asked to de-
scribe their action, and only if they can. Approaching the samples
this way is more holistic and allows the use of intermodal, more
complex and idiosyncratic semantics.
When choosing the method for a specific problem, one must
consider the method-specific differences rather than choosing a
method just because it can be performed rapidly. It would be a
shame to ‘waste’ resources on panel training and four repetitions
if a rapid approach would be sufficient to cover ones needs.
Although it is clear that faster approaches are sufficient to reveal
configurational main dimensions, the largest difference lies in the
semantics.
When precise definitions of sensory product differences are
needed, the CP may be the method of choice. The CP may also give
the possibility to describe smaller sensory nuances in the products
due to the longer training period. In FP, the analyst partly lets go of
a certain degree of control. A vocabulary is still present and the
number of terms is known. But assessors might apply different
attributes and some attributes will surely not be significant. If
the analyst wants a degree of control over which attributes come
into play, these approaches can be followed. It could be a common
situation that the orderer of the evaluation expects a differentia-
tion among products on predecided attributes. It would then make
sense to train on them. In other cases, the desire is to examine mar-
kets including unknown products or to promote the development
of new and creative vocabularies. Here, a rapid evaluation with a
holistic approach, e.g. GN or PN, would be suitable. In cases were
a holistic approach is favoured but focus need to be directed, the
PNs performed by modality is an advantage. It is then easy to make
three distinct models based on Euclidean distances and with
superimposed vocabulary for just a single sensory modality. After
examining individual modalities, they can be combined to a model
presenting an overall picture, and, as we see in this study, shows
configurations similar to that of the CP. Another adaption would
be, when the analyst is interested in a specific type of information
only, e.g. textural differences. Then it is only necessary to perform a
PN on texture. But bear in mind, that the amount of recorded data
is little per assessor. In this study, the three PNs were performed
within a total of 80 min, which is twice as much as the 40 min
spent in the GN, but still rapid, indeed, and providing three times
more data to model then GN. In cases where the product set con-
sists of products with minor differences, e.g. small changes of
ingredient content, an un-guided approach like GN would not be
expected to show significant descriptions between products. In
that case, a CP might show a few significant attributes. But as a ra-
pid approach, it could be meaningful to try out a restricted PN
beforehand to examine the eventual discriminability.
Evaluations with holistic sample approaches seem to promote
any kind of discriminator, explainable or not. A vocabulary devel-
opment session can have different purposes and can be directed to-
wards, for example, product development usage or marketing
usage. For marketing purposes, it could be an advantage to apply
a holistic approach on a group of creative people. Other examples
of industrial applications of sensory evaluations are found in new
product development (NPD). It would be natural and straightfor-
ward to implement rapid approaches, e.g. Napping, in the NPD idea
phase and a more reliable fixed vocabulary type evaluation, e.g. CP,
in the later NPD documentation phase.
5. Conclusions
A CP was compared with the FP, FMS, GN and PNs performed
by several panels. All methods examined show similar configura-
tions, though CP and PN are significantly more highly related than
others within and between panels. Regarding two-dimensional
model configurations, the highest RV coefficients and similarities
between panels are found for the CP (0.94, p < 0.001) followed by
PN (0.82, p < 0.001), then FMS (0.71, p = 0.005) and finally GN
(0.48, p = 0.044). Regarding panels individually, Panel A shows
high RV coefficients for all methodologies compared with the
CP, while Panel B shows a larger variation. When comparing dif-
ferent methods to the CP, both panels show the PN to have higher
RV coefficients and other methods to have lower RV coefficients.
Also, both panels show the same declining rank of methods.
Based on resampling, confidence ellipses for each sample show
larger ellipses, especially for FMS and GN, illustrating noisier data.
Although the configurations are similar, the applied semantics are
different. Applying conservative semantic analysis, four out of
nine products (Products 2, 5, 7 and 9) have some descriptors, ap-
plied through all methodologies, in common. Comparing seman-
tics between panels, most common descriptors are found when
using the CP, though common descriptors are also found among
the other methods.
 C. Dehlholm et al. / Food Quality and Preference 26 (2012) 267–277
PN is introduced, and the result is found to be very comparable
to the CP, even though PN can be performed within a total of
80 min. It is recommended to allow the same subjects to perform
PNs on multiple sensory modalities, making it possible to obtain
both a modality-based and an overall sensory picture of the prod-
uct in question. Compared to GN, multiple PNs provide a larger
data set but may limit the holistic perception by restricting the
ability to apply cross-modal discriminators.
Choosing the right method for an evaluation depends on the
aim and expected results of the evaluation, but, indeed, rapid
methodologies are suitable to describe differences among
products.
Acknowledgements
This work was financially supported by the Danish Food Indus-
try Agency (Ministry of Food, Agriculture and Fisheries) and the
Danish Pig Levy Fund.
References
Blancher, G., Chollet, S., Kesteloot, R., Hoang, D. N., Cuvelier, G., & Sieffermann, J. M.
(2007). French and Vietnamese: How do they describe texture characteristics of
the same food? A case study with jellies. Food Quality and Preference, 18,
560–575.
Cairncross, S. E., & Sjôstrom, L. B. (1950). Flavor profiles: A new approach to flavor
problems. Food Technology, 4, 308–311.
Civille, G. V., & Lyon, B. (1996). ASTM lexicon vocabulary for descriptive analysis.
Philadelphia: American Society for Testing and Materials.
Dairou, V., & Sieffermann, J. M. (2002). A comparison of 14 jams characterized by
conventional profile and a quick original method, the flash profile. Journal of
Food Science, 67, 826–834.
Dehlholm, C., Brockhoff, P. B., & Bredie, W. L. P. (2012). Confidence ellipses: A
variation based on parametric bootstrapping applicable on Multiple Factor
Analysis results for rapid graphical evaluation. Food Quality and Preference, 26,
278–280.
Delarue, J., & Sieffermann, J. M. (2004). Sensory mapping using flash profile.
comparison with a conventional descriptive method for the evaluation of the
flavour of fruit dairy products. Food Quality and Preference, 15, 383–392.
Escofier, B., & Pagès, J. (1994). Multiple factor-analysis (Afmult package).
Computational Statistics & Data Analysis, 18, 121–140.
Faye, P., Bremaud, D., Daubin, M. D., Courcoux, P., Giboreau, A., & Nicod, H. (2004).
Perceptive free sorting and verbalization tasks with naive subjects: an
alternative to descriptive mappings. Food Quality and Preference, 15, 781–791.
Husson, F., Josse, J., Le, S., & Mazet, J. (2010). FactoMineR: Multivariate exploratory
data analysis and data mining with R. R package version 1.14. Available at
http://www.CRAN.R-project.org/package=FactoMineR.
Husson, F., & Lê, S. (2010). SensoMineR: Sensory data analysis with R. R package
version 1.11. http://CRAN.R-project.org/package=SensoMineR.
Husson, F., & Pagès, J. (2003). Comparison of sensory profiles done by trained and
untrained juries: Methodology and results. Journal of Sensory Studies, 18,
453–464.
Ihaka, R., & Gentleman, R. (1996). R: A language for data analysis and graphic.
Journal of Computational and Graphical Statistics, 5, 299–314.
ISO 5492 (2008). International Standard 5492. Sensory analysis – Vocabulary. Ref.
No. ISO 5492:2008 (E). International Organization for Standardization. Genève.
277
ISO 8589 (2007). International Standard 8589. Sensory analysis - General guidance
for the design of test rooms. Ref. No. ISO 8589:2007 (E). International
Organization for Standardization. Genève.
Josse, J., Pagès, J., & Husson, F. (2008). Testing the significance of the RV coefficient.
Computational Statistics & Data Analysis, 53, 82–91.
Kennedy, J., & Heymann, H. (2009). Projective mapping and descriptive analysis of
milk and dark chocolates. Journal of Sensory Studies, 24, 220–233.
King, M. C., Cliff, M. A., & Hall, J. W. (1998). Comparison of projective mapping and
sorting data collection and multivariate methodologies for identification of
similarity-of-use of snack bars. Journal of Sensory Studies, 13, 347–358.
Lawless, H. T., & Heymann, H. (2010). Principles of good practice. In Sensory
evaluation of food – principles and practices, pp. 57–77). New York: Springer.
Lawless, H. T., Sheng, N., & Knoops, S. S. C. P. (1995). Multidimensional-scaling of
sorting data applied to cheese perception. Food Quality and Preference, 6, 91–98.
Lê, S., & Husson, F. (2008). Sensominer: A package for sensory data analysis. Journal
of Sensory Studies, 23, 14–25.
Lê, S., Josse, J., & Husson, F. (2008). FactoMineR: An R package for multivariate
analysis. Journal of Statistical Software, 25, 1–18.
Lelievre, M., Chollet, S., Abdi, H., & Valentin, D. (2008). What is the validity of the
sorting task for describing beers? A study using trained and untrained assessors.
Food Quality and Preference, 19, 697–703.
Nestrud, M. A., & Lawless, H. T. (2010). Perceptual mapping of apples and cheeses
using projective mapping and sorting. Journal of Sensory Studies, 25, 390–405.
Pagès, J. (2003). Direct collection of sensory distances: application to the evaluation
of ten white wines of the Loire Valley. Sciences des Aliments, 23, 679–688.
Pagès, J. (2005). Collection and analysis of perceived product inter-distances using
multiple factor analysis: Application to the study of 10 white wines from the
Loire Valley. Food Quality and Preference, 16, 642–649.
Pagès, J., Asselin, C., Morlat, R., & Robichet, J. (1987). Multiple factor-analysis and its
use in the sensory data-processing - application to red wines from the Loire
Valley. Sciences des Aliments, 7, 549–571.
Pagès, J., Cadoret, M., & Lê, S. (2010). The sorted napping: A new holistic approach in
sensory evaluation. Journal of Sensory Studies, 25, 637–658.
Perrin, L., Symoneaux, R., Maitre, I., Asselin, C., Jourjon, F., & Pages, J. (2008).
Comparison of three sensory napping (R) procedure: Case of methods for use
with the ten wines from Loire valley. Food Quality and Preference, 19, 1–11.
R Development Core Team (2010). R: A language and environment for statistical
computing. 3-900051-07-0. Vienna, Austria: R Foundation for Statistical
Computing.
Risvik, E., McEwan, J. A., Colwill, J. S., Rogers, R., & Lyon, D. H. (1994). Projective
mapping: A tool for sensory analysis and consumer research. Food Quality and
Preference, 5, 263–269.
Risvik, E., McEwan, J. A., & Rodbotten, M. (1997). Evaluation of sensory profiling and
projective mapping data. Food Quality and Preference, 8, 63–71.
Robert, P., & Escoufier, Y. (1976). Unifying tool for linear multivariate statistical-
methods - rv-coefficient. Journal of the Royal Statistical Society Series C – Applied
Statistics, 25, 257–265.
Rosenberg, S., & Kim, M. P. (1975). Method of sorting as a data-gathering procedure
in multivariate research. Multivariate Behavioral Research, 10, 489–502.
Rosenberg, S., Nelson, C., & Vivekana, P. S. (1968). A multidimensional approach to
structure of personality impressions. Journal of Personality and Social Psychology,
9, 283.
Steinberg, D. D. (1967). The Word Sort: An instrument for semantic analysis.
Psychonomic Science, 8, 541–542.
Stone, H., Sidel, J., Oliver, S., Woolsey, A., & Singleto, R. C. (1974). Sensory evaluation
by quantitative descriptive analysis. Food Technology, 28, 24–34.
Williams, A. A., & Langron, S. P. (1984). The use of free-choice profiling for the
evaluation of commercial ports. Journal of the Science of Food and Agriculture, 35,
558–568.
Worch, T., Lê, S., & Punter, P. (2010). How reliable are the consumers? Comparison
of sensory profiles from consumers and experts. Food Quality and Preference, 21,
309–318.