Evaluation and Program Planning 35 (2012) 236–245

Contents lists available at SciVerse ScienceDirect

Evaluation and Program Planning

journal homepage: www.elsevier.com/locate/evalprogplan

Quality and rigor of the concept mapping methodology: A pooled study analysis
Scott R. Rosas *, Mary Kane
Concept Systems, Inc., 136 East State Street, Ithaca, NY 14850, United States

A R T I C L E I N F O A B S T R A C T

Article history: The use of concept mapping in research and evaluation has expanded dramatically over the past 20
Received 8 June 2011 years. Researchers in academic, organizational, and community-based settings have applied concept
Received in revised form 28 September 2011 mapping successfully without the benefit of systematic analyses across studies to identify the features of
Accepted 5 October 2011
a methodologically sound study. Quantitative characteristics and estimates of quality and rigor that may
Available online 12 October 2011
guide for future studies are lacking. To address this gap, we conducted a pooled analysis of 69 concept
mapping studies to describe characteristics across study phases, generate specific indicators of validity
Keywords:
and reliability, and examine the relationship between select study characteristics and quality indicators.
Concept mapping
Pooled analysis
Individual study characteristics and estimates were pooled and quantitatively summarized, describing
Quality the distribution, variation and parameters for each. In addition, variation in the concept mapping data
Benchmarking collection in relation to characteristics and estimates was examined. Overall, results suggest concept
Validity mapping yields strong internal representational validity and very strong sorting and rating reliability
Reliability estimates. Validity and reliability were consistently high despite variation in participation and task
completion percentages across data collection modes. The implications of these findings as a practical
reference to assess the quality and rigor for future concept mapping studies are discussed.
ß 2011 Elsevier Ltd. All rights reserved.

1. Introduction theoretical questions. As concept mapping has gained in populari-

More than 20 years ago, Trochim and colleagues published a
series of papers on concept mapping in a special issue of Evaluation
and Program Planning (Trochim, 1989a). In this seminal work, the
theoretical and practical features of concept mapping were
outlined, making a case for its utility in planning, evaluation,
and research. Since then, concept mapping has been applied in a
number of fields and contexts, including public and community
health (Rao et al., 2005; Risisky et al., 2008; Trochim, Cabrera,
Milstein, Gallagher, & Leischow, 2006; Trochim, Milstein, Wood,
Jackson, & Pressler, 2004), social work (Petrucci & Quinlan, 2007;
Ridings et al., 2008), health care (Trochim & Kane, 2005), human
services (Pammer et al., 2001; Paulson & Worth, 2002), and
biomedical research and evaluation (Kagan, Kane, Quinlan, Rosas,
& Trochim, 2009; Robinson & Trochim, 2007; Trochim, Marcus,
Masse, Moser, & Weld, 2008). The publication of the book Concept
Mapping for Planning and Evaluation (Kane & Trochim, 2007)
provided concept mapping practitioners with a comprehensive
methodological resource.
Over the course of two decades, concept mapping has
demonstrated value in addressing a variety of practical and
* Corresponding author. Tel.: +1 607 272 1206.
E-mail addresses: srosas@conceptsystems.com, scott.rosas@cortland.edu
(S.R. Rosas), mkane@conceptsystems.com (M. Kane).
ty, so too has the need to define and examine the characteristics of
the method’s methodological quality. No published research exists
that has systematically assessed the degree to which concept
mapping produces valid and reliable results across an array of
different studies. The absence of such information limits research-
ers’ ability to articulate, assess and improve the methodological
quality of concept mapping studies. To address this need, we
accessed a large sample of concept mapping studies to: (a)
quantitatively describe study characteristics across different
phases of the process; (b) quantitatively describe specific
indicators of validity and reliability; and (c) examine the
relationship between select study characteristics and quality
indicators. As a context for this study, we provide a succinct
overview of concept mapping, followed by a rationale for
examining the quality of concept mapping as a mixed-method
approach. Finally, we briefly outline an explanation of validity and
reliability, as they pertain to concept mapping.
1.1. Concept mapping overview
Concept mapping is a type of structured conceptualization
method designed to organize and represent ideas from an
identified group. A participatory mixed-methods approach,
concept mapping integrates qualitative individual and group
processes with multivariate statistical analyses to help a group of
individuals describe ideas on any topic of interest and represent

0149-7189/$ – see front matter ß 2011 Elsevier Ltd. All rights reserved.
doi:10.1016/j.evalprogplan.2011.10.003
 S.R. Rosas, M. Kane / Evaluation and Program Planning 35 (2012) 236–245
these ideas visually through a series of related two-dimensional
maps (Kane & Trochim, 2007; Trochim, 1989a). Concept mapping
is used frequently in evaluation as a practical means of addressing
stakeholder participation in ways that enhance the relevance,
ownership and utilization of evaluation (Cousins & Whitmore,
1998). The multi-phase concept mapping process typically
requires participants to first brainstorm a large set of statements
relevant to the topic of interest. Second, each participant sorts
these statements into piles based on perceived similarity, and rates
each statement on one or more scales. Third, multivariate analyses
are conducted that include two-dimensional multidimensional
scaling (MDS) of the unstructured sort data, a hierarchical cluster
analysis of the MDS coordinates, and the computation of average
ratings for each statement and cluster of statements. The maps that
result show the individual statements in two-dimensional (x, y)
space with more similar statements located nearer each other, and
show how the statements are grouped into clusters that partition
the space on the map. Finally, the group interprets the maps that
result from the analyses through a structured interpretation
process designed to help them understand the maps and label
them in a substantively meaningful way. The quantitative maps
reveal how a group discerns the interrelationships between and
among items and assigns values to ideas and concepts, thus
constructing a basis for further discussion, interpretation, and
action. We refer readers to Kane and Trochim (2007) for a more
detailed description of the entire concept mapping process.
1.2. Quality in mixed-method applications
Consistent with the arguments for combining qualitative and
quantitative methods in a single study (Creswell & Plano Clark,
2007; Sale, Lohfeld, & Brazil, 2002; Tashakkori & Teddlie, 1998)
concept mapping blends the two in a complementary and additive
manner. Rather than data remaining distinct but connected as with
some mixed-method applications, concept mapping integrates
data at multiple points of the process. Qualitative and quantitative
methods are combined in ways that challenge the distinction
between the two, and may suggest they may be more deeply
intertwined (Kane & Trochim, 2007). Given the presence of several
design typologies that emphasize a range of sequencing and
mixing decisions (e.g. Creswell & Plano Clark, 2007; Tashakkori &
Teddlie, 1998) addressing the quality of concept mapping is
pertinent. However, the absence of a comprehensive set of criteria
for critically appraising mixed-method studies (Tashakkori &
Teddlie, 1998; Sale & Brazil, 2004) and the conceptual variation of
mixed-method quality among evaluators and researchers (Car-
acelli & Riggin, 1994) further compound how concept mapping
quality should be operationalized. Although generic criteria have
been used to assess the quality of mixed-method studies, the need
for more specific evaluation criteria, depending on the design and
approach, is warranted (Sale & Brazil, 2004). This perspective
supports the need to address the methodological quality of concept
mapping in ways unique to the approach.
1.3. Validity in concept mapping
The traditional notions of external and internal validity are
challenging to operationalize for concept mapping, and are
frequently overlooked. As defined by Cook and Campbell (1979),
validity is the best available approximation of truth or falsity, both
externally and internally, of a given inference, proposition, or
conclusion. Because validity can be operationalized differently, we
posit that external representational validity and internal representa-
tional validity may be analogues for concept mapping. External
representational validity is concerned with the extent to which a
conceptualized model mirrors the reality it is purported to
237
represent. Analytical strategies for concept mapping to assess
the degree to which the conceptual model is recognized as the
modal representation for a group have been suggested (Cacy,
1996). These techniques, however, are exploratory and not yet
practical. Typically, the assessment of external representational
validity is generally managed as a function of each concept
mapping study by seeking verification that the brainstormed
statement set represents the topic under inquiry, using multiple
data collection and analysis methods, and including independent
participants with diverse perspectives. Because uniform data
relevant to external validity are not routinely available for
individual studies to include in a pooled analysis, external
representational validity is not considered in this study.
Internal representational validity however, is particularly
germane to this study. Internal representational validity refers
to the degree to which the conceptualized model reflects the
judgments made by participants in organizing information to
produce the model. In that sense, the question of whether the
conceptualized model reveals the same distinctions among
groupings made by the average participant is of particular
importance. A case has been made that the analytic approach
that anchors concept mapping results represents the best fit of the
various cognitive structures of participants (Forgas, 1979).
Questions have been raised, however, as to whether the final
model may obscure some of the finer details; due perhaps to
variations in how participants approach the structuring task
(Keith, 1989). Thus, determining the overall match between the
participant-structured input and the mathematically generated
output is central to assessing internal representational validity.
Several data elements common to all concept mapping studies
can be used to evaluate the correspondence of the represented
model to the original participant structures. First, early work by
Dumont (1989) and Trochim (1989b) suggests that the degree of
configural similarity between input and output matrices can be
measured by computing a Pearson’s Product–Moment correlation
coefficient. A second measure, the stress value, is a goodness-of-fit
indicator between a given set of dissimilarities as input and the
resultant distances in a configuration (Kruskal & Wish, 1978).
Finally, the individual sorting input (i.e. the number of sorted piles)
can be examined relative to the number of clusters, to understand
the relationship between the groupings from each participant and
the final partitioning of the content represented in the map.
Collectively, these measures computed from data routinely
produced for each concept mapping study can be used to estimate
the internal representational validity of the conceptualized model.
1.4. Reliability in concept mapping
For concept mapping, the consistency of participant input can
be assessed using the sorting and rating data. Reliability of
participant ratings on a chosen scale for each of the final
statements can be assessed by computing conventional item
and rater reliability estimates. The reliability of participant input of
the perceived relationships between statements can be assessed by
computing a set of estimates that are specific to concept mapping
sort data. As suggested by Trochim (1993), the traditional theory of
reliability, as typically applied in social science research, does not
readily conform to sort data in the concept mapping model.
Conventional means for assessing reliability focus on estimating
the repeatability of test items or total scores, based on some known
or assumed correct response. Sort data in concept mapping is
different. Instead of estimating the reliability of items or overall
scores of a measure, sorting reliability assessment is more
appropriately focused on determining the extent to which the
structural arrangements, both individually and collectively, reflect
an assumed normatively typical arrangement. Thus, the individual
238 S.R. Rosas, M. Kane / Evaluation and Program Planning 35 (2012) 236–245
and aggregated sort configurations (similarity matrices used as
input by multidimensional scaling), as well as the resulting
distance matrices (the between-item Euclidean distances output
generated by multidimensional scaling), provide information to
calculate reliability estimates. We refer to Trochim’s (1993)
recommended procedures for calculating a set of reliability
statistics specific to concept mapping input and output to estimate
consistency of sorting within and across specific studies.
2. Method
2.1. Overall study approach
The standardized procedures for data collection, data organi-
zation, analysis, and representation in the concept mapping
process yield a set of common quantitative data, which can be
configured in a comparable statistical form. This uniformity allows
for the same quality constructs and relationships to be examined,
and thus produces results that are more objective and exact than a
narrative review. Our first step in conducting this pooled study
analysis was to generate quantitative characteristics and estimates
of the common data elements for each concept mapping study in a
sample of studies meeting specific inclusion criteria. These
characteristics and estimates were then aggregated across the
sample of studies describing the distribution, variation, and
parameters for each characteristic and set of estimates. Finally,
we conducted quantitative analyses to examine further the
relationships between project characteristics.
2.2. Sample
The sample for this quantitative pooled study analysis was
sixty-nine (69) individual concept mapping studies conducted
within the past 10 years. This set is part of an archive of completed
concept mapping studies conducted over that time period by
Concept Systems, Inc., the sole proprietor and licensor of the
Concept System1.1 The criteria for sample selection were general
and meant to include a wide range of studies. For inclusion in this
pooled study analysis, each concept mapping study needed to have
a final computed map (i.e. each had to have a final multidimen-
sional scaling analysis result and cluster solution generated), and
each study needed to have at least one rating completed by
participants.
2.3. Procedure
Once the list of acceptable studies was generated, we extracted
data from each specific concept mapping study database.
Descriptive characteristics for each study included: data on the
number of participants (overall and by concept mapping task),
completion rates by task, and the number of statements resulting
from brainstorming. Categorical information for each study
included: the general field of study, related organizational or
institutional support, the general purpose of the study, and the
primary data collection mode. For each study, we obtained data for
assessing the reliability and validity as it pertains to concept
mapping and computed estimates.
After generating each concept mapping study’s specific
characteristics and estimates, the results were configured,
aggregated, and analyzed for the entire sample of studies.
Measures of dispersion, central tendency, and interval estimates
were computed for each study element for the pooled studies in
the sample. We analyzed further these estimates to examine
1
Details and availability of the technology can be found at http://www.con-
ceptsystems.com.
patterns and relationships between different concept mapping
study characteristics. Finally, we computed several correlation
analyses between concept mapping study characteristics that were
expected to be related. Specifically, configural similarity and stress
values were correlated with other concept mapping data, including
the number of statements, the final number of clusters, and the sort
reliability estimates.
3. Results
A majority of the concept mapping studies in the sample were
classified as public health oriented (59.4%). Others were in the
fields of human services (20.3%), biomedical research (5.8%), social
science research (2.9%), and business or human resources (2.9%).
Twenty-eight studies were supported through Federal sources
(41.5%), with several receiving support from foundations or not-
for-profit organizations (20.3%), universities or colleges (17.4%),
and state government sources (11.6%). The stated purpose of each
study varied considerably, confirming the broad use of concept
mapping. Twenty-eight (40.6%) were used for strategic planning
purposes, defined as an initiative of an organization to establish a
specific strategy or direction and make decisions regarding
resources to pursue this strategy. Twelve (17.4%) were conducted
for the purpose of developing an action or research agenda, defined
as a collaborative effort to outline a strategic direction in a field
that extends across organization boundaries. Project purposes also
included: evaluation (14.5%), primarily for framework and design
development; research (7.2%), primarily for conceptualization and
theory development; needs assessment (8.7%); and program or
intervention development (8.7%).
Three main types of data collection modes were identified in
the sample. Face-to-face or traditional means, whereby the
researcher interacts directly with the participants for brainstorm-
ing, sorting, and rating tasks, accounted for 14.5% of the studies.
Web-based means, whereby information is gathered exclusively
through the use of the Internet without direct interaction of the
researchers with participants, accounted for 34.8% of the studies.
Multi-method means, whereby information is collected through a
variety of means including paper forms, face-to-face interaction,
and web-based platforms, accounted for 50.7% of the studies.
All pooled analysis results for the sample of concept mapping
studies by study characteristic, described in subsequent sections,
are presented in Table 1.
3.1. Participants
The total number of participants is the unduplicated count of
individuals who provided input during the brainstorming, sorting,
or rating tasks. The average total number of participants was
155.78 with a range of 20–649 across studies. The total number of
participants for each study varied depending upon the concept
mapping data collection mode, and a significant group effect was
detected, F(2, 66) = 13.25, p < .001. The average total number of
study participants for the web-based data collection mode was
significantly larger (M = 243.42, SD = 172.14) than both the face-
to-face (M = 62.10, SD = 49.14), p < .001 and multi-method
(M = 122.46, SD = 45.76) modes, p < .001. For specific concept
mapping tasks, numbers are captured for both sorting and rating
participation. Overall, the number of sorters averaged 24.62
(SD = 15.29), well over the recommended number of 15 (Jackson &
Trochim, 2002), and nearly 1.7 times (M = 14.62) larger than what
was found in Trochim (1993). The smallest study in the sample had
6 sorters and the largest had 90. For the ratings task, the number of
participants on the first rating (rating 1) averaged 81.77
(SD = 69.83) and 65.82 (SD = 43.32) on the second (rating 2).
The second rating typically has fewer participants, due to attrition,
 S.R. Rosas, M. Kane / Evaluation and Program Planning 35 (2012) 236–245 239

Table 1
Participants and completion percentages by concept mapping data collection mode.

Data collection mode Average number of participants Average percent completing task

Sorting Rating 1 Rating 2 Sorting Rating 1 Rating 2

Face-to-face 25.7 44.7 33.6 74.1 80.3 72.8

Web-based 27.9 112.8 75.6 52.4 68.7 48.0
Multi-method 22.1 71.1 63.9 43.7 61.1 54.0
Overall 24.6 81.8 65.8 50.1 65.9 51.6

level of participant knowledge, or fatigue by those completing the
first rating. The average number of participants for the first rating
is nearly 5.8 times (M = 13.94) what was found in Trochim’s (1993)
study.
Differences in the number of participants by the data collection
mode were observed in this sample and are displayed in Table 2.
No meaningful difference was found in the average number of
sorters by data collection mode. However, a significant group effect
was observed, F(2, 66) = 4.62, p < .05, for rating 1, with significant
mean differences in the number of participants between the web-
based mode (M = 164.04, SD = 116.70) and both face-to-face
(M = 116.31, SD = 43.17), p < .01 and multi-method (M = 55.70,
SD = 45.31) modes, p < .05. Interestingly, no group differences
were detected in the average number of participants for rating 2 by
the data collection mode. On close inspection, the largest decrease
in the average number of participants between ratings 1 and 2 was
observed for the web-based mode. These findings suggest that the
use of the web for concept mapping facilitates greater participation
with respect to the rating task. However, the level of attrition is
greater from completion of rating 1 to rating 2, when the web is
used exclusively for data collection.
Despite the availability of the Internet to increase access to the
concept mapping process, sorting remains a fairly intensive
activity, intended to capture participant judgments about the
relationships between all items in a typically large set. Sorting
participation was observed to be fairly consistent across the
various data collection modes, and may be perceived to be limited
more by the demanding nature of the task, rather than the means
of participation.
3.2. Completion rates for sorting and rating
As shown in Table 2, the average percent completion for sorting
and rating tasks indicates that, overall more than half of those who
initially agreed to complete the task, did so in a manner that
produced usable data. The percent completion for sorting differed
by data collection mode and a significant group effect was
detected, x2(2) = 111.36, p < .001. As expected, the average
percent completion for sorting was highest when done face-to-
face (M = 74.06, SD = 19.93) compared to web-based (M = 52.38,
SD = 23.88), z = 7.17, p < .001 and multi-method (M = 43.69,
SD = 19.86) modes, z = 10.28, p < .001. The average percent
completion for web-based was also significantly higher than the
multi-method mode, z = 4.70, p < .001. The average percent
completion of rating tasks followed a similar pattern with a
significant group effect found for rating 1, x2(2) = 107.50, p < .001,
and rating 2, x2(2) = 66.75, p < .001. As with sorting, the average
percent of completion for rating 1 was significantly higher for

Table 2
Concept mapping study characteristics and estimates.

Common study elements M SE SD Mdn Min Max 95% CI for mean

Lower Upper

Number of statements 96.32 2.07 17.23 98.00 45 132 92.18 100.46

Number of sorters 24.62 1.84 15.30 20.00 6 90 20.95 28.30
Number of raters 1 81.77 8.04 69.83 62.00 18 485 64.99 98.54
Number of raters 2 65.82 5.84 43.32 57.00 5 247 54.11 77.53
Total number of participants 155.78 15.21 126.34 118.00 20 649 125.43 186.13
Percent completing sorting 50.07 2.84 23.59 56.86 10.58 100 48.39 51.75
Percent completing rating 1 65.87 2.43 20.24 70.27 12.79 100 64.87 66.87
Percent completing rating 2 51.64 2.83 20.84 56.00 10.50 100 50.47 52.81
Stress value .28 .00 .04 .29 .17 .34 .27 .29
r (configural similarity)a .66 .01 .07 .66 .53 .83 .64 .68
r2 .44 .01 .09 .43 .28 .68 .42 .46
Stress value split-half 1 .30 .00 .03 .30 .20 .36 .29 .31
Stress value split-half 2 .30 .00 .04 .31 .19 .35 .29 .31
rII .87 .01 .06 .88 .69 .96 .85 .88
rIT .96 .00 .02 .96 .90 .99 .95 .96
rIM .91 .00 .04 .92 .80 .98 .90 .92
rSHT .86 .01 .07 .87 .65 .97 .85 .88
rSHM .63 .02 .17 .61 .26 .95 .59 .67
a for rating 1 .97 .00 .02 .97 .91 .99 .96 .97
a for rating 2 .97 .00 .02 .97 .91 .99 .96 .97
AICC for rating 1 .89 .01 .07 .92 .69 .99 .88 .91
AICC for rating 2 .87 .01 .10 .90 .42 .97 .84 .90
Number of map clusters 8.93 .19 1.55 9 6 14 8.56 9.30
Average statements per cluster 11.10 .31 2.58 11.11 5.63 20.67 10.43 11.67
Statements in largest cluster 18.64 .59 4.94 18.00 9 32 17.45 19.82
Statements in smallest cluster 5.49 .23 1.94 5.00 1 10 5.03 5.96
Average number of sorted piles 10.93 .23 1.88 10.90 6.55 15.76 10.47 11.38
Median number of sorted piles 9.93 .27 2.22 10.00 6.00 16.00 9.40 10.46
Largest pile of statements 23.25 1.09 9.04 21.00 11 61 21.0 25.42
Smallest pile of statements 4.68 .15 1.28 5.00 2 8 4.37 4.99
a
Absolute values are reported for this characteristic.
240 S.R. Rosas, M. Kane / Evaluation and Program Planning 35 (2012) 236–245
face-to-face (M = 80.25, SD = 21.61) than web-based (M = 68.73,
SD = 24.61), z = 5.51, p < .001 and multi-method (M = 61.13,
SD = 14.17) modes, z = 8.74, p < .001. Similarly, the average
percent of completion for rating 2 was significantly higher for
face-to-face (M = 72.72, SD = 30.02) than web-based (M = 48.02,
SD = 24.34), z = 7.21, p < .001 and multi-method (M = 53.95,
SD = 12.58) modes, z = 5.48, p < .001). Interestingly, in both cases
the average percent completion for web-based ratings was
significantly higher than the multi-method mode (z = 7.09,
p < .001 for rating 1 and z = 4.86, p < .001 for rating 2). It is not
surprising that the face-to-face mode of data collection yielded
higher completion percentages, as the researcher manages the
process directly. It may also be that the variety of forms and
options to manage in the multi-method mode contributed to the
consistently lowest percent completion. Nevertheless, the percent
completion for data collected through the web are well above
those found in other on-line activities, such as internet-based
surveys where completion rates of 20% to 30% are common (Cook,
Health, & Thompson, 2000; Kaplowitz, Hadlock, & Levine, 2004).
3.3. Statements
The sample of studies averaged 96.32 statements (SD = 17.23)
with a range of 45 to 132. This average represents approximately a
20% increase initially found by Trochim (1993). No difference was
found in the number of statements by data collection mode. Kane
and Trochim (2007) report that with the availability of the web-
based platform, concept mapping studies commonly yields
brainstormed statements sets well over 100 items. Guidance on
selection of an appropriate statement set size has emphasized the
need to consider participant burden, at the same time working to
ensure saturation of the topic (Kane & Trochim, 2007; Trochim,
1989a). These authors recommend a structured process for
synthesizing and reducing the set to a manageable size that
minimizes burden and maximizes breadth. Thus, despite the
propensity for statement sets to be very large when collected via
the Internet, size consistency in the final statement set used for
sorting and rating was evident.
3.4. Sorting and clusters
The sorting task requests each participant to arrange the set of
statements into piles or groups, based on participant-perceived
similarity. It is an unstructured sort procedure; there is no pre-
determined number of piles into which the participants are
expected to sort the statements. Procedurally, concept mapping
participants receive a set of instructions and minimum expecta-
tions to guide the sorting task. Each participant is directed to sort
the statements into an arrangement that makes sense to her or him
(Kane & Trochim, 2007). The mean and median number of piles for
each concept mapping study was identified and then summarized
for the entire sample. The mean average number of piles was 10.93
(SD = 1.88) and the average median number of piles was 9.93
(SD = 2.22). No difference was found in the number of individual
participant sorted piles by data collection mode.
The number of clusters for each concept map is selected
through a combination of statistical analysis, expert judgment, and
participant feedback. There is no single correct number of clusters
or a set of mathematical decision criteria for determining the final
cluster solution (Kane & Trochim, 2007). The average number of
clusters selected for the final concept map in the sample was 8.93
(SD = 1.55) with a range of 6–14. Again, no difference was found in
the number of clusters selected for the final concept map in
relation to the data collection mode. The distribution of statements
across the final number of clusters for each study was also
calculated. For each study, an average distribution was computed
by dividing the number of statements by the final number of
clusters found in the map. The mean average number of statements
per cluster was 11.10 (SD = 2.58) for the sample. The number of
statements in the largest and smallest clusters for each map was
identified for each study, and was averaged to identify the upper
and lower levels across the sample. The average number of
statements in the largest cluster for each of the studies was 18.64
(SD = 4.94, range: 9–32) and the average number of statements in
the smallest cluster for each study was 5.49 (SD = 1.94, range: 1–
10), suggesting considerable variation in cluster density across
sample.
Because sorting reflects the judgments made by each partici-
pant about the relationships among the statements in a set, and the
number of clusters selected reflects the aggregated relationship
representation, it is useful to determine the degree of correspon-
dence. A significant Pearson’s Product–Moment correlation of
r = .43, p < .001 was found, indicating a moderate relationship
between the median number of piles and the final number of
clusters in which the map is partitioned. Fig. 1 represents a
bivariate plot of the relationship between the mean number of
piles and the final number of clusters for each study. This portrays
the correspondence between the structural arrangement of the
sort data from each participant, on average, and the final
partitioning of the multidimensional scaling analysis structure
across the sample. Collectively these findings suggests a positive
relationship and conceptual consistency between the aggregated
groupings from participants and the final groupings found in the
map, despite the multiple ways the final number of clusters may be
selected.
3.5. Stress, fit, and similarity
Stress is a statistic routinely generated and reported in
multidimensional scaling (MDS) analyses, reflecting the goodness
of fit of the final representation with the original similarity matrix
used as input. Stress is the normalized residual variance for a
perfect relationship of a monotone regression of distance upon
dissimilarity or similarity (Kruskal, 1964). Thus, for any given
configuration the stress indicates how well that configuration
matches the data.
The average stress value for the sample was .28 (SD = .04, range:
.17–.34, 95% CI [.27, .29]). The literature on multidimensional
scaling suggests lower stress values are preferred and reflect better
congruence between the raw data and the processed data
(Davison, 1983; Kruskal, 1964). Stress values found in concept
mapping analyses are typically higher than those recommended in
16
Average Number of Sorted Piles
14
12
10
8
6
4
4 5 6 7 8 9 10 11 12 13 14 15
Final Number of Clusters
Fig. 1. Plot of average piles by final number of clusters for 69 concept mapping
studies.
 S.R. Rosas, M. Kane / Evaluation and Program Planning 35 (2012) 236–245
the literature on MDS. Several reasons for the discrepancy have
been presented by Trochim (1993) and Kane and Trochim (2007).
Comparatively, the stress values across the sample were very
similar to those found by Trochim (1993). In fact, nearly the entire
set of concept mapping studies in this sample (96%) had a stress
value that fell within the 95% CI [.21, .37] originally estimated by
Trochim (1993) and reported in Kane and Trochim (2007). Hence,
in two pooled analyses using independent samples, nearly
identical patterns of stress were observed. It should be noted that
a group effect was found, F(2, 66) = 3.62, p < .05, when examining
the average stress values by data collection mode. However, the
mean difference between the web-based (M = .27, SD = .04) and
multi-method (M = .29, SD = .03) modes, p < .05 was very small,
and no difference was seen between face-to-face and web-based
sorting.
In terms of judging the acceptability of the stress values found
across studies in this sample, a previous simulation study by
Sturrock and Rocha (2000) can serve as a guide. Based on the
distributions of over a half a million randomly created and scaled
matrices, these authors found that for two-dimensional MDS
solutions where 100 objects have been scaled, there is a 1% chance
the arrangement of the objects in the matrix is random if the stress
value is below a upper limit of .39. Thus, multidimensional maps
with a stress statistic below this threshold have less than a 1%
probability of having either no structure or a random configura-
tion. Since none of the studies produced a stress value above .39
(even those with over 100 objects in the input matrix), it is likely
that none of the two-dimensional configurations included in this
study were random or without structure.
As a second measure of validity, the configural similarity was
calculated for each concept mapping study, reflecting the
congruence between data used as input and the final repre-
sented form. Configural similarity was estimated by computing
the Pearson’s Product–Moment correlation between the original
aggregated similarity matrix from participants and the final
matrix of Euclidean distances between points on the map. A
squared correlation coefficient, r2 was also calculated for each
study to assess the proportion of shared variance of the input
and output data. The average squared correlation of the input
similarities and the scaled distances from the MDS coordinates
was .44 and statistically significant, t(68) = 5.38, p < .05.
This estimate indicates that on average 44% of the variation
in the aggregated participant sort was accounted for by the
conceptualized model. No difference was found in the propor-
tion of shared variance of the input and output by data
collection mode.
Fig. 2 represents a bivariate plot that illustrates the relationship
between the stress values and the r2 values for each study in the
sample. As observed in the figure, the better the fit of the sort data
with the statistically represented model (i.e. lower stress values)
the greater the proportion the input data are accounted for in the
model. Taken together, these results suggest that concept mapping
performs well in representing a complex set of multivariate data in
two-dimensional space.
An examination of the correlation between the stress value and
the number of sorters for each study revealed no linear
relationship between the two variables. Nonetheless, it is
important to understand further the relationship between sorting
participation and stress. To model how stress is affected by the
number of sorters, a subset of five studies with the most sorters
was identified. Within this subset, sort data from an individual
participant was randomly selected from the list of participants
who completed the sort task in that study. A second randomly
selected sort was then aggregated with the previous, and the stress
value calculated. The process continued, and resulted in the
inclusion of sort data from all study participants. At every addition
241
0.4
0.35
Stress Values
0.3
0.25
0.2
0.15
0.20 0.30 0.40 0.50 0.60 0.70 0.80
r Values
2
Fig. 2. Plot of stress values by coefficient of determination values for 69 concept
mapping studies.
of one randomly selected participant, the fit of the map was
calculated and assessed.
The results for each of the five studies are displayed in Fig. 3,
with number of sorters plotted on the x-axis and stress values
plotted on the y-axis. Trends across the five studies reveal a
consistent pattern of stress as the number of sorters increase. The
variability in the stress values for each study was dramatic when
about 15 or fewer sorters were included. As the number of sorters
for each of the five studies reached about 35 sorters, substantial
improvements in stress (i.e. lower stress values) were observed.
However, beyond 40 sorters, only marginal improvements in stress
were detected. These findings suggest between 20 and 30 sorters is
warranted to maximize the consistency of fit in the concept
mapping representation by minimizing the variability in the stress
value found with smaller groups of sorters. This range is about
twice the number recommended by Trochim (1989a) and Jackson
and Trochim (2002). Our observation comports with previous card
sorting studies examining adequate sample sizes needed to
produce high quality representations (Tullis & Wood, 2004; Wood
& Wood, 2008). While smaller numbers of sort participants may
still yield acceptable stress values, the likelihood of generating a
higher stress value is greater with smaller groups. Thus,
consideration of the appropriate sorting sample in designing
concept mapping studies is critical.
3.6. Sorting reliability estimates
For each study in the sample, five unique reliability estimates
for the sort data were generated using the procedures outlined in
Trochim (1993), then averaged. First, the set of sort data from each
study was randomly divided into two halves (for odd-numbered
groups, one group was randomly assigned one more person than
the other). Separate concept maps were computed for each group.
The total sort matrices for each split-half group were correlated
and the Spearman–Brown correction applied to obtain the split-
half reliability of the sorts (rSHT). The Euclidean distances between
all pairs of points on the two maps from the split-half samples were
also correlated and the Spearman–Brown correction applied to
obtain the split-half reliability of the map (rSHM). Second, the sort
matrices for each individual were correlated and averaged. The
Spearman–Brown correction was applied to yield the Individual-
to-Individual Sort Reliability (rII). Third, the sort matrix for each
individual was correlated with the total similarity matrix. These
correlations were averaged and the Spearman–Brown correction
applied to produce the Individual-to-Total Matrix Reliability (rIT).
Finally, the sort matrix for each individual was correlated with the
242 S.R. Rosas, M. Kane / Evaluation and Program Planning 35 (2012) 236–245

0.36
0.35
0.34
0.33
0.32
0.31
0.3
0.29
0.28
Stress value

0.27
0.26
0.25
0.24
0.23
0.22
0.21
0.2
0.19
0.18
0.17
0.16
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89
Sorters

Study 1 Study 2 Study 3 Study 4 Study 5

Fig. 3. Increase in sorters by stress values: five largest concept mapping projects studies in the pooled sample.

Euclidean distances from the overall, final map. These correlations
were averaged and the Spearman–Brown correction applied to
produce the individual-to-map reliability (rIM). Of the five
reliability estimates calculated, three rely on analysis of the sort
data used as input. Overall, the average reliability estimates for the
sort data for the sample were high. The average individual-to-
individual sort reliability value (rII) was .87, the average individual-
to-total matrix value (rIT) was .96, and the average split-half total
matrix reliability (rSHT) was .87. No differences were found in the
reliability estimates of the sort data in relation to the data
collection mode. Two of the reliability estimates include informa-
tion from the final map for each study. The average reliability
between individuals’ sort matrices and the final map configuration
(rIM) was .91. The average split-half reliability of the final map
configuration (rSHM) was .67. The lower value for rSHM was
expected as this reliability estimate is calculated on the split-half
sample of analyzed or processed data, rather than the raw data
used to generate rSHT (Trochim, 1993). Nevertheless, the average
reliability estimates found here were high and slightly above those
found in Trochim’s (1993) previous study.
The interrelationships between the sorting reliability estimates
were assessed by calculating the Pearson’s Product–Moment
correlations for the pairs of estimates. All correlations were
significant at the .001 level. The correlations between the sort data
estimates (rII, rIT, rIM, rSHT) were strongly positive (range: .94–.99),
while correlations between the split-half map reliabilities and all
other reliabilities were lower (range: .62–.72). This suggests strong
associations between all reliability estimates, even among those
assessing the reliability of the output data.
The five sort data reliability estimates were also correlated with
the number of statements and the number of sorters, in order
further to examine relationships between sort reliability and other
study characteristics that presumably affect the estimates. The
number of statements was marginally correlated with the
reliability estimates, although the correlations were negative.
Conversely, the number of sorters was significantly correlated with
the reliability estimates, ranging from .44 to .71(all significant at
p < .001). This finding suggests that more sorters may yield more
reliable sorting results although the differences may be minor.
3.7. Rating reliability estimates
Reliability estimates of the ratings data were calculated for each
of the concept mapping studies in the sample (54 studies had two
ratings). For each rating, the internal consistency was assessed by
computing the average correlation among items using Cronbach’s
alpha. We were also interested in the reliability of different raters
averaged together. The reliability of averaged ratings is a more
useful statistic in practice due to the considerable variation in
reliability among raters which cannot be assessed (MacLennan,
1993). The average measure intraclass correlation (AICC) was
calculated to produce an inter-rater reliability coefficient, which is
equivalent to the average correlation between all pairs of raters
with the Spearman–Brown correction for the number of raters. The
average Cronbach’s alpha coefficients for both ratings 1 and 2 were
above .96, suggesting the items on the scale are highly
intercorrelated and internally consistent (DeVellis, 1991), even
for studies where two ratings were administered. A significant
group effect was found for rating 1, F(2, 66) = 4.29, p < .05 and
rating 2, F(2, 51) = 3.20, p < .05 when examining the average
alphas by data collection mode. For rating 1, mean differences were
found between the web-based (M = .97, SD = .02) and face-to-face
(M = .95, SD = .02) modes, p < .05. Similarly, for rating 2, mean
differences were found between the web-based (M = .97, SD = .02)
and face-to-face (M = .95, SD = .02) modes, p < .05. However, these
differences were very slight and the alphas by group were high
enough so as to not be meaningful. The average inter-rater
reliability coefficients (AICC) were also high, suggesting that across
raters the mean ratings are stable, although the inter-rater
reliability coefficients for the second rating were slightly lower.
No differences were found in the AICC reliability estimates for
rating 1 or rating 2, relative to the data collection mode.
 S.R. Rosas, M. Kane / Evaluation and Program Planning 35 (2012) 236–245
These reliability estimates were correlated with the number of
statements and the number of raters to further examine relation-
ships between rating reliability and other concept mapping
characteristics that presumably affect the estimates. Moderately
strong correlations between the number of statements and
Cronbach’s alphas for rating 1 (r = .49, p < .001) and rating 2
(r = .55, p < .001) were detected. This suggests that larger
statement sets yield higher internal consistency estimates.
Because ratings used in concept mapping are uni-dimensional in
nature; that is, they measure a single construct across a large
number of items (e.g. importance, feasibility, readiness, etc.), it is
not surprising that higher alphas were found. Moreover, there is a
tendency for large numbers of items to produce higher alpha
coefficients (DeVellis, 1991). Similarly, the number of raters was
moderately correlated with the inter-rater reliability coefficients
for both rating 1 (r = .53, p < .001) and rating 2 (r = .51, p < .001),
suggesting larger numbers of raters yield higher inter-rater
reliability estimates.
4. Discussion
As with similar mixed-method applications, the concept
mapping studies that are the basis of this pooled analysis were
conducted to understand complex realities using data from
multiple perspectives to combine and present practical informa-
tion. This quantitative analysis generated useful baseline infor-
mation to address questions regarding the methodological quality
of concept mapping. The study approach suggested several means
for determining the quality of concept mapping that are
appropriate, defensible, and relevant. In particular, this work
considers the significance of validity and reliability as it relates to
concept mapping, and reports on these critical aspects. With this
emphasis in mind, concept mapping as an integrated mixed-
method approach for planning, evaluation, and research appears
to generate valid and reliable results. Although the representation
of a complex set of input data was limited to two dimensions, the
internal representational validity across the set of studies was
found to be good, supported by multiple measures of fit and
similarity. While better fit and greater similarity between the
input data and output representation might be observed using
more than two dimensions, the current approach appears
appropriate for generating the most parsimonious and interpret-
able results. The reliability of the sort data was observed to be
high, both between sorters and among sets of aggregated sorters.
Likewise, the consistency of the rating data, between individuals
and among items, was very high. These findings comport with an
earlier, more limited, study of the reliability of concept mapping
by Trochim (1993), where similar patterns were found using the
same reliability calculations.
As observed in this sample, the advent of a web-based platform
for conducting concept mapping asynchronously in a virtual
environment has expanded the level of participation across all
phases of the process. This presents concept mapping users with
unique benefits for expanded participation, and notable challenges
to establishing and maintaining quality. For example, the use of the
web affects the percent of completion across different tasks based
on the ability to invite a greater number to take part in the study.
However, despite the utilization of web technology and increased
access to concept mapping for a broader set of participants,
estimates of reliability and validity appear consistent. Indeed, no
meaningful differences were found between multiple data
collection modes and the estimates of reliability and validity
calculated in this study. While questions persist as to how data
collected through group processes like brainstorming are affected
when generated individually (Dugosh, Paulus, Roland, & Yang,
243
2000), it appears that quality and rigor can be maintained for
concept mapping data collected via the Internet.
Although this sample does not represent the totality of concept
mapping studies, this pooled study analysis is the largest and most
comprehensive conducted to date. The studies combined for this
analysis were diverse in scope and topic, and typical of those found
in the published research and evaluation literature. In fact, the
results of several concept mapping studies included in this analysis
were published previously across a variety of content areas and
disciplines. Each study in this sample followed the same
procedural steps and was subject to similar constraints during
implementation. This process consistency, coupled with data
collection and computation standardization at the study level,
enabled an analysis that produced findings configured in a
comparable statistical form. This study’s systematic approach
enabled the identification and analysis of patterns that might
otherwise be obscured in a case by case assessment.
Notwithstanding the strengths of the analytical strategy,
several limitations to the study are important to note. First, as
with any analysis that pools the results of multiple studies, the
variability in random and non-random error is of concern. To
minimize error, and subsequent overestimation or underestima-
tion of estimates, we employed several strategies. A detailed
protocol for calculating the indicators was consistently applied for
each study in the sample. In addition, strict inclusion criteria were
used to ensure methodological homogeneity in the pooled sample.
Nonetheless, for this analysis, advanced statistical methods were
not employed when analyzing the sample to mitigate the potential
compounding of error found at the study level. Second, the analysis
does not capture qualitative distinctions across studies. Each of the
individual studies included in this study sample contain extensive
detail that offers insight into a particular topic considered in
context. The qualitative variation in participant experiences,
settings, content, interpretation, and uses of the results were
not considered in this pooled study analysis. Furthermore, the
assessment of quality of elements that are directly influenced by
the qualitative judgments of the researcher was not undertaken.
Cluster selection and labeling, for example, is one of those areas
that would benefit from further study to better delineate some
notion of intra-observer agreement in the determination of
clusters and names of concepts represented by each cluster. The
typical approach is for the researcher to conduct the analysis and
arrive at options for cluster arrays, and then discuss and confirm or
change with key study stakeholders. Given that the concept
mapping process calls for cooperation and negotiation in the final
structure and labeling, capturing the degree to which two
reasonable people with knowledge in the field agree or disagree
has important implications for quality. Third, relative to the entire
set of concept mapping projects completed in the past 20 years, it is
possible that this study includes potentially problematic studies.
However, this risk is somewhat mitigated by results that were
consistent with those found in previous pooled analyses (cf.
Trochim, 1993). In addition, the sample consisted of studies
completed by the originators and providers of the concept
mapping technique, and as such are an exclusive set. Implemented
by experts in the conduct of the concept mapping process, these
studies may not be representative of studies conducted by those
with less experience with the method. A case could be made that
studies in the sample represent a high level of quality due to the
adherence to the concept mapping process, as outlined by the
developers. Finally, this pooled study analysis is fundamentally
correlational. This study examined only the linear relationships
between select characteristics found in concept mapping. Ques-
tions as to whether varying certain features of the concept
mapping process, such as participant engagement, may result in
changes in other characteristics remain unanswered.
244 S.R. Rosas, M. Kane / Evaluation and Program Planning 35 (2012) 236–245
Despite these limitations, three implications of the findings
warrant attention. First, the results of this analysis offer a practical
reference for researchers and evaluators to judge the quality of
concept mapping studies and support their choices related to data
collection, analysis, and representation. In establishing a basis for
comparison, this study generated empirical data for several
characteristics of the concept mapping process, providing realistic
estimates of what one might expect in typical field applications of
the method. This study establishes a set of benchmarks and ranges
that can be used for individual concept mapping studies to gauge
the reliability and validity of the results, and can provide concept
mapping users a basis for confirming the practical issues of fidelity
and integrity related to their work (Bradbury & Reason, 2001).
Second, the results provide critical information that helps to
establish expectations for the quality of concept mapping as a
social science research method; upon which other researchers,
journal reviewers, editors, and dissertations committees can
evaluate the utility of the methodology’s processes and outcomes
in future studies. Except for basic counts of the number of
statements and participants, few published concept mapping
studies report critical elements included in this study, such as
stress values. This is likely due, in part, to the lack of information
about standards for reporting and how the data is generated. By
extension, the absence of clear expectations as to what constitutes
quality for concept mapping hinders peer-reviewers in their
appraisal of submissions. Thus, the results of this study provide
both researchers, and those who review research, information to
help evaluate and ensure the rigor of concept mapping studies
across a broad base of literature.
Third, the results provide a set of empirically grounded
recommendations for different activities within the concept
mapping process. Most of the current recommendations for
concept mapping found in the literature are general heuristics
based on professional experience or logic. The results of this study
provide systematically derived information that practitioners can
use to make key decisions in the concept mapping process, which
will affect quality. Recommendations regarding the appropriate
number of sorters or suggestions for interpreting stress values are
based on examining the variation of these elements in relation to
other indicators. Furthermore, criteria that emerged from this
study for judging the integrity of concept mapping are more
consistent with the assumptions and realities of the approach. For
example, present criteria for determining the acceptability of fit for
multidimensional scaling (MDS) applications are based on
experimental and synthetic data (Kruskal, 1964). For applied field
studies like concept mapping where MDS is used, it seems more
appropriate and reasonable to assess the acceptability of fit in
relation to results from the study of similar practical applications.
Thus, using the results of this analysis as a reference, concept
mapping practitioners should routinely report on importance
indices like stress values, to allow others to judge the relative
quality of their studies.
This work represents a foundational step in building a base of
evidence to support the methodological quality and expectations
of the concept mapping approach. However, several areas of
inquiry remain incomplete and suggest opportunities for future
pooled study investigation, including: content analysis in areas
where multiple concept maps have been produced; examination of
concept mapping processes and procedures; and inquiry into
participant characteristics across different studies. Future analyses
might also include other studies completed by a broader pool of
concept mapping practitioners. Pooled study analyses of studies
with smaller numbers of participants or those with relatively
smaller statement sets might provide useful information not
observed in this study. Moreover, the development and standardi-
zation of methods for assessing external representational validity
and consistency in subjective decisions are needed to further the
dialogue on validity in concept mapping.
Ultimately, defining appropriate expectations of quality and
rigor for concept mapping, established through systematic study,
has value for users of the approach. This study attempted to
address academic questions of validity, reliability, and quality at
the same time accounting for the practical and participatory
concerns for concept mapping. This study supports those who use
concept mapping in their work, to help establish value in
participation, measurement, data, and conclusions within a
mixed-method, participatory application.
Acknowledgements
The authors would like to acknowledge the support of the staff
at Concept Systems, Inc., Specifically we thank Brenda Pepe for
information related to the concept mapping studies included in the
analysis, Perry Slack for study level data extraction, and Marie Cope
for review and comments on the manuscript. We reference and
acknowledge the foundational thinking and work of William
Trochim in the methodology’s early applications and summary
reliability estimates of almost 20 years ago.
References
Bradbury, H., & Reason, P. (2001). Broadening the bandwidth of validity: Issues and
choice-points for improving the quality of action research. In P. Reason & H.
Bradbury (Eds.), Handbook of action research: Participative inquiry and practice
(pp. 447–456). Thousand Oaks, CA: Sage Publications.
Cacy, J.R. (1996). The reality of stakeholder groups: A study of the validity and reliability of
concept maps. Ph.D. dissertation, University of Oklahoma.
Caracelli, V. J., & Riggin, L. J. C. (1994). Mixed-method evaluation: Developing quality
criteria through concept mapping. Evaluation Practice, 15(2), 139–152.
Cook, C., Health, F., & Thompson, R. L. (2000). A meta-analysis of response rates in
web- or Internet-based surveys. Educational and Psychological Measurement,
60(6), 821–836.
Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues
for field settings. Houghton Mifflin.
Cousins, J. B., & Whitmore, E. (1998). Framing participatory evaluation. New Directions
for Evaluation, 80, 5–23.
Creswell, J. W., & Plano Clark, V. L. (2007). Designing and conducting mixed methods
research. Thousand Oaks, CA: Sage Publications.
Davison, M. L. (1983). Multidimensional scaling. New York, NY: John Wiley and Sons.
DeVellis, R. F. (1991). Scale development: Theory and applications. Newbury Park, CA:
Sage.
Dugosh, K. L., Paulus, P. B., Roland, E. J., & Yang, C.-H. (2000). Cognitive stimulation in
brainstorming. Journal of Personality and Social Psychology, 79(5), 722–735.
Dumont, J. (1989). Validity of multidimensional scaling in the context of structured
conceptualization. Evaluation and Program Planning, 12, 81–86.
Forgas, J.P. (1979). Multidimensional scaling: A discovery method in social psychology.
In G. P. Ginsburg (Ed.), Emerging strategies in social psychological research (pp.
253–288). New York: Wiley.
Jackson, K., & Trochim, W. (2002). Concept mapping as an alternative approach for the
analysis of open-ended survey responses. Organizational Research Methods, 5(4),
307–336.
Kagan, J. M., Kane, M., Quinlan, K. M., Rosas, S., & Trochim, W. M. K. (2009). Developing a
conceptual framework for an evaluation system for the NIAID HIV/AIDS clinical
trials networks. Health Research Policy and Systems 7(12).
Kane, M., & Trochim, W. M. K. (2007). Concept mapping for planning and evaluation.
Thousand Oaks, CA: Sage Publications.
Kaplowitz, M. D., Hadlock, T. D., & Levine, R. (2004). A comparison of web and mail
survey response rates. Public Opinion Quarterly, 68(1), 94–101.
Keith, D. (1989). Refining concept maps: Methodological issues and an example.
Evaluation and Program Planning, 12(1), 75–80.
Kruskal, J. B. (1964). Multidimensional scaling by optimizing goodness of fit to a
nonmetric hypothesis. Psychometrika, 29(1), 1–27.
Kruskal, J. B., & Wish, M. (1978). Multidimensional scaling. Beverly Hills, CA: Sage.
MacLennan, R. N. (1993). Interrater reliability with SPSS for Windows 5.0. The American
Statistician, 47(4), 292–296.
Pammer, W., Haney, M., Wood, B. M., Brooks, R. G., Morse, K., Hicks, P., et al. (2001). Use
of telehealth technology to extend child protection team services. Pediatrics,
108(3), 584–590.
Paulson, B. L., & Worth, M. (2002). Counseling for suicide: Client perspectives. Journal of
Counseling & Development, 80, 86–93.
Petrucci, C. J., & Quinlan, K. M. (2007). Bridging the research-practice gap: Concept
mapping as a mixed methods strategy in practice-based research and evaluation.
Journal of Social Services Research, 34(2), 25–42.
 S.R. Rosas, M. Kane / Evaluation and Program Planning 35 (2012) 236–245
Rao, J. K., Alongi, J., Anderson, L. A., Jenkins, L., Stokes, G. A., & Kane, M. (2005).
Development of public health priorities for end-of-life initiatives. American Journal
of Preventive Medicine, 29(5), 453–460.
Ridings, J. W., Powell, D. M., Johnson, J. E., Pullie, C. J., Jones, C. M., Jones, R. L., et al.
(2008). Using concept mapping to promote community building: The African
American initiative at Roseland. Journal of Community Practice, 16(1),
39–63.
Risisky, D., Hogan, V. K., Kane, M., Burt, B., Dove, C., & Payton, M. (2008). Concept
mapping as a tool to engage a community in health disparity identification.
Ethnicity & Disease, 18, 77–83.
Robinson, J. M., & Trochim, W. M. K. (2007). An examination of community members’,
researchers’ and health professionals’ perceptions of barriers to minority partici-
pation in medical research: An application of concept mapping. Ethnicity and
Health, 12(5), 521–539.
Sale, J. E., & Brazil, K. (2004). A strategy to identify critical appraisal criteria for primary
mixed-method studies. Quality and Quantity, 38, 352–365.
Sale, J. E., Lohfeld, L. H., & Brazil, K. (2002). Revisiting the quantitative–qualitative
debate: Implications for mixed methods research. Quality and Quantity, 36,
43–53.
Sturrock, K., & Rocha, J. (2000). A multidimensional scaling stress evaluation table. Field
Methods, 12(1), 49–60.
Tashakkori, A., & Teddlie, C. (1998). Mixed methodology: Combining qualitative and
quantitative approaches. Applied Social Research Methods Series (46, pp. ). ).
Thousand Oaks, CA: Sage Publications.
Trochim, W. M. K. (1989a). An introduction to concept mapping for planning and
evaluation. Evaluation and Program Planning, 12(1), 1–16.
Trochim, W. M. K. (1989b). Concept mapping: Soft science or hard art? Evaluation and
Program Planning, 12(1), 87–110.
Trochim, W. M. K. (1993, November). The reliability of concept mapping. Paper
presented at the Annual Conference of the American Evaluation Association.
Trochim, W. M. K., Cabrera, D. A., Milstein, B., Gallagher, R. S., & Leischow, S. J. (2006).
Practical challenges of systems thinking and modeling in public health. American
Journal of Public Health, 96(3), 538–546.
245
Trochim, W., & Kane, M. (2005). Concept mapping: An introduction to structured
conceptualization in health care. International Journal for Quality in Health Care,
7(3), 187–191.
Trochim, W. M. K., Marcus, S. E., Masse, L. C., Moser, R. P., & Weld, P. C. (2008). The
evaluation of large research initiatives: A participatory integrative mixed-methods
approach. American Journal of Evaluation, 29, 8–28.
Trochim, W. M. K., Milstein, B., Wood, B. J., Jackson, S., & Pressler, V. (2004). Setting
objectives for community and systems change: An application of concept mapping
for planning a statewide health improvement initiative. Health Promotion Practice,
5(1), 8–19.
Tullis, T., & Wood, L. (2004 June). How many users are enough for a card-sorting study?
The Proceedings of Usability Professionals Association Conference.
Wood, J., & Wood, L. (2008). Card sorting: Current practices and beyond. Journal of
Usability Studies, 4(1), 1–6.
Scott R. Rosas, PhD is a Senior Consultant at Concept Systems, Inc. where he specializes
in the design and use of the concept mapping methodology. His work has focused on
conceptualization and measurement in evaluation using concept mapping, with
attention to the validity and reliability of the approach. He received his PhD in Human
Development and Family Studies from the University of Delaware, with an emphasis
on program evaluation. He previously served as Associate Faculty at the Bloomberg
School of Public Health at Johns Hopkins University and is currently Adjunct Faculty in
the Department of Health at SUNY-Cortland.
Mary Kane, MSLIS is the Chief Executive and Principal Consultant at Concept Systems,
Inc. Her consulting experience includes strategic and operational planning, product
and program development, education and training design, and program needs assess-
ment and evaluation. She has coauthored several articles on the application of concept
mapping across several content areas, including public and community health. Ms.
Kane is co-author of the definitive volume on concept mapping: Concept Mapping for
Planning and Evaluation. She holds a Masters degree in Library and Information
Sciences from Columbia University.