See

discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/260110526

How to review journal manuscripts: A lesson

learnt from the world's excellent reviewers
Article in Tourism Management Perspectives April 2014
DOI: 10.1016/j.tmp.2014.01.003

CITATION

READS

64

4 authors, including:
Daniel Leung

Deniz Kucukusta

MODUL University Vienna

The Hong Kong Polytechnic University

31 PUBLICATIONS 271 CITATIONS

25 PUBLICATIONS 222 CITATIONS

SEE PROFILE

SEE PROFILE

Basak Denizci Guillet

The Hong Kong Polytechnic University
62 PUBLICATIONS 442 CITATIONS
SEE PROFILE

All in-text references underlined in blue are linked to publications on ResearchGate,

letting you access and read them immediately.

Available from: Daniel Leung

Retrieved on: 26 July 2016

Tourism Management Perspectives 10 (2014) 4656

Contents lists available at ScienceDirect

Tourism Management Perspectives

journal homepage: www.elsevier.com/locate/tmp

How to review journal manuscripts: A lesson learnt from the world's

excellent reviewers
Daniel Leung a,, Rob Law b,1, Deniz Kucukusta b,2, Basak Denizci Guillet b,3
a
b

Department of Tourism and Service Management, MODUL University Vienna, Vienna, Austria
School of Hotel and Tourism Management, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, China

a r t i c l e

i n f o

Article history:
Received 15 July 2013
Accepted 18 January 2014
Keywords:
Peer review process
Reviewers
Manuscript
Academic journals

a b s t r a c t
Considering the lack of a common and structured approach to evaluate manuscripts for academic journals, this
study examines top-quality reviewers' perceptions of peer review and synthesizes their approaches to produce
a reference guide to help reviewers, especially inexperienced scholars, understand the process. Using qualitative
data collected from 24 recipients of excellent reviewer awards in tourism, hospitality, social sciences, and management, the ndings of this study show that reviewers across different elds commonly regard the peer review
process positively as an objective assessment mechanism that can improve the quality of research output.
Though it does not identify a single and structured approach to reading and reviewing manuscripts, this study
derives 12 content and 15 process issues considered by top-quality reviewers during the process. It is also
shown that excellent reviewers adopt one of three general procedures, namely the content- or processoriented approach, or a mixture of both, when they review manuscripts.
2014 Elsevier Ltd. All rights reserved.

1. Introduction
Being one of the most important determinants of recruitment, promotion, salary adjustment, and tenure decisions, publication in academic journals is generally used as an indicator of the research performance
and qualications of an academician (Ortinau, 2011; Yuksel, 2003). Publications are one of the important means through which university faculty members and doctoral students develop their careers (Law & Chon,
2007). However, getting published in academic journals has never been
an easy task. Every publication decision is made by following the recommendations of reviewers who are considered to be qualied to rigorously assess the quality and appropriateness of the manuscript (Winck,
Fonseca, Azevedo, & Wedzicha, 2011). Since the peer review process
guarantees that each submission is scrutinized by reviewers who are
considered knowledgeable about the subject, the process can help
screen out papers with irrelevant, trivial, weak, misleading or potentially harmful content while also improving the clarity, transparency,
accuracy and utility of potential papers (Jefferson, Wager, & Davidoff,
2002, p. 2788). Although peer review has been criticized as ineffective

Corresponding author. Tel.: +43 1 3203555 422; fax: +43 1 3203555 903.
E-mail addresses: daniel.leung@modul.ac.at (D. Leung), rob.law@polyu.edu.hk
(R. Law), deniz.kucukusta@polyu.edu.h (D. Kucukusta), basak.denizci@polyu.edu.hk
(B.D. Guillet).
1
Tel.: +852 3400 2181; fax: +852 2362 9362.
2
Tel.: +852 3400 2296; fax: +852 2362 9362.
3
Tel.: +852 3400 2173; fax: +852 2362 9362.
2211-9736/$ see front matter 2014 Elsevier Ltd. All rights reserved.
http://dx.doi.org/10.1016/j.tmp.2014.01.003

and expensive (Jefferson, Alderson, Wager, & Davidoff, 2002), the process still plays a prominent role in journal publication because it helps
journal editors to fulll their gatekeeping role and also assists authors
in improving their research. Scholars have argued that peer review
guarantees journal quality (Bailey, Hair, Hermanson, & Crittenden,
2012; Laband, 1990; Rosenfeld, 2010).
Like scientic journals in other disciplines, tourism journals have
adopted peer review, particularly in the form of double-blind assessment, as their principal quality assurance mechanism (McKercher,
2002). Goodman, Berlin, Fletcher, and Fletcher (1994) suggest and empirically demonstrate that the quality of accepted manuscripts is improved by peer review. Though peer review has been proven to help
improve worthwhile manuscripts to a publishable level, an increasing
body of research also demonstrates and discusses its errors, inconsistencies, and methodological weaknesses (Bordage & Caelleigh, 2001;
Moizer, 2009; Snell & Spencer, 2005). For instance, Moizer (2009) contends that reviewers generally pay too much attention to the technical
quality of a manuscript rather than evaluating its inherent contribution
to the discipline. In Gans and Shepherd's (1994) study on rejected classic articles by leading economists, interviewees claimed that their masterpieces were rejected because the reviewers often failed to read their
works with the care and understanding that it merited. These analyses
raise a concern about the credibility of peer review. As well as technical
deciencies, many researchers have suggested that a major limitation of
the peer review process is the absence of a structured approach and/or
universal guidance or procedures for evaluating journal manuscripts
(Moizer, 2009; Provenzale & Stanley, 2005; Sternberg, 1985).

D. Leung et al. / Tourism Management Perspectives 10 (2014) 4656

Among the extant studies which have discussed ways of reviewing

manuscripts for academic journals, most of them focus primarily on putting forward various lists of evaluative criteria (Beatty, Bandyopadhyay,
Chae, & Tarasingh, 1992; Rosenfeld, 2010; Winck et al., 2011) and practical tips (Benos, Kirk, & Hall, 2003; Heddle & Ness, 2009; Lee, 2008,
Provenzale & Stanley, 2005; Rogers, 2002). However, the majority of
these discussions take the form of editorials and commentaries based
on personal observations rather than being based on systematic data
gathering (Larson & Chung, 2012). Furthermore, despite the existence
of guidelines and practical tips for evaluating journal manuscripts,
Brackbill and Korten (1970) note that reviewers should stress the significance of an article rather than its technical soundness. Echoing this,
Kliewer, Freed, DeLong, Pickhardt, and Provenzale (2005) add that a reviewer's job is to judge whether a manuscript is scientically valid, coherent, readable, and appropriate for a particular journal. Other than
maintaining a checklist for what must appear in a manuscript, reviewers
should undertake their evaluation using a systematic and exible approach. However, no research has yet been conducted that proposes
or discusses a structured approach to reviewing manuscripts for academic journals in general, and tourism journals in particular. Most recently, Law, Leung, and Li (2013) examined the article review process
from the perspective of excellent reviewers in different disciplines.
Their study, however, failed to identify a systematic way for reviewing
journal manuscripts. In other words, an answer to the research question
What would a structured approach to reviewing journal manuscripts
look like? is still largely unknown.
An evaluation of a manuscript for publication is generally based on
the reviewer's experience and personal background Winck et al.
(2011). Kachelmeier (2004) claims that the quality of reviews improves
with experience. Nylenna, Riis, and Karlsson (1994) conrm that the experience of reviewing is signicantly associated with the ability to produce a high-quality assessment. In view of the absence of structured
approaches to reviewing manuscripts for tourism journals, an investigation of the methods used by experienced referees is expected to serve as
a good prototype and reference for novice reviewers. As such, this study
examines and synthesizes the journal manuscript reviewing approaches
adopted by the world's top-quality reviewers for academic journals in
tourism, hospitality, social sciences, and management. On the other
hand, though peer review has long been applied for identifying research
that is worthy of publication in academic journals, the existing literature
provides limited information about a reviewer's perception of the peer
review process (Bailey et al., 2012; Beatty et al., 1992). Considering the
crucial role of reviewers in journal publication and thereby the knowledge creation in tourism academia, this study also examines perception
of the double-blind review process from the referee's perspective.
Cheng, Li, Petrick, and O'Leary (2011) reported that the number of
English language tourism journals has increased approximately fourfold
between 1994 and 2004. Due to the rapid growth in the number of tourism journals, a study synthesizing the structured approach by professionals in manuscript reviewing is desperately needed in tourism
journals given that more tourism academics have been asking to referee
journal manuscripts. To address the suggested research question, interviews were conducted with reviewers who have been recognized by
merit awards in the elds of tourism, hospitality, social sciences, and
management, in view of their outstanding contributions to journal quality. Drawing on a synthesis and analysis of the structured approaches to
reviewing journal manuscripts by professionals, this study is expected
to improve reviewers' expertise in reviewing and academic writing, as
well as their own scientic research endeavors.
2. Literature review
2.1. Peer review in academic journals
Editorial peer review, which emerged in the early eighteenth century and has been institutionalized since the 1940s, developed mainly in

47

response to editors' need for expert advice to help them select quality
articles from the numerous submissions made (Rosenfeld, 2010). According to Rennie (1999), the rst description of the peer review process dates back to 1731, with the preface to the Royal Society of
Edinburgh's Medical Essays and Observations. This describes the peer review as a process whereby memoirs sent by correspondence are distributed according to the subject matter to those members who are
most versed in these matters. The report of their identity is not known
to the authors (Rennie, 1999, p. 2). Though the peer review process
was introduced in 1731, this process was rstly adopted in academia
21 years later. In 1752, the Royal Society of London took over the responsibility of reviewing submissions for the Philosophical Transactions.
All materials sent to the Society for publications were inspected by the
Committee on Papers with members who were knowledgeable in
such matters. Many subsequent societies followed and even recognized
peer review as a disclaimer to the accuracy of the published material
(Spier, 2002).
For a long time, the urgent need to ll the pages of these journals discouraged editors from adopting any rigorous process of weeding out
poor-quality material. Rennie (1999) reports that journal editors had
no incentive to use peer review in the nineteenth century and the rst
half of the twentieth century, especially given that editors often wrote
much of the journal content themselves. Starting from the 1960s, the
number of people working to generate new science increased considerably (Spier, 2002). The previous excess of space in journals vanished
and there was an increasing need to scrutinize what to be published.
Due to the evolving complexity of the subject matter and growing concern over the quality of content, more use was gradually made of peer
review and by the end of the Second World War the process had become institutionalized. Nowadays, nearly all academic journals rely on
the peer review process to guarantee the scientic quality of the manuscripts that they publish (Winck et al., 2011); Kachelmeier (2004) even
suggests that a top-tier journal might lose prestige and readership if it
decided to dispense with the process.
Despite the inevitability of the peer review process in journal publication, research on how it operates is surprisingly scarce. In a systematic
review of the literature, Larson and Chung (2012) identify a total of 37
relevant articles from a database search. Table 1 summarizes the ndings of selected studies on peer review in medicine, marketing, and
other disciplines. Ethics (for both reviewers and authors) was the
most prevalent theme (see for example Pollock & Ewer, 2010; Reider,
2010). Larson and Chung (2012) emphasize that personal bias and conicts of interest (nancial and nonnancial) are core areas of ethical
concern in the peer review process. Discussion of different versions of
the peer review process is another frequently researched theme (see
for example Groves, 2010; Tierney, 2008). The peer review process
can generally be classied into ve different versions. The single-blind
system means that reviewers know the identity as well as the institution of the author who submitted the work, but the author/s does/do
not know who the reviewers are. In the double-blind system, neither
the author/s nor the reviewers know each other's identity. In contrast,
the open system makes available the identities of both, and in the
peer agreement system, authors and reviewers work together to rene
the research protocol, which means that they complete the manuscripts
collaboratively. Lastly, the author suggestion-based system requires the
author/s to provide a list of potential reviewers for the journal editor's
consideration. This approach assumes that authors are best placed to
determine who is most qualied to review their work.
Regardless of the system used, a peer review process depends on reviewers' sense of professional obligation, generosity, and understanding
(Biggs, 1990). In other words, how well reviewers perform their tasks
may inuence the quality of the articles published and ultimately the
reputation of the journal (Rowley, 1999). If a good paper is rejected because of poor reviewing (a Type II error), the reputation of the journal
may suffer. Likewise, publishing a poor quality paper (a Type I error)
may also harm the reputation of a journal. Since a reviewer's work is

48

D. Leung et al. / Tourism Management Perspectives 10 (2014) 4656

Table 1
Summary of selected studies on the peer review process.
Author/s (year)

Discipline

Data source

Key ndings

Laband (1990)

Economics

89 papers published in six economic journals during

19761980

Day and Peters (1994)

Library and information 72 reviewer reports provided by 20 different

studies
journals; 271 contributors to MCB University Press
journals

Goodman et al. (1994)

Medicine (internal
medicine)

258 evaluative reports by 44 physicians and

epidemiologists

Black et al. (1998)

Medicine (general
medicine)

670 editors' and authors' analysis on reviews for the

British Medical Journal

Kliewer et al. (2005)

Medicine (radiology)

714 reviewers for the American Journal of

Roentgenology

Snell and Spencer (2005)

Education (medical
education)

143 reviewers for Medical Education

Lee (2008)

Gastroenterology

Editorial

Schroter et al. (2008)

Medicine (general
medicine)

1380 review reports from all British Medical Journal

reviewers resident in the United Kingdom

Heddle and Ness (2009)

Hematology

Commentary

Christensen and
Yokomizo (2010)
Rosenfeld (2010)

Urology

Editorial

Otolaryngology

Commentary; relevant published references

Winck et al. (2011)

Pulmonology

Commentary; relevant published references

Bailey et al. (2012)

Marketing

653 marketing faculty members in American and

Canadian universities

Larson and Chung (2012)

Medical (hand surgery) 37 published references on peer review

Length of referee reports demonstrated a signicantly positive impact on subsequent citations of a paper;
Comments by the editors showed no signicant inuence on subsequent citations of a paper.
Editors and publishers should be explicit with their reviewers as to
what they expect, and explicit about the review criteria to their
contributors;
Contributors generally suggested that the quality and mix of articles
is the most important factor that determines the quality of a journal.
The second and third most important factors were prestige of authors, editors and editorial board as well as presentation of the journal, respectively.
Thirty-three of the 34 journal quality assessment items improved
after peer review and editing;
Five items showed statistically signicant improvements (at the 10%
level) after peer review and editing, which are 1) discussion of limitations; 2) acknowledgement and justication of generalizations; 3)
appropriateness of the strength or tone of the conclusions; 4) use of
condence intervals, and 5) description of the setting.
Two characteristics of reviewers were signicantly associated with
editors' assessment of review quality: 1) younger age; and 2) having
received training in epidemiology or statistics;
Two characteristics of reviewers were signicantly associated with
authors' assessment of review quality: 1) having received training in
epidemiology or statistics; and 2) nonmember of the editorial board.
The average quality score of reviewers is strongly correlated with age.
Older reviewers received lower quality scores (p b 0.01);
Reviewers from academia (M = 3.41) typically rated higher than
those in private practice (M = 3.26, p b 0.01);
The quality scores of reviewers with more than 25 years of service (M
= 3.19) was lower than those with shorter periods (M = 3.37).
Average time spent on the review was 184.3 min;
Two-thirds (66%) of reviewers would like to receive formal training
in reviewing;
74% of reviewers would be willing to sign their reviews in the spirit of
creating a more transparent process;
90% of reviewers thought that the comments of other reviewers on
the same paper might provide a benchmark or point of comparison,
as well as increasing reliability and consistency.
From the editor's perspective, a good reviewer has seven common
characteristics. He/she should be fair, fast, fearless, frugal and friendly. Besides, he/she should be Russian and not be egocentric.
Training led to some (albeit trivial) improvements in error detection;
The errors that were detected more frequently after training dealt
with technical aspects such as response rate, sample size calculation
and randomization procedure.
Discusses a reviewer's responsibilities as well as the considerations
involved in deciding to accept or reject a paper;
Describes the format of evaluative reports used by Transfusion and
discusses their content in detail.
Lists a total of 14 questions that reviewers should ask when they review a journal manuscript.
Discusses the structure of a review and criteria for explicit
consideration;
Discusses the way of assessing manuscript composition,
recommending manuscript disposition, and dealing with revised
manuscript;
Discusses the role and responsibilities of reviewers
Based on published references, reviews the history and rationale of
the peer review process in journal publication;
Shares good editorial practice, practical tips on manuscript reviewing,
and essential content of an evaluative report.
Marketing academics are likely to consider the peer review process as
fair/unbiased (M = 72.3) and double-blind (M = 71.3);
Top-tier journals are perceived to be relatively less blind than highlevel or other journals (p b 0.01);
Top-tier journals are perceived to be more timely than high-level or
other journals (p b 0.001);
The most common suggestion to improve the peer review process is
to speed it up by meeting deadlines and achieving faster turnaround.
The four main themes of the literature on peer review are ethics of
reviewers and authors; discussion of different versions of peer review; reviewers' criteria for analyzing manuscripts; as well as process
and structure of peer review.

D. Leung et al. / Tourism Management Perspectives 10 (2014) 4656

49

relevance to the journal theme. Table 2 summarizes the evaluation

criteria for manuscript reviewing proposed in the literature.
When reviewing manuscripts, Kachelmeier (2004) claims that reviewers see themselves as guardians bearing the responsibility for
protecting the gates of scholarship from shoddy research. Hence, reviewers often stress on weaknesses in the manuscripts while describing
their positive features in less detail (Boice, Pecker, Zaback, & Barlow,
1985; Fiske & Fogg, 1990). Considering that reviewers are responsible
for judging whether or not a manuscript is scientically valid and appropriate to a particular journal, a number of senior researchers, all of
whom are current or former editors and editorial board members of academic journals, have delineated a step-by-step reviewing process in
order to equip their less-experienced counterparts with advice and
guidance (Heddle & Ness, 2009; Hoppin, 2002; Provenzale & Stanley,
2005). Drawing on his extensive experience as an Associate Editor and
reviewer, Hoppin (2002) recommends that reviewers should assess
whether the research topic is of interest to him/her, if it is within his/
her expertise, and if he/she can commit the time before accepting an invitation to review an article. After deciding to do so, the reviewer should
rstly skim through the abstract to set himself/herself up, and then read
the article closely, focusing primarily on understanding the scientic
content. Hoppin (2002) regularly made notes about both substantive
and trivial issues in the margin during the reviewing process. The manuscript may then be put aside for a day or so, and the reviewer should
then focus on exploring the novelty of the ideas, methodology, and conclusion when rereading.
Instead of suggesting a xed method, Provenzale and Stanley (2005)
provide a list of criteria that reviewers should consider when dealing
with each manuscript. For instance, in the introduction section, reviewers must assess whether the research topic is signicant and
whether the questions have been previously answered. In the methodology section, reviewers are responsible for checking on the articulation
of the rationale and justication for adopting specic methodological
choices. Apart from sharing systematic approaches to reviewing, academic researchers have also discussed other relevant issues including
the necessity of correcting misspellings (Rogers, 2002; Rosenfeld,
2010), the time spent on reviewing a manuscript (Beatty et al., 1992;
Black et al., 1998), and the content of a good evaluative report
(Heddle & Ness, 2009; Kachelmeier, 2004). Despite the signicant contribution of these prior studies, as shown in Table 1, they are primarily
based on personal experience and draws on the editor's perspective.
That is, the suggestions offered may not reect the opinions of

crucial in fostering good scholarship and knowledge dissemination

within a discipline (McKercher, 2002), it is necessary to have a thorough
understanding of how journal manuscripts should be systematically
reviewed, resulting in a better approach to knowledge creation and dissemination in academia.

2.2. Evaluation criteria and approaches to reviewing

By its very nature, manuscript reviewing involves making both a
subjective and objective assessment of someone else's work. The publication decision and the quality of reviewing reports can thus be important variables when subjectivity dominates objectivity (Brinn, Jones, &
Pendlebury, 2000). Though the peer review process describes a system
whereby papers are scrutinized by the people who are most qualied to
do so, it is actually not easy to predict the quality of reviews based solely
on researchers' academic productivity or qualications (Rogers, 2002).
Sternberg (1985) adds that the low reliability of assessments across different reviewers results from the lack of objective criteria for dening
good scientic work. To enhance reliability, a number of academic
scholars have regularly discussed the criteria and approaches that
should be used when analyzing journal manuscripts.
Beyer (1978) pioneered the development of evaluation criteria for
journal manuscript reviewing. Beyer's work identies and contrasts
the importance of 10 criteria (originality, logical rigor, statistical rigor,
clarity and conciseness of writing style, theoretical signicance, positive
ndings, relevance to current areas of research, replicability, coverage of
literature, and applicability to problems) for reviewers in physics,
chemistry, sociology, and political science. The empirical ndings
show that sociologists and political scientists give the highest ranking
to logical rigor, whereas physicists perceive originality as the most important criterion, and for chemists it is replicability. Subsequently,
Beatty et al. (1992) replicate the methodology and show that signicance of contribution is the most important criterion from marketing reviewers' perspective. Among the relevant studies in medical journals,
Black, van Rooyen, Godlee, Smith, and Evans (1998) note that reviewers
have to assess the originality of the question, importance of the question, the strengths and weaknesses of the method, the presentation of
the paper, and the interpretation of the results in each review. In addition to examining the internal and external validity of manuscripts,
Rosenfeld (2010) suggests that the decision to accept or reject is determined by a manuscript's contribution to academia as well as its
Table 2
List of evaluation criteria cited in prior literature.
Evaluation criteria
Research topic
Signicance of the research topic
Originality of the research topic
Conceptual rigor
Literature review
Adequacy of literature
Methodology
Methodological rigor
Adequacy of data analysis
Findings
Presentation of the ndings
Interpretation of the ndings
Presentation
Logical organization
Readability
Rhetoric and language
Presentation of gures and tables
Contribution
Relevance to target audience
Signicant contribution
Other
Ethical concerns

Beatty et al.
(1992)

Black et al.
(1998)

Rogers
(2002)

Benos et al.
(2003)

Yuksel
(2003)

Lee
(2008)

Rosenfeld
(2010)

Winck et al.
(2011)

50

D. Leung et al. / Tourism Management Perspectives 10 (2014) 4656

reviewers. Laband (1990) argued that the quality of a published paper is

a function of the quality of the original manuscript and the further collaborative efforts of the author, editor and reviewers. Though editors
are the ones who decide whether to publish a manuscript, their decisions are primarily based on the recommendations from reviewers
(Armstrong, 1997; Spier, 2002; Winck et al., 2011). Since reviewers
are the gatekeeper and executor of the evaluation process, an understanding of how they go about constructing a review would help other
reviewers, especially those who are less experienced, to carry out good
reviews and indeed to improve their own scientic writing in the future.
3. Methodology
Crittenden and Wilson (2006) state that surveying is the predominant mode of data collection in studies of the peer review process. However, given that this exploratory study sets out to examine and
synthesize the approach taken by the world's excellent reviewers, a
qualitative methodology was deemed to be more appropriate in order
to gain richer and more insightful data from respondents. Based on a
comprehensive literature review (see for example Beatty et al., 1992;
Hoppin, 2002; Provenzale & Stanley, 2005; Rosenfeld, 2010; Winck
et al., 2011), an interview instrument was designed using an online survey software program named Qualtrics (http://www.qualtrics.com).
The instrument comprised three open-ended questions focusing on
perceptions of the double-blind peer review process (Q1: Do you
agree that a double-blind peer review process can increase the quality
of a manuscript? Please elaborate on your response.), its advantage/s
and disadvantage/s (Q2: What is/are the advantage/s and disadvantage/s of the double-blind peer review process?), and how they
assessed articles for academic journals (Q3: Please describe the ow
(that is, the steps taken) of the way that you review articles for academic journals.). Before the data collection, a pilot test was conducted with
two other academics, who had received excellent reviewer awards,
working in the tourism eld to ensure content validity of the interview
instrument. Apart from a few minor amendments on the wording of the
questions, no major change was suggested.
As the guardians of the body of knowledge, peer reviewers serve a
vital role in guaranteeing the genuine contribution of every published
paper. In recognition of those reviewers with outstanding contribution
to journal quality and knowledge development, some academic journals
(e.g., Journal of Service Management) and academic associations (e.g., the
International Council on Hotel, Restaurant, and Institutional Education)
honor them with the excellent/outstanding reviewer awards on a regular basis. Since these awards accredit not only the recipients' dedication
to the journals but also their professionalism in reviewing journal manuscripts, an examination of the methods used by the recipients of these
excellent/outstanding awards is thus expected to gain more insights.
The recipients of awards for excellence in reviewing for tourism, hospitality, social sciences, and management journals were the target respondents of this study. Social sciences and management were included due
to their interdisciplinary relationship with tourism and hospitality (Xiao
& Smith, 2006). Having approached different editors and publishers,
one of the leading publishers of global research in business, society,
and education agreed to assist in the data collection. However, for
data protection, the names and contact details of the reviewers could
not be provided by the publisher. Instead, the publisher passed an
email invitation containing a hyperlink to the interview instrument to
a list of 163 reviewers who had received excellent/outstanding reviewer
awards in 2010, 2011, and 2012. The data collection was conducted at
the end of 2012, and a total of 24 valid responses (response rate:
14.7%) were received at the end of the data collection period.
Content analysis was used to analyze the qualitative data received
from the respondents. As noted by Merriam (1998), content analysis
involves the simultaneous coding of raw data and the construction of
categories that capture relevant characteristics of the documents' content (p. 160). According to Hsieh and Shannon (2005), conventional

content analysis, directed content analysis and summative content analysis are three general approaches to qualitative content analysis. Directed content analysis is usually employed to validate or extend a theory or
theoretical framework, whereas the focus of summative content analysis is on discovering the underlying meanings of the data. Different from
directed and summative content analysis, conventional content analysis
is generally used with a study design whose aim is to describe a phenomenon. Considering that the objective of this study is to synthesize
the journal manuscript reviewing approaches adopted by the world's
top-quality reviewers, conventional content analysis is deemed to be
the appropriate method in the current study. Instead of using preconceived categories for coding, conventional content analysis suggests
dening codes and categories from the data (Kondracki & Wellman,
2002). As such, all responses were rstly read word by word to derive
codes and capture key thoughts. After identifying all keywords and
codes, codes were sorted into categories based on how they are related
and linked. Themes and patterns were then derived into meaningful
clusters, and preliminary conclusions were drawn according to the
themes. Lincoln and Guba (1985) noted that cross-validation can help
establish the credibility of the coding scheme. As such, all researchers
independently coded the data, and then compared the results to ensure
consistency and inter-coder reliability.
4. Reviewers' perceptions of the peer review process
4.1. The potential of the double-blind process to increase manuscript quality
Respondents were rstly asked their views on the statement a
double-blind peer review process can increase the quality of a manuscript. Not surprisingly, 21 out of 24 respondents agreed. Of the three
who did not, one suggested that the double-blind peer review process
does not always increase the quality of manuscript (r4), and another
commented that it can improve quality but this is not necessarily
true (r7). That respondent also highlighted the potentially distortive
effect of reviewing by peers.
The proponents of double-blind reviewing stated that they trusted
the process due to its anonymous and objective nature. One stated
that knowledge of an author's reputation may cause someone to ignore
a shortcoming of the manuscript (r10). Another commented that
concealing the author's identity helps reviewers concentrate on the
text and what is really there (r24). Given that the identities of both authors and reviewers are undisclosed in the double-blind peer review
process, favoritism can be avoided (r20). Several respondents felt
that reviewers can provide unbiased, objective, and honest evaluations
of the work without the need to worry who the author is (r1). One
even claimed that researchers also trust the review process where
both the authors and reviewers remain anonymous (r3).
The benet of integrating the collective and diverse knowledge of reviewers was another reason given for respondents trusting the doubleblind process. In such a reviewing method, every submission is read by
multiple experts who often have broader/different perspectives on the
work (r2). As the paper is therefore reviewed through multiple lenses
and perspectives (r13), respondents pointed out that reviewers' comments can acknowledge mistakes which the author/s may not have realized were there, and identify issues that researchers have not
considered (r2). Furthermore, given that every decision about publication in academic journals is determined by multiple reviews from different perspectives, the accuracy and validity of the reviewing can be
enhanced (r11). Also, informed and less arbitrary decision can result
(r3, r9).
4.2. Advantages and disadvantages of the double-blind process
Though a number of studies have examined reviewers' perceptions
of the peer review process (see for example Bailey et al., 2012, Snell &
Spencer, 2005), this research is one of the rst attempts to investigate

D. Leung et al. / Tourism Management Perspectives 10 (2014) 4656

its advantages and disadvantages from the perspective of acknowledged excellent reviewers. Based on the descriptions provided by respondents in the second question of the survey, three major
advantages were identied, namely, 1) the assurance of objectivity; 2)
a collective effort to enhance manuscript quality; and 3) a collective effort to enhance the rigor of reviewing. The two major disadvantages
identied were: 1) the time-consuming nature of the process; and 2)
personal bias.
4.2.1. Advantages of double-blind peer review
4.2.1.1. Assurance of objectivity. Objectivity assurance was the core advantage frequently mentioned among all responses. Some respondents
suggested that double-blind peer reviewing can avoid the halo effect
based on the name of the authors (r22). Its anonymous nature means
that the work is assessed on its merit rather than the status of the author (r10) or a reputational advantage given to a person or university
(r2). As a result, some respondents deemed that an anonymous peer review is fair to lesser names in a eld (r22). Authors can thus obtain an
objective assessment of the quality of their work. On the other hand,
some respondents suggested that reviewers are protected by this
form of peer review (r10). Since the identity of the reviewer is veiled
as well, one respondent commented that you [reviewers] can tell
what you [they] really think without worrying too much (r15). Compared with other forms of peer review, respondents generally agreed
that the double-blind method helps enhance fairness for both authors
and reviewers.
4.2.1.2. Collective effort to enhance manuscript quality. Quality improvement was another positive aspect of the double-blind review process
perceived by respondents. Given that reviewers may have a different
background and knowledge base, each may have a different way to
criticize and correct the manuscript (r8). In addition, reviewers may
analyze the work from multiple perspectives. Researchers may thus acquire more suggestions (r9) and get to know their blind side of things
(r23), resulting in an improvement in the quality of manuscripts which
may make them more relevant to a wider audience.
4.2.1.3. Collective effort to enhance the rigor of reviewing. As well as
beneting contributors, respondents mentioned the collective effort
to enhance the rigor of reviewing as another intrinsic advantage of
the double-blind review process. As more people than the editor
judge the manuscript quality (r21), one respondent commented that
the combined views from multiple reviewers will facilitate the editor
to make informed decisions for publication consideration (r3). In addition, one respondent stated that reviewers can learn to write and research through the review process (r21). Rogers (2002) suggests that
reviewers can stay abreast of the latest developments in their specic
eld of interest. By acquiring insights into scientic writing and research, Rosenfeld (2010) also notes that reviewers can develop their
own critical thinking skills and hence improve research and teaching.
4.2.2. Disadvantages of double-blind peer review
4.2.2.1. Amount of time required. A number of respondents discussed the
issue of peer review as a time-consuming process. One felt that the
process appears to be very slow and does not help in bringing out research results early enough (r23). Another posited that the author
has to wait until the reviewers have completed their work (r13).
Given that contributors and reviewers may have different views of the
rationale and contribution of the research, another respondent was concerned that it may take longer to get manuscripts published, especially
when the research questions or methods used are debatable (r3).
Many respondents raised concerns about ensuring publications remain
current when the peer review process is used.

51

4.2.2.2. Personal bias. Personal bias as perceived by reviewers and editors

was another commonly identied disadvantage. Two respondents
noted that editors could select reviewers in a way which could affect
the outcome. One described that editors could send a paper to a reviewer who is positively or negatively disposed towards a certain perspective or methodology (r2). Editors might give more weight to
comments by one reviewer over those of another, resulting in a different publication decision. One respondent thought that editorial bias
might even confound publication (r11).
Even though experts with relevant knowledge and professional experience are normally selected as reviewers, many respondents suggested that reviewers sometimes focus on what they think should
have been done, rather than assessing the work in regard to its objectives (r2). On the other hand, professionals from other disciplines
might be invited to review submissions, in which case one respondent
was concerned that the reviewers may not fully understand why
some work was done in a particular manner because those things
might not be important in their social environment at all (r19).
Apart from these two major disadvantages, respondents also
discussed other issues including inconsistent quality of reviews (r23),
unfriendly design of online review systems (r4), lack of openness in
the review process (r6), and the possibility of guessing the identity of
the reviewers (r1).
4.3. Structured approach to reviewing journal manuscripts
Sternberg (1985) suggests that there is no universal procedure for
reviewers to use when assessing the quality of a manuscript. Based on
an analysis of the respondents' descriptions of their approach, the current research conrms Sternberg's (1985) proposition that the procedures used by excellent reviewers vary. Although no identical
sequence of reading and assessing manuscripts was identied, a number of content issues and process issues were identied as common
concerns for respondents during the reviewing process. Table 3 summarizes the concepts and the frequencies with which they were mentioned
by respondents. Despite the variance of frequencies, the frequency does
not reect the relative importance of each process since a limited number of reviewers were included for analysis.
4.3.1. Content issues
In line with its literal meaning, content issues denote issues relevant
to locating and examining the existence and quality of particular aspects
of content while reviewing. In other words, it means that reviewers specically search for and examine certain parts of submitted manuscripts.
As can be seen in Table 3, a total of 12 content issues were extracted
from reviewers' survey responses. These cover all sections of a typical
research-based manuscript, suggesting that excellent reviewers are
concerned with all and not part of the content when they consider
other people's work.
4.3.1.1. Examining potential contribution to the literature/academia. Five
respondents said that they examined a study's potential contribution
to academic knowledge. Two stated that they looked at what are the
contributions of this paper to the literature (r5) and sought to determine if it adds to the literature review on the area (r20) when assessing
every submission. Two other respondents also reported that they were
interested in what the contribution currently is (r15) and the potential contributions it will make to the theory (r3). Bailey et al. (2012)
note that peer review is central to the creation and dissemination of
new academic knowledge. Indeed, academic research is conducted
within a knowledge system that purposes to increase knowledge
(Preece, 1994). Its output is thus expected to provide meaningful contributions to the body of knowledge and assist the scholarly community.
Moreover, given that the ndings of, and discussion presented in, academic research might serve as a theoretical basis and guide for future
work by other researchers, it is easy to understand why many reviewers

52

D. Leung et al. / Tourism Management Perspectives 10 (2014) 4656

Table 3
List of content and process issues.
Respondents
Content issues
Examine the potential contribution to the literature/
r3
academia
Examine how the research design is proposed
r3
Examine how the data analysis helps achieve the
r3
research objective/s
Check cited references (occasionally)
r2
Examine how the problem statement/research question
r3
is formulated
Examine the author's views on the study limitations
r3
and future research
Check that in-text citations match the reference list
r1
Check the accuracy of the ndings and the discussion
r3
Examine the potential contribution to the industry
r3
Examine the meaningfulness of the study rationale
Examine the cohesiveness of the literature review
r3
Check whether the author has a global context in mind

N
r5

r9

r5

r15

r7 r8
r8

r20

4
4

r13 r14
r15

r5

r19 r20

4
3

r20

r7

r5

Process issues
Read the manuscript more than once
r4
Mark notations while reading the manuscript
r1 r2
Coach the author to strengthen the contribution of the
paper
Read additional and relevant references
r2
Read through article quickly to obtain a general
r3
understanding of the study
Read the abstract to obtain a general understanding of
r3
the study
Examine readability (such as grammar, writing style)
r3
Focus on issues that are not clear
r2
Provide feedback according to the review pro forma or
r2
other system
Provide suggested readings/references for supporting
r2
issues
Write an evaluative report with constructive comments
Talk to colleagues if there are issues that they are not
certain about
Determine if the paper ts the scope of the journal
Coach the author to x problems with method or logic
Summarize the main issues for the editor
r2
Reviewing approacha
M M M P C

r8
r8

r14

3
2
2
2
2
1

r20
r8

r13

r7
r13

r6 r7 r8
r10 r11 r12
r6
r10
r12
r8 r9
r12

r16

r21
r22
r22

r15

r10 r11
r10 r11

r23 r24 11
r24 7
5

r21

4
4

r14

r8

r14

r20

r14

r20
r17

3
3
3

r23

r6

r22
r21
r10

r24

r15

2
1

r16
r20

1
1
1

r9
P

M M M P

C represents Content-oriented approach; M represents Mixed approach; P represents Process-oriented approach.

placed an emphasis on evaluating how the work that they were

assessing can contribute to knowledge in the sense of testing a theory
and extending the applicability of research methodologies.
4.3.1.2. Examining the research design. Four content issues were dealt
with occasionally by these reviewers, including an investigation of
how the research design is described. Four respondents mentioned
this issue when they described their reviewing workow, and one of
them claimed that he/she was very concerned about how the research
is proposed (r3). According to Ortinau (2011), the methodology section is intended to set out the data collection procedures and describe
the degree of scientic rigor used in conducting the investigation.
Since this section represents the blueprint by which another researcher
can reproduce the study, Provenzale and Stanley (2005) recommend
that authors should provide a rationale for their specic methodological
choices and explain the study design. As it is not possible for publishable
research output to be reproduced if the method is awed, reviewers
have to rigorously investigate the technical concerns involved in the
choice and use of a methodology as well as its ability to answer the research questions proposed.
4.3.1.3. Examining how the data analysis helps achieve the research objectives. The issue of whether the data analysis methods chosen would
meet the research objectives was another content-related concern
from the reviewers' perspective. One respondent suggested that he/
she would want to ensure that methodology used is aligned with the

kind of research questions that are being addressed (r13). Another respondent reported examining the measurement and data analyses recommended to achieve the research objectives (r3). McKercher, Law,
Weber, Song, and Hsu (2007) suggest that a sound methodology
forms the foundation of good research whereas the use of an unsound
methodology usually produces poor or unreliable research results
(p. 460). As a methodology which is not scientically sound may seriously impair the credibility of the manuscript, reviewers have to ensure
that the analytical approach chosen enables the authors to answer the
research question and achieve the study objectives.
4.3.1.4. Checking cited references. Provenzale and Stanley (2005) recommend that researchers scan and determine whether important
articles have not been included in the citations of a manuscript. Similarly, four respondents reported that they occasionally checked
cited references. One said that he/she would pull a few (references)
out if it looks central to the argument of the authors (r15). Another
two stated that they looked at the references to see how the work is
justied (r19) or make sure they say what is purported (r2).
In their study of the reason why referees reject manuscripts,
McKercher et al. (2007) note that a strong review of the literature
will result in a sound research idea or theoretical framework and is
therefore more likely to produce work that adds new knowledge to
the discipline. The prominent inuence of references on the study's
theoretical contribution may partly explain why some reviewers
will examine and verify the citations.

D. Leung et al. / Tourism Management Perspectives 10 (2014) 4656

4.3.1.5. Other content issues. Three respondents said that they would examine how the problem statement has been formulated. One
commented that he/she would look at the way that the paper links
the present with previous research (r5), placing an emphasis on
whether or not the research rationale and problem statement is convincing. Identifying a credible and cogent problem statement is a core
concern for every academic research project and drives decisions
about the research design, ndings, and contribution to both academia
and industry. Hence, it is essential for reviewers to scrutinize the clarity
and completeness of the problem statement. Authors' thoughts on the
limitations of their studies and ideas for future research also received attention from these respondents. Three respondents reported that they
would look at what are the limitations and future research potential
(r5) and determine if areas for future research are included and [if so,
that they are] clear (r20). Moreover, three respondents would check
that the in-text citations matched those listed in the references. One respondent reported that a lack of consistency in this area may bring up
trust issues for me [him/her] with the author and might color his/her
whole view of the work (r8).
Two respondents said that they would check the accuracy of the
ndings and the discussion of the results. The assessment of the industrial implications and generalizability of a study was, perhaps surprisingly, not a topic frequently discussed by these reviewers. Only two
respondents suggested that they would examine what the contribution is for the practitioner community (r20). Similarly, only one reported that he/she would check whether a global context is in mind when
presented (r13). Issues pertinent to the meaningfulness of the study
rationale and the cohesiveness of literature review also received
some, albeit limited, attention from respondents.
4.3.2. Process issues
Distinct from content issues, process issues primarily comprise general reading and reviewing practices which are not directly relevant to
any part of the manuscript content itself, and accordingly reect individual reviewing style and patterns. Table 3 shows that 15 process issues were identied by the excellent reviewers. These largely describe
aspects of the process by which they receive, review, and then return
a manuscript.
4.3.2.1. Reading more than once. Eleven out of the 24 respondents said
that they would read the manuscript twice or more. In general, a number of reviewers rstly read through the paper to gain a general understanding of the study, then conducted their substantive assessment and
critique during a reread taking place a few hours, days, or even weeks
afterward. One respondent reported that he/she specically read the
manuscript again to check if it ows well (r8). Hoppin (2002) also reports that he focused on understanding the science of the study in the
rst reading. After putting the manuscript aside for a day, he would
then reread it and focus this time on exploring the novelty of the
ideas, conclusions, data, or methodology. Hoppin (2002) stresses that
he frequently miss[ed] important insights on [his] rst reading (p.
1019). As human error is unavoidable, one respondent identied the advantage of reading the work twice so that he/she can see things more
clearly (r12) and hence manage the likelihood of making mistakes. In
this regard, reading a manuscript more than once would be a way of
minimizing the risk of overlooking some aspects of its content or
contribution.
4.3.2.2. Making notations while reading. Making notations in the margin
or text was another frequently used technique. Seven respondents
claimed to use this method to mark down comments (r1) or make
notes about specic issues (r10). In his discussion of reviewing scientific articles, Hoppin (2002) also reports that he regularly made notations
on both substantial and trivial issues, as well as jotting down citations
he wanted to check. The notations may help reviewers retrieve the
key content and construct their evaluative reports, as one respondent

53

reported my notes help me construct a good and constructive critique

(r12). Another respondent added that at the end I review these notes
to make sure that my views were correct and edit the comments as necessary (r2).
4.3.2.3. Coach the author to strengthen the contribution. Beatty et al. (1992)
report that the ability to recognize and maximize a manuscript's contribution is the most important characteristic of an outstanding reviewer.
In the present study, ve of the excellent reviewers empirically veried
this proposition and reported that they would coach the author/s to
strengthen the contribution of the paper after they had evaluated it.
Two respondents said that they would rstly examine the current contribution, and would then make further suggestions if there is more to
elaborate (r8) or feedback to the author/s about how to improve it
[the contribution] (r15). One respondent said that he/she would usually
try to give advice on how to improve articles, even if my recommendation
is rejection (r12). Another stated that if the paper is poor and heading
for rejection, I write overall comments suggesting how the writer could
improve. van Rooyen, Black, and Godlee (1999) note that providing authors with advice and guidance on how to improve the manuscript quality is one of the reviewer's major responsibilities. Reviewers should
provide constructive comments regardless of their nal decision.
4.3.2.4. Read additional and relevant references. In order to provide constructive comments for improving manuscript quality, these reviewers
often read additional references. One survey respondent contended
that he/she would read a few other papers on the topic to be able to
better comment on the work (r2). Other three respondents felt that
looking up and reading related articles for background reading is a
necessary part of reviewing (r21). Scientic theories and professional
knowledge are constantly evolving over time. As reviewers need to
equip themselves with all-round and up-to-date knowledge on the subject before assessing someone else's work, reading additional and relevant references may help them improve their expertise so as to better
perform their gatekeeper role. Furthermore, since reviewers' comments and decisions are associated with the reputation of a journal,
their assessments have to be supported with evidence in order to verify
the objectivity of the process. Reading additional references may help
reviewers construct credible arguments and identify ways of enhancing
the contribution of the work being assessed.
4.3.2.5. Other process issues. Four reviewers reported that they would
read through the article quickly to pick up a general understanding.
One respondent described that he/she would read the abstract, glance
over the gures and exhibits, and then skim through to get an idea of
what the paper is trying to accomplish in his/her rst reading (r14). Instead of skimming through the whole article, four respondents chose
rstly to read the abstract or article summary to get an idea of the
work. In terms of the comments provided, three respondents checked
and provided their feedback according to the requirements of the review pro forma. One respondent mentioned that his/her feedback
would vary depending on the reviewing system used as many online
systems have boxes requesting feedback on specic points (r2).
Three respondents said they would focus on issues that are not clear
or points where issues need to be addressed (r2). Another respondent
commented that he/she might challenge the paper in places where it
isn't clear, is contradictory, is unintelligible, or unsupported (r17).
Three respondents would sometimes provide suggested reading or references to help authors clarify issues in the original manuscript. From a
readability perspective, three respondents reported that they examined
this aspect, including grammar, writing style, and ow. However, no respondent reported that he/she would copyedit the manuscript for the
contributor/s. Two respondents would write an evaluative report with
constructive comments after concluding their review, while one
would talk to colleagues if the paper raised issues about which he/she
was not certain. Last but not the least, other process issues, including

54

D. Leung et al. / Tourism Management Perspectives 10 (2014) 4656

summarizing the main points for the editor, determining if the paper ts
the scope of the journal, as well as coaching the author/s to x problems
with method or logic, were discussed by respondents.
4.3.3. General approaches to manuscript reviewing
As discussed previously, a number of content and process issues
were identied by these reviewers. Though no respondents described
an identical sequence of reading and assessing manuscripts, three general approaches used by excellent reviewers can be identied according
to the issues extracted from their responses. The last row of Table 3
summarizes the approach employed by each respondent.
4.3.3.1. Content-oriented approach. The rst is the content-oriented approach. This focuses on locating and examining the quality of a single
or multiple parts of the manuscript content. In other words, reviewers
who adopt this approach generally concentrate on content issues and
on examining all or part of the manuscript content. The approach
employed by r5 is a prototype of this, given that all issues he/she raised
dealt with different aspects of content. According to r5:
I look at: /How the paper link[s] the present with previous research/
The methodology used/What are the contributions of this paper to
the literature/What are the limitations and future research potential.
[r5]
Though the content-oriented approach is a straightforward and
standardized one, it was not frequently used by these excellent reviewers. Table 3 shows that only three of them adopted it.
4.3.3.2. Process-oriented approach. The process-oriented approach is the
second, and also the most popular, approach taken by these reviewers.
As Table 3 shows, it was adopted by 11 out of the 24 respondents. This
approach is based on reading and reviewing the manuscript without
clearly identifying particular content. In other words, it is primarily concerned with process issues and the reviewer will address the content of
the paper according to their own reading and advising style. r10 exemplies this approach. In particular, r10 stated that he/she would:
Read the article quickly and lightly to get a general picture of the
study. Then, read more carefully, making notes about specic issues.
Sometimes [I will] read a related article as background. Write review
and then revisit the review to assure it is not too harsh, inconsistent,
or not supported with evidence.
[r10]
Throughout his/her description, r10 does not name particular sections of the manuscript for further investigation, but describes general
reading and reviewing behavior.
4.3.3.3. Mixed approach. This is a combination of the content- and
process-oriented approaches. It means that reviewers might be particularly concerned about certain parts of the manuscript, while reading the
submission according to their own styles. r14 is one of the nine respondents who adopted the mixed approach. He/she suggested the following process:
First, I read the abstract, glance over the gures and exhibits, and
then skim through to get an idea of what the paper is trying to accomplish. Second, I go through a rst time just checking for reference and citation agreement between the text and the reference
list. If it is lacking, it adversely colors my whole view of the work,
since it brings up a trust issue for me with the author(s). Any lack
of attention to detail in this area is troubling. Third, I read the paper
line for line for content and quality, of both the communication and
the analysis (if it is not a purely conceptual work). Then I write my
review.
[r14]

This respondent mentions both examining a particular part of content (such as checking that the in-text citations match the reference
list) and using particular techniques to evaluate the manuscript overall
(such as reading the article quickly to get a general understanding of the
study).
5. Discussions and conclusions
The absence of any structured approach to evaluating manuscripts
for academic journals has long been criticized as a major limitation of
the peer review process (Moizer, 2009, Provenzale & Stanley, 2005).
To explore the nature and form that such a structured approach might
take, this study has examined and synthesized the journal manuscript
reviewing approach adopted by the world's top-quality reviewers in
order to generate a prototype for use by novice reviewers. Drawing on
the responses of an online survey from the 24 top-quality reviewers in
tourism, hospitality, social sciences, and management, the ndings of
this study conrm that no single identical approach was taken to reading and assessing manuscripts. Despite this, 12 content issues and 15
process issues of concern to these excellent reviewers have been identied. Based on the analysis of issues of concern, three general
methods, namely the content- and process-oriented and the mixed approach, used by these excellent reviewers have been constructed. The
ndings show that a high-quality review does not result from the single
application of any clear-cut approach to do the evaluation. Moreover,
the ndings reect the diversity of the issues and approaches dealt
with by reviewers.
From the content perspective, although there are discrepancies
across the content issues in terms with the frequency with which they
were mentioned by respondents, it can be seen that these excellent reviewers generally examined the whole manuscript rather than particular parts of it when assessing someone's work. Since reviewers have to
determine if the work is conceptually sound, methodologically robust,
and will contribute to the development of the eld (van Rooyen et al.,
1999), all components of the manuscript content should be rigorously
scrutinized. Lewis and Pizam (1986) suggest that papers are often
rejected because they fail to advance the knowledge of the discipline.
As the need to examine the potential contribution, a paper makes to
the body of academic knowledge was a topic frequently mentioned by
respondents. Authors are thus advised to pay more attention to communicating how their work contributes to the literature. One interesting
nding is that an inconsistency in citations between the text and the reference list may negatively affect a reviewer's overall impression of the
manuscript. Since authors are responsible for ensuring that their papers
are free of errors, they should proofread the manuscript before
submission.
Provenzale and Stanley (2005) suggest that a reviewer serves two
major functions in the peer review process. The rst is to judge whether
or not the manuscript merits publication, by providing a global rating.
The second is to provide constructive criticism for the author, regardless
of whether the manuscript is deemed acceptable for publication. As
tourism faculties and graduate students face increasing pressure to
meet research and publishing productivity requirements, it is not surprising that competition for space in academic journals is increasing.
However, though authors and reviewers may be virtual rivals for limited
publication space, the ndings of this study empirically conrm
Provenzale and Stanley's (2005) proposition that excellent reviewers
are generally able to coach authors to strengthen their study contributions. Benos et al. (2003, p. 48) recommends that all reviewers treat
all manuscripts in the same manner that you would want your own
treated. Accordingly, reviewers should provide constructive criticism
based on evidence in order to help authors enhance the quality of
their work.
Given the drive towards productivity, we may expect to see the proliferation of submissions to tourism journals and also a higher demand
for reviewers to assess the quality of submissions (McKercher, 2002).

D. Leung et al. / Tourism Management Perspectives 10 (2014) 4656

Though this study does not identify a uniform approach to evaluating

manuscripts as adopted by some of the world's best reviewers,
Table 4, which summarizes the 27 issues and three general approaches
derived from the respondents' feedback, might serve as a prototype or
guideline for novice reviewers. In addition, the issues discussed here
can provide scholars with further insights into the demands of scientic
writing and how to ensure that their contributions make it through the
journal reviewing process to publication.
Another focus of this study is the examination of reviewers' perception of the double-blind review process. In general, reviewers trusted
the process due to its anonymous and objective nature. Moreover, the
benet of integrating the collective and diverse knowledge of reviewers
was another reason given for respondents trusting the double-blind
process. Based on a survey of 653 marketing academics, Bailey et al.
(2012) show that respondents ranked the fairness and genuinely
double-blind nature as the most signicant qualities of the peer review
process. Although this study focuses on different academic disciplines,
the ndings show that reviewers across different elds share favorable
impressions of the peer review process as an objective or unbiased
mechanism for assessment which can improve the quality of research.
The paper has some limitations which may restrict the extent to
which the ndings can be generalized. Firstly, content analysis was
adopted to understand the approach taken by survey respondents. As
Hsieh and Shannon (2005) suggest, personal bias in coding, categorizing, and interpreting may arise during content analysis because of the
subjective judgment of researchers. The limited reliability of the pilot
test result should also be acknowledged as only two academics were invited to ensure the content validity of the interview instrument. Moreover, this study cannot claim to be widely generalizable as all
participating reviewers were from only one single source (i.e. from
one publisher). Incomplete information might be provided by the participating reviewers as no probing questions could be asked during
the data collection process. The ndings may thus not represent the perceptions of all journal reviewers. Future studies should therefore attempt to reproduce these ndings using a larger sample size in order
to generate more insights. Comparing and contrasting the differences
in journal manuscript reviewing approaches by reviewers from different disciplines would be another direction for future research. Given
that the stereotype in tourism research is of positivist researchers
from the USA and Asia, a more interpretivist style from the UK and
Europe and a more pragmatist/mixed methods approach from
Australia and New Zealand, future research may examine if reviewers'

55

reviewing approaches vary across the geographical regions of their

afliations.
Acknowledgment
The authors would like to thank Professor Chris Ryan and the anonymous reviewers for their constructive comments on improving an
early version of this paper. This project is partly supported by a research
grant funded by the Hong Kong Polytechnic University.
References
Armstrong, J. S. (1997). Peer review for journals: Evidence on quality control, fairness, and
innovation. Science and Engineering Ethics, 3(1), 6384.
Bailey, C. D., Hair, J. F., Hermanson, D. R., & Crittenden, V. L. (2012). Marketing academics'
perceptions of the peer review process. Marketing Education Review, 22(3), 263278.
Beatty, S. E., Bandyopadhyay, S., Chae, M. S., & Tarasingh, P.S. (1992). A closer look at manuscript reviewing in marketing. Journal of Marketing Education, 14(3), 314.
Benos, D. J., Kirk, K. L., & Hall, J. E. (2003). How to review a paper? Advances in Physiology
Education, 27(2), 4752.
Beyer, J. M. (1978). Editorial policies and practices among leading journals in four scientic elds. Sociological Quarterly, 19(1), 6888.
Biggs, M. (1990). The impact of peer review on intellectual freedom. Library Trends,
39(12), 145167.
Black, N., van Rooyen, S., Godlee, F., Smith, R., & Evans, S. (1998). What makes a good reviewer and a good review for a general medical journal? Journal of the American Medical Association, 280(3), 231233.
Boice, R., Pecker, G., Zaback, E., & Barlow, D. H. (1985). A challenge to Peters and Ceci's
conclusions with an examination of editorial les for reviewer appropriateness.
Behavioral and Brain Sciences, 8(4), 744745.
Bordage, G., & Caelleigh, A. S. (2001). A tool for reviewers: Review criteria for research
manuscripts. Academic Medicine, 76(9), 904908.
Brackbill, Y., & Korten, F. (1970). Journal reviewing practices: Authors' and APA members'
suggestions for revision. American Psychologist, 25, 937940.
Brinn, T., Jones, M., & Pendlebury, M. (2000). Measuring research quality: Peer review 1,
citation indices 0. Omega, 28(2), 237239.
Cheng, C. K., Li, X. R., Petrick, J. F., & O'Leary, J. T. (2011). An examination of tourism journal
development. Tourism Management, 32(1), 5361.
Christensen, N.B., & Yokomizo, A. (2010). How to peer review. International Journal of
Urology, 17(9), 754.
Crittenden, V. L., & Wilson, E. (2006). An exploratory study of cross-functional education
in the undergraduate marketing curriculum. Journal of Marketing Education, 28(1),
8186.
Day, A., & Peters, J. (1994). Quality indicators in academic publishing. Library Review,
43(7), 472.
Fiske, D. W., & Fogg, L. F. (1990). But the reviewers are making different criticisms of my
paper! Diversity and uniqueness in reviewer comments. American Psychologist, 45(5),
591598.
Gans, J. S., & Shepherd, G. B. (1994). How are the mighty fallen: Rejected classic articles by
leading economists. Journal of Economic Perspectives, 8(1), 165179.

Table 4
A reference guide for journal manuscript reviewing.
Mixed approach
Content-oriented approach

Process-oriented approach

Introduction section
How is the problem statement/research question formulated?
How meaningful is the study rationale?
Literature review section
How cohesive is the literature review section?
Method section
How is the research design proposed?
How does the data analysis help achieve the research objective/s?
Results section
How accurate is the presentation of the results?
Discussions section
How accurate is the discussion of the results?
What is the potential contribution to the literature/academia?
What is the potential contribution to the industry?
Does the author have a global context in mind?
Conclusions section
What is the author's view on the study limitations?
What is the author's view on the direction for future research?
References section
Are the cited references relevant?
Do the in-text citations match those in the reference list?

Before reviewing the manuscript

Read through the article quick to obtain a general understanding of the study
Read the abstract to obtain a general understanding of the study
While reviewing the manuscript
Mark notations on the unclear areas
Read additional and relevant references
Examine the readability
Talk to colleagues if there are uncertain issues
Determine if the paper ts the scope of the journal
Before submitting the evaluative reports
Read the manuscript more than once
Focus on issues that are not clear
Coach the author to x problems with method or logic
Coach the author to strengthen the contribution of the paper
Provide feedback according to the review pro forma or other system
Provide suggested readings/references for supporting issues
Write an evaluative reports with constructive comments
Summarize the main issues for the editor

56

D. Leung et al. / Tourism Management Perspectives 10 (2014) 4656

Goodman, S. N., Berlin, J., Fletcher, S. W., & Fletcher, R. H. (1994). Manuscript quality before and after peer review and editing at Annals of Internal Medicine. Annals of
Internal Medicine, 121(1), 1121.
Groves, T. (2010). Is open peer review the fairest system? Yes. British Medical Journal, 341,
c6424.
Heddle, N. M., & Ness, P.M. (2009). Reviewing manuscripts: Tips and responsibilities.
Transfusion, 49(11), 22652268.
Hoppin, F. G. (2002). How I review an original scientic article. American Journal of Respiratory and Critical Care Medicine, 166(8), 10191023.
Hsieh, H. F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis.
Qualitative Health Research, 15(9), 12771288.
Jefferson, T., Alderson, P., Wager, E., & Davidoff, F. (2002). Effects of peer review: A systematic review. Journal of the American Medical Association, 287(21), 27842786.
Jefferson, T., Wager, E., & Davidoff, F. (2002). Measuring the quality of editorial peer review. Journal of the American Medical Association, 287(21), 27862790.
Kachelmeier, S. J. (2004). Reviewing the review process. Journal of the American Taxation
Association, 26(1), 143154.
Kliewer, M.A., Freed, K. S., DeLong, D.M., Pickhardt, P. J., & Provenzale, J. M. (2005).
Reviewing the reviewers: Comparison of review quality and reviewer characteristics
at the American Journal of Roentgenology. American Journal of Roentgenology, 184(6),
17311735.
Kondracki, N. L., & Wellman, N. S. (2002). Content analysis: Review of methods and their
applications in nutrition education. Journal of Nutrition Education and Behavior, 34(4),
224230.
Laband, D. N. (1990). Is there value-added from the review process in economics?: Preliminary evidence from authors. The Quarterly Journal of Economics, 105(2), 341352.
Larson, B. P., & Chung, K. C. (2012). A systematic review of peer review for scientic manuscripts. Hand, 7(1), 3744.
Law, R., & Chon, K. (2007). Evaluating research performance in tourism and hospitality:
The perspective of university program heads. Tourism Management, 28(5),
12031211.
Law, R., Leung, D., & Li, G. (2013). Reviewing articles for tourism journals. Annals of
Tourism Research, 43, 643645.
Lee, S. S. (2008). How to be a great reviewer: An editor's view. Liver International, 28(2),
158159.
Lewis, C. R., & Pizam, A. (1986). Designing research for publication. Cornell Hotel and
Restaurant Administration Quarterly, 27(2), 5661.
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage Publication.
McKercher, B. (2002). The privileges and responsibilities of being a referee. Annals of
Tourism Research, 29(3), 859862.
McKercher, B., Law, R., Weber, K., Song, H., & Hsu, C. (2007). Why referees reject manuscripts. Journal of Hospitality and Tourism Research, 31(4), 455470.
Merriam, S. B. (1998). Qualitative research and case study applications in education. San
Francisco, CA: Jossey-Bass.
Moizer, P. (2009). Publishing in accounting journals: A fair game? Accounting,
Organizations and Society, 34(2), 285304.
Nylenna, M., Riis, P., & Karlsson, Y. (1994). Multiple blinded reviews of the same two
manuscripts: Effects of referee characteristics and publication language. Journal of
the American Medical Association, 272(2), 149151.
Ortinau, D. J. (2011). Writing and publishing important scientic articles: A reviewer's
perspective. Journal of Business Research, 64(2), 150156.
Pollock, R. E., & Ewer, M. S. (2010). The integrity of authorship: Doing the right thing.
Cancer, 116(17), 39863987.
Preece, R. (1994). Starting research. London: Pinter.
Provenzale, J. M., & Stanley, R. J. (2005). A systematic guide to reviewing a manuscript.
Journal of Nuclear Medicine Technology, 34(2), 9299.
Reider, B. (2010). Fabrication, falsication et al. American Journal of Sports Medicine, 38(3),
445447.
Rennie, D. (1999). Editorial peer review: Its development and rationale. In F. Godlee, & T.
Jefferson (Eds.), Peer Review in Health Sciences (pp. 113). London, UK: BMJ Books.
Rogers, L. F. (2002). Reviewing manuscripts for the AJR. American Journal of Roentgenology, 178(5), 10511052.
Rosenfeld, R. M. (2010). How to review journal manuscripts. Otolaryngology-Head and
Neck Surgery, 142(4), 472486.
Rowley, J. (1999). Developing research capacity: The second step. International Journal of
Educational Management, 13(4), 208212.
Schroter, S., Black, N., Evans, S., Godlee, F., Osorio, L., & Smith, R. (2008). What errors do
peer reviewers detect, and does training improve their ability to detect them?
Journal of the Royal Society of Medicine, 101(10), 507514.
Snell, L., & Spencer, J. (2005). Reviewers' perceptions of the peer review process for a
medical education journal. Medical Education, 39(1), 9097.
Spier, R. (2002). The history of the peer-review process. Trends in Biotechnology, 20(8),
357358.
Sternberg, R. J. (1985). Tacit agreements between authors and editors. Behavioral and
Brain Science, 8(4), 746747.

Tierney, A. J. (2008). Reviewers support blinding in peer review. Journal of Advanced

Nursing, 64(2), 113.
van Rooyen, S., Black, N., & Godlee, F. (1999). Development of the Review Quality Instrument (RQI) for assessing peer reviews of manuscripts. Journal of Clinical Epidemiology,
52(7), 625629.
Winck, J. C., Fonseca, J. A., Azevedo, L. F., & Wedzicha, J. A. (2011). To publish or perish:
How to review a manuscript. Pneumologia, 17(2), 96103.
Xiao, H. G., & Smith, S. L. J. (2006). The making of tourism research: Insights from a social
sciences journal. Annals of Tourism Research, 33(2), 490507.
Yuksel, A. (2003). Writing publishable papers. Tourism Management, 24(4), 437446.

Daniel Leung is a Researcher and Lecturer in the Department

of Tourism and Service Management at MODUL University
Vienna.

Rob Law, Ph.D., is a Professor of Technology Management in

the School of Hotel and Tourism Management at The Hong
Kong Polytechnic University.

Deniz Kucukusta, Ph.D., is an Assistant Professor in the

School of Hotel and Tourism Management at The Hong Kong
Polytechnic University.

Basak Denizci Guillet, Ph.D., is an Associate Professor in the

School of Hotel and Tourism Management at The Hong Kong
Polytechnic University.