Health Environments Research

& Design Journal

1-16
How to Write a Systematic ª The Author(s) 2017
Reprints and permission:
Review of the Literature sagepub.com/journalsPermissions.nav
DOI: 10.1177/1937586717747384
journals.sagepub.com/home/her

Debajyoti Pati, PhD, FIIA, IDEC, LEED AP1,

and Lesa N. Lorusso, MBA, MS Arch2

Abstract
This article provides a step-by-step approach to conducting and reporting systematic literature reviews
(SLRs) in the domain of healthcare design and discusses some of the key quality issues associated with
SLRs. SLR, as the name implies, is a systematic way of collecting, critically evaluating, integrating, and
presenting findings from across multiple research studies on a research question or topic of interest. SLR
provides a way to assess the quality level and magnitude of existing evidence on a question or topic of
interest. It offers a broader and more accurate level of understanding than a traditional literature review.
A systematic review adheres to standardized methodologies/guidelines in systematic searching, filtering,
reviewing, critiquing, interpreting, synthesizing, and reporting of findings from multiple publications on a
topic/domain of interest. The Cochrane Collaboration is the most well-known and widely respected
global organization producing SLRs within the healthcare field and a standard to follow for any researcher
seeking to write a transparent and methodologically sound SLR. Preferred Reporting Items for Sys-
tematic Reviews and Meta-Analysis (PRISMA), like the Cochrane Collaboration, was created by an
international network of health-based collaborators and provides the framework for SLR to ensure
methodological rigor and quality. The PRISMA statement is an evidence-based guide consisting of a
checklist and flowchart intended to be used as tools for authors seeking to write SLR and meta-analyses.

Keywords
systematic literature review, healthcare design, evidence based design

As the name suggests, systematic literature
review (SLR) is a systematic way of collecting,
critically evaluating, integrating, and presenting
findings from across multiple research studies on
a research question or topic of interest. It is
“systematic” since it adopts a consistent, widely
accepted, methodology. The methodology should
address concerns regarding quality issues, such as
bias, replicability, credibility, et cetera.
SLR provides a way to assess the quality level
and magnitude of existing evidence on a question
or topic of interest. It offers a broader and more
accurate level of understanding than a traditional
literature review. Imagine a single cross-sectional
study as a single snapshot in time and a longitu-
dinal study as a single video. As the number of
snapshots and videos available on a question or
topic increases, it offers a better and more accu-
rate grasp on the phenomena of interest. Since a
single research is just one snapshot or video,
1
Texas Tech University, Lubbock, TX, USA
2
University of Florida, Gainesville, FL, USA
Corresponding Author:
Debajyoti Pati, PhD, FIIA, IDEC, LEED AP, Texas Tech
University, Lubbock, TX 79409, USA.
Email: d.pati@ttu.edu
2 Health Environments Research & Design Journal XX(X)
basing design or policy decisions on a single
research is not as robust as basing decisions on
findings from multiple studies. There are, of
course, issues related to the quality of snapshots
or videos, and the viewpoint from which those
were taken. Poor quality could result in less clarity
and, hence, less confidence. Changes in viewpoint
could offer a different perspective of a phenomena
or introduce bias. Even with appropriate measures
for quality, findings from individual studies may
vary owing to differences in population, setting,
instruments, random chance, and so on. A properly
conducted SLR considers the quality of issues and
incorporates multiple viewpoints, providing a
broader and more complete picture.
SLRs are particularly gaining importance in
the healthcare design field. Starting from about
600 published studies in 2004 (Ulrich, Zimring,
Quan, Joseph, & Choudhary, 2004), the number
of studies in the field has increased to a level that
facilitates SLRs on important questions. This
growing body of knowledge is making SLRs in
the healthcare design field both possible and cri-
tically important. From single-patient rooms to
decentralization, there is a wide range of topics
of importance to efficiency, safety, care quality,
cost, patient/family/staff experience, and so on,
which could benefit from SLRs. Articles on SLRs
are available in other fields of studies. The pur-
pose of this article is to provide a step-by-step
approach to conducting and reporting SLRs in the
domain of healthcare design and discuss some of
the key quality issues associated with SLRs.
The purpose of this article is to provide a
step-by-step approach to conducting and
reporting SLRs in the domain of
healthcare design and discuss some of the
key quality issues associated with SLRs.
Types of SLRs
SLR began within the realm of healthcare and can
be traced back to the work of Scottish naval surgeon
James Lind who, in 1753, conducted the first ran-
domized controlled trial (RCT) and recognized sys-
tematic methods as a key component in avoiding
bias, which remains a pivotal goal of systematic
research (Dunn, 1997). By the 1970s, meta-
analysis was formalized as a process of research
synthesis (Glass, 1976) and evidence-based policy
and practice gained momentum in the 1990s.
SLRs are only one type of review of literature.
The type of review one needs to conduct depends
on the objective of the review. Broadly speaking,
the two most commonly known types of literature
reviews are the “traditional” or narrative review
and the “gold standard” or SLR (Booth, Sutton, &
Papaioannou, 2016). Traditional reviews, also
known as conventional or nonsystematic reviews
(Greenhalgh, 2014), are generally faster and eas-
ier to conduct and are sometimes appropriate due
to a short time frame or lack of resources. Grant
and Booth (2009) provide a more comprehensive
classification of 14 different types of reviews in
the fields of medicine and library sciences. In gen-
eral, reviews differ in the question being asked,
including, among others: What is the current the-
oretical framework/understanding on a topic/
domain—based on either just peer-reviewed pub-
lications or all sources? What are some gaps, in
theory or evidence, in existing published litera-
ture? Is there sufficient justification for conducting
a study? What is the most appropriate study ques-
tion/topic? Which research question/topic will pro-
vide the best return on investment? What is the
effect size of an intervention, based on quantitative
findings from previous studies? How did a certain
past intervention (design, policy, practice, etc.)
work? What is the volume of literature available
on a topic/domain of interest?
The key variations among these review types are
in the primary audience group being targeted
(intended end use of the review), scope, level of
interpretation, level of evaluation of individual
study quality, type of information included (pub-
lished literature vs. other information sources), type
of data included (qualitative, quantitative, or both),
depth of analysis of published data/findings, level
of synthesis, level of investigator bias, level of pub-
lication bias, extent of time commitment by the
review team, and the type and amount of resources
needed to conduct a review, among others.
Among these 14 different review types articu-
lated by Grant and Booth (2009), 3 are of relevance
to the current discussion, namely: (1) meta-analysis,
(2) qualitative systematic review or qualitative
Pati and Lorusso 3

evidence synthesis, and (3) systematic review. Key Steps in Conducting SLRs
These three types of reviews share several common
The first and most crucial step in conducting
attributes: need for systematic search, need for com-
an SLR is to familiarize oneself with generally
prehensiveness, need for study quality assessment,
accepted standards on the subject. Today, the
need for interpretation, and, most importantly, the
Cochrane Collaboration is the most well-
need for transparency of the methodology such that
known and widely respected global organization
the study can be replicated. The variations between
producing SLRs within the healthcare field and a
the three types of review originate from the long
standard to follow for any researcher seeking to
history and maturation of science and publication
write a transparent and methodologically sound
in medicine and library sciences, resulting in more
SLR. Founded in 1993, Cochrane consists of
targeted review types.
over 37,000 contributors from more than 130
Meta-analysis uses statistical analysis proce-
countries who make up over 50 subject review
dures on quantitative data to estimate the effect
groups (Cochrane, n.d.). Together, this network
size of a specific intervention, where the input
of contributors has transformed the healthcare
data are the end products of the statistical proce-
industry by publishing SLRs that gather and
dures of a number of studies with vastly similar
summarize high-quality evidence to aide
questions, setting, population, intervention, and
researchers, physicians, caregivers, and others
time period. The authors believe that the health-
in making well-informed decisions that positively
care design research field does not currently
impact human health globally. Cochrane provides
enjoy sufficient volume of quantitative studies
guidance on methodological standards which are
with the above similarities, which may, however,
available at methods.cochrane.org/mecir and are
change as the field matures. The intent of a meta-
compliant with the Preferred Reporting Items for
analysis is similar to those of a systematic review.
Systematic Reviews and Meta-Analysis
The objective of a qualitative systematic review
(PRISMA) which provides the framework for
or a qualitative evidence synthesis is similar to
SLR (Moher, 2009; PRISMA, n.d.).
meta-analysis, except that these reviews deal pri-
PRISMA, like the Cochrane Collaboration,
marily with qualitative studies, and hence the out-
was created by an international network of
comes and resulting end uses are different.
healthcare-based collaborators seeking to
A systematic review adheres to standardized
strengthen and streamline the methodological
methodologies/guidelines in systematic searching,
rigor and quality of SLRs. First published
filtering, reviewing, critiquing, interpreting, synthe-
in 2009 as the PRISMA statement, it is an
sizing, and reporting of findings from multiple pub-
evidence-based guide consisting of a checklist
lications on a topic/domain of interest. It attempts to
and flowchart intended to be used as tools for
capture the broadest set of available literature on the
authors seeking to write SLR and meta-analyses
topic of interest. After assessing the quality of indi-
(Moher, 2009; PRISMA, n.d.). Several exten-
vidual studies, SLRs may eliminate low-quality
sions to the original document were published
studies from further consideration. Moreover,
in 2015 and can all be found at www.prisma-sta
owing to the extensive documentation and reporting
tement.org/extensions/default.aspx. Although
of the steps and assumptions, SLRs renders itself
originally created for the review of randomized
amenable for replication. It demands a team effort
trials, PRISMA is broadly applicable to myriad
(at least two) to eliminate bias, among other issues.
types of research. The actual SLR then proceeds
SLRs are inherently time intensive. Owing to the
systematically through various sequential steps,
aforementioned quality demands, SLRs is regarded
including question formulation, team formula-
as the most robust level of evidence available on a
tion, identification of search domains and publi-
topic or question of interest.
cation sources, systematic search, systematic
SLRs is regarded as the most robust level critical analysis, systematic interpretation, and
of evidence available on a topic or systematic reporting. Table 1 outlines key steps
in the SLR process.
question of interest.
4 Health Environments Research & Design Journal XX(X)

Table 1. Key Steps in Conducting an SLR.

Step Description Helpful Hints

1 Familiarize yourself with the PRISMA framework.

2 Develop your study question. It should be clear and unambiguous.
3 Develop your study team. SLRs work best when the team has at least three
members, where disagreement between two members is resolved by a
third member. Make sure that each member has the necessary
qualification and experience to conduct your specific SLR.
4 Identify the concepts to be included in your search. For SLRs focused on
interventional and/or comparative effectiveness studies, complete the
PICO search model. For other study questions, in general, one should
articulate at least three concepts of interest: (1) population(s) of interest,
(2) setting(s) of interest, and (3) issue(s) of interest.
5 Define each term in the study question and the identified concepts of If definitions already exist in
interest. Do not assume that any term is unequivocally understood either published literature, one
by the team members or by the readers. could adopt those.
6 Select databases relevant to your topical area to conduct your search (at PubMed, CINAHL,
least one, typically no more than five). Typical healthcare-based databases PsycINFO, MEDLINE,
are listed on the right. You may also include other databases as well, Web of Science
depending on your topic, such as ERIC, JSTOR, etc. It is a good idea to
enlist the expertise of a reference librarian at this point.
7 Guided by the terms identified and defined in Step 5, now finalize your plan
for the systematic search. The first substep is to develop a series of key
words to be used for the search.
8 Next, decide the inclusion and exclusion criteria. All team members must Limits may include Specific
understand these criteria. Terms in the inclusion/exclusion criteria must date ranges, human
also be defined. Define the limits you will use for your systematic search. subjects, participant
Decide whether or not you will include gray literature (non-peer- descriptive, publication
reviewed publications, dissertations, thesis, conference proceedings, etc.) type, language, etc.
and whether or not you will incorporate reference list searching
(snowballing) or hand searches. See Appendix for example.
9 Next, go into each database individually to tailor your key words and search The overall yield amount
terms specifically to each search engine. Each database will have its own considered manageable will
system for terms. Some use MeSH and truncated terms and some do not. depend on the size of your
MeSH are Medical Subject Headings and is a vocabulary thesaurus analytic team. Aim for
managed by the National Library of Medicine and are tied to key words initial yields no more than
embedded within published manuscripts. When used, they allow the 500 per database as a rule
researcher to search very specifically. This step has an ebb and flow to it of thumb.
as you develop your search terms and key words. Too specific and your
search will yield nothing, too broad and your search will yield will be
unmanageable. Use the key word generators within the database to
create a formula of your search terms, using “or,” “and” as appropriate
between terms. Document in a table for transparency and replicability.
See Appendix for example.
10 Title review: Now conduct your search for all databases. Your initial review Reference citation managers:
can be done using titles within the actual database to screen out Zotero, EndNote,
irrelevant articles. Be generous in your inclusion at this point, making sure RefWorks, etc.
to keep your yield manageable based on your team, time, and resources.
Export the article yield into a reference citation manager and then from
the citation manager into an excel spreadsheet as a .csv file. This will
become your journal citation report (JCR) which is a vital tool for the
screening process and overall search documentation.

(continued)
Pati and Lorusso 5

Table 1. (continued)

Step Description Helpful Hints

11 Within the JCR, create a tab for each database, including snowballing, hand You can use viewing panes
searching, and gray literature on their own tabs, if included in the search. within Excel to help
Organize headers for relevant information including author name, abstract, condense the cells to only
publication journal, DOI, volume, year of publication, etc. Include columns at what is relevant during this
the right pertaining to your exclusion criteria that can be answered by yes or stage (abstract and
no. Remove any duplicates by striking through and hiding the cells. This way if screening questions).
you make a mistake, the information is still there and you can reincorporate
it into your JCR if needed. In case you are not sure whether to include or
exclude an article based on the title, either retain it or consult the other
team members.
12 Abstract review: Up to this point you have screened only titles, so now you Note, once you have deleted
will begin to read through the abstracts of each article and screen them articles from inclusion in
against the exclusion criteria. Instead of inserting “yes” or “no,” it is helpful the JCR you can remove
to create separate columns for the two and insert a number “1” (for yes or them from your reference
for no, as applicable) so that you can run a sum total at the bottom of each citation manager for
spreadsheet to keep track of how many duplicates were removed, how consistency.
many were excluded and how many were included per database. Once you
have completed this for each separate database, create a tab for all included
articles. Copy and paste all the included articles into this sheet and sort by
title to quickly find and remove any remaining duplicates. This information
will be important data for the PRIMSA flowchart.
13 Abstract phase screening quality check: To strengthen validity and reliability You may review the research
for your search results, work with at least one other researcher to question, exclusion
conduct an inter-rater reliability assessment. For instance, you can give criteria, and definitions
the second reviewer 20% of the total number of abstracts you reviewed, again to ensure uniformity
randomly chosen, and leave your exclusion fields blank so that the second of understanding.
reviewer can run a parallel screening process on the subset of abstracts.
Compare your findings and aim for at least 90% agreement. Discuss any
areas of disagreement with the third team member reviewer until you
reach an inclusion/exclusion agreement.
13 On the last tab in the Excel spreadsheet, organize your process totals in the PRISMA flowchart can be
JCR and update your PRISMA flowchart with the information. Keep the found at http://www.prisma-
JCR as a tool and record of your process. statement.org/
14 Critical review of full articles: Once you have the list of final included articles,
conduct in-depth critical review of each article. Extract all relevant
information from each article and insert in the appropriate place in the Excel
document. As part of the critical review, rate the level of quality of evidence
for each article using the framework you have adopted for your study.
15 Critical review assessment quality check: To strengthen validity and reliability You may review the research
for your critical review findings, work with at least one other researcher to question, the quality of
conduct an interrater reliability assessment. For instance, you can give the evidence scale, quality
secondary reviewer 20% of the total number of articles you critically issues, and definitions again
reviewed, randomly chosen, and leave your assessment fields blank, so that to ensure uniformity of
the second reviewer can run a parallel critical evaluation process on the understanding.
subset of articles. Compare your findings and aim for complete agreement.
Closer the agreement to 100%, more robust is the critical review process.
A level of at least 90% agreement may be acceptable. Discuss any areas of
disagreement with the third team member.

(continued)
6 Health Environments Research & Design Journal XX(X)

Table 1. (continued)

Step Description Helpful Hints

16 Update your PRISMA flowchart with any changes from the inter-rater You can also calculate
reliability analysis. Now you will create a new, clean spreadsheet that will interrater reliability with
become your matrix of included studies. This spreadsheet will be simpler the level of quality of
than the JCR with headings that are tailored specifically to the purpose of evidence to further
your overall SLR. These might include first author, year, level of evidence, strengthen validity and
methods, participants, environmental intervention, outcomes, setting, reliability of your SLR.
assessments, results, etc. This will likely be published with your article
and should be well organized. Now you are ready to compose your
report, following the guidelines from the Cochrane Collaboration and the
PRISMA checklist, and using Table 2.
Note. SLRs ¼ systematic literature reviews; PRISMA ¼ Preferred Reporting Items for Systematic Reviews and Meta-Analysis.

A Suggested Model for
Systematically Composing
an SLR Report
Key Sections in Writing an SLR Following the
PRISMA Checklist
Contents within the 27 PRISMA checklist items
include guidelines regarding key steps in the pro-
cess and are broken into title, abstract, methods,
results, discussion, and funding related to the SLR
(Moher, 2009; PRISMA, n.d.). According to the
PRISMA checklist, authors should first clearly
identify the report as a systematic review, meta-
analysis, or both. This is the only aspect of PRISMA
not required by the Cochrane Collaboration but is a
good rule to follow for healthcare design research-
ers who wish to make the nature of their work
easily identifiable for publishers and readers. The
following is a summary of the PRISMA checklist,
which can be found and downloaded at http://
www.prisma-statement.org/ for use as a guide.
Introduction. The two critical elements within the
introduction are the author’s rationale and objec-
tives. The rationale for the review should be
written in a way that explains the topic in light
of what is currently understood in the existing lit-
erature. The objectives should follow and clearly
identify the question(s) to be addressed and how
they relate to the various concepts of interest. In
interventional studies, a typical articulation of con-
cepts of interest is participants/interventions/com-
parisons/outcomes (PICO) and study design
(Figure 1). The PICO process comes from the field
of evidence-based medicine (EBM) to address a
healthcare-based question. In SLRs, PICO can be
used to frame the search terms and key words to
the specific topic being studied (Centre for
Evidence-Based Medicine, 2014; Centre for
Reviews and Dissemination, 2009; Lorusso &
Bosch, 2016). Investigators should identify or
develop their specific list of concepts of interest,
such as population/setting/issue, based on the
study objective and report it in the introduction.

Abstract. The abstract of the report should consist
of a detailed summary including the following ele-
ments, as applicable to each report: background,
objectives, data sources, study eligibility criteria,
participants and interventions, study appraisal and
synthesis methods, results, limitations, conclusions
and implications of key findings, and inclusion of
a systematic review registration number if appli-
cable. The overall goal is to be clear and concise in
the abstract so as to transparently communicate the
contents of the paper to readers.
Methods. The methods section should outline the
overall process followed within the SLR. It is
important to maintain clarity and transparency
throughout the description of the process fol-
lowed in the SLR in order to reduce bias and
improve the translation of the overall findings.
This section will include information on the pro-
tocol and registration, eligibility criteria, infor-
mation sources, search strategy, study selection
and data collection process, data items studied,
risk of bias in individual studies, a summary of
Pati and Lorusso
P Patient, Population or Problem
I Intervention
C Comparison of Intervention (if appropriate)
O Outcome to Measure or Achieve
Figure 1. Explanation of participants/interventions/comparisons/outcomes (PICO) search model (Center for
Evidence-Based Medicine).
measures, synthesis of results, risk of bias across
studies, and any additional analysis. This section
provides an overview explanation preceding the
results section which will provide information
culminating from a deeper dive into many of the
same topics for the studies included in the final
review. Protocol and registration, if it exists and
where it can be found, should be reported. Elig-
ibility criteria should be provided of included
studies such as study characteristics, time frame
of studies, language, publication, and so on, and a
rationale for why the author used these specific
eligibility criteria. Information sources included
in the review should be clearly described. These
are typically the databases targeted for the review
with an explanation of the dates used within the
review and an explanation of any contact with
authors for additional included studies. The
search strategy should be clearly defined for one
or more databases. This should be provided in
great detail to allow for future updates or replica-
tion and should include any limits used by the
author. The study selection process should be out-
lined regarding screening of findings, ratings of
eligibility, and a breakdown of what articles were
included in the review. This is supported by the
PRISMA flowchart, which provides a clear, gra-
phic representation of the process from included
articles through to the final selection of included
articles (Figure 2). A template is available at
http://www.prisma-statement.org/PRISMAState
ment/FlowDiagram.aspx
The data collection process and precise
method used by the authors to gather information
from the literature should be clearly provided.
This is often done through an excel document
known as the journal citation report (JCR) that
collates information extracted from reference
managers in an organized way (author name,
7
e.g. Dementia
e.g. Multisensory Environments
NA
e.g. Patient Behavior
publication, publication date, volume, abstract,
etc.). It is often used as a tool by the author or
team writing the SLR to rate the found articles
against the inclusion criteria and keeps a record of
duplicates, excluded and included articles. Data
items are listed within the methods section and
include variables and all assumptions or simplifi-
cations made by the SLR author(s). Risk of bias
in individual studies is provided and discussed in
terms of how the risk was assessed for each article
and how it will be used in the data synthesis of the
SLR. A summary of measures should be outlined in
terms of the primary measures used within the arti-
cles included in the SLR. This helps to clarify the
overall analysis and helps readers understand how
outcomes were achieved. A synthesis of results is
explained by a description of the methods used by
the SLR author(s) to review data and evaluate the
level of evidence. Risk of bias across studies is
discussed that may impact the overall findings. This
is different from the risk of bias within individual
studies because it discusses a broader range of risk
including publication bias or limited reporting of
data throughout studies. Additional analyses are
important particularly in SLRs, which might
include sensitivity or subgroup analyses, meta-
regression, and so on. It may also include any other
additional analysis conducted by the SLR author or
team relevant to the synthesis of findings.
Results. The results section of the SLR is presented
in descriptive, graphic, and detailed tabular for-
mats. Combined, these representations should
carefully describe the study selection, characteris-
tics, risk of bias within and of individual studies,
synthesis of results, risk of bias across studies, and
any additional analysis of findings. Whereas the
methods section provides an introduction, and
outline of how these topics will be discussed, the
8 Health Environments Research & Design Journal XX(X)

Figure 2. Example of Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) flow
diagram.

results section provides great detail specifically on
the articles included in the final review. Study
selection, supported by the PRISMA flowchart,
provides detailed information on the number of
studies screened, assessed for eligibility, and
excluded or included for review with an explana-
tion for exclusion throughout the process. Study
characteristics are provided for each included arti-
cle along with information on the data that were
extracted. This could include participant or site
descriptions, intervention, method, time frames,
and so on. This information is supported graphi-
cally in a matrix of the included studies. Risk of
bias within studies is identified with outcome-level
assessments. Results of individual studies are pro-
vided for all outcomes including information on
the intervention and any group differences. This
can be done in written format, separated into com-
mon themes or areas of affinity, and supported
graphically by the SLR in a matrix of the included
studies. A synthesis of results will provide infor-
mation on meta-analysis of included articles and
Pati and Lorusso
assessed levels of evidence. Risk of bias across
studies and additional analysis provides informa-
tion on broad risk of bias across all included stud-
ies and detail any additional analysis conducted on
included literature.
Discussion. Lastly, the discussion section of the
article will bring all the information together into
a final summary of the evidence, listing limita-
tions of the SLR and offering conclusions stemming
from the review. In healthcare design SLRs, this
section may also include a description of possible
implications for practice that can be used by health-
care designers, researchers, and administrators in
future applications. The summary of evidence will
restate the main findings and discuss the overall
quality level and magnitude of evidence and how
each finding relates to key user groups within the
healthcare industry. A discussion of limitations is
an important component of the SLR to overcome
bias. All limitations including any risk at any level
of the review process should be identified. All
reviews will have some form of risk of bias and so
this is not a section that should be avoided or seen as
a weakness to the overall report. To the contrary, an
open and straightforward discussion of any risk of
bias serves to increase the credibility of an SLR.
Conclusions. The concluding section of the discus-
sion will include the author(s) perspective on the
findings and their role related to other reported find-
ings and how they may impact future research or
healthcare design overall. Any sources of funding
should be reported including a description of any
other support from sources supplying data and the
subsequent role of funding sources related to the SLR.
Table 2 provides a suggested step-by-step,
generic model for composing an SLR report. It
includes important questions that need to be
responded to in each section of an SLR report.
One may ignore questions that do not apply to
one’s specific SLR study.
Explanation of Key Elements Within
the Review
As discussed earlier, critical factors separating
SLR from traditional reviews are clarity, validity,
and auditability (Booth et al., 2016). These
9
elements are best achieved by following the
Cochrane Collaboration’s guidelines and the
PRISMA statement checklist closely. Adhering
to these frameworks helps ensure a level of uni-
formity in the SLRs, supporting transparency, and
reducing overall risk of bias. This uniformity
incorporates key elements including an assessment
of the evidence, the concepts of interest and
PRISMA flow diagrams, a JCR, and a matrix of
included studies.
Both the quality level and magnitude of evi-
dence need to be analyzed in an SLR. The quality
level of evidence pertains to the quality of an indi-
vidual study. The magnitude of evidence pertains
to the volume of quality evidence available on a
topic or question of interest. Referring back to our
analogy of snapshots in the introduction, the qual-
ity level of evidence can be viewed as the quality
of each photograph whereas the magnitude of evi-
dence can be viewed as the number of good quality
photos available for stitching together.
The purpose of critically assessing the evidence
in an SLR is important because it provides the
reader with a description of its quality. This assess-
ment is often tied directly to the inclusion/exclusion
criteria when an author will state that only certain
quality of evidence will be included. This may be
done in an SLR of a topic where the author(s) goal is
to report only the highest quality of evidence and
there is enough available literature to be selective.
Sometimes, however, this is not possible particu-
larly in the healthcare design domain, currently,
because the body of knowledge regarding a topic
may not yet be so extensive as to allow it. Whatever
the decision of the author or team conducting the
SLR, it is important to rate the quality level of evi-
dence. Examples can be found in EBM (Balshem
et al., 2011) and also in healthcare design literature
(Hamilton, 2011; Marquardt, Bueter, & Motzek,
2014; Pati, 2011; Stichler, 2010) and are commonly
described in a simple table format outlining the
descriptions of each quality level.
Other key elements within the review are the
concepts of interest (such as PICO) and PRISMA
flowchart both of which relate to the overall
search strategy of the SLR. It is important to note
that when designing the specific search strategy
for each SLR, it is incredibly valuable to enlist the
help of a research librarian with expertise within
10 Health Environments Research & Design Journal XX(X)

Table 2. A Step-by-Step Model for Composing an SLR Report.

Main
Sections Subtopics Key Questions to Respond to

Abstract [as required by the targeted journal]

Introduction What is the SLR study question?
Background: What was already known on the topic/question of interest?
What drove the need for conducting this specific SLR?
What is the objective of this SLR?
What is the significance of this SLR study?
Method Team composition How many investigators were on the team? What were their qualifications,
expertise, and levels of experience?
Search preparation What databases were targeted?
What key words/phrases were used in each database search?
What were the inclusion and exclusion criteria?
Search process How and when were the database searches conducted? How many phases of
search were involved?
Was a hand search conducted? How was it conducted?
How was it determined whether to include or exclude a specific publication?
How many steps were involved in arriving at the final list of publications?
What was involved in each step?
How was interrater reliability ensured during the abstract review phase?
Critical review How was the critical review conducted?
What information was extracted and recorded?
How was the quality level of each study determined? What quality
framework was used? How were qualitative, quantitative, and mixed-
method studies evaluated?
What method was used to develop thematic areas of relevance? How were
studies within each thematic area synthesized?
Were any statistical analysis/meta-analysis conducted on quantitative
findings? If not, why?
How was interrater reliability ensured during the critical review phase?
Results Description of How many articles were found in each sequential search step? How many
final list were retained in each step and how many were discarded? What was the
final number of articles on which critical review was performed?
Content of final list How many studies were empirical?
Of the empirical studies, how many were causal/experimental, how many
were relational, and how many were descriptive?
Of the empirical studies, how many were qualitative, how many were
quantitative, and how many used mixed methods?
In which countries were the studies conducted?
How many studies were classified at each level on the quality of evidence
scale?
Content of each What thematic areas were identified?
thematic area For each thematic area (separately), what are the key descriptions of the
contents (respond to the same questions in the “content of final list,” but
for each thematic area)?
What were the key findings in each thematic area? What are some patterns
observable? Are there any converging/diverging findings?
What is the overall confidence (magnitude of evidence) within each thematic
area?
Within each thematic area, how many studies and what proportion of
studies fall under each level of the quality of evidence scale?

(continued)
Pati and Lorusso 11

Table 2. (continued)

Main
Sections Subtopics Key Questions to Respond to

Discussions Summarization What new information was generated from this systematic review?
What is the general level (quality and magnitude) of evidence in various
thematic areas? Which of the thematic areas enjoy higher quality and
magnitude of evidence? Which areas currently lack quality and/or
magnitude of evidence?
Limitations What are the limitations of this SLR?
Future studies What topics/questions should be examine in future studies?
Conclusions What are the key take-away messages of this SLR for your target readers?
References [include complete list of articles reviewed and/or cited in the SLR, denote
articles included in the SLR matrix with an asterisk*]
Appendix [as appropriate]

the specific topic to be reviewed. Research librar-
ians with a focus on healthcare, for example, have
extensive knowledge of the broad range of liter-
ature and are adept in assisting in the develop-
ment of the PICO model serving as a guide for the
search terms and key words relating the patient,
population or problem, intervention, and outcome
to be evaluated in the SLR, if the SLR is focused
on interventional studies. Once established, the
PICO search model for the SLR is often illu-
strated with a table and it is helpful to document
the exact key words and terms used in a separate
appendix for future revision or replication of the
search. In case of SLRs not restricted to interven-
tional studies, librarians can also assist in devel-
oping the right search terms for population,
settings, and issues of interest. The PRISMA flow
diagram is also used to document the search strat-
egy. A template is available from the PRISMA
statement’s website that can be edited and
inserted directly into a Word document. The
flowchart is simple and easy to follow and pro-
vides a consistent way to communicate the results
of the search process from the initial findings,
inclusion of gray or snowballed literature, exclu-
sions, and final inclusion counts for the SLR.
One of the most important elements within the
SLR is the matrix of included studies. The matrix is
a tabular representation of the results section of the
paper made in a spreadsheet document that sum-
marizes the details of each included study and the
overall findings of the SLR. It is important to note
that the articles included in the final review are
included in the reference list of the paper and are
marked with an asterisk to clearly identify them for
the reader. The matrix will be tailored specifically to
each SLR based on the unique objectives of the
paper. Typical components of the matrix include the
last name of the first author and year of publication,
assigned quality level of evidence, methods, parti-
cipants’ description, intervention summary, list of
outcomes, description of setting and assessments,
and a brief recap of the results of each study. The
intention of the matrix is to serve as a tool providing
an easily digestible format for readers to understand
the purpose and outcomes of the SLR.
Important Issues in
Conducting SLRs
There are several issues that warrant close atten-
tion while conducting and reporting an SLR. The
key question one should ask while conducting an
SLR is whether the intended audience will con-
sider the findings credible (irrespective of the
actual nature of SLR findings). If trust is not gen-
erated among the readers, then findings may not be
perceived as credible, which will be counterpro-
ductive to the objectives of the SLR study. The
following subsections outline some of the issues
that investigators need to accord special attention
to in order to generate trust and confidence.
The key question one should ask while
conducting an SLR is whether the intended
audience will consider the findings
credible.
12
Systematicity
One distinguishing feature of SLRs that separates it
from most other types of reviews is systematicity.
Being systematic includes a systematic process of
searching for literature, filtering results of literature
searches, critical analysis of the studies, evaluating
quality of identified studies, evaluating the level of
evidence, interpretive analysis of findings across mul-
tiple studies, and reporting for the intended audience.
Documentation
One simple strategy for generating trust and con-
fidence is documentation. Each step and results
should be documented and reported in a clear,
unambiguous manner.
Study Question
The most critical aspect of any research study is
the study question. It should be noted that SLR,
owing to its interpretive component, could be
deemed as a research study. The study questions
impact every subsequent step in a study. It should
be clear and precise.
Definitions
One way of achieving clarity and precision in the
study question is developing clear, unambiguous,
and documented definitions of all terms associ-
ated with the study. Definitions are important in
bringing all the investigators and the audience on
the same page. Table 3 provides the definitions of
key terms used in this article.
Search Strategy
The objective of a typical SLR is to include all
literature associated with a study question, includ-
ing literature in peer-reviewed publications and
other sources, such as dissertations. Numerous
non-peer-reviewed sources, also referred to as
“gray literature” should also be brought into scru-
tiny, such as white papers and industry/trade arti-
cles. Search may (or should) include manual
review of the reference section of key articles.
In some fields, quantitative studies that do not
result in a statistically significant findings face
Health Environments Research & Design Journal XX(X)
greater difficulties in getting published in peer-
reviewed journals (also known as “publication
bias”), whereas from a theoretical perspective and
in numerous real-life situations knowledge
regarding nonsignificant findings could also be valu-
able. As a result, it is essential to search and retrieve
associated literature from all types of sources.
How a search is conducted could lead to bias,
whether intentional or otherwise. The appropriate
way to develop trust in the audience of an SLR is
to demonstrate that a broad, complete, and thor-
ough search was conducted, wherein the prob-
ability of missing any important work related to
the study question is minimized.
Inclusion and Exclusion Criteria
Another source of potential bias and ambiguity is
in the way literature found in the searchers are
included or excluded from further analysis. Clear
documentation of criteria used for inclusion or
exclusion of a study will result in a higher trust
and credibility in the SLR findings.
Critical Review
Producing a good SLR is dependent on critical
review of the quality of each study. There are two
approaches to assessing study quality. One
approach focuses purely on the research design,
wherein studies with higher internal validity are
considered higher in quality. Quantitative studies
have typically been considered higher in quality
than qualitative studies since assessing internal
validity is easier in quantitative study design. For
instance, an experimental study design would be
considered higher in quality than a relational or
descriptive study or a qualitative study design.
However, research design is not the only factor
(and should not be) to assess in determining study
quality. Perhaps, when time is limited, a classifica-
tion based just on research design is justified. It
should never be assumed that a good research
design necessarily results in a good quality study.
Ideally, each study should be assessed deeper on all
quality issues—construct, internal, external, and
conclusion validity as well as reliability. Issues
such as definitions, study setting, tools and mea-
sures, sampling strategy, sample characteristics,
Pati and Lorusso 13

Table 3. Definition of Terms Found in Article.

Term Definition

Areas of affinity Common themes found in the SLR literature.

Auditability Ability for the SLR to be externally reviewed and replicated.
Bias Prejudice, predisposition, or partiality to a topic or viewpoint.
Clarity Related to the level of transparency, openly displaying the process.
Cochrane Collaboration Global healthcare research organization producing SLRs.
Conformity Consistency in design a method or process.
Credibility Reliability or trustworthiness of a method or process.
Cross-sectional Study Study that analyzes data at a specific point in time.
Databases Online archives (e.g., PubMed, CINAHL, MEDLINE, etc.)
Dependability Related to the soundness of the findings.
Dissertations A five-chapter report written as a requirement for a PhD degree.
Effect size Statistical measure of strength of relationship between variables.
Eligibility criteria Criteria or boundaries that must be shared by all articles included.
Gray literature Unpublished or published noncommercially, not peer reviewed.
Industry/trade articles Articles published in a commercial magazine, targeting industry.
Instruments Measurement tools to gather data (e.g., survey, observations, etc.)
Intervention That which is being tested in an experimental study.
Investigator bias Occurs when investigators influence results due to predisposition.
Journal citation report Document used in an SLR to organize and analyze literature found.
Limits Boundaries within which search is conducted (e.g., date range, species, etc.)
Longitudinal study Data is gathered from the same subjects over a long period of time.
Magnitude Relating to the size or extent of existing, published literature.
MeSH Medical Subject Headings, thesaurus terms embedded in literature.
Meta-analysis Statistical approach combining results from multiple quantitative studies.
Mixed-methods Studies Studies that employ both qualitative and quantitative methods.
Peer-reviewed Evaluation prior to publication by others within the same field.
publications
Population Describes the people at the focus of a study.
PRISMA Preferred Reporting Items for Systematic Review and Meta-Analyses
Protocol A document outlining rationale and detailed procedure of a study.
Publication bias Occurs when a publication has a known partiality to a topic or findings.
Qualitative Research involving nonnumeric data.
Quality level Relating to the value and perceived excellence of a study.
Quantitative Research involving numeric data.
Randomized Studies where subjects are assigned randomly to receive one of several interventions
controlled trial where one is a base-line of comparison.
Reference manager Online software used to organize and manage literature findings.
Replicability The ability for a study to be repeated with similar results.
Research question The question that a researcher attempts to find an answer.
Sample characteristics The defining features of a study sample (e.g., gender, race, age, etc.)
Sample size The number of participants in a study.
Sampling strategy The researcher’s approach to finding participants for a study.
Setting The location of a particular study (e.g., hospital, nursing home, etc.)
Snowballed literature Literature found through the reference list of included articles.
Statistically significant A result or finding between variables not attributed to chance.
Subgroup analyses Exploration of how people respond differently to an intervention.
Synthesis Combination of findings.
Themes Similarities or commonalities found between studies.
Theoretical framework Interrelated concepts.
Topic of interest Subject matter that a researcher seeks to find deeper understanding on.
Transferability How easily a finding can be applied to other settings.

(continued)
14 Health Environments Research & Design Journal XX(X)

Table 3. (continued)

Term Definition

Transparency Relates to the clarity and openness in the design of a study.

Truncated terms Search technique that broadens to various spellings with an asterisk
Validity Closeness to the truth.

sample size, analytical procedures, measures Conclusion

adopted to remove bias, among many others, should
be evaluated in order to accurately, systematically,
and transparently gauge the quality of a study.
One fundamental requirement for conducting
critical quality review of a study at this depth is to
develop a team with appropriate qualifications to
enable it. One should carefully select the team of
a proposed SLR to achieve a robust level of crit-
ical review. It should also be noted that qualita-
tive methods also have quality issues similar to
those in quantitative methods, such as credibility,
transferability, dependability, and conformity
(Healey, 2014). In case of qualitative or mixed-
method studies, team members need to have suf-
ficient knowledge and experience to evaluate the
deeper quality issues in both types of methods.
Reporting
Reporting findings from SLRs should not be a
simple additive task, where findings from various
studies are reported as a summative narrative.
One of the strengths of SLRs is in the interpretive
analysis of findings from multiple studies. This
requires in-depth domain knowledge and exper-
tise of the investigating team.
A critical issue in reporting is investigator bias.
As mentioned previously, any hint of investigator
bias could lead to an erosion of trust and credibility
of SLR findings. Since every individual has some
degree of bias, it is necessary to take precautionary
measures to eliminate any. Having a team (instead
of one investigator) is one of the strategies to elim-
inate or reduce bias in the literature search and
selection phases. The ideal solution is to include
team members who have little bias on a topical area.
In the reporting phase, if one’s strong bias is known,
it may be advisable to exclude that individual from
the interpretation and reporting phases, or subject
the interpretation and reporting to scrutiny by mul-
tiple team members or external experts.
With a growing body of evidence pertaining to the
influence of physical design on clinical, beha-
vioral, operational, and business processes out-
comes, there will be a growing demand for SLRs
on healthcare design topics. Key factors support-
ing SLRs are clarity, validity, and auditability
(Booth et al., 2016). It is important to note that
although bias is significantly reduced in SLRs, its
existence is not eradicated. Also, despite their mer-
its, SLRs are the most demanding types of litera-
ture reviews to conduct and involve a structured
process for conducting and reporting. Although the
process is significantly more involved than a tra-
ditional literature review, there is a great deal of
support available through organizations like the
Cochrane Collaborative and the PRISMA state-
ment. As with any literature review, SLRs are
undoubtedly strengthened through the additional
support of a research librarian with expertise in
your field of study. Overall, adhering to the
requirements described in this article, and/or other
similar SLR guides available in medical literature,
will ensure a credible, high-quality SLR and con-
tribute vital information to the ever-growing
healthcare design knowledge base.
Implications for Practice
 SLRs provide more comprehensive and holistic
evidences than single research studies. Use SLRs
to inform decision making when available.
 Use SLRs published in peer-reviewed journals
only. SLRs not subjected to peer-review may
or may not have investigators’ biases.
 Always check that your definition/understand-
ing of terms used in practice are identical to
those used in the SLR. Frequently, differences
in the way terms are defined lead to erroneous
conclusions, with potentially undesirable
downstream outcomes.
Pati and Lorusso 15

Appendix
Table A1. Search Strategy Template.

Research Question(s) Databases, Resources, and Limits

Core Databases Gray Literature Limits

þReference List Searching
þHand Searching Journals
-Clinical Trial Registries 1. Language:
-National Clearinghouse 2. Years:
-Google Scholar 3. Publication Types:
-Dissertation Sources 4. Species:
-Conference Abstracts

Table B1. Primary Literature Searching Template. guidelines: 3. Rating the quality of evidence. Journal
Population Intervention Outcome of Clinical Epidemiology, 64, 401–406.
Concept: Concept Concept Booth, A., Sutton, A., & Papaioannou, D. (2016). Sys-
tematic approaches to a successful literature review
Mesh/ (2nd ed.). Los Angeles, CA: Sage.
subheadings Centre for Evidence-Based Medicine. (2014, June 10).
Text words
Asking focused questions. Retrieved February 22,
Note. Database: (Do this for each of your databases and gray 2016, from http://www.cebm.net/asking-focused-
literature). questions/
Centre for Reviews and Dissemination. (Ed.). (2009).
PubMed Search Strategies CRD’s guidance for undertaking reviews in healthcare
(3rd ed.). York, England: York Publishing Services.
POPULATION (Example below, do the same Cochrane. (n.d.). Cochrane Collaboration. Retrieved
for intervention/outcome as well) October 10, 2017, from www.cochrane.org/about-us
(“Dementia/drug therapy”[Mesh] OR “Dementia/ Dunn, P. M. (1997). James Lind (1716-94) of Edin-
rehabilitation”[Mesh] burgh and the treatment of scurvy. Archives of Dis-
INTERVENTION ease in Childhood—Fetal and Neonatal Edition, 76,
OUTCOME F64–F65. Retrieved from https://doi.org/10.1136/
Yield: fn.76.1.F64
Duplicates Removed: Glass, G. V. (1976). Primary, secondary, and meta-
Total Added analysis of research. Educational Researcher, 5,
3–8. Retrieved from https://doi.org/10.3102/
0013189X005010003
Declaration of Conflicting Interests
Grant, M., & Booth, A. (2009). A typology of reviews:
The author(s) declared no potential conflicts of An analysis of 14 review types and associated meth-
interest with respect to the research, authorship, odologies. Health Information and Libraries Jour-
and/or publication of this article. nal, 26, 91–108.
Greenhalgh, T. (2014). How to read a paper: The
Funding basics of evidence-based medicine (5th ed.). Chi-
The author(s) received no financial support for chester, England: John Wiley.
the research, authorship, and/or publication of Hamilton, D. K. (2011). What constitutes best practice
this article. in healthcare design? Health Environments
Research & Design Journal, 4, 121–126. Retrieved
References from https://doi.org/10.1177/193758671100400210
Balshem, H., Helfand, M., Schunemann, H., Oxman, A., Healey, J. (2014). Statistics: A tool for social research
Kunz, R., Brozek, J. . . . Guyatt, G. (2011). GRADE (10th ed.). Belmont, CA: Wadsworth.
16
Lorusso, L., & Bosch, S. (2016). Impact of multi-
sensory environments on behavior for people with
dementia: A systematic literature review. The Ger-
ontologist. Retrieved from https://doi.org/10.1093/
geront/gnw168
Marquardt, G., Bueter, K., & Motzek, T. (2014).
Impact of the design of the built environment on
people with dementia: An evidence-based review.
Health Environments Research & Design Journal,
8, 127–157.
Moher, D. (2009). Preferred reporting items for sys-
tematic reviews and meta-analyses: The PRISMA
statement. Annals of Internal Medicine, 151, 264.
Retrieved from https://doi.org/10.7326/0003-4819-
151-4-200908180-00135
Pati, D. (2011). A framework for evaluating evidence
in evidence-based design. Health Environments
Research & Design Journal, 4, 50–71.
PRISMA. (n.d.). PRISMA transparent reporting of sys-
tematic reviews and meta-analyses. Retrieved
October 10, 2017, from www.prisma-statement.org
Stichler, J. (2010). Weighing the evidence. Health
Environments Research & Design Journal, 3, 3–7.
Ulrich, R., Zimring, C., Quan, X., Joseph, A., &
Choudhary, R. (2004). The role of the physical envi-
ronment in the hospital of the 21st century: A once-
in-a-lifetime opportunity. Concord, CA: The Center
for Health Design.
Health Environments Research & Design Journal XX(X)
Author Biographies
Debajyoti Pati is currently a professor of environmen-
tal design and Rockwell Endowment Chair at Texas
Tech University. He has over 28 years of experience
in research, practice, and teaching in the United States,
Canada, and India. He was twice voted among the 25
most influential people in healthcare design and has
received over ten awards for research excellence. His
areas of interest include health, healthcare, healthy
environments, sustainability, public architecture, judi-
cial facilities, performance-based design, organiza-
tional learning, and developing indicators of facility
performance. He is on the editorial board of HERD and
on the board of directors of the NIBS-AIA BRIK
program.
Lesa Lorusso is a licensed interior designer, with over
15 years of experience in both academic and profes-
sional practice. Her educational background includes a
BS in interior design, MBA and MS in architecture.
She practices in Florida and is currently a doctoral
candidate and research assistant within the College of
Design, Construction and Planning at the University of
Florida and researcher with the Geriatric Research
Education and Clinical Center (GRECC) at the
Malcolm Randall Veteran’s Administration Medical
Center. The focus of her doctoral research is a colla-
boration between UF and the VA and investigates the
impact of multisensory environments on behavior for
Veterans with dementia.